Greedy Algorithms1

Simple Knapsack Problem

Greedy Algorithms form an important class of algorithmic techniques. We
illustrate the idea by applying it to a simplified version of the Knapsack
Problem. Informally, the problem is that we have a knapsack that can only
hold weight C, and we have a bunch of items that we wish to put in the
knapsack; each item has a specified weight, and the total weight of all the
items exceeds C; we want to put items in the knapsack so as to come as close
as possible to weight C, without going over. More formally, we can express
the problem as follows.
Let w1 , . . . , wd N beP
weights, and let C N be a weight. For each S
{1, . . . , d} let K(S) = iS wi . (Note that K() = 0.)
Find:
M = max {K(S)|K(S) C}
S{1,...,d}

For large values of d, brute force search is not feasible because there are 2d
subsets of {1, . . . , d}.
We can estimate M using the Greedy method:
We first sort the weights in decreasing (or rather nonincreasing order)
w1 w2 ... wd
We then try the weights one at a time, adding each if there is room. Call
this resulting estimate M .
It is easy to find examples for which this greedy algorithm does not give the
optimal solution; for example weights {501, 500, 500} with C = 1000. Then
M = 501 but M = 1000. However, this is just about the worst case:
Lemma 1 M 21 M
1
These notes are based on lecture notes by Stephen Cook and Michael Soltys in CSC
364, University of Toronto, Spring 2003

Proof:
We first show that if M 6= M , then M > 21 C; this is left as an exercise. Since
C M , the Lemma follows.
Unfortunately the greedy algorithm does not necessarily yield an optimal
solution. This brings up the question: is there any polynomial-time algorithm that is guaranteed to find an optimal solution to the simple knapsack
problem?
However, a positive answer to this question would show that NP=P, since
we can show that Knapsack problem is NP-hard. Let us consider a decision
version of Knapsack, in which there is one more parameter B; now, rather
than trying to get S with maximal possible sum, we are just asking if there
exists an S {1, . . . , n} with K(S) B
Lemma 2 The decision version of Simple Knapsack is NP-complete.
Proof:
The valid instances are of the form {w1 , . . . , wn , C, B}, and an instance is in
the language if there exists an S {1, . . . , n} such that B iS wi C.
To see that this problem is in NP, note that given a certificate S it takes
polynomial time to compute the sum of elements in S and compare it to two
numbers B and C.
To see that it is NP-hard, we will use a reduction from SubsetSum problem;
that is, we will show SubsetSum p DecisionSimpleKnapsack. Recall that
instances of SubsetSum are of the form {a1 , . . . , an , t}, and an instance is
in the language iff there is an S {1, . . . , n} such that iS ai = t. Now,
f ({a1 , . . . , an , t}) = {a1 , . . . , an , t, t}. That is, set wi = ai for i {1, . . . , n}
and set both B and C equal to t. Now, if there is a set S 0 such that the sum
of elements in it is both and to t, then it must equal to t. Similarly for
the other direction, a set S with sum of its elements equal to t satisfies both
B and C conditions for B = C = t. .
So unless something drastic (that is, P=NP) happens, we have no hope of
getting a polynomial-time algorithm for Knapsack. However, we can change
the problem in such a way that the new problem is indeed solvable in polynomial time. Let us consider a variation called FractionalKnapsack: in this
2

case, we will again look at the optimization (search) problem, as opposed to

decision problem. Also, rather than modifying SimpleKnapsack in which we
only had weights, we will now assign a weight and a profit to each element.
Let FractionalKnapsack to be defined as follows. The input is {(w1 , p1 ), . . . , (wn , pn ), C}
where wi are weights, pi are profits, and C is the capacity. But now we allow
fractions of elements to be put in the Knapsack. So S becomes a list of numbers (s1 , . . . , sn ), where each 0 si 1 and si corresponds to the fraction
of ith element in the set. Now, the capacity condition becomes ni=1 wi si C
and we are trying to maximize ni=1 pi si .
(To make it easier to think about this problem, consider a situation when
a company is trying to load a container with goods. If they are required to
put each item as a whole, then it is the usual Knapsack definition. However,
suppose they are transporting something that can be split e.g., coal or
grain. Now, if the whole batch of grain does not fit, they can just take as
much of it as fits, and leave the rest).
For the case of FractionalKnapsack, the following simple greedy algorithm
solves the problem optimally. The idea is to sort the elements by their unit
value (that is, total profit divided by the total weight of each element). Now,
start adding them in that order; when cannot put the whole element put as
much as fits.
Fractional Knapsack Algorithm:
Sort the items so that: p1 /w1 p2 /w2 . . . pn /wn
S = (0, 0, . . . , 0)
M =0
for i = 1 to n do
if C M wi then
S(i) = 1; M = M + wi
else if M < C
S(i) = (C M )/wi ; M = C.
else return S
end if
end for

Minimum Spanning Trees

An undirected graph G is a pair (V, E); V is a set (of vertices or nodes); E
is a set of (undirected) edges, where an edge is a set consisting of exactly
two (distinct) vertices. For convenience, we will sometimes denote the edge
between u and v by [u, v], rather than by {u, v}.
The degree of a vertex v is the number of edges touching v. A path in G
between v1 and vk is a sequence v1 , v2 , . . . , vk such that each {vi , vi+1 } E.
G is connected if between every pair of distinct nodes there is a path. A cycle
(or simple cycle) is a closed path v1 , . . . , vk , v1 with k 3, where v1 , ..., vk
are all distinct. A graph is acyclic if it has no cycle. A tree is a connected
acyclic graph. A spanning tree of a connected graph G is a subset T E of
the edges such that (V, T ) is a tree. (In other words, the edges in T must
connect all nodes of G and contain no cycle.)
If a connected G has a cycle, then there is more than one spanning tree for
G, and in general G may have exponentially many spanning trees, but each
spanning tree has the same number of edges.
Lemma 3 Every tree with n nodes has exactly n 1 edges.
The proof is by induction on n, using the fact that every (finite) tree has a
leaf (i.e. a node of degree one).
We are interested in finding a minimum cost spanning tree for a given connected graph G, assuming that each edge e is assigned a cost c(e). (Assume
for now that the cost c(e) is a nonnegative real number.) In this case, the
cost c(T ) is defined to be the sum of the costs of the edges in T . We say that
T is a minimum cost spanning tree (or an optimal spanning tree) for G if T
is a spanning tree for G, and given any spanning tree T 0 for G, c(T ) c(T 0 ).
Given a connected graph G = (V, E) with n vertices and m edges e1 , e2 , . . . , em ,
where c(ei ) = cost of edge ei , we want to find a minimum cost spanning
tree. It turns out (miraculously) that in this case, an obvious greedy algorithm (Kruskals algorithm) always works. Kruskals algorithm is the following: first, sort the edges in increasing (or rather nondecreasing) order of
costs, so that c(e1 ) c(e2 ) . . . c(em ); then, starting with an initially
empty tree T , go through the edges one at a time, putting an edge in T if it
will not cause a cycle, but throwing the edge out if it would cause a cycle.
4

Kruskals Algorithm:
Sort the edges so that: c(e1 ) c(e2 ) . . . c(em )
T
for i : 1..m
(*) if T {ei } has no cycle then
T T {ei }
end if
end for
But how do we test for a cycle (i.e. execute (*))? After each execution of
the loop, the set T of edges divides the vertices V into a collection V1 . . . Vk
of connected components. Thus V is the disjoint union of V1 . . . Vk , each Vi
forms a connected graph using edges from T , and no edge in T connects Vi
and Vj , if i 6= j.
A simple way to keep track of the connected components of T is to use an
array D[1..n] where D[i] = D[j] iff vertex i is in the same component as
vertex j. So our initialization becomes:
T
for i : 1..n
D[i] i
end for
To check whether ei = [r, s] forms a cycle with T , check whether D[r] = D[s].
If not, and we therefore want to add ei to T , we merge the components
containing r and s as follows:
k D[r]
l D[s]
for j : 1..n
if D[j] = l then
D[j] k
end if
end for
5

The complete program for Kruskals algorithm then becomes as follows:

Sort the edges so that: c(e1 ) c(e2 ) . . . c(em )
T
for i : 1..n
D[i] i
end for
for i : 1..m
Assign to r and s the endpoints of ei
if D[r] 6= D[s] then
T T {ei }
k D[r]
l D[s]
for j : 1..n
if D[j] = l then
D[j] k
end if
end for
end if
end for
We wish to analyze the running of Kruskals algorithm, in terms of n (the
number of vertices) and m (the number of edges);
keep in mind that n1 m


(since the graph is connected) and m n2 < n2 . Let us assume that the
graph is input as the sequence n, I1 , I2 , . . . , Im where n represents the vertex
set V = {1, 2, . . . , n}, and Ii is the information about edge ei , namely the
two endpoints and the cost associated with the edge. To analyze the running
time, lets assume that any two cost values can be either added or compared
in one step. The algorithm first sorts the m edges, and that takes O(m log m)
steps. Then it initializes D, which takes time O(n). Then it passes through
the m edges, checking for cycles each time and possibly merging components;
this takes O(m) steps, plus the time to do the merging. Each merge takes
O(n) steps, but note that the total number of merges is the total number
of edges in the final spanning tree T , namely (by the above lemma) n 1.
Therefore this version of Kruskals algorithm runs in time O(m log m + n2 ).
Alternatively, we can say it runs in time O(m2 ), and we can also say it runs

in time O(n2 log n). Since it is reasonable to view the size of the input as n,
this is a polynomial-time algorithm.
This running time can be improved to O(m log m) (equivalently O(m log n))
by using a more sophisticated data structure to keep track of the connected
components of T ; this is discussed on page 570 of CLRS (page 505 of CLR).
Correctness of Kruskals Algorithm
It is not immediately clear that Kruskals algorithm yields a spanning tree
at all, let alone a minimum cost spanning tree. We will now prove that it
does in fact produce an optimal spanning tree. To show this, we reason that
after each execution of the loop, the set T of edges can be expanded to an
optimal spanning tree using edges that have not yet been considered. Hence
after termination, since all edges have been considered, T must itself be a
minimum cost spanning tree.
We can formalize this reasoning as follows:
Definition 1 A set T of edges of G is promising after stage i if T can be expanded to a optimal spanning tree for G using edges from {ei+1 , ei+2 , . . . , em }.
That is, T is promising after stage i if there is an optimal spanning tree Topt
such that T Topt T {ei+1 , ei+2 , . . . , em }.
Lemma 4 For 0 i m, let Ti be the value of T after i stages, that is,
after examining edges e1 , . . . , ei . Then the following predicate P (i) holds for
every i, 0 i m:
P (i) : Ti is promising after stage i.
Proof:
We will prove this by induction. P (0) holds because T is initially empty.
Since the graph is connected, there exists some optimal spanning tree Topt ,
and
T0 Topt T0 {e1 , e2 , . . . , em }.
For the induction step, let 0 i < m, and assume P (i). We want to show
P (i + 1). Since Ti is promising for stage i, let Topt be an optimal spanning
7

tree such that

Ti Topt Ti {ei+1 , ei+2 , . . . , em }. If ei+1 is rejected, then Ti {ei+1 }
contains a cycle and Ti+1 = Ti . Since Ti Topt and Topt is acyclic, ei+1
/ Topt .
So
Ti+1 Topt Ti+1 {ei+2 , . . . , em }.
Now consider the case that Ti {ei+1 } does not contain a cycle, so we have
Ti+1 = Ti {ei+1 }. If ei+1 Topt , then we have Ti+1 Topt Ti+1
{ei+2 , . . . , em }.
So assume that ei+1
/ Topt . Then according to the Exchange Lemma below
(letting T1 be Topt and T2 be Ti+1 ), there is an edge ej Topt Ti+1 such
0
0

= Topt {ei+1 } {ej } is a spanning tree. Clearly Ti+1 Topt

that Topt
0
Ti+1 {ei+2 , . . . , em }. It remains to show that Topt is optimal. Since Topt
Ti {ei+1 , ei+2 , . . . , em } and ej Topt Ti+1 , we have j > i + 1. So (because
0
) = c(Topt ) + c(ei+1 ) c(ej )
we sorted the edges) c(ei+1 ) c(ej ), so c(Topt
0
0
) = c(Topt ), and Topt
c(Topt ). Since Topt is optimal, we must in fact have c(Topt
is optimal.
This completes the proof of the above lemma, except for the Exchange
Lemma.
Lemma 5 (Exchange Lemma) Let G be a connected graph, let T1 be any
spanning tree of G, and let T2 be be a set of edges not containing a cycle.
Then for every edge e T2 T1 there is an edge e0 T1 T2 such that
T1 {e} {e0 } is a spanning tree of G.
Proof:
Let T1 and T2 be as in the lemma, and let e T2 T1 . Say that e = [u, v].
Since there is a path from u to v in T1 , T1 {e} contains a cycle C, and it
is easy to see that C is the only cycle in T1 {e}. Since T2 is acyclic, there
must be an edge e0 on C that is not in T2 , and hence e0 T1 T2 . Removing
a single edge of C from T1 {e} leaves the resulting graph acyclic but still
connected, and hence a spanning tree. So T1 {e} {e0 } is a spanning tree
of G.
We have now proven Lemma 4. We therefore know that Tm is promising
after stage m; that is, there is an optimal spanning tree Topt such that Tm
Topt Tm = Tm , and so Tm = Topt . We can therefore state:
8

Theorem 1 Given any connected edge weighted graph G, Kruskals algorithm

outputs a minimum spanning tree for G.

Discussion of Greedy Algorithms

Before we give another example of a greedy algorithm, it is instructive to
give an overview of how these algorithms work, and how proofs of correctness
(when they exist) are constructed.
A Greedy algorithm often begins with sorting the input data in some way.
The algorithm then builds up a solution to the problem, one stage at a
time. At each stage, we have a partial solution to the original problem
dont think of these as solutions to subproblems (although sometimes they
are). At each stage we make some decision, usually to include or exclude
some particular element from our solution; we never backtrack or change our
mind. It is usually not hard to see that the algorithm eventually halts with
some solution to the problem. It is also usually not hard to argue about the
running time of the algorithm, and when it is hard to argue about the running
time it is because of issues involved in the data structures used rather than
with anything involving the greedy nature of the algorithm. The key issue
is whether or not the algorithm finds an optimal solution, that is, a solution
that minimizes or maximizes whatever quantity is supposed to be minimized
or maximized. We say a greedy algorithm is optimal if it is guaranteed to
find an optimal solution for every input.
Most greedy algorithms are not opitmal! The method we use to show that a
greedy algorithm is optimal (when it is) often proceeds as follows. At each
stage i, we define our partial solution to be promising if it can be extended to
an optimal solution by using elements that havent been considered yet by the
algorithm; that is, a partial solution is promising after stage i if there exists
an optimal solution that is consistent with all the decisions made through
stage i by our partial solution. We prove the algorithm is optimal by fixing
the input problem, and proving by induction on i 0 that after stage i is
performed, the partial solution obtained is promising. The base case of i = 0
is usually completely trivial: the partial solution after stage 0 is what we
start with, which is usually the empty partial solution, which of course can
be extended to an optimal solution. The hard part is always the induction
step, which we prove as follows. Say that stage i + 1 occurs, and that the
9

partial solution after stage i is Si and that the partial solution after stage
i + 1 is Si+1 , and we know that there is an optimal solution Sopt that extends
0
Si ; we want to prove that there is an optimal solution Sopt
that extends
Si+1 . Si+1 extends Si by making only one decision; if Sopt makes the same
0
decision, then it also extends Si+1 , and we can just let Sopt
= Sopt and we
are done. The hard part of the induction step is if Sopt does not extend Si+1 .
In this case, we have to show either that Sopt could not have been optimal
(implying that this case cannot happen), or we show how to change some
0
parts of Sopt to create a solution Sopt
such that
0
Sopt
extends Si+1 , and
0
has value (cost, profit, or whatever it is were measuring) at least
Sopt
0
is
as good as Sopt , so the fact that Sopt is optimal implies that Sopt
optimal.

For most greedy algorithms, when it ends, it has constructed a solution that
cannot be extended to any solution other than itself. Therefore, if we have
proven the above, we know that the solution constructed must be optimal.

10