MODULE 4

Dynamic Programming:
3 Basic Examples
Example 1: Coin-Row Problem
Problem Statement

• Given: A row of n coins with values c1, c2, ..., cn.

• Goal: Pick up the maximum amount of money such that no two adjacent coins
are picked up.

Solution Approach

• Use dynamic programming to solve this problem efficiently.

• Define F(i) as the maximum amount of money that can be picked up from the
first i coins.
• Recurrence relation: F(i)=max(F(i−1),ci+F(i−2))
o F(i-1): Maximum amount without picking the i-th coin.
o c_i + F(i-2): Maximum amount picking the i-th coin and adding it to the
solution of the first i-2 coins.

Algorithm

1. Initialize an array F of size n+1.

2. Set F(0) = 0 and F(1) = c1.
3. For each coin from 2 to n, calculate F(i).
4. Return F(n) as the maximum amount of money.

C Code Snippet
#include <stdio.h>
#include <stdlib.h>

// Function to find the maximum amount of money

int coinRow(int coins[], int n) {
if (n == 0) return 0;
if (n == 1) return coins[0];

int *F = (int *)malloc((n + 1) * sizeof(int));

F[0] = 0;
F[1] = coins[0];

for (int i = 2; i <= n; i++) {

 F[i] = (F[i-1] > (coins[i-1] + F[i-2])) ? F[i-1] : (coins[i-1] + F[i-
2]);
}

int result = F[n];

free(F);
return result;
}

int main() {
int coins[] = {5, 1, 2, 10, 6, 2};
int n = sizeof(coins) / sizeof(coins[0]);
printf("Maximum amount of money: %d\n", coinRow(coins, n));
return 0;
}

Example 2: Change-Making Problem

Problem Statement

• Given: An amount n and coin denominations d1 < d2 < ... < dm.
• Goal: Make change for n using the minimum number of coins.

Solution Approach

• Use dynamic programming to solve this problem.

• Define C(n) as the minimum number of coins needed to make change for
amount n.
• Recurrence relation

C(n)=min(C(n−dj))+1 for all j such that dj≤n

Algorithm

1. Initialize an array C of size n+1 with a large value.

2. Set C(0) = 0.
3. For each amount from 1 to n, compute C(i) by considering all denominations dj.
4. Return C(n) as the minimum number of coins

C Code Snippet:

#include <stdio.h>
#include <limits.h>
#define INF INT_MAX
// Function to find the minimum number of coins
int minCoins(int denominations[], int m, int n) {
int *C = (int *)malloc((n + 1) * sizeof(int));
 for (int i = 0; i <= n; i++) {
C[i] = INF;
}
C[0] = 0;

for (int i = 1; i <= n; i++) {

for (int j = 0; j < m; j++) {
if (denominations[j] <= i) {
int sub_res = C[i - denominations[j]];
if (sub_res != INF && sub_res + 1 < C[i]) {
C[i] = sub_res + 1;
}
}
}
}

int result = C[n];

free(C);
return result;
}

int main() {
int denominations[] = {1, 2, 5};
int m = sizeof(denominations) / sizeof(denominations[0]);
int n = 11;
printf("Minimum number of coins: %d\n", minCoins(denominations, m, n));
return 0;
}

Example 3: Coin-Collecting Problem

• Problem: Given an n x m board with coins placed in some cells, a robot starting
at the upper left cell (0,0) needs to collect the maximum number of coins and
reach the bottom right cell (n-1, m-1), moving only right or down.
• Approach: Dynamic programming.

Algorithm

1. Define F(i, j) as the maximum number of coins collected when reaching cell (i,
j).
2. Recurrence relation:

F(i,j)=max(F(i−1,j),F(i,j−1))+coins(i,j)

o If we come from the top (F(i-1, j)) or from the left (F(i, j-1)), add the
coin at cell (i, j) if it exists.
 3. Base case:

F(0,0)=coins(0,0)

C Code Snippet:

#include <stdio.h>
#include <stdlib.h>

int max(int a, int b) {

return (a > b) ? a : b;
}

int coin_collecting(int** board, int n, int m) {

int** F = (int**)malloc(n * sizeof(int*));
for (int i = 0; i < n; i++) {
F[i] = (int*)malloc(m * sizeof(int));
}

F[0][0] = board[0][0];

for (int i = 1; i < n; i++) {

F[i][0] = F[i-1][0] + board[i][0];
}
for (int j = 1; j < m; j++) {
F[0][j] = F[0][j-1] + board[0][j];
}

for (int i = 1; i < n; i++) {

for (int j = 1; j < m; j++) {
F[i][j] = max(F[i-1][j], F[i][j-1]) + board[i][j];
}
}
int result = F[n-1][m-1];
for (int i = 0; i < n; i++) {
free(F[i]);
}
free(F);
return result;
}
int main() {
int n = 3, m = 4;
int* board[] = {
(int[]){0, 1, 2, 3},
(int[]){1, 2, 0, 4},
(int[]){0, 6, 1, 2}
};
 printf("Maximum coins collected: %d\n", coin_collecting(board, n, m));
return 0;
}

The Knapsack Problem

Problem Statement

• Objective: Given n items with known weights w1, w2, ..., wn and values v1,
v2, ..., vn, and a knapsack with capacity W, find the most valuable subset of
items that fit into the knapsack.
• Assumptions: All weights and the knapsack capacity are positive integers. Item
values do not have to be integers.

Dynamic Programming Approach

To design a dynamic programming algorithm, derive a recurrence relation to express the

solution in terms of smaller subproblems.

1. Definitions:
o Let F(i, j) be the maximum value of a subset of the first i items that fit
into a knapsack of capacity j.
2. Recurrence Relation:
o Consider the subsets of the first i items:
1. Subsets that do not include the i-th item: The value of the
optimal subset is F(i-1, j).
2. Subsets that include the i-th item: If j - wi >= 0, the value is vi
+ F(i-1, j-wi).

3. Initial Conditions:

4. Goal: Find F(n, W), the maximum value of a subset of the n items that fit into the
knapsack of capacity W, and determine the composition of this optimal subset.

Example

Consider the following instance:

 • Items:
oItem 1: weight = 2, value = 12
oItem 2: weight = 1, value = 10
oItem 3: weight = 3, value = 20
oItem 4: weight = 2, value = 15
• Knapsack capacity: W = 5

Dynamic programming table:

0 1 2 3 4 5

0 0 0 0 0 0 0

1 0 0 12 12 12 12

2 0 10 12 22 22 22

3 0 10 12 22 30 32

4 0 10 15 25 30 37

Maximal value: F(4, 5) = 37

• Optimal subset: {item 1, item 2, item 4}

Memory Functions

Combines strengths of top-down and bottom-up approaches by maintaining a table for

already computed values.

1. ALGORITHM

#include <stdio.h>

#include <stdlib.h>

#define MAX(a, b) ((a) > (b) ? (a) : (b))

int** F;
int* Weights;
int* Values;
int n, W;

int MFKnapsack(int i, int j) {

if (F[i][j] < 0) {
if (j < Weights[i])
F[i][j] = MFKnapsack(i - 1, j);
else
F[i][j] = MAX(MFKnapsack(i - 1, j), Values[i] + MFKnapsack(i - 1,
j - Weights[i]));
 }
return F[i][j];
}

int main() {
n = 4;
W = 5;
Weights = (int[]) {0, 2, 1, 3, 2};
Values = (int[]) {0, 12, 10, 20, 15};

F = (int**) malloc((n + 1) * sizeof(int*));

for (int i = 0; i <= n; i++) {
F[i] = (int*) malloc((W + 1) * sizeof(int));
for (int j = 0; j <= W; j++) {
F[i][j] = (i == 0 || j == 0) ? 0 : -1;
}
}

printf("Maximum value: %d\n", MFKnapsack(n, W));

for (int i = 0; i <= n; i++) {

free(F[i]);
}
free(F);

return 0;
}

2. Efficiency:
o Time: O(nW)
o Space: O(nW)
3. Example: For the same instance as above, the table filled using memory functions
will compute only necessary subproblems, making the approach more efficient for
larger instances.

Summary

• Dynamic Programming: Solve the knapsack problem using a recurrence

relation, filling a table to find the maximum value subset.
• Memory Functions: Combine top-down and bottom-up approaches to optimize
the computation of necessary subproblems.
Floyd-Warshall Algorithm in C

#include <stdio.h>

// defining the number of vertices

#define nV 4

#define INF 999

void printMatrix(int matrix[][nV]);

// Implementing floyd warshall algorithm

void floydWarshall(int graph[][nV]) {
int matrix[nV][nV], i, j, k;

for (i = 0; i < nV; i++)

for (j = 0; j < nV; j++)
matrix[i][j] = graph[i][j];

// Adding vertices individually

for (k = 0; k < nV; k++) {
for (i = 0; i < nV; i++) {
for (j = 0; j < nV; j++) {
if (matrix[i][k] + matrix[k][j] < matrix[i][j])
matrix[i][j] = matrix[i][k] + matrix[k][j];
}}}
printMatrix(matrix);
}

void printMatrix(int matrix[][nV]) {

for (int i = 0; i < nV; i++) {
for (int j = 0; j < nV; j++) {
if (matrix[i][j] == INF)
printf("%4s", "INF");
else
printf("%4d", matrix[i][j]);
}
printf("\n");
}}

int main() {
int graph[nV][nV] = {{0, 3, INF, 5},
{2, 0, INF, 4},
{INF, 1, 0, INF},
{INF, INF, 2, 0}};
floydWarshall(graph);
}
 Design & Analysis
A of Algorithms | Module 4: Dynamic Programming

2. Transitive Closure using Warshall’s Algorithm,

Definition: The transitive closure of a directed graph with n vertices can be defined as the n
× n boolean matrix T = {tij }, in which the element in the ith row and the jth column is 1 if
there exists a nontrivial path (i.e., directed path of a positive length) from the ith vertex to the
jth vertex; otherwise, tij is 0.
Example: An example of a digraph, its adjacency matrix, and its transitive closure is given
below.

(a) Digraph. (b) Its adjacency matrix. (c) Its transitive closure.

We can generate the transitive closure of a digraph with the help of depthfirst search or
first search. Performing either traversal starting at the ith vertex gives the information
breadth-first
about the vertices reachable from it and hence the columns that contain 1’s in the ith row of
the transitive closure. Thus, doing such a traversal for every vertex as a starting point yields
the transitive closure in its entirety.
Since this method traverses the same digraph several times, we can use a better algorithm
called Warshall’s algorithm.. Warshall’s algorithm constructs the transitive closure through
a series of n × n boolean matrices:

Each of these matrices provides certain information about directed paths in the digraph.
Specifically, the element in the ith row and jth column of matrix R(k) (i, j = 1, 2, . . . , n, k
= 0, 1, . . . , n) is equal to 1 if and only if there exists a directed path of a positive length from
the ith vertex to the jth vertex with each intermediate vertex, if any, numbered not higher than
k.
Thus, the series starts with R(0) , which does not allow any intermediate vertices in its paths;
hence, R(0) is nothing other than the adjacency matrix of the digraph.
digraph R(1) contains the
information about paths thatat can use the first vertex as intermediate. The last matrix in the
(n)
series, R , reflects paths that can use all n vertices of the digraph as intermediate and hence
is nothing other than the digraph’s transitive closure.
This means that there exists a path from the ith vertex vi to the jth vertex vj with each
intermediate vertex numbered not higher than k:
vi, a list of intermediate vertices each numbered not higher than k, vj . --- (*)
Two situations regarding this path are possible.
 Design & Analysis
A of Algorithms | Module 4: Dynamic Programming

1. In the first, the list of its intermediate vertices does not contain the kth vertex. Then this
−1. i.e. r
path from vi to vj has intermediate vertices numbered not higher than k− 1
2. The secondd possibility is that path (*)(* does contain the kth vertex vk among the
intermediate vertices. Then path (*) can be rewritten as;
vi, vertices numbered ≤ k − 1, vk, vertices numbered ≤ k − 1, vj .

i.e r 1 and r 1

Thus, we have the following formula for generating the elements of matrix R(k) from the
elements of matrix R(k−1)

The Warshall’ss algorithm works based on the above formula.

As an example, the application of Warshall’s algorithm to the digraph is shown below. New
1’s are in bold.
 Design & Analysis
A of Algorithms | Module 4: Dynamic Programming

Analysis
Its time efficiency is Θ(n3). We can make the algorithm to run faster by treating matrix rows
as bit strings and employ the bitwise or operation available in most modern computer
languages.
Space efficiency: Although separate matrices for recording intermediate results of the
algorithm are used, that can be avoided.

3. All Pairs Shortest Paths using Floyd's Algorithm,

Problem definition: Given a weighted connected graph (undirected or directed), the all-pairs
all
shortest paths problem asks to find the distances—i.e.,
distances the lengths of the shortest paths - from
each vertex to all other vertices.
Applications: Solution to this problem finds applications in communications, transportation
networks,
tworks, and operations research. Among recent applications of the all--pairs shortest-path
problem is pre-computing
computing distances for motion planning in computer games.
We store the lengths of shortest paths in an n x n matrix D called the distance matrix: the
element dij in the ith row and the jth column of this matrix indicates the length of the shortest
path from the ith vertex to the jth vertex.

(a) Digraph. (b) Its weight matrix. (c) Its distance matrix
We can generate the distance matrix with an algorithm that is very similar to Warshall’s
algorithm. It is called Floyd’s algorithm.
Floyd’s algorithm computes the distance matrix of a weighted graph with n vertices through a
series of n × n matrices:
 Design & Analysis
A of Algorithms | Module 4: Dynamic Programming

The element in the ith row and the jth column of matrix D(k) (i, j = 1, 2, . . . , n, k = 0, 1,
he length of the shortest path among all paths from the i vertex to the jth
. . . , n) is equal to the th

vertex with each intermediate vertex, if any, numbered not higher than k.
As in Warshall’s algorithm, we can compute all the elements of each matrix D(k) from its
immediate predecessor D(k−1)

If 1,, then it means that there is a path;

vi, a list of intermediate vertices each numbered not higher than k, vj .

We can partition all such paths into two disjoint subsets: those that do not use the kth vertex vk
as intermediate and those that do.
i. Since the paths of the first subset have their intermediate vertices numbered not higher
than k − 1, the shortest of them is, by definition of our matrices, of length
ii. In the second subset the paths are of the form
vi, vertices numbered ≤ k − 1, vk, vertices numbered ≤ k − 1, vj .

The situation is depicted symbolically in Figure,

Figure which shows
the underlying
nderlying idea of Floyd’s algorithm.

Taking into account the lengths of the shortest paths in both subsets leads to the following
recurrence:

Analysis: Its time efficiency is Θ(n3), similar to the warshall’s algorithm.

 Design & Analysis
A of Algorithms | Module 4: Dynamic Programming

Application of Floyd’s algorithm to the digraph is shown below.. Updated elements are shown
in bold.

4. Optimal Binary Search Trees

A binary search tree is one of the most important data structures in computer science. One of
its principal applications is to implement a dictionary, a set of elements with the operations of
searching, insertion, and deletion.
If probabilities of searching for elements of a set are known e.g., from accumulated data
about past searches it is natural to pose a question about an optimal binary search tree for
which the average number of comparisons in a search is the smallest possible.
As an example, consider four keys A, B, C, and D
to be searched for with probabilities 0.1, 0.2, 0.4,
and 0.3, respectively. The figure depicts two out of
14 possible binary search trees containing these
keys.
2. Minimum cost spannin
anning trees
Definition: A spanning tree ee of a connected graph is its connected acycl
clic subgraph (i.e., a
tree) that contains all the ver
ertices of the graph. A minimum spanningg tree t of a weighted
connected graph is its spanni ning tree of the smallest weight, where thee weight
w of a tree is
defined as the sum of the weig
eights on all its edges. The minimum spannin
ning tree problem is
the problem of finding a minim
nimum spanning tree for a given weighted conn
nnected graph.

2.1. Prim’s Algorithm

Prim's algorithm constructs a minimum spanning tree through a sequence ce of expanding sub-
trees. The initial subtree in such
s a sequence consists of a single vertexx selected arbitrarily
from the set V of the graph'sh's vertices. On each iteration it expands thee current tree in the
greedy manner by simply atta ttaching to it the nearest vertex not in that tree.
tre (By the nearest
vertex, we mean a vertex not ot in the tree connected to a vertex in the tree
ee by an edge of the
smallest weight. Ties can be broken arbitrarily.) The algorithm stops after af all the graph's
vertices have been included in the tree being constructed. Since the algori rithm expands a tree
by exactly one vertex on eachach of its iterations, the total number of such
ch iterations is n - 1,
where n is the number of vertices
v in the graph. The tree generated by b the algorithm is
obtained as the set of edges.
Correctness
Prim’s algorithm always yield
lds a minimum spanning tree.

Example: An example of pri rim’s algorithm is shown below.

The parenthesized labels off a vertex in the middle column
indicate the nearest tree vert
ertex and edge weight; selected
vertices and edges are shownn in
i bold.

Tree vertices Remaining vertices Illustratio

tion
Analysis of Efficiency
The efficiency of Prim’s algor
orithm depends on the data structures chosenn for
f the graph itself
and for the priority queue of the set V − VT whose vertex priorities aree the
t distances to the
nearest tree vertices.
1. If a graph is represente
nted by its weight matrix and the priority queueue is implemented
as an unordered arra Θ 2). Indeed, on
ray, the algorithm’s running time will be in Θ(|V|
each of the |V| − 1itera
erations, the array implementing the priority queue
qu is traversed to
find and delete the minimum
m and then to update, if necessary,, the
th priorities of the
remaining vertices.
We can implement the priorit
rity queue as a min-heap. (A min-heap is a complete
co binary tree
in which every element is less
ess than or equal to its children.) Deletion of the
th smallest element
from and insertion of a new element
el into a min-heap of size n are O(log n) operations.
2. If a graph is represente
nted by its adjacency lists and the priority que
ueue is implemented
as a min-heap, the run
unning time of the algorithm is in O(|E| log |V
V |).
This is because the algorithmm performs |V| − 1 deletions of the smallestt element
e and makes
|E| verifications and, possibly
bly, changes of an element’s priority in a min-heap
mi of size not
exceeding |V|. Each of thesee operations,
o as noted earlier, is a O(log |V|) operation.
op Hence, the
running time of this implemenentation of Prim’s algorithm is in
(|V| − 1+ |E|) O (log |V |) = O(|E| log |V |) because, in a connected grap
aph, |V| − 1≤ |E|.

2.2. Kruskal’s Algorithm

Background
Kruskal's algorithm is another
er greedy algorithm for the minimum spanning ing tree problem that
also always yields an optimal
al solution. It is named Kruskal's algorithm, afte
fter Joseph Kruskal.
Kruskal's algorithm looks att a minimum spanning tree for a weighted connected
con graph G =
(V, E) as an acyclic sub graph
aph with |V | - 1 edges for which the sum of the
th edge weights is
the smallest. Consequently, y, the algorithm constructs a minimum spanning
spa tree as an
expanding sequence of subb graphs, which are always acyclic but are a not necessarily
connected on the intermediate
te stages of the algorithm.

Working
The algorithm begins by sortirting the graph's edges in non decreasing orde rder of their weights.
Then, starting with the empty ty sub graph, it scans this sorted list adding the
th next edge on the
list to the current sub graph if such an inclusion does not create a cycle and
an simply skipping
the edge otherwise.
The fact that ET ,the set of edges
edg composing a minimum spanning tree of graph G actually a
tree in Prim's algorithm but generally
ge just an acyclic sub graph in Kruskal's
l's algorithm.

Kruskal’s algorithm is not simpler

sim because it has to check whether the addition
a of the next
edge to the edges already selec
lected would create a cycle.

We can consider the algorith rithm's operations as a progression through a series of forests
containing all the vertices off a given graph and some of its edges. The initia
itial forest consists of
|V| trivial trees, each comprisi
ising a single vertex of the graph. The finall forest
f consists of a
single tree, which is a min inimum spanning tree of the graph. On each e iteration, the
algorithm takes the next edge ge (u, v) from the sorted list of the graph's edges,
ed finds the trees
containing the vertices u andd v, and, if these trees are not the same, uniteites them in a larger
tree by adding the edge (u, v).).

Analysis of Efficiency
The crucial check whether two
wo vertices belong to the same tree can be foun
und out using union-
find algorithms.
Efficiency of Kruskal’s algori
orithm is based on the time needed for sorting ng the edge weights
of a given graph. Hence, withith an efficient sorting algorithm, the time effic
fficiency of Kruskal's
algorithm will be in O (|E| log
og |E|).
Illustration
An example of Kruskal’s alg lgorithm is shown below. The
selected edges are shown in bold.
bo
3. Single source shortest
rtest paths
Single-source shortest-pathss problem is defined as follows. For a givenen vertex called the
source in a weighted connect
ected graph, the problem is to find shortest paths
pa to all its other
vertices. The single-source shortest-paths
sh problem asks for a family off paths,
p each leading
from the source to a different
ent vertex in the graph, though some paths may,
ma of course, have
edges in common.
3.1. Dijkstra's Algorithm
Dijkstra's Algorithm is the
he best-known algorithm for the single-sou ource shortest-paths
problem. This algorithm is applicable
ap to undirected and directed graphs
hs with nonnegative
weights only.
Working - Dijkstra's algorithm
ithm finds the shortest paths to a graph's vertic
tices in order of their
distance from a given source.
e.
First, it finds the shor
ortest path from the source to a vertex neare
arest to it, then to a
second nearest, and so on.
In general, before its i ith iteration commences, the
algorithm has alreadyy identified the shortest paths to i-1
other vertices nearest
st to the source. These vertices, the
source, and the edgess of
o the shortest paths leading to them
from the source form rm a subtree Ti of the given graph
shown in the figure.
Since all the edge wei
eights are nonnegative, the next vertex nearesrest to the source can
be found among the vertices
ve adjacent to the vertices of Ti. The sett of
o vertices adjacent
to the vertices in Ti can
c be referred to as "fringe vertices"; theyy are the candidates
from which Dijkstra's
's algorithm
a selects the next vertex nearest to the
th source.
To identify the ith nea
earest vertex, the algorithm computes, for eve very fringe vertex u,
ce to the nearest tree vertex v (given by the weight
the sum of the distance we of the edge (v,
u)) and the length d., of
o the shortest path from the source to v (prev
reviously determined
by the algorithm) andd then
t selects the vertex with the smallest such
ch sum. The fact that
it suffices to compare
are the lengths of such special paths is the he central insight of
Dijkstra's algorithm.
To facilitate the algorit
rithm's operations, we label each vertex with two
tw labels.
o The numeric labebel d indicates the length of the shortest pathth from the source to
this vertex foundd by the algorithm so far; when a vertex is added
a to the tree, d
indicates the lengt
gth of the shortest path from the source to that
at vertex.
o The other label indicates
in the name of the next-to-last vertex ono such a path, i.e.,
the parent of thee vertex
v in the tree being constructed. (It cann be left unspecified
for the source s and
an vertices that are adjacent to none of the current
cur tree vertices.)
 With such labeling,, finding
f the next nearest vertex u* becomes es a simple task of
finding a fringe vertex
ex with the smallest d value. Ties can be broken
en arbitrarily.
After we have identifie
ified a vertex u* to be added to the tree, we need
ne to perform two
operations:
o Move u* from m the fringe to the set of tree vertices.
o For each remaaining fringe vertex u that is connected to u* by an edge of
weight w (u*,, u)
u such that d u*+ w(u*, u) <d u, update the labels of u by u*
and du* + w(u*
u*, u), respectively.
o
Illustration: An example of Dijkstra's algorithm is shown
below. The next closest vertex
tex is shown in bold.

The shortest paths (identified

ed by following nonnumeric labels backward rd from a destination
vertex in the left column to the
th source) and their lengths (given by numeri
eric labels of the tree
vertices) are as follows:
The pseudocode of Dijkstra tra’s algorithm is given below. Note that at in the following
pseudocode, VT contains a giv iven source vertex and the fringe contains thee vertices
v adjacent to
it after iteration 0 is completed
ted.

Analysis:
The time efficiency of Dijk ijkstra’s algorithm depends on the data structures
st used for
implementing the priority queue
qu and for representing an input graphh itself. For graphs
represented by their adjacency
cy lists and the priority queue implemented as a min-heap, it is in
O ( |E| log |V| )
Applications
Transportation plannin
ing and packet routing in communication netwtworks, including the
Internet
Finding shortest paths
hs in social networks, speech recognition, doc
ocument formatting,
robotics, compilers, and
an airline crew scheduling.
4. Optimal Tree problem
blem
Background
Suppose we have to encode a text
t that comprises characters from some n-ccharacter alphabet
by assigning to each of the text's th codeword.There
tex characters some sequence of bits called the
are two types of encoding: Fix
ixed-length encoding, Variable-length encodin
ing
Fixed-length encoding: This
is method assigns to each character a bit string
ng of the same length
m (m >= log2 n). This is exac
actly what the standard ASCII code does. Onne way of getting a
coding scheme that yields a shorter bit string on the average is basedd on the old idea of
assigning shorter code-words
rds to more frequent characters and longerr code-words
c to less
frequent characters.
Variable-length encoding: This Th method assigns code-words of differentt lengths
l to different
characters, introduces a proble
blem that fixed-length encoding does not have
ve. Namely, how can
we tell how many bits of an encoded text represent the first (or, more re generally, the ith)
character? To avoid this compplication, we can limit ourselves to prefix-free
ee (or simply prefix)
codes. In a prefix code, no codeword
co is a prefix of a codeword of anothe
her character. Hence,
with such an encoding, we can simply scan a bit string until we get the firs
irst group of bits that
is a codeword for some character,
cha replace these bits by this characte
cter, and repeat this
operation until the bit string's
's end
e is reached.
If we want to create a binaryary prefix code for some alphabet, it is natur tural to associate the
alphabet's characters with leav
eaves of a binary tree in which all the left edge
ges are labelled by 0
and all the right edges are labe
abelled by 1 (or vice versa). The codeword off a character can then
be obtained by recording thee labels on the simple path from the root too the t character's leaf.
Since there is no simple path th to a leaf that continues to another leaf, noo codeword
c can be a
prefix of another codeword; hence,
he any such tree yields a prefix code.
Among the many trees thatt can c be constructed in this manner for a given
giv alphabet with
known frequencies of the character
ch occurrences, construction of such
ch a tree that would
assign shorter bit strings too high-frequency characters and longer ones es to low-frequency
characters can be done by thee following greedy algorithm, invented by Dav
avid Huffman.
4.1 Huffman Trees and Codes
Huffman's Algorithm
Step 1: Initialize n one-nodee trees
t and label them with the characters of the th alphabet. Record
cter in its tree's root to indicate the tree's weigh
the frequency of each characte ght. (More generally,
the weight of a tree will be equal
equ to the sum of the frequencies in the tree's 's leaves.)
Step 2: Repeat the followingg operation
o until a single tree is obtained. Find
nd two trees with the
smallest weight. Make them the t left and right subtree of a new tree and nd record the sum of
their weights in the root of the
he new tree as its weight.
ove algorithm is called a Huffman tree. It def
A tree constructed by the abov efines-in the manner
described-a Huffman code.
Example: Consider the five-ssymbol alphabet {A, B, C, D, _} with the following
fol occurrence
frequencies in a text made upp of
o these symbols:

The Huffman tree construction

ion for the above problem is shown below:

The resulting codewords are as

a follows:

Hence, DAD is encoded as 011101,

01 and 10011011011101 is decoded as BAD_AD.
BA
With the occurrence frequen
encies given and the codeword lengths obta
btained, the average
number of bits per symboll in this code is
2 * 0.35 + 3 * 0.1+ 2 * 0.2 + 2 * 0.2 + 3 * 0.15 = 2.25.