Data Structures and Algorithms

Algorithm: Outline, the essence of a

computational procedure, step-by-step
instructions
Program: an implementation of an
algorithm in some programming language
Data structure: Organization of data
needed to solve the problem

Algorithmic problem
Specification
of input

Specification
of output as
a function of
input

Infinite number of input instances satisfying the

specification. For eg: A sorted, non-decreasing
sequence of natural numbers of non-zero, finite
length:
1, 20, 908, 909, 100000, 1000000000.
3.

Algorithmic Solution
Input instance,
adhering to
the
specification

Algorithm

Output
related to
the input as
required

Algorithm describes actions on the input instance

Infinitely many correct algorithms for the same
algorithmic problem

What is a Good Algorithm?

Efficient:
Running time
Space used

Efficiency as a function of input size:

The number of bits in an input number
Number of data elements (numbers, points)

Measuring the Running Time

t (ms)
60

How should we measure the

running time of an algorithm?

50
40
30
20
10

Experimental Study

n
50

100

Write a program that implements the algorithm

Run the program with data sets of varying size
and composition.
Use a method like System.currentTimeMillis() to
get an accurate measure of the actual running
time.

Limitations of Experimental Studies

It is necessary to implement and test the
algorithm in order to determine its running time.
Experiments can be done only on a limited set of
inputs, and may not be indicative of the running
time on other inputs not included in the
experiment.
In order to compare two algorithms, the same
hardware and software environments should be
used.

Beyond Experimental Studies

We will develop a general methodology for
analyzing running time of algorithms. This
approach
Uses a high-level description of the algorithm
instead of testing one of its implementations.
Takes into account all possible inputs.
Allows one to evaluate the efficiency of any
algorithm in a way that is independent of the
hardware and software environment.

Pseudo-Code
A mixture of natural language and high-level
programming concepts that describes the main
ideas behind a generic implementation of a data
structure or algorithm.
Eg: Algorithm arrayMax(A, n):
Input: An array A storing n integers.
Output: The maximum element in A.
currentMax A[0]
for i 1 to n-1 do
if currentMax < A[i] then currentMax A[i]
return currentMax

Pseudo-Code
It is more structured than usual prose but
less formal than a programming language
Expressions:
use standard mathematical symbols to
describe numeric and boolean expressions
use for assignment (= in Java)
use = for the equality relationship (== in
Java)

Method Declarations:
Algorithm name(param1, param2)

Pseudo Code
Programming Constructs:
decision structures: if ... then ... [else ... ]
while-loops: while ... do
repeat-loops: repeat ... until ...
for-loop: for ... do
array indexing: A[i], A[i,j]

Methods:
calls: object method(args)
returns: return value

Analysis of Algorithms
Primitive Operation: Low-level operation
independent of programming language.
Can be identified in pseudo-code. For eg:
Data movement (assign)
Control (branch, subroutine call, return)
arithmetic an logical operations (e.g. addition,
comparison)

By inspecting the pseudo-code, we can

count the number of primitive operations
executed by an algorithm.

Example: Sorting
OUTPUT

INPUT

a permutation of the
sequence of numbers

sequence of numbers

a1, a2, a3,.,an

2

10

b1,b2,b3,.,bn

Sort
2

Correctness
Correctness(requirements
(requirementsfor
forthe
the
output)
output)
For
Forany
anygiven
giveninput
inputthe
thealgorithm
algorithm
halts
haltswith
withthe
theoutput:
output:
bb1 <<bb2 <<bb3 <<.
.<< bbnn
1
2
3
bb1,,bb2,,bb3,,.,
b is a
1
2
3 ., bnn is a
permutation
permutationof
ofaa1,,aa2,,aa3,.,a
,.,an
1

10

Running
Runningtime
time
Depends
Dependson
on
number
numberof
ofelements
elements(n)
(n)
how
how(partially)
(partially)sorted
sorted
they
theyare
are
algorithm
algorithm

Insertion Sort
A

2
j

5 1
n

i
Strategy
Strategy
Start
Startempty
emptyhanded
handed
Insert
Insertaacard
cardininthe
theright
right
position
positionof
ofthe
thealready
alreadysorted
sorted
hand
hand
Continue
Continueuntil
untilall
allcards
cardsare
are
inserted/sorted
inserted/sorted

INPUT:
INPUT:A[1..n]
A[1..n]an
anarray
arrayof
ofintegers
integers
OUTPUT:
OUTPUT:aapermutation
permutationof
ofAAsuch
suchthat
that
A[1]
A[2]
A[n]
A[1] A[2] A[n]
for
for j2
j2 to
to nn do
do
key
key
A[j]
A[j]
Insert
InsertA[j]
A[j]into
intothe
thesorted
sortedsequence
sequence
A[1..j-1]
A[1..j-1]
ij-1
ij-1
while
while i>0
i>0 and
and A[i]>key
A[i]>key
do
do A[i+1]A[i]
A[i+1]A[i]
i-i-A[i+1]key
A[i+1]key

Analysis of Insertion Sort

for j2 to n do
keyA[j]
Insert A[j] into the sorted
sequence A[1..j-1]
ij-1
while i>0 and A[i]>key
do A[i+1]A[i]
i-A[i+1] key

cost
c1
c2
0
c3
c4
c5
c6
c7

times
n
n-1
n-1
n-1
n

j =2 j
n
j =2
n
j =2

(t j 1)
(t j 1)

n-1

Total time = n(c1+c2+c3+c7) + nj=2 tj (c4+c5+c6)

(c2+c3+c5+c6+c7)

Best/Worst/Average Case
Total time = n(c1+c2+c3+c7) + nj=2 tj (c4+c5+c6)
(c2+c3+c5+c6+c7)

Best case: elements already sorted; tj=1,

running time = f(n), i.e., linear time.
Worst case: elements are sorted in
inverse order; tj=j, running time = f(n2), i.e.,
quadratic time
Average case: tj=j/2, running time = f(n2),
i.e., quadratic time

Best/Worst/Average Case (2)

For a specific size of input n, investigate
running times for different input instances:

Best/Worst/Average Case (3)

For inputs of all sizes:
worst-case
average-case

Running time

6n
5n

best-case

4n
3n
2n
1n
1 2 3 4 5

6 7 8

9 10 11 12 ..

Input instance size

Best/Worst/Average Case (4)

Worst case is usually used: It is an upperbound and in certain application domains (e.g.,
air traffic control, surgery) knowing the worstcase time complexity is of crucial importance
For some algorithms worst case occurs fairly
often
Average case is often as bad as the worst
case
Finding average case can be very difficult

Asymptotic Analysis
Goal: to simplify analysis of running time by
getting rid of details, which may be affected by
specific implementation and hardware
like rounding: 1,000,001 1,000,000
3n2 n2

Capturing the essence: how the running time of

an algorithm increases with the size of the input
in the limit.
Asymptotically more efficient algorithms are best for
all but small inputs

Asymptotic Notation
The big-Oh O-Notation

Used for worst-case

analysis

c g(n)
f (n)

Running Time

asymptotic upper bound

f(n) = O(g(n)), if there
exists constants c and
n0, s.t. f(n) c g(n) for n
n0
f(n) and g(n) are
functions over nonnegative integers

n0

Input Size

Example
For functions f(n) and g(n) there are positive
constants c and n0 such that: f(n) c g(n) for n n0
f(n) = 2n + 6

conclusion:
2n+6 is O(n).

c g(n) = 4n

g(n) = n
n

Another Example

On the other hand

n2 is not O(n) because there is
no c and n0 such that:
n2 cn for n n0
The graph to the right
illustrates that no matter how
large a c is chosen there is an n
big enough that n2 > cn ) .

Asymptotic Notation
Simple Rule: Drop lower order terms and
constant factors.
50 n log n is O(n log n)
7n - 3 is O(n)
8n2 log n + 5n2 + n is O(n2 log n)

Note: Even though (50 n log n) is O(n5), it

is expected that such an approximation be
of as small an order as possible

Asymptotic Analysis of Running Time

Use O-notation to express number of primitive
operations executed as function of input size.
Comparing asymptotic running times
an algorithm that runs in O(n) time is better
than one that runs in O(n2) time
similarly, O(log n) is better than O(n)
hierarchy of functions: log n < n < n2 < n3 < 2n
Caution! Beware of very large constant factors.
An algorithm running in time 1,000,000 n is still
O(n) but might be less efficient than one running
in time 2n2, which is O(n2)

Example of Asymptotic Analysis

Algorithm prefixAverages1(X):
Input: An n-element array X of numbers.
Output: An n-element array A of numbers such that
A[i] is the average of elements X[0], ... , X[i].

for i 0 to n-1 do
a0
n iterations
for j 0 to i do
i iterations
a a + X[j]
1 step with
i=0,1,2...n-1
A[i] a/(i+1)
return array A
Analysis: running time is O(n2)

A Better Algorithm
Algorithm prefixAverages2(X):
Input: An n-element array X of numbers.
Output: An n-element array A of numbers such
that A[i] is the average of elements X[0], ... , X[i].

s0
for i 0 to n do
s s + X[i]
A[i] s/(i+1)

return array A
Analysis: Running time is O(n)

Asymptotic Notation (terminology)

Special classes of algorithms:
Logarithmic: O(log n)
Linear: O(n)
Quadratic: O(n2)
Polynomial: O(nk), k 1
Exponential: O(an), a > 1

Relatives of the Big-Oh

(f(n)): Big Omega -asymptotic lower bound
(f(n)): Big Theta -asymptotic tight bound

Asymptotic Notation
The big-Omega
Notation

Used to describe bestcase running times or

lower bounds of
algorithmic problems

E.g., lower-bound of
searching in an unsorted
array is (n).

f (n )
c g ( n)

Running Time

asymptotic lower bound

f(n) = (g(n)) if there exists
constants c and n0, s.t.
c g(n) f(n) for n n0

n0

Input Size

Asymptotic Notation
The big-Theta Notation
asymptotically tight bound
f(n) = (g(n)) if there exists
constants c1, c2, and n0, s.t.
c1 g(n) f(n) c2 g(n) for n
n0

f (n )

Running Time

f(n) = (g(n)) if and only if

f(n) = (g(n)) and f(n) =
(g(n))
O(f(n)) is often misused
instead of (f(n))

c 2 g (n )
c 1 g (n )

n0

Input Size

Asymptotic Notation
Two more asymptotic notations
"Little-Oh" notation f(n)=o(g(n))
non-tight analogue of Big-Oh
For every c, there should exist n0 , s.t. f(n)
c g(n) for n n0
Used for comparisons of running times.
If f(n)=o(g(n)), it is said that g(n)
dominates f(n).

"Little-omega" notation f(n)=(g(n))

non-tight analogue of Big-Omega

Asymptotic Notation
Analogy with real numbers
f(n) = O(g(n))
f(n) = (g(n))
f(n) = (g(n))
f(n) = o(g(n))
f(n) = (g(n))

fg
fg
f=g
f<g
f>g

Abuse of notation: f(n) = O(g(n)) actually

means f(n) O(g(n))

Comparison of Running Times

Running
Time

Maximum problem size (n)

1 second 1 minute

1 hour

400n

2500

150000

9000000

20n log n 4096

166666

7826087

2n2

707

5477

42426

n4

31

88

244

2n

19

25

31

Correctness of Algorithms
The algorithm is correct if for any legal
input it terminates and produces the
desired output.
Automatic proof of correctness is not
possible
But there are practical techniques and
rigorous formalisms that help to reason
about the correctness of algorithms

Partial and Total Correctness

Partial correctness
IF this point is reached,

Algorithm

Any legal input

THEN this is the desired output

Output

Total correctness
INDEED this point is reached, AND this is the desired output

Any legal input

Algorithm

Output

Assertions
To prove correctness we associate a number of
assertions (statements about the state of the
execution) with specific checkpoints in the
algorithm.
E.g., A[1], , A[k] form an increasing sequence

Preconditions assertions that must be valid

before the execution of an algorithm or a
subroutine
Postconditions assertions that must be valid
after the execution of an algorithm or a
subroutine

Loop Invariants
Invariants assertions that are valid any
time they are reached (many times during
the execution of an algorithm, e.g., in
loops)
We must show three things about loop
invariants:
Initialization it is true prior to the first
iteration
Maintenance if it is true before an iteration,
it remains true before the next iteration
Termination when loop terminates the
invariant gives a useful property to show the
correctness of the algorithm

Example of Loop Invariants (1)

Invariant: at the start of
each for loop, A[1j-1]
consists of elements
originally in A[1j-1] but
in sorted order

for
for jj 22 to
to length(A)
length(A)
do
do key
key A[j]
A[j]
ii j-1
j-1
while
while i>0
i>0 and
and A[i]>key
A[i]>key
do
do A[i+1]
A[i+1] A[i]
A[i]
i-i-A[i+1]
A[i+1] key
key

Example of Loop Invariants (2)

Invariant: at the start of
each for loop, A[1j-1]
consists of elements
originally in A[1j-1] but
in sorted order

for
for jj 22 to
to length(A)
length(A)
do
do key
key A[j]
A[j]
ii j-1
j-1
while
while i>0
i>0 and
and A[i]>key
A[i]>key
do
do A[i+1]
A[i+1] A[i]
A[i]
i-i-A[i+1]
A[i+1] key
key

Initialization: j = 2, the invariant trivially holds

because A[1] is a sorted array

Example of Loop Invariants (3)

Invariant: at the start of
each for loop, A[1j-1]
consists of elements
originally in A[1j-1] but
in sorted order

for
for jj 22 to
to length(A)
length(A)
do
do key
key A[j]
A[j]
ii j-1
j-1
while
while i>0
i>0 and
and A[i]>key
A[i]>key
do
do A[i+1]
A[i+1] A[i]
A[i]
i-i-A[i+1]
A[i+1] key
key

Maintenance: the inner while loop moves elements

A[j-1], A[j-2], , A[j-k] one position right without
changing their order. Then the former A[j] element is
inserted into k-th position so that A[k-1] A[k]
A[k+1].

A[1j-1] sorted + A[j] A[1j] sorted

Example of Loop Invariants (4)

Invariant: at the start of
each for loop, A[1j-1]
consists of elements
originally in A[1j-1] but
in sorted order

for
for jj 22 to
to length(A)
length(A)
do
do key
key A[j]
A[j]
ii j-1
j-1
while
while i>0
i>0 and
and A[i]>key
A[i]>key
do
do A[i+1]
A[i+1] A[i]
A[i]
i-i-A[i+1]
A[i+1] key
key

Termination: the loop terminates, when j=n+1. Then

the invariant states: A[1n] consists of elements
originally in A[1n] but in sorted order

Math You Need to Review

Properties of logarithms:
logb(xy) = logbx + logby
logb(x/y) = logbx - logby
logb xa = a logb x
logb a = logxa/logxb
Properties of exponentials:
a(b+c) = abac ; abc = (ab)c
ab /ac = a(b-c) ; b = aloga b

Floor:

x = the largest integer x

Ceiling: x = the smallest integer x

Math Review
Geometric progression
given an integer n0 and a real number 0< a 1
1 a n +1
a = 1 + a + a + ... + a =

1 a
i =0
n

geometric progressions exhibit exponential

growth

Arithmetic progression
n2 + n
i = 1 + 2 + 3 + ... + n =

2
i =0
n

Summations
The running time of insertion sort is
determined by a nested loop
for
for j2
j2 to
to length(A)
length(A)
keyA[j]
keyA[j]
ij-1
ij-1
while
while i>0
i>0 and
and A[i]>key
A[i]>key
A[i+1]A[i]
A[i+1]A[i]
ii-1
ii-1
A[i+1]key
A[i+1]key

Nested loops correspond to summations

2
(
j

1)
=
O
(
n
)
j =2
n

Proof by Induction
We want to show that property P is true for
all integers n n0
Basis: prove that P is true for n0
Inductive step: prove that if P is true for
all k such that n0 k n 1 then P is also
true for n
n
n(n + 1)
=
=
S
(
n
)
i
for n 1
Example

i =0
1

Basis

S (1) = i =
i =0

1(1 + 1)
2

Proof by Induction (2)

Inductive Step
k ( k + 1)
S (k ) = i =
for 1 k n 1
2
i =0
k

n 1

i =0

i=0

S ( n ) = i = i + n =S ( n 1) + n =
( n 1 + 1)
(n 2 n + 2n)
= ( n 1)
+n=
=
2
2
n ( n + 1)
=
2

Stacks
Abstract Data Types (ADTs)
Stacks
Interfaces and exceptions
Java implementation of a stack
Application to the analysis of a
time series
Growable stacks
Stacks in the Java virtual
machine

Abstract Data Types (ADTs)

ADT is a mathematically specified entity

that defines a set of its instances, with:
a

specific interface a collection of signatures

of operations that can be invoked on an
instance,
a set of axioms (preconditions and
postconditions) that define the semantics of the
operations (i.e., what the operations do to
instances of the ADT, but not how)

Abstract Data Types (ADTs)

Types of operations:
Constructors
Access functions
Manipulation procedures

Abstract Data Types

Why do we need to talk about ADTs in a DS

course?
They

serve as specifications of requirements for the

building blocks of solutions to algorithmic problems
Provides a language to talk on a higher level of
abstraction
ADTs encapsulate data structures and algorithms that
implement them
Separate the issues of correctness and efficiency

Example - Dynamic Sets

We will deal with ADTs, instances of which

are sets of some type of elements.
Operations

are provided that change the set

We call such class of ADTs dynamic sets

Dynamic Sets (2)

An example dynamic set ADT

Methods:

New():ADT
Insert(S:ADT, v:element):ADT
Delete(S:ADT, v:element):ADT
IsIn(S:ADT, v:element):boolean

and Delete manipulation operations

IsIn Access method method
Insert

Dynamic Sets (3)

Axioms that define the methods:

v) = false
IsIn(Insert(S, v), v) = true
IsIn(Insert(S, u), v) = IsIn(S, v), if v u
IsIn(Delete(S, v), v) = false
IsIn(Delete(S, u), v) = IsIn(S, v), if v u
IsIn(New(),

Other Examples

Simple ADTs:
Queue
Deque
Stack

Stacks
A stack is a container of objects that are
inserted and removed according to the
last-in-first-out (LIFO) principle.
Objects can be inserted at any time, but
only the last (the most-recently inserted)
object can be removed.
Inserting an item is known as pushing
onto the stack. Popping off the stack is
synonymous with removing an item.

Stacks (2)

A PEZ dispenser as an analogy:

Stacks(3)

A stack is an ADT that supports four main

methods:
Creates a new stack
push(S:ADT, o:element):ADT - Inserts object o onto
top of stack S
pop(S:ADT):ADT - Removes the top object of stack
S; if the stack is empty an error occurs
top(S:ADT):element Returns the top object of the
stack, without removing it; if the stack is empty an
error occurs
new():ADT

Stacks(4)

The following support methods could also

be defined:
size(S:ADT):integer

- Returns the number of

objects in stack S
isEmpty(S:ADT): boolean - Indicates if stack
S is empty

Axioms
v)) = S
Top(Push(S, v)) = v
Pop(Push(S,

Java Stuff
Given the stack ADT, we need to code the ADT in
order to use it in the programs. We need two
constructs: interfaces and exceptions.
An interface is a way to declare what a class is to
do. It does not mention how to do it.

For

an interface, we just write down the method

names and the parameters. When specifying
parameters, what really matters is their types.
Later, when we write a class for that interface, we
actually code the content of the methods.
Separating interface and implementation is a useful
programming technique.

A Stack Interface in Java

The stack data structure is a built-in class of Javas
java.util package. But we define our own stack interface:
public interface Stack {
// accessor methods
public int size();
public boolean isEmpty();
public Object top() throws StackEmptyException;
// update methods
public void push (Object element);
public Object pop() throws StackEmptyException;
}

Exceptions
Exceptions are yet another programming
construct, useful for handling errors. When we
find an error (or an exceptional case), we just
throw an exception.
As soon as the exception is thrown, the flow of
control exits from the current method and goes to
where that method was called from.
What is the point of using exceptions? We
can delegate upwards the responsibility of
handling an error. Delegating upwards means
letting the code who called the current code deal
with the problem.

More Exceptions
public void eatPizza() throws
StomachAcheException
{...
if (ateTooMuch)
So when
throw new
StomachAcheException(Ouch); StomachAcheException
...}
is thrown, we exit from
private void simulateMeeting()
method eatPizza() and
{...
go to TA.eatpizza().
try
{TA.eatPizza();}
catch (StomachAcheException e)
{system.out.println(somebody has a stomach ache);}
...}

Even More Exceptions

The try block and the catch block means that we
are listening for exceptions that are specified in
the catch parameter.
Because catch is listening for
StomachAcheException, the flow of control will
now go to the catch block. And
System.out.println will get executed.
Note that a catch block can contain anything. It
does not have to do only System.out.println. We
can handle the caught error in any way you like;
we can even throw them again.

Exceptions (finally)

Note that if somewhere in your method, you

throw an exception, you need to add a throws
clause next to your method name.
If you never catch an exception, it will propagate
upwards and upwards along the chain of
method calls until the user sees it.
What exactly are exceptions in Java? Classes.

public class StomachAcheException extends

RuntimeException {
public StomachAcheException (String err)
{super(err);}
}

Array-Based Stack in Java

Create a stack using an array by specifying a

maximum size N for our stack.
The stack consists of an N-element array S and
an integer variable t, the index of the top
element in array S.

Array indices start at 0, so we initialize t to -1

Array-Based Stack in Java (2)

public class ArrayStack implements Stack {
// Implementation of the Stack interface using an array
public static final int CAPACITY = 1024;
// default capacity of stack
private int N;
// maximum capacity of the stack
private Object S[ ];
// S holds the elements of the stack
private int t = -1;
// the top element of the stack
public ArrayStack( ) // Initialize the stack with default capacity
{ this(CAPACITY) }
public ArrayStack(int cap)
// Initialize the stack with given capacity
{N = cap; S = new Object[N]}

Array-Based Stack in Java (3)

public int size( )
{return (t + 1)}

//Return the current stack size

public boolean isEmpty( ) //Return true iff the stack is empty

{return (t < 0)}
public void push(Object obj) throws StackFullException{
//Push a new element on the stack
if (size() == N)
throw new StackFullException(Stack overflow.)
S[++t] = obj;}

Array-Based Stack in Java (4)

public Object top( ) throws StackEmptyException {
// Return the top stack element
if (isEmpty( ))
throw new StackEmptyException(Stack is empty.);
return S[t]}
public Object pop() throws StackEmptyException {
// Pop off the stack element
Object elem;
if (isEmpty( ))
throw new StackEmptyException(Stack is Empty.);
elem = S[t];
S[t--] = null;
// Dereference S[top] and decrement top
return elem}}

Array-Based Stack in Java (5)

The

array implementation is simple and

efficient (methods performed in O(1)).
There is an upper bound, N, on the size of
the stack. The arbitrary value N may be
too small for a given application, or a
waste of memory.
StackEmptyException is required by the
interface.
StackFullException is particular to this
implementation.

Application: Time Series

The span si of a stocks price on a certain day i is

the maximum number of consecutive days (up to
the current day) the price of the stock has been
less than or equal to its price on day i

An Inefficient Algorithm
Algorithm computeSpans1(P):
Input: an n-element array P of numbers such that P[i] is
the price of the stock on day i
Output: an n-element array S of numbers such that S[i]
is the span of the stock on day i
for i 0 to n - 1 do
k 0; done false
repeat
if P[i - k] P[i] then k k + 1
else done true
until (k = i) or done
S[i] k
return S

The running time of this algorithm is O(n2). Why?

A Stack Can Help

si

can be easily computed if we know the

closest day preceding i, on which the price is
greater than the price on day i. If such a day
exists, lets call it h(i), otherwise, we
conventionally define h(i) = -1

In the figure, h(3)=2,

h(5)=1 and h(6)=0.
The span is now
computed as si = i - h(i)

What to do with the Stack?

What are possible values of h(7)?
Can it be 1 or 3 or 4?
No, h(7) can only be 2 or 5 or 6.
1

We

store indices 2,5,6 in the stack.

To determine h(7) we compare the price on day 7 with
prices on day 6, day 5, day 2 in that order.
The

first price larger than the price on day 7 gives h(7)

The stack should be updated to reflect the price of day 7
It should now contains 2,5,7

An Efficient Algorithm
Algorithm computeSpans2(P):
Let D be an empty stack
for i 0 to n - 1 do
k 0; done false
while not (D.isEmpty() or done) do
if P[i] P[D.top()] then D.pop()
else done true
if D.isEmpty() then h -1
else h D.top()
S[i] i - h
D.push(i)
return S

A Growable Array-Based Stack

Instead of giving up with a StackFullException, we can

replace the array S with a larger one and continue
processing push operations.
Algorithm push(o)
if size() = N then A new array of length f(N)
for i 0 to N 1
A[i] S[i]
S A; t t + 1
S[t] o

How large should the new array be?

tight strategy (add a constant): f(N) = N + c
growth strategy (double up): f(N) = 2N

Tight vs. Growth Strategies:

a comparison
To compare the two strategies, we use the
following cost model:
A regular push operation: adds one
element and cost one unit.
A special push operation: create an array
of size f(N), copy N elements, and add one
element. Costs f(N)+N+1 units

Tight Strategy (c=4)

start with an array of size 0. cost of a special push is 2N + 5
a

4+1

a b

a b c

a b c d

Phase 1

a b c d e

8+4+1

a b c d e f

a b c d e f g

a b c d e f g h

a b c d e f g h i
a b c d e f g h i k
a b c d e f g h i k l
a b c d e f g h i k l m
a b c d e f g h i k l m n

Phase 2

12+8+1
1
1
1
16+12+1

Phase 3

Performance of the Tight Strategy

In phase i the array has size ci
Total cost of phase i is

ci

is the cost of creating the array

c(i-1) is the cost of copying elements into new array
c is the cost of the c pushes.

Hence, cost of phase i is 2ci

In each phase we do c pushes. Hence for n
pushes we need n/c phases. Total cost of these
n/c phases is
= 2c (1 + 2 + 3 + ... + n/c) t O(n2/c)

Growth Strategy
start with an array of size 0. cost of a special push is 3N + 1
a

1+0+1

a b

2+1+1

a b c

4+2+1

a b c d

Phase 0
Phase 1
Phase 2

a b c d e

8+4+1

a b c d e f

a b c d e f g

a b c d e f g h

Phase 3

a b c d e f g h i

16+8+1

a b c d e f g h i k

a b c d e f g h i k l

a b c d e f g h i k l m

1
1

a b c d e f g h i k l m n

Phase 4

Performance of the Growth Strategy

In phase i the array has size 2i
Total cost of phase i is

2i

is the cost of creating the array

2i-1 is the cost of copying elements into new array
2i-1 is the cost of the 2i-1 pushes done in this phase

Hence, cost of phase i is 2i+1.

If we do n pushes, we will have log n phases.
Total cost of n pushes
= 2 + 4 + 8 + ... + 2log n+1 = 4n 1
The growth strategy wins!

Stacks in the Java Virtual Machine

Each

process running in a Java program

has its own Java Method Stack.
Each time a method is called, it is pushed
onto the stack.
The choice of a stack for this operation
allows Java to do several useful things:
Perform

recursive method calls

Print stack traces to locate an error

Java Method Stack

Queues and Linked Lists

Queues
Linked Lists
Double-Ended Queues

Queues
A queue differs from a stack in that its insertion
and removal routines follows the first-in-first-out
(FIFO) principle.
Elements may be inserted at any time, but only
the element which has been in the queue the
longest may be removed.
Elements are inserted at the rear (enqueued) and
removed from the front (dequeued)

Front

Queue

Rear

Queues (2)

The queue supports three fundamental

methods:
New():ADT

Creates an empty queue

Enqueue(S:ADT, o:element):ADT - Inserts object o
at the rear of the queue
Dequeue(S:ADT):ADT - Removes the object from the
front of the queue; an error occurs if the queue is
empty
Front(S:ADT):element - Returns, but does not
remove, the front element; an error occurs if the
queue is empty

Queues (3)

These support methods should also be

defined:
Size(S:ADT):integer
IsEmpty(S:ADT):boolean

Axioms:
Front(Enqueue(New(),

v)) = v
Dequeque(Enqueue(New(), v)) = New()
Front(Enqueue(Enqueue(Q, w), v)) =
Front(Enqueue(Q, w))
Dequeue(Enqueue(Enqueue(Q, w), v)) =
Enqueue(Dequeue(Enqueue(Q, w)), v)

An Array Implementation
Create a queue using an array in a circular
fashion
A maximum size N is specified.
The queue consists of an N-element array
Q and two integer variables:

f,

index of the front element (head for

dequeue)
r, index of the element after the rear one (tail
for enqueue)

An Array Implementation (2)

wrapped around configuration

what does f=r mean?

An Array Implementation (3)

Pseudo code

Algorithm size()
return (N-f+r) mod N
Algorithm isEmpty()
return (f=r)

Algorithm dequeue()
if isEmpty() then
return Queueemptyexception
Q[f] null
f (f+1)modN

Algorithm front()
if isEmpty() then
Algorithm enqueue(o)
return Queueemptyexception
if size = N - 1 then
return Q[f]
return Queuefullexception
Q[r] o
r (r +1)modN

Implementing Queue with Linked List

Nodes (data, pointer) connected in a chain by

links

The head of the list is the front of the queue, the

tail of the list is the rear of the queue. Why not
the opposite?

Linked List Implementation

Dequeue - advance head reference

Inserting at the head is just as easy

Linked List Implementation (2)

Enqueue - create a new node at the tail

chain it and move the tail reference

How about removing at the tail?

Double-Ended Queue
A double-ended queue, or deque, supports
insertion and deletion from the front and back
The deque supports six fundamental methods

InsertFirst(S:ADT, o:element):ADT - Inserts e at the

beginning of deque
InsertLast(S:ADT, o:element):ADT - Inserts e at end of
deque
RemoveFirst(S:ADT):ADT Removes the first element
RemoveLast(S:ADT):ADT Removes the last element
First(S:ADT):element and Last(S:ADT):element
Returns the first and the last elements

Doubly Linked Lists

Deletions at the tail of a singly linked list cannot be
done in constant time
To implement a deque, we use a doubly linked
list

A node of a doubly linked list has a next and a

prev link
Then, all the methods of a deque have a constant
(that is, O(1)) running time.

Doubly Linked Lists (2)

When implementing a doubly linked lists, we add

two special nodes to the ends of the lists: the
header and trailer nodes
The header node goes before the first list element. It
has a valid next link but a null prev link.
The trailer node goes after the last element. It has a
valid prev reference but a null next reference.

The header and trailer nodes are sentinel or

dummy nodes because they do not store
elements

Stacks with Deques

Implementing ADTs using

implementations of other ADTs as building
blocks
Stack Method
size()
isEmpty()
top()
push(o)
pop()

Deque
Implementation
size()
isEmpty()
last()
insertLast(o)
removeLast()

Queues with Deques

Queue Method

Deque
Implementation

size()

size()

isEmpty()

isEmpty()

front()

first()

enqueue(o)

insertLast(o)

dequeue()

removeFirst()

The Adaptor Pattern

Using

a deque to implement a stack or queue is an

example of the adaptor pattern. Adaptor patterns
implement a class by using methods of another class
In general, adaptor classes specialize general
classes
Two such applications
Specialize

a general class by changing some methods eg

implementing a stack with a deque.
Specialize the types of objects used by a general class eg
defining an IntegerArrayStack class that adapts ArrayStack
to only store integers.

Circular Lists

No end and no beginning of the list, only one

pointer as an entry point
Circular doubly linked list with a sentinel is an
elegant implementation of a stack or a queue

Sequences
Vectors
Positions
Lists
General Sequences

The Vector ADT

A sequence S (with n elements) that supports the following
methods:
elemAtRank(r): Return the element of S with rank r; an
error occurs if r < 0 or r > n -1
replaceAtRank(r,e): Replace the element at rank r with e
and return the old element; an error condition occurs if
r < 0 or r > n - 1
insertAtRank(r,e): Insert a new element into S which will
have rank r; an error occurs if r< 0 or r > n
removeAtRank(r): Remove from S the element at rank r;
an error occurs if r < 0 or r > n - 1

Array-Based Implementation
Algorithm insertAtRank(r,e):
for i = n 1 downto r do
S[i+1] S[i]
S[r] e
nn+1

Algorithm removeAtRank(r):
e S[r]
for i = r to n - 2 do
S[i] S[i + 1]
nn-1
return

Array-Based Implementation (contd.)

Time complexity of the various methods:

Implem. with a Doubly Linked List

the list before insertion

creating a new node for

insertion:

the list after insertion

Java Implementation
public void insertAtRank (int rank, Object element)
throws BoundaryViolationException {
if (rank < 0 || rank > size())
throw new BoundaryViolationException(invalid rank);
DLNode next = nodeAtRank(rank);
// the new node will be right before this
DLNode prev = next.getPrev();
// the new node will be right after this
DLNode node = new DLNode(element, prev, next);
// new node knows about its next & prev.
// Now we tell next & prev about the new node.
next.setPrev(node);
prev.setNext(node);
size++;
}

Implementation with a Doubly Linked

List
the list before
deletion

deleting a node

after deletion

Java Implementation
public Object removeAtRank (int rank)
throws BoundaryViolationException {
if (rank < 0 || rank > size()-1)
throw new BoundaryViolationException(Invalid rank.);
DLNode node = nodeAtRank(rank); // node to be removed
DLNode next = node.getNext();
// node before it
DLNode prev = node.getPrev();
// node after it
prev.setNext(next);
next.setPrev(prev);
size--;
return node.getElement();
// returns the element of the deleted node
}

Java Implementation (contd.)

private DLNode nodeAtRank (int rank) {
//auxiliary method to find node of element with given rank
DLNode node;
if (rank <= size()/2) {
//scan forward from head
node = header.getNext();
for (int i=0; i < rank; i++)
node = node.getNext();
} else {
//scan backward from the tail
node = trailer.getPrev();
for (int i=0; i < size()-rank-1 ; i++)
node = node.getPrev();
}
return node;
}

Nodes
Linked lists support the efficient execution of
node-based operations:
removeAtNode(Node v) and
insertAfterNode(Node v, Object e), would be
O(1).
However, node-based operations are not
meaningful in an array-based implementation
because there are no nodes in an array.
Nodes are implementation-specific.
Dilemma:

If

we do not define node based operations, we are not

taking full advantage of doubly-linked lists.
If we do define them, we violate the generality of ADTs.

From Nodes to Positions

We

introduce the Position ADT

Intuitive notion of place of an element.
Positions have only one method:
element(): Returns the element at this
position
Positions are defined relative to other
positions (before/after relation)
Positions are not tied to an element or
rank

The List ADT

ADT with position-based methods
generic methods: size(), isEmpty()
query methods: isFirst(p), isLast(p)
accessor methods: first(), last(), before(p), after(p)
update methods
swapElements(p,q), replaceElement(p,e)
insertFirst(e), insertLast(e)
insertBefore(p,e), insertAfter(p,e)
remove(p)
each method takes O(1) time if implemented with
a doubly linked list

The Sequence ADT

Combines the Vector
and List ADT (multiple
inheritance)
Adds methods that
bridge between ranks
and positions

atRank(r)

returns a

position
rankOf(p) returns an
integer rank

An array-based implementation needs to use

objects to represent the positions

Comparison of Sequence
Implementations

Iterators
Abstraction of the process of scanning through a
collection of elements
Encapsulates the notions of place and next
Extends the position ADT
Generic and specialized iterators
ObjectIterator: hasNext(), nextObject(), object()
PositionIterator: nextPosition()
Useful methods that return iterators: elements(),
positions()

Dictionaries
the

dictionary ADT
binary search
Hashing

Dictionaries
Dictionaries

store elements so that they

can be located quickly using keys
A dictionary may hold bank accounts
each

account is an object that is identified by

an account number
each account stores a wealth of additional
information
including

the current balance,

the name and address of the account holder, and
the history of deposits and withdrawals performed
an

application wishing to operate on an

account would have to provide the account
number as a search key

The Dictionary ADT

A dictionary is an abstract model of a database
A dictionary stores key-element pairs
The main operation supported by a dictionary
is searching by key
simple container methods: size(), isEmpty(),
elements()
query methods: findElem(k), findAllElem(k)
update methods: insertItem(k,e), removeElem(k),
removeAllElem(k)
special element: NIL, returned by an
unsuccessful search

The Dictionary ADT

Supporting

order (methods min, max,

successor, predecessor ) is not required,
thus it is enough that keys are
comparable for equality

The Dictionary ADT

Different data structures to realize dictionaries
arrays, linked lists (inefficient)
Hash table (used in Java...)
Binary trees
Red/Black trees
AVL trees
B-trees
In Java:
java.util.Dictionary abstract class
java.util.Map interface

Searching
INPUT
sequence of numbers
(database)
a single number (query)
a1, a2, a3,.,an; q

OUTPUT
index of the found
number or NIL

10

7;

10

7;

NIL

Binary Search
Idea: Divide and conquer, a key design technique
narrow down the search range in stages
findElement(22)

A recursive procedure
Algorithm BinarySearch(A, k, low, high)
if low > high then return Nil
else mid (low+high) / 2
if k = A[mid] then return mid
elseif k < A[mid] then
return BinarySearch(A, k, low, mid-1)
else return BinarySearch(A, k, mid+1, high)
2

12

low
2

14

17

19

22

27

28

33

12

14

12

14

37
high

mid
17

19

22

low
2

25

17

25

27

28

33

high

mid

19

low mid

22

25

37

27

28

33

37

An iterative procedure
INPUT: A[1..n] a sorted (non-decreasing) array of
integers, key an integer.
OUTPUT: an index j such that A[j] = k.
NIL, if j (1 j n): A[j] k
2

low 1
high n
do
mid (low+high)/2
if A[mid] = k then return mid
else if A[mid] > k then high mid-1
else low mid+1
while low <= high
return NIL

12

low
2

14

17

19

22

25

27

28

33

high

mid
12

14

17

19

22

low
12

14

37

17

25

27

28

33

high

mid

19

low mid

22

25

37

27

28

33

37

Running Time of Binary Search

The range of candidate items to be searched is

halved after each comparison

In the array-based implementation, access by

rank takes O(1) time, thus binary search runs in
O(log n) time

Searching in an unsorted array

INPUT: A[1..n] an array of integers, q an integer.
OUTPUT: an index j such that A[j] = q. NIL, if j (1jn):
A[j] q
j1
while j n and A[j] q
do j++
if j n then return j
else return NIL

Worst-case running time: O(n), average-case:

O(n)
We cant do better. This is a lower bound for the
problem of searching in an arbitrary sequence.

The Problem
T&T is a large phone company, and they
want to provide caller ID capability:
given a phone number, return the callers
name
phone numbers range from 0 to r = 108 -1
There are n phone numbers, n << r.
want to do this as efficiently as possible

Using an unordered sequence

searching

and removing takes O(n) time

inserting takes O(1) time
applications to log files (frequent
insertions, rare searches and removals)

Using an ordered sequence

searching

takes O(log n) time (binary

search)
inserting and removing takes O(n) time
application to look-up tables (frequent
searches, rare insertions and removals)

Other Suboptimal ways

direct addressing: an array indexed by key:
takes O(1) time,
O(r) space where r is the range of numbers
(108)
huge amount of wasted space
(null)

(null)

Ankur

(null)

(null)

0000-0000 0000-0000 9635-8904 0000-0000 0000-0000

Another Solution
Can do better, with a Hash table -- O(1) expected
time, O(n+m) space, where m is table size
Like an array, but come up with a function to map
the large range into one which we can manage
e.g., take the original key, modulo the (relatively
small) size of the array, and use that as an index
Insert (9635-8904, Ankur) into a hashed array
with, say, five slots. 96358904 mod 5 = 4

(null)

(null)

(null)

(null)

Ankur

An Example
Let keys be entry nos of students in
CSL201. eg. 2004CS10110.
There are 100 students in the class. We
create a hash table of size, say 100.
Hash function is, say, last two digits.
Then 2004CS10110 goes to location 10.
Where does 2004CS50310 go?
Also to location 10. We have a collision!!

Collision Resolution
How to deal with two keys which hash to the
same spot in the array?
Use chaining
Set up an array of links (a table), indexed by
the keys, to lists of items with the same key

Most

efficient (time-wise) collision resolution

scheme

Collision resolution (2)

To find/insert/delete an element
using h, look up its position in table T
Search/insert/delete the element in the
linked list of the hashed slot

Analysis of Hashing
An

element with key k is stored in slot

h(k) (instead of slot k without hashing)
The hash function h maps the universe U
of keys into the slots of hash table
T[0...m-1]
h : U {0,1,..., m 1}
Assume

time to compute h(k) is (1)

Analysis of Hashing(2)
An good hash function is one which distributes
keys evenly amongst the slots.
An ideal hash function would pick a slot,
uniformly at random and hash the key to it.
However, this is not a hash function since we
would not know which slot to look into when
searching for a key.
For our analysis we will use this simple uniform
hash function
Given hash table T with m slots holding n
elements, the load factor is defined as =n/m

Analysis of Hashing(3)
Unsuccessful search
element is not in the linked list
Simple uniform hashing yields an average
list length = n/m
expected number of elements to be
examined
search time O(1+) (includes computing
the hash value)

Analysis of Hashing (4)

Successful search
assume that a new element is inserted at the
end of the linked list
upon insertion of the i-th element, the expected
length of the list is (i-1)/m
in case of a successful search, the expected
number of elements examined is 1 more that the
number of elements examined when the soughtfor element was inserted!

Analysis of Hashing (5)

The expected number of elements examined is

1 i 1
1
thus
+
=
+
1
1

( i 1)

i =1

nm

i =1

1 ( n 1) n

nm
2
n 1
=1+
2m
n
1
=1+

2m 2m
1
1+
2 2m
=1+

Considering the time for computing the hash

function, we obtain
(2 + / 2 1/ 2m) = (1 + )

Analysis of Hashing (6)

Assuming the number of hash table slots is
proportional to the number of elements in
the table
n=O(m)
= n/m = O(m)/m = O(1)
searching takes constant time on average
insertion takes O(1) worst-case time
deletion takes O(1) worst-case time when
the lists are doubly-linked

Hashing

Hash functions
Hash-code

maps
Compression maps

Open addressing
Linear

Probing
Double hashing

Hash Functions
Need

to choose a good hash function

quick

to compute
distributes keys uniformly throughout the table
good hash functions are very rare birthday
paradox
How

to deal with hashing non-integer keys:

find

some way of turning keys into integers

eg.

remove hyphen in 9635-8904 to get 96358904!

for a string, add up ASCII values of the characters of
your string (e.g., java.lang.String.hashCode())
then

use standard hash function on the integers

From Keys to Indices

The mapping of keys to indices of a hash table
is called a hash function
A hash function is usually the composition of
two maps, a hash code map and a
compression map.
An essential requirement of the hash function
is to map equal keys to equal indices
A good hash function minimizes the
probability of collisions

Popular Hash-Code Maps

Integer

cast: for numeric types with 32 bits or

less, we can reinterpret the bits of the
number as an int
Component sum: for numeric types with
more than 32 bits (e.g., long and double), we
can add the 32-bit components.
Why is the component-sum hash code bad
for strings?

Hash-Code Maps (2)

Polynomial accumulation: for strings of a natural
language, combine the character values (ASCII
or Unicode) a0a1 ... an-1 by viewing them as the
coefficients of a polynomial:
a0 + a1x + ...+ xn-1an-1
The polynomial is computed with Horners rule,
ignoring overflows, at a fixed value x:
a0 + x (a1 +x (a2+ ... x (an-2+ x an-1) ... ))
The choice x = 33, 37, 39, or 41 gives at most 6
collisions on a vocabulary of 50,000 English
words

Compression Maps
Use the remainder
h(k) = k mod m, k is the key, m the size of the
table
Need to choose m
m = be (bad)
if m is a power of 2, h(k) gives the e least
significant bits of k
all keys with the same ending go to the same
place
m prime (good)
helps ensure uniform distribution
primes not too close to exact powers of 2

Compression Maps (2)

Example
hash

table for n = 2000 character strings

we dont mind examining 3 elements
m = 701
a prime near 2000/3
but not near any power of 2

Compression Maps (3)

Use

= m (k A mod 1)
k is the key, m the size of the table, and A
is a constant
0<A<1
h(k)

The

steps involved

map

0...kmax into 0...kmax A

take the fractional part (mod 1)
map it into 0...m-1

Compression Maps (4)

Choice of m and A
value of m is not critical, typically use m =
2p
optimal choice of A depends on the
characteristics of the data
5 1
A
=
Knuth says use
(conjugate of the
2
golden ratio) Fibonacci hashing

Compression Maps (5)

Multiply, Add, and Divide (MAD):
h(k) = |ak + b| mod N
eliminates patterns provided a is not a
multiple of N
same formula used in linear congruential
(pseudo) random number generators

Universal Hashing

For any choice of hash function, there exists a

bad set of identifiers
A malicious adversary could choose keys to be
hashed such that all go into the same slot
(bucket)
Average retrieval time is (n)
Solution
a

random hash function

choose hash function independently of keys!
create a set of hash functions H, from which h can
be randomly selected

Universal Hashing (2)

A collection H of hash functions is

universal if for any randomly chosen f
from H (and two keys k and l),
Pr{f(k) = f(l)} 1/m

More on Collisions
A

key is mapped to an already occupied

table location
what

to do?!?

Use

a collision handling technique

Weve seen Chaining
Can also use Open Addressing
Probing
Double

Hashing

Open Addressing
All

elements are stored in the hash table

(can fill up!), i.e., n m
Each table entry contains either an element
or null
When searching for an element,
systematically probe table slots

Open Addressing (2)

Modify

hash function to take the probe

number i as the second parameter
h : U {0,1,..., m 1} {0,1,..., m 1}

Hash

function, h, determines the

sequence of slots examined for a given
key
Probe sequence for a given key k given by
h(k ,0), h(k ,1),..., h(k , m 1) - a permutation of 0,1,..., m 1

Linear Probing

If the current location is used, try the next table

location
LinearProbingInsert(k)
if (table is full) error
probe = h(k)
while (table[probe] occupied)
probe = (probe+1) mod m
table[probe] = k

Uses less memory than chaining as one does not

have to store all those links
Slower than chaining since one might have to
walk along the table for a long time

Linear Probing Example

h(k)

= k mod 13
insert keys: 18 41 22 44 59 32 31 73

31
44 32
41

73

18 44 59 32 22 31 73

Lookup in linear probing

41
0

18 44 59 32 22 31 73
3

10

11

12

To search for a key k we go to (k mod 13) and

continue looking at successive locations till we
find k or encounter an empty location.
Successful search: To search for 31 we go to
(31 mod 13) = 5 and continue onto 6,7,8 till we
find 31 at location 10
Unsuccessful search: To search for 33 we go to
(33 mod 5 = 7) and continue till we encounter an
empty location (12)

Deletion in Linear Probing

41
0

18 44 59 32 22 31 73
3

10

11

12

To delete key 32 we first search for 32.

32 is found in location 8. Suppose we set this
location to null.
Now if we search for 31 we will encounter a null
location before seeing 31.
Lookup procedure would declare that 31 is not
present.

Deletion (2)
41
0

18 44 59 X 22 31 73
3

10

11

12

Instead of setting location 8 to null place a

tombstone (a marker) there.
When lookup encounters a tombstone it ignores
it and continues with next location.
If Insert comes across a tombstone it puts the
element at that location and removes the
tombstone.
Too many tombstones degrades lookup
performance.
Rehash if there are too many tombstones.

Double Hashing
Uses two hash functions, h1, h2
h1(k) is the position in the table where we first
check for key k
h2(k) determines the offset we use when
searching for k
In linear probing h2(k) is always 1.
DoubleHashingInsert(k)
if (table is full) error
probe = h1(k); offset = h2(k)
while (table[probe] occupied)
probe = (probe+offset) mod m
table[probe] = k

Double Hashing(2)
If m is prime, we will eventually examine
every position in the table
Many of the same (dis)advantages as
linear probing
Distributes keys more uniformly than linear
probing

Double Hashing Example

h1(k) = k mod 13
h2(k) = 8 (k mod 8)
insert keys: 18 41 22 44 59 32 31 73

18 41 22 44 59 32 31 73

10

11

12

Analysis of Double Hashing

load factor is less than 1.
We assume that every probe looks at a
random location in the table.
1- fraction of the table is empty.
Expected number of probes required to
find an empty location (unsuccessful
search) is 1/(1- )
The

Analysis (2)
Average

no of probes for a successful

search = average no of probes required to
insert all the elements.
To insert an element we need to find an
empty location.
inserting
First m/2
Next m/4
Next m/8

Avg no of probes
<= 2
<= 4
<= 8

Total no of probes
m
m
m

Analysis(3)
No of probes required to insert
m/2+m/4+m/8++m/2i elements = number of
probes required to leave 2-i fraction of the table
empty = m x i.
No of probes required to leave 1- fraction of
the table empty = - m log (1- )
Average no. of probes required to insert n
elements is - (m/n) log (1- ) = -(1/) log (1- )

Expected Number of Probes

Load factor < 1 for probing

Analysis of probing uses uniform hashing
assumption any permutation is equally likely
What

about linear probing and double hashing?

unsuccessful

successful

chaining

O(1 + )

O(1 + )

probing

1
O

1
1
O ln

Expected Number of Probes (2)

Trees

trees
binary trees
data structures for trees

Trees: Definitions

A is the root node.

B is parent of D & E.
A is ancestor of D & E.
D and E are
descendants of A.
C is the sibling of B
D and E are the
children of B.
D, E, F, G, I are
leaves.
2

Trees: Definitions (2)

A, B, C, H are internal nodes
The depth (level) of E is 2
The height of the tree is 3
The degree of node B is 2

level 0
level 1
level 2
level 3

Trees

A tree represents a
hierarchy, for e.g. the
organization structure of a
corporation

Or table of contents
of a book
4

Another Example
Unix or DOS/Windows file system

Binary Tree

An ordered tree is one in which the children of

each node are ordered.
Binary tree: ordered tree with all nodes having
at most 2 children.

Recursive definition of binary tree

A binary tree is either a

leaf or

An internal node (the root) and one/two binary

trees (left subtree and/or right subtree).

Examples of Binary Trees

arithmetic expressions

Examples of Binary Trees

decision trees

Complete Binary tree

i has 2i nodes
In a tree of height h
leaves are at level h
No. of leaves is 2h
h
No. of internal nodes = 1+2+22++2h-1 = 2 -1
No of internal nodes = no of leaves -1
Total no. of nodes is 2h+1-1 = n
In a tree of n nodes
No of leaves is (n+1)/2
Height = log2 (no of leaves)
level

10

Binary Tree

A Binary tree can be obtained from an

appropriate complete binary tree by
pruning

11

Minimum height of a binary tree

A binary tree of height h has

most 2i nodes at level i
At most 1+2+22++2h = 2h+1-1 nodes
At

If the tree has n nodes then

<= 2h+1-1
Hence h >= log2 (n+1)/2
n

12

Maximum height of a binary tree

A binary tree on n nodes has height at
most n-1
This is obtained when every node (except
the leaf) has exactly one child

13

No of leaves in a binary tree

no of leaves <= 1+ no of internal nodes.
Proof: by induction on no of internal nodes

Tree

with 1 node has a leaf but no internal

node.
Assume stmt is true for tree with k-1 internal
nodes.
A tree with k internal nodes has k1 internal
nodes in left subtree and (k-k1-1) internal
nodes in right subtree.
No of leaves <= (k1+1)+(k-k1) = k+1
14

leaves in a binary tree (2)

For a binary tree on n nodes
No of leaves + no of internal nodes = n
No of leaves <= no of internal nodes + 1
Hence, no of leaves <= (n+1)/2
Minimum no of leaves is 1

15

ADTs for Trees

generic container methods: size(), isEmpty(),

elements()
positional container methods: positions(),
swapElements(p,q), replaceElement(p,e)
query methods: isRoot(p), isInternal(p),
isExternal(p)
accessor methods: root(), parent(p),
children(p)
update methods: application specific
16

ADTs for Binary Trees

accessor methods: leftChild(p),
rightChild(p), sibling(p)
update methods:

expandExternal(p),

removeAboveExternal(p)
other application specific methods

17

The Node Structure

Each node in the tree
contains
key[x]

key
left[x] pointer to left
child
right[x] pt. to right
child
p[x] pt. to parent node

18

Representing Rooted Trees

BinaryTree:

Root

Parent:

BinaryTree
LeftChild: BinaryTree
RightChild: BinaryTree

19

Unbounded Branching
UnboundedTree:

Root

Parent:

UnboundedTree
LeftChild: UnboundedTree
RightSibling: UnboundedTree

20

Tree Walks/Traversals
A

tree walk or traversal is a way of visiting

all the nodes in a tree in a specified order.
A preorder tree walk processes each node
before processing its children
A postorder tree walk processes each
node after processing its children

Traversing Trees (preorder)

preorder traversal
Algorithm preOrder(v)
visit node v
for each child w of v do
recursively perform preOrder(w)

reading a document from beginning to end

2

Traversing Trees (postorder)

postorder traversal
Algorithm postOrder(v)
for each child w of v do
recursively perform postOrder(w)
visit node v

du (disk usage) command in Unix

Traversals of Binary Trees

preorder(v)
if (v == null) then return
else visit(v)
preorder(v.leftchild())
preorder(v.rightchild())
postorder(v)
if (v == null) then return
else postorder(v.leftchild())
postorder(v.rightchild())
visit(v)
4

Examples of pre and postorder

assume that we
are only printing the
data in a node when
we visit it.
Preorder
Postorder

We

a b c d f g e

e
d

c
f

c f g d be a

Evaluating Arithmetic Expressions

specialization of a
postorder traversal

Algorithm evaluate(v)
if v is a leaf
return the variable stored at v
else
let o be the operator stored at v
x evaluate(v.leftChild())
y evaluate(v.rightChild())
return x o y

Traversing Trees

Besides preorder and postorder, a third

possibility arises when v is visted between the
visit to the left ad right subtree.

Algorithm inOrder(v)
if (v == null) then return
else inOrder(v.leftChild())
visit(v)
inOrder(v.rightChild())

Inorder Example
a

Inorder
b

c b f d ga e
d

c
f

Euler Tour Traversal

generic traversal of a binary tree

the preorder, inorder, and postorder traversals are special
cases of the Euler tour traversal
walk around the tree and visit each node three times:

on the left
from below
on the right

Printing an arithmetic expression

Printing an arithmetic
expression - so called
Eulers walk:
Print ( before
traversing the left
subtree, traverse it
Print the value of a
node
Traverse the right
subtree, print ) after
traversing it

10

Template Method Pattern

generic computation
mechanism that can be
specialized by redefining
certain steps
implemented by means of
an abstract Java class
with methods that can be
redefined by its
subclasses

11

Specializing Generic Binary Tree Traversal

printing an arithmetic expression

public class PrintExpressionTraversal extends
BinaryTreeTraversal {
...
protected void external(Position p, TraversalResult r)
{ System.out.print(p.element()); }
protected void left(Position p, TraversalResult r)
{ System.out.print("("); }
protected void below(Position p, TraversalResult r)
{ System.out.print(p.element()); }
protected void right(Position p, TraversalResult r)
{ System.out.print(")"); }
}

12

Building tree from pre- and in- order

Given the preorder and
inorder traversals of a binary
tree we can uniquely
determine the tree.
Preorder
Inorder

abcdfge

cbfdgae

bcdfg

cbfdg

d f g

f d g

a
b

e
d

c
f

13

Building tree from post and inorder

In

place of preorder we can use postorder.

The last node visited in the postorder
traversal is the root of the binary tree.
This can then be used to split in the
inorder traversal to identify the left and
right subtrees.
Procedure is similar to the one for
obtaining tree from preorder and inorder
traversals.
14

Insufficiency of pre & postorder

Given

the pre and postoder traversal of a

binary tree we cannot uniquely identify the
tree.
This is because there can be two trees
with the same pre and postorder
traversals.
a

Preorder: a b c
Postorder: c b a

b
c

c
15

A special case
If each internal node of the
binary tree has at least two
children then the tree can
be determined from the pre
and post order traversals.
Preorder
postorder

abcdfge
bcdfg
d f g

cfgdbea

a
b

e
d

c
f

cfgdb
f g d
16

Ordered Dictionaries
In

addition to dictionary functionality, we

want to support following operations:
Min()
Max()
Predecessor(S,
Successor(S,

k)

k)

For

this we require that there should be a

total order on the keys.
1

A List-Based Implementation
Unordered

list

searching

takes O(n) time

inserting takes O(1) time
Ordered

list

searching

takes O(n) time

inserting takes O(n) time
Using array would definitely improve search
time.
2

Binary Search
Narrow

down the search range in stages

findElement(22)

Running Time
The range of candidate items to be searched is
halved after comparing the key with the middle
element
Binary search runs in O(lg n) time (remember
recurrence...)
What about insertion and deletion?

Binary Search Trees

A binary search tree is a binary tree T such that

each internal node stores an item (k,e) of a dictionary
keys stored at nodes in the left subtree of v are less
than or equal to k
keys stored at nodes in the right subtree of v are
greater than or equal to k

Example sequence 2,3,5,5,7,8

Searching a BST

To

find an element with key k in a tree T

compare

k with key[root[T]]
if k < key[root[T]], search for k in left[root[T]]
otherwise, search for k in right[root[T]]
6

Search Examples
Search(T,

11)

Search Examples (2)

Search(T,

6)

Pseudocode for BST Search

Recursive version
Search(T,k)
01
02
03
04
05
06

x root[T]
if x = NIL then return NIL
if k = key[x] then return x
if k < key[x]
then return Search(left[x],k)
else return Search(right[x],k)

Iterative version
Search(T,k)
01 x root[T]
02 while x NIL and k key[x] do
03
if k < key[x]
04
then x left[x]
05
else x right[x]
06 return x
9

Analysis of Search
Running

time on tree of height h is O(h)

After the insertion of n keys, the worstcase running time of searching is O(n)

10

BST Minimum (Maximum)

Find

the minimum key in a tree rooted at x

TreeMinimum(x)
01 while left[x] NIL
02
do x left[x]
03 return x

Running

time O(h), i.e., it is proportional to

the height of the tree

11

Successor
Given x, find the node with the smallest key
greater than key[x]
We can distinguish two cases, depending on the
right subtree of x
Case 1

right subtree of x is nonempty

successor is leftmost node in the right subtree (Why?)
this can be done by returning TreeMinimum(right[x])

12

Successor (2)
Case

the

right subtree of x is empty

successor is the lowest ancestor of x whose
left child is also an ancestor of x (Why?)

13

Successor Pseudocode
TreeSuccessor(x)
01
02
03
04
05
06
03

if right[x] NIL
then return TreeMinimum(right[x])
y p[x]
while y NIL and x = right[y]
x y
y p[y]
return y

For

a tree of height h, the running time is

O(h)

14

BST Insertion
The

basic idea is similar to searching

take

an element z (whose left and right

children are NIL) and insert it into T
find place in T where z belongs (as if
searching for z),
and add z there
The

running on a tree of height h is O(h),

i.e., it is proportional to the height of the
tree
15

BST Insertion Example

Insert

16

BST Insertion Pseudo Code

TreeInsert(T,z)
01
02
03
04
05
06
07
08
09
10
11
12
13

y NIL
x root[T]
while x NIL
y x
if key[z] < key[x]
then x left[x]
else x right[x]
p[z] y
if y = NIL
then root[T] z
else if key[z] < key[y]
then left[y] z
else right[y] z

17

BST Insertion: Worst Case

In

what sequence should insertions be

made to produce a BST of height n?

18

Deletion
Delete

node x from a tree T

We can distinguish three cases
x

has no children
x has one child
x has two children

Deletion Case 1
If

x has no children just remove x

Deletion Case 2
If

x has exactly one child, then to delete x,

simply make p[x] point to that child

Deletion Case 3
If

x has two children, then

to delete it we have to
find

its successor (or

predecessor) y
remove y (note that y has
at most one child why?)
replace x with y

Delete Pseudocode
TreeDelete(T,z)
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16

if left[z] = NIL or right[z] = NIL

then y z
else y TreeSuccessor(z)
if left[y] NIL
then x left[y]
else x right[y]
if x NIL
then p[x] p[y]
if p[y] = NIL
then root[T] x
else if y = left[p[y]]
then left[p[y]] x
else right[p[y]] x
if y z
then key[z] key[y] //copy all fileds of y
return y
5

In order traversal of a BST

ITW

can be thought of as a projection of

the BST nodes onto a one dimensional
interval

BST Sorting
Use

TreeInsert and InorderTreeWalk to

sort a list of n elements, A
TreeSort(A)
01 root[T] NIL
02 for i 1 to n
03
TreeInsert(T,A[i])
04 InorderTreeWalk(root[T])

BST Sorting (2)

Sort the following numbers
5 10 7 1 3 1 8
Build a binary search tree

Call InorderTreeWalk
1 1 3 5 7 8 10
8

In what order should we insert?

We

want to sort numbers {1,2,,n}

Total time taken to insert these numbers
equals the sum of the level numbers of the
nodes.
Thus if numbers were inserted in
ascending order we would get a tree of
height n-1 in which there is one node at
each level.
So total time for insertion in this case is
1+2+3++n-1 = O(n2).
9

Inserting a random permutation

Suppose we take a random permutation of the
keys and inserted them in this order.
The total time required for insertion is now a
random variable.
We want to compute the expected value of this
r.v.
Recall that the expected value of a r.v. is the
average value it takes over a large number of
trials.

10

Expected insertion time for a random

permutation
We

will compute the average time taken to

insert keys in the order specified by the n!
permutations.
In other words, for each of the n!
permutations we will compute the time
taken to insert keys in that order and then
compute the average.
Let T(n) denote this quantity.
11

Inserting a random permutation (2)

Of

the n! permutations, there are (n-1)!

permutations in which the first element is i.
The tree formed in these instances has i as
the root. The left subtree has keys 1..(i-1)
and the right subtree has keys (i+1)..n
Consider the ordering of keys 1..(i-1) in the
(n-1)! permutations. All (i-1)! permutations
appear and each occurs (n-1)!/(i-1)! times.
12

Inserting a random permuation(3)

Recall

that if we only had keys 1..(i-1) then

average time taken to insert them is T(i-1).
The average is taken over all permutations
of 1..(i-1).
hence total time to insert all (i-1)!
permutations is (i-1)!T(i-1).

13

Inserting a random permutation(4)

When

inserting keys 1..(i-1) into the left

subtree, each key has to be compared with
the root.
This leads to an additional unit cost for each
key.
So total time to insert all (i-1)! permutations
is (i-1)!(T(i-1)+(i-1)).
Since each permutation appears (n-1)!/(i-1)!
times, total time to insert keys 1..(i-1) is
(n-1)!(T(i-1)+(i-1))
14

Inserting a random permutation(5)

Time

to insert keys 1..(i-1) is

(n-1)!(T(i-1)+(i-1))
Similarly, time to insert keys (i+1)..n is
(n-1)!(T(n-i)+(n-i))
Total time to insert all n keys in
permutations where the first key is i is
(n-1)! (T(i-1)+T(n-i)+ n-1)
Total time to insert all n keys in all n!
permutations is
(n-1)! i=1n (T(i-1)+T(n-i)+ n-1).
15

Building the recurrence

We are expressing the value of function T() at

point n in terms of the value of T() at points 0..n1. This is called a recurrence relation.
16

Solving the recurrence

17

Solving the recurrence(2)

18

Summing the harmonic series

What

is the sum + 1/3 + +1/n?

1/x

1/2
1/3
1/41/5
1/6

1/n
x

1 2 3 4 5 6

It is at most the area under the curve f(x)=1/x

between limits 1 and n

19

The expected time for insertion

Thus the expected time for inserting a

randomly chosen permutation of n keys is
O(n log n)
20

Minimum time to insert n keys

The

time required to insert n keys is

minimum when the resulting tree has the
smallest possible height.
A binary tree on n nodes has height at
least log2 n
To insert the n/2 nodes at level log2 n we
require at least (n/2)log2 n time.
Hence inserting n keys requires (nlog2 n)
time.
21

Summary of Running times

To

insert n keys into a binary search tree

which is initially empty requires
O(n2) time in the worst case.
O(nlog n) time in the best case.
O(nlog n) time in the average case; the
average is taken over the n! different
orders in which the n keys could be
inserted.
22

Quick Sort
Characteristics
sorts

almost in "place," i.e., does not require

an additional array
very practical, average sort performance O(n
log n) (with small constant factors), but worst
case O(n2)

Quick Sort the Principle

To

understand quick-sort, lets look at a

high-level description of the algorithm
A divide-and-conquer algorithm
Divide:

partition array into 2 subarrays such

that elements in the lower part <= elements in
the higher part
Conquer: recursively sort the 2 subarrays
Combine: trivial since sorting is done in place
2

Partitioning
Linear

time partitioning procedure

Partition(A,p,r)
01
02
03
04
05
06
07
08
09
10
11

j j

i i

xA[r]
17 12
ip-1
X=10
i
jr+1
while TRUE
10 12
repeat jj-1
until A[j] x
repeat ii+1
until A[i] x
10 5
if i<j
then exchange A[i]A[j]
else return j

10

19 23

10

j
6

19 23

19 23

17

12 17

23 19 12 17
3

Quick Sort Algorithm

Initial

call Quicksort(A, 1, length[A])

Quicksort(A,p,r)
01 if p<r
02
then q Partition(A,p,r)
03
Quicksort(A,p,q)
04
Quicksort(A,q+1,r)

Analysis of Quicksort
Assume

that all input elements are distinct

The running time depends on the
distribution of splits

Best Case

If we are lucky, Partition splits the array evenly

T (n) = 2T (n / 2) + (n)

Worst Case
What

is the worst case?

One side of the parition has only one element
T (n) = T (1) + T (n 1) + (n)
= T (n 1) + (n)
=

( k )
k =1

= ( k )
k =1

= ( n 2 )
7

Worst Case (2)

Worst Case (3)

When

does the worst case appear?

input

is sorted
input reverse sorted
Same

recurrence for the worst case of

insertion sort
However, sorted input yields the best case
for insertion sort!

Analysis of Quicksort
Suppose

the split is 1/10 : 9/10

T (n) = T (n /10) + T (9n /10) + (n) = (n log n)!

10

An Average Case Scenario

Suppose, we alternate
lucky and unlucky
cases to get an
average behavior

(n)

L(n) = 2U (n / 2) + (n) lucky

U (n) = L(n 1) + (n) unlucky
we consequently get
L(n) = 2( L(n / 2 1) + (n / 2)) + (n)
= 2 L(n / 2 1) + (n)
= (n log n)

n-1

1
(n-1)/2

(n-1)/2

(n-1)/2+1

( n )
(n-1)/2
11

An Average Case Scenario (2)

How can we make sure that we are usually

lucky?
Partition around the middle (n/2th) element?
Partition around a random element (works well in
practice)

Randomized algorithm
running time is independent of the input ordering
no specific input triggers worst-case behavior
the worst-case is only determined by the output of the
random-number generator

12

Randomized Quicksort
Assume

all elements are distinct

Partition around a random element
Consequently, all splits (1:n-1, 2:n-2, ..., n1:1) are equally likely with probability 1/n
Randomization

is a general tool to improve

algorithms with bad worst-case but good
average-case complexity
13

Randomized Quicksort (2)

Randomized-Partition(A,p,r)
01 i Random(p,r)
02 exchange A[r] A[i]
03 return Partition(A,p,r)
Randomized-Quicksort(A,p,r)
01 if p < r then
02
q Randomized-Partition(A,p,r)
03
Randomized-Quicksort(A,p,q)
04
Randomized-Quicksort(A,q+1,r)

14

Randomized Quicksort Analysis

Let T(n) be the expected number of
comparisons needed to quicksort n numbers.
Since each split occurs with probability 1/n, T(n)
has value T(i-1)+T(n-i)+n-1 with probability 1/n.
Hence,

15

Randomized Quicksort Analysis(2)

We

have seen this recurrence before.

It is the recurrence for the expected
number of comparisons required to insert
a randomly chosen permutation of n
elements.
We proved that T(n) = O(nlog2 n).
Hence expected number of comparisons
required by randomized quicksort is
O(nlog2 n)
16

Randomized Quicksort running times

Worst

case running time of quicksort is

O(n2)
Best case running time of quicksort is
O(nlog2 n)
Expected running time of quicksort is
O(nlog2 n)

17

What does expected running time mean?

The running time of quicksort does not depend
on the input. It depends on the random numbers
provided by the generator.
Thus for the same input the program might take
3sec today and 5sec tomorrow.
The average time taken over many different runs
of the program would give us the expected time.
Same as saying that we are taking average over
all possible random number sequences provided
by the generator.

18

Analysis of insertion in BST

When creating a binary search tree on n
elements the running time does depend on the
order of the elements.
Our algorithm for insertion did not employ an
random bits.
Given a specific input order the algorithm takes
the same time each day.
However, the time taken is different for different
input orders.
The average time taken over all possible input
orders is O(nlog2 n).

19

AVL Trees

AVL Trees

AVL Tree
AVL trees are
balanced.
An AVL Tree is a
binary search tree
such that for every
internal node v of T,
the heights of the
children of v can differ
by at most 1.

4
44
2

3
17

78
1

32

1
88

50
1
48

62

An example of an AVL tree where the

heights are shown next to the nodes:
2

Height of an AVL Tree

Proposition: The height of an AVL tree T storing
n keys is O(log n).
Justification: The easiest way to approach this
problem is to find n(h): the minimum number of
nodes in an AVL tree of height h.
We see that n(1) = 1 and n(2) = 2
For h 3, an AVL tree of height h contains the
root node, one AVL subtree of height h-1 and the
other AVL subtree of height h-1 or h-2.
i.e. n(h) = 1 + n(h-1) + n(h-2)

Height of an AVL Tree (2)

Knowing n(h-1) >= n(h-2), we get
n(h) = n(h-1) + n(h-2) + 1 > 2n(h-2)
n(h) > 2n(h-2)
> 4n(h-4)
> 8n(h-6)

> 2in(h-2i)
When i = h/2-1 we get: n(h) > 2h/2-1n(2) = 2h/2
Taking logarithms: h < 2log n(h)
Thus the height of an AVL tree is O(log n)

A sharper bound
We

will show how to obtain a sharper

bound on the height of an AVL tree.
We prove using induction that the minimum
number of nodes in an AVL tree of height h,
n(h) >= ch, where c is some number >1.
Base case: h=1. Now n(h) >= c > 1.
Suppose claim is true for all h < k
We have to show that n(k) >= ck
5

Sharper bound (2)

n(k) = n(k-1)+n(k-2)+1
>= ck-1 + ck-2 (by induction hypothesis)
We will be able to show that n(k) >= ck if we can
show that ck-1 + ck-2 >= ck.
So c should be such that c2-c-1 <= 0.
The quadratic equation c2-c-1=0 has
roots
and
.
Hence we can take c as
which is roughly
1.63
Hence AVL tree on n nodes has height atmost
log1.63 n

Structure of an AVL tree

Consider

an AVL tree on n nodes.

Consider a leaf which is closest to the
root.
Suppose this leaf is at level k.
We will show that the height of the tree is
at most 2k-1.

Structure of an AVL tree (2)

Claim: Since closest leaf is at level k all nodes at
levels 1..k-2 have 2 children.
Proof is by contradiction
Suppose node u at level k-2 has only 1 child, v.
v is at level k-1 and so cannot be a leaf.
Hence subtree rooted at v has height at least 2.
Height-balanced property is violated at u

u
v

k-2
k-1

Structure of an AVL tree (3)

By

previous claim, all levels 1 to k-1 are

full.
Hence tree has at least 2k-1 nodes.
Since height of tree is at most 2k-1 it has
at most 22k-1 nodes.
Thus 2k-1 <= n <= 22k-1
Substituting h for 2k-1 we get
2(h-1)/2 <= n <= 2h
9

Summary of AVL tree structure

In

an AVL tree of height h, the leaf closest

to the root is at level at least (h+1)/2.
On the first (h-1)/2 levels the AVL tree is a
complete binary tree.
After (h-1)/2 levels the AVL tree may start
thinning out.
Number of nodes in the AVL tree is at
least 2(h-1)/2 and at most 2h
10

Insertion
A binary tree T is called height-balanced if for
every node v, height of vs children differ by atmost
one.
Inserting a node into an AVL tree changes the
heights of some of the nodes in T.
If insertion causes T to become unbalanced, we
travel up the tree from the newly created node until
we find the first node x such that its grandparent z
is unbalanced node.
Let y be the parent of node x.

11

Insertion (2)
5
44

To rebalance the subtree

rooted at z, we must
perform a rotation.

2
17
32

78

4
1

50

88

1
48

62

54

44

2
17
2

1
32

62

78

50

z
2

1
48

54

1
88

12

AVL Trees

AVL Trees

Insertion
Inserting a node, v, into an AVL tree changes the
heights of some of the nodes in T.
The only nodes whose heights can increase are
the ancestors of node v.
If insertion causes T to become unbalanced, then
some ancestor of v would have a heightimbalance.
We travel up the tree from v until we find the first
node x such that its grandparent z is unbalanced.
Let y be the parent of node x.

Insertion (2)
To rebalance the subtree rooted at z, we must
perform a rotation.
44
17

z
78
y
50

32

44

48

88
62 x

62

17

y
50

32
48

x
78 z

54

88

54
3

Rotations
Rotation is a way of locally reorganizing a BST.
Let u,v be two nodes such that u=parent(v)
Keys(T1) < key(v) < keys(T2) < key (u) < keys(T3)

u
v
T1

T3
T2

Insertion
Insertion happens in subtree T1.
ht(T1) increases from h to h+1.
Since x remains balanced ht(T2)
is h or h+1 or h+2.
h+1 to h+2

If ht(T2)=h+2 then x is originally

x
unbalanced
If ht(T2)=h+1 then ht(x) does not
T1
increase.
Hence ht(T2)=h.
h to h+1

z
y

T4

T3
T2
h

So ht(x) increases from h+1 to

h+2.
5

Insertion(2)

h+3

Since y remains balanced,

ht(T3) is h+1 or h+2 or h+3.
If ht(T3)=h+3 then y is originally
unbalanced.
If ht(T3)=h+2 then ht(y) does not
increase.
So ht(T3)=h+1.

h+2 to h+3

T4
h+1

h+1 to h+2

T3
h+1

T1
h to h+1

T2
h

So ht(y) inc. from h+2 to h+3.

Since z was balanced ht(T4)
is h+1 or h+2 or h+3.
z is now unbalanced and so
ht(T4)=h+1.

Single rotation
h+3
h+2 to h+3

h+1 to h+2

rotation(y,z)

T4
T3
h+1

T1
h to h+1

T2
h

h+3

h+1
h+2

z h+2

T1

T2

T3

T4

h+1

h+1

h+1

The height of the subtree remains the same after

rotation. Hence no further rotations required
7

Double rotation
h+3
h+2 to h+3

h+1

h+4 z
h+3 x

rotation(x,y)

h+2 y

T4

T3

h+1

x h+1 to h+2

T1

T4

T1

T2

h+1

h+1

h+1

rotation(x,z)
h to h+1

T2

T3

h
x

Final tree has same height

as original tree. Hence we
need not go further up the
tree.

h+3

h+2 y

h+2

T1

T2

T3

T4

h+1

h+1

h+1
8

Restructuring

The four ways to rotate nodes in an AVL tree, graphically

represented
-Single Rotations:
a=z

single rotation
a=z

b=y

b=y
c=x

c=x
T0

T3

T1

T3

T0

T1

T2

T2
c=z

b=y

single rotation

b=y

a=x

c=z

a=x
T3
T0

T2
T1

T3
T2

T1

T0

Restructuring (contd.)

double rotations:
double rotation

a=z
c=y

b=x

a=z

c=y

b=x
T0

T3

T2

T0

T2

T1

T3

T1
double rotation

c=z
a=y

b=x
a=y

c=z

b=x
T3

T0
T2

T3

T2

T1

T0

T1
10

Deletion
When

deleting a node in a BST, we either

delete a leaf or a node with only one child.
In an AVL tree if a node has only one child
then that child is a leaf.
Hence in an AVL tree we either delete a
leaf or the parent of a leaf.
Hence deletion can be assumed to be at a
leaf.
11

Deletion(2)
Let w be the node deleted.
Let z be the first unbalanced node encountered
while travelling up the tree from w. Also, let y be the
child of z with larger height, and let x be the child of
y with larger height.
We perform rotations to restore balance at the
subtree rooted at z.
As this restructuring may upset the balance of
another node higher in the tree, we must continue
checking for balance until the root of T is reached

12

Deletion(3)
h+2
h+1
h

T1
h-1 or h-2

T2

Suppose

deletion
happens in subtree T4
z
and its ht. reduces from
h to h-1.
T4 h to h-1
Since z was balanced
but
is
now
unbalanced,
h
or
h-1
T3
ht(y) = h+1.
h-1 or h-2
x has larger ht. than T3
and so ht(x)=h.
Since y is balanced
ht(T3)= h or h-1
13

Deletion(4)
h+2
h+1
h

T1
h-1 or h-2

Since

z
T4

h to h-1

T3 h or h-1

ht(x)=h, and x is
balanced ht(T1), ht(T2)
is h-1 or h-2.
However, both T1 and
T2 cannot have ht. h-2

T2
h-1 or h-2

14

Single rotation (deletion)

h+2
h+1

rotation(y,z)
T4

h to h-1

h+1 or h+2
h or h+1

T1
h-1 or h-2

T3

h or h-1

T2 h-1 or h-2

T1

z
T2

h-1 or h-2 h-1 or h-2

T3
h or h-1

T4
h-1

After rotation height of subtree might be 1 less than

original height. In that case we continue up the tree
15

Deletion: another case

h+2
h+1

y
x

T1
h-1

T2
h-1 or h-2

As

before we can claim

that ht(y)=h+1 and
T4 h to h-1
ht(x)=h.
h
Since y is balanced
ht(T1) is h or h-1.
T3
If ht(T1) is h then we
h-1 or h-2
would have picked x as
the root of T1.
So ht(T1)=h-1
16

Double rotation
h+2

z
h+1

rotation(x,y)

T4
x h

h-1

T3 h-1 or h-2

h to h-1
T1

T1

T4

h+1

h+2

T2

h-1

h-1 or h-2

h-1

rotation(x,z)
T2

h-1 or h-2

T3
x

h-1 or h-2

Final tree has height less

than original tree. Hence we
need to continue up the tree

T1
h-1

h+1

z
T2

h-1 or h-2

T3

T4 h-1

h-1 or h-2
17

Running time of insertion & deletion

Insertion
We

perform rotation only once but might have to

go O(log n) levels to find the unbalanced node.
So time for insertion is O(log n)
Deletion
We

need O(log n) time to delete a node.

Rebalancing also requires O(log n) time.
More than one rotation may have to be
performed.
18

(2,4) Trees

What are they?

They are search Trees (but not binary search
trees)
They are also known as 2-4, 2-3-4 trees

Multi-way Search Trees

Each internal node of a multi-way search tree T:

has at least two children
stores a collection of items of the form (k, x), where k
is a key and x is an element
contains d - 1 items, where d is the number of children
Has pointers to d children

Children of each internal node are between

items
all keys in the subtree rooted at the child fall
between keys of those items.

Multi-way Searching

Similar to binary searching

If search key s<k1 search
the leftmost child
If s>kd-1 , search the
rightmost child

Thats it in a binary tree;

what about if d>2?

Find two keys ki-1 and ki

between which s falls, and
search the child vi.

What would an in-order

traversal look like?

Searching
for s = 8
5 10

33 44

6 8

22
Searching
for s = 12

14

11 13
Not found!

25

23 24

27

17 18 19 20 21

(2,4) Trees

Properties:
At most 4 children
All leaf nodes are at
the same level.
Height h of (2,4) tree
is at least log4 n and
atmost log2 n

12

How is the last fact

useful in searching?

5 10

3 4

6 8

15

11

13 14

17

Insertion

21

23 40

29

No problem if the node

has empty space

13 22 32

18

3 8 10

1 2

4 5 6

11 12

14 15

25

20

24

35

26 28 30 33

37 39

Insertion(2)

29

Nodes get split if there is

insufficient space.

13 22 32

18

3 8 10

1 2

4 5 6

11 12

14 15

25

20 21

23 24

35

26 28 30 33

37 39 40

Insertion(3)

One key is promoted to

parent and inserted in there

13 22 32

18

3 8 10

1 2

4 5 6

11 12

14 15

25

20 21

35

33

23 24
26 28

29 30

37 39 40

Insertion(4)
If parent node does not have sufficient space
then it is split.
In this manner splits can cascade.

13 22 32

18

3 8 10

1 2

9
4 5

6 7

11 12

14 15

25 28

20 21

35

33

23 24
26

29 30

37 39 40

Insertion(5)
Eventually we may have to create a new root.
This increases the height of the tree

5 13

22 32

8 10
18

1 2

9
4

6 7

11 12

14 15

25 28

20 21

35

33

23 24
26

29 30

37 39 40

Time for Search and Insertion

A search visits O(log N) nodes

An insertion requires O(log N) node splits
Each node split takes constant time
Hence, operations Search and Insert each take time
O(log N)

Deletion

Delete 21.
No problem if key to be deleted is in a leaf with at least 2
keys
13

22

8 10
18

1 2

9
4

6 7

11 12

14 15

25 28

20 21

23 24
26

29 30

Deletion(2)
If key to be deleted is in an internal node then
we swap it with its predecessor (which is in a
leaf) and then delete it.
Delete 25

13

22

8 10
18

1 2

9
4

6 7

11 12

14 15

25 28

20

23 24
26

29 30

Deletion(3)
If after deleting a key a node becomes empty
then we borrow a key from its sibling.
Delete 20

13

22

8 10
18

1 2

9
4

6 7

11 12

14 15

24 28

20

23
26

29 30

Deletion(4)
If sibling has only one key then we merge with it.
The key in the parent node separating these two
siblings moves down into the merged node.
13
Delete 23

22

8 10
24 28

15

1 2

9
4

6 7

11 12

14

18

23
26

29 30

Delete(5)
Moving a key down from the parent corresponds
to deletion in the parent node.
The procedure is the same as for a leaf node.
13
Can lead to a cascade .
Delete 18
5
22

8 10
28

15

1 2

9
4

6 7

11 12

14

18
24 26

29 30

(2,4) Conclusion
The

height of a (2,4) tree is O(log n).

Split, transfer, and merge each take O(1).
Search, insertion and deletion each take
O(log n) .
Why are we doing this?
(2,4)

trees are fun! Why else would we do it?

Well, theres another reason, too.
Theyre pretty fundamental to the idea of
Red-Black trees as well.

Red-Black Trees
What are they?
Their relation to 2-4 trees
Deletion in red-black trees

Red-Black Trees
A red black tree is a binary search tree in which
each node is colored red or black.
The root is colored black.
A red node can have only black children.
If a node of the BST does not have a left and/or
right child then we add an external node.
External nodes are not colored.
The black depth of an external node is defined
as the number of black ancestors it has.
In a red-black tree every external node has the
same black depth.

Examples of red-black trees

Black height of
tree is 2

Black height of
tree is 2

Trees which are not red-black

Double red

Black height not uniform

Height of a red-black tree

Let

h be the black height of a red-black

tree on n nodes.
n is smallest when all nodes are black. In
this case tree is a complete binary tree of
height h and n=2h -1.
n is largest when alternate levels of tree
are red. Then height of tree is 2h and
n=22h-1.
Hence, log4 n < h < 1+log2 n

Red-black trees to 2-4 trees

Any red-black tree can be converted into a 2-4
tree.
Take a black node and its red children (at most 2)
and combine them into one node of a 2-4 tree.
Each node so formed has at least 1 and at most 3
keys.
Since black height of all external nodes is same,
in the resulting 2-4 tree all leaves are at same
level.

Example
9
4

13
7

2
1

19

11

17

4 9 13
1 2 3

5 7

11

17 19

2-4 trees to red-black trees

Any

2-4 tree can be converted into a redblack tree.

We replace a node of the 2-4 tree with one
black node and 0/1/2 red nodes which are
children of the black node.
The height of the 2-4 tree is the black
height of the red-black tree created.
Every red node has a black child.

Example

13

18

3 8 10

1 2

4 5 6

11 12

14 15

20

13
8

18
10

3
2
1

5
4

15
12 14

11

20

Black height is same as

height of 2-4 tree

Deletion: preliminaries
To

delete a node we proceed as in a BST.

Thus the node which is deleted is the
parent of an external node.
Hence it is either a leaf or the parent of a
leaf.
19

11
17

17

Deletion: preliminaries(2)
Hence we can assume that the node deleted is a
black leaf.
Removing this reduces black depth of an
external node by 1.
Hence, in a general step, we consider how to
reorganize the tree when the black height of
some subtree goes down from h to h-1.

Deletion: the cases

a

a is red

a is black
a

b is red

b is black

a
Both c
are black

b
c

a
b
c

Some c is
red

a
b

a
b

b
c

c
d

Some c is
red

Both c
are
black

Both d
are black

d
d
Some d is red

b
c

a
b

Deletion: case1.1
If parent is a red node (a).
Then it has a child (b) which must be black
If b has a red child (c)

a
b

h to h-1

h-1

c
h-1

h-1

h-1

h-1

h-1

h-1

a
a

b
c

h-1

h to h-1
h-1

h-1

b c

Deletion: case 1.2

If parent is a red node (a).
Then it has a child (b) which must be black
If b has no red child

h to h-1

h-1
h-1

h-1

h-1

h-1

b a

Deletion: case 2.1.1

If parent is a black node (a).
If a has a red child (b)
Consider right child of b (c) which must be black.
If (c) has a red child (d).
c

a
b
c

h-1

h to h-1

h-1

h-1
h-1

h-1
b c

b a

d c

h-1

h-1

Deletion: case 2.1.2

If parent is a black node (a).
If a has a red child (b)
Consider right child of b (c) which must be black.
If (c) has no red child.
b

h to h-1

h-1

c
h

h-1
h-1

h-1
b a

c a

h-1

Deletion: case 2.2.1

If parent is a black node (a).
If the child of node a (c) is black.
If (c) has a red child (d)

h to h-1

h-1

h-1
h-1

h-1

h-1

h-1
a

c d

h-1

Deletion: case 2.2.2

If parent is a black node (a).
If the child of node a (c) is black.
If (c) has no red child.

a
c

h to h-1

h-1
h-1

h-1

h-1
a

c a

h-1

Deletion: Summary
In

all cases, except 2.2.2, deletion can be

completed by a simple rotation/recoloring
In case 2.2.2, the height of the subtree
reduces and so we need to proceed up the
tree.
But in case 2.2.2 we only recolor nodes.
Thus, if we proceed up the tree then we
only need to recolor. Eventually we would
do a rotation.

Insertion in red-black trees

a-b trees

What are they?

Insertion and deletion in a-b trees

Insertion in red-black trees

Let

k be the key being inserted

As in the case of a BST we first search for
k; this gives us the place where we have
to insert k.
We create a new node with key k and
insert it at this place.
The new node is colored red.

Insertion(2)
Since

inserted node is colored red, the

black height of the tree remains
unchanged.
However, if the parent of inserted node is
also red then we have a double red
problem.

No problem

Double red problem

Insertion: case 1
Parent of inserted node (a) is red.
Parent of (a) must be black (b)
The other child of (b) is black (c).

b
c

a
k

The 2-4 tree node contains {b,a,k} and is malformed.

The rotation corrects the defect.

Insertion: Case 2
Parent of inserted node (a) is red.
Parent of (a) must be black (b)
The other child of (b) is also red (c).

b
c

b
c b a
k

a k

Insertion: case 2 (contd)

The parent of b could also be red. In that case,
the double red problem moves up a level.
We repeat this process at the next level.
Eventually, we might color the root red.
In this case we recolor the root black. This
increases the black depth of every external node
by 1.
In the 2-4 tree this corresponds to splitting the
root.

Insertion and Deletion: Summary

In

both insertion and deletion we need to

make at most one rotation.
We might have to move up the tree but in
doing so we only recolor nodes.
Time taken is O(log n)

(a,b) Trees
A multiway search tree.
Each node has at least a
and at most b children.
Root can have less than
a children but it has at
least 2 children.
All leaf nodes are at the
same level.
Height h of (a,b) tree is
at least logb n and at
most loga n.

12

5 10

3 4

6 8

15

11

13 14

17

Insertion

21

23

29

No problem if the node

has empty space

13 22

18

3 8

1 2

4 5

14 15

25

20

24

26 28

Insertion(2)
29

Nodes get split if there is

insufficient space.
The median key is promoted to the
parent node and inserted there

13 22

1 2

4 5

25

18

3 8

14 15

20

23 24

26 28

Insertion(3)
A

node is split when it has exactly b keys.

One of these is promoted to the parent
and the remaining are split between two
nodes.
Thus one node gets
and the other
keys.
This

implies that a-1 >=

Deletion
If after deleting a key a node becomes empty
then we borrow a key from its sibling.
Delete 20

13
5

22

8 10
18

1 2

6 7

11 12

14 15

24 28

20

23

26

29 30

Deletion(2)
If sibling has only one key then we merge with it.
The key in the parent node separating these two
siblings moves down into the merged node.
Delete 23

13
5

22

8 10
24 28

18

1 2

6 7

11 12

14 15

20

23

26

29 30

Deletion(3)
In

an (a,b) tree we will merge a node with

its sibling if the node has a-2 keys and its
sibling has a-1 keys.
Thus the merged node has 2(a-1) keys.
This implies that 2(a-1) <= b-1 which is
equivalent to a-1 <=
.
Earlier too we argued that a-1 <=
This implies b >= 2a-1
For a=2 are b >= 3

Conclusion
The

height of a (a,b) tree is O(log n).

b >= 2a-1.
For insertion and deletion we take time
proportional to the height.

Disk Based Data Structures

So far search trees were limited to main

memory structures

Counter-example: transaction data of a bank >

1 GB per day

Assumption: the dataset organized in a search tree

fits in main memory (including the tree overhead)

use secondary storage media (punch cards, hard

disks, magnetic tapes, etc.)

Consequence: make a search tree structure

secondary-storage-enabled

Hard Disks
Large amounts of
storage, but slow
access!
Identifying a page
takes a long time (seek
time plus rotational
delay 5-10ms),
reading it is fast

It pays off to read or

write data in pages (or
blocks) of 2-16 Kb in
size.

Algorithm analysis

The running time of disk-based algorithms is

measured in terms of
computing time (CPU)
number of disk accesses

sequential reads
random reads

Regular main-memory algorithms that work one

data element at a time can not be ported to
secondary storage in a straight-forward way

Principles
Pointers

in data structures are no longer

addresses in main memory but
locations in files
If x is a pointer to an object
if

x is in main memory key[x] refers to it

otherwise DiskRead(x) reads the object
from disk into main memory (DiskWrite(x)
writes it back to disk)

Principles (2)
A

typical working pattern

01
02
03
04
05
06
07

x a pointer to some
DiskRead(x)
operations that access
DiskWrite(x) //omitted
other operations, only

object
and/or modify x
if nothing changed
access no modify

Operations:
DiskRead(x:pointer_to_a_node)
DiskWrite(x:pointer_to_a_node)
AllocateNode():pointer_to_a_node

Binary-trees vs. B-trees

Size of B-tree nodes is determined by the page
size. One page one node.
A B-tree of height 2 may contain > 1 billion keys!
Heights of Binary-tree and B-tree are logarithmic

B-tree: logarithm of base, e.g., 1000

Binary-tree: logarithm of base 2

1 node
1000 keys

1000
1001

1000

1000

1001

1000
1001

1001

1000

1000

1001 nodes,
1,001,000 keys

1000

1,002,001 nodes,
1,002,001,000 keys

B-tree Definitions

Node x has fields

n[x]: the number of keys of that the node
key1[x] keyn[x][x]: the keys in ascending order
leaf[x]: true if leaf node, false if internal node
if internal node, then c1[x], , cn[x]+1[x]: pointers to
children

Keys separate the ranges of keys in the subtrees. If ki is an arbitrary key in the subtree ci[x]
then ki keyi[x] ki+1

B-tree Definitions (2)

Every

leaf has the same depth

In a B-tree of a degree t all nodes except
the root node have between t and 2t
children (i.e., between t1 and 2t1 keys).
The root node has between 0 and 2t
children (i.e., between 0 and 2t1 keys)

Height of a B-tree

B-tree T of height h, containing n 1 keys and

minimum degree t 2, the following restriction
on the height holds:
n +1
#of
h log t
depth
nodes
2
1
0 1
t-1

t-1

t-1

2t

t-1

t-1
h

t-1

t-1

n 1 + (t 1) 2t i 1 = 2t h 1
i =1

t-1

B-tree Operations
An

implementation needs to suport the

following B-tree operations
Searching

(simple)
Creating an empty tree (trivial)
Insertion (complex)
Deletion (complex)

Searching
Straightforward

generalization of a binary

tree search
BTreeSearch(x,k)
01
02
03
04
05
06
08
09
10

i 1
while i n[x] and k > keyi[x]
i i+1
if i n[x] and k = keyi[x] then
return(x,i)
if leaf[x] then
return NIL
else DiskRead(ci[x])
return BTtreeSearch(ci[x],k)

Creating an Empty Tree

Empty

B-tree = create a root & write it to

disk!
BTreeCreate(T)
01
02
03
04
05

x AllocateNode();
leaf[x] TRUE;
n[x] 0;
DiskWrite(x);
root[T] x

Splitting Nodes
Nodes

fill up and reach their maximum

capacity 2t 1
Before we can insert a new key, we have
to make room, i.e., split nodes

Splitting Nodes (2)

Result:

one key of x moves up to parent +

2 nodes with t-1 keys
]
[1 x [x]
y i- ey i
e
k
k

... N W ...

... N S W ...

y = ci[x]

y = ci[x]

P Q R S T V W

T1

]
]
[1 x
[1 x [x]
y i- ey i ey i+
e
k
k
k

...

P Q R

T8

z = ci+1[x]
T V W

Splitting Nodes (2)

BTreeSplitChild(x,i,y)
z AllocateNode()
leaf[z] leaf[y]
n[z] t-1
for j 1 to t-1
keyj[z] keyj+t[y]
if not leaf[y] then
for j 1 to t
cj[z] cj+t[y]
n[y] t-1
for j n[x]+1 downto i+1
cj+1[x] cj[x]
ci+1[x] z
for j n[x] downto i
keyj+1[x] keyj[x]
keyi[x] keyt[y]
n[x] n[x]+1
DiskWrite(y)
DiskWrite(z)
DiskWrite(x)

x: parent node
y: node to be split and child of x
i: index in x
z: new node
]
[1 x [x]
y i- ey i
e
k
k

... N W ...
y = ci[x]
P Q R S T V W

T1

...

T8

Split: Running Time

A

local operation that does not traverse

the tree
(t) CPU-time, since two loops run t times
3 I/Os

Inserting Keys
Done

recursively, by starting from the root

and recursively traversing down the tree to
the leaf level
Before descending to a lower level in the
tree, make sure that the node contains <
2t 1 keys:
so

that if we split a node in a lower level we

will have space to include a new key

Inserting Keys (2)

Special

case: root is full (BtreeInsert)

BTreeInsert(T)
r root[T]
if n[r] = 2t 1 then
s AllocateNode()
root[T] s
leaf[s] FALSE
n[s] 0
c1[s] r
BTreeSplitChild(s,1,r)
BTreeInsertNonFull(s,k)
else BTreeInsertNonFull(r,k)

Splitting the Root

Splitting

the root requires the creation of a

new root
root[T]

root[T]

A D F H L N P

r
T1
The

...

T8

A D F

L N P

tree grows at the top instead of the

bottom

Inserting Keys
BtreeNonFull

tries to insert a key k into

a node x, which is assumed to be nonfull when the procedure is called
BTreeInsert and the recursion in
BTreeInsertNonFull guarantee that this
assumption is true!

Inserting Keys: Pseudo Code

BTreeInsertNonFull(x,k)
01 i n[x]
02 if leaf[x] then
03
while i 1 and k < keyi[x]
04
keyi+1[x] keyi[x]
05
i i - 1
06
keyi+1[x] k
07
n[x] n[x] + 1
08
DiskWrite(x)
09 else while i 1 and k < keyi[x]
10
i i - 1
11
i i + 1
12
DiskRead ci[x]
13
if n[ci[x]] = 2t 1 then
14
BTreeSplitChild(x,i,ci[x])
15
if k > keyi[x] then
16
i i + 1
17
BTreeInsertNonFull(ci[x],k)

leaf insertion

internal node:
traversing tree

Insertion: Example
initial tree (t = 3)
G M P X
A C D E

J K

N O

R S T U V

Y Z

B inserted
G M P X
A B C D E

J K

Q inserted
A B C D E

N O

R S T U V

Y Z

G M P T X
J K

N O

Q R S

U V

Y Z

Insertion: Example (2)

P

L inserted
G M
A B C D E

J K L

T X
N O

C G M
D E F

U V

Y Z

U V

Y Z

F inserted

A B

Q R S

J K L

T X
N O

Q R S

Insertion: Running Time

Disk

I/O: O(h), since only O(1) disk

accesses are performed during recursive
calls of BTreeInsertNonFull
CPU: O(th) = O(t logtn)
At any given time there are O(1) number
of disk pages in main memory

Deleting Keys
Done recursively, by starting from the root and
recursively traversing down the tree to the leaf
level
Before descending to a lower level in the tree,
make sure that the node contains t keys (cf.
insertion < 2t 1 keys)
BtreeDelete distinguishes three different
stages/scenarios for deletion

Case 1: key k found in leaf node

Case 2: key k found in internal node
Case 3: key k suspected in lower level node

Deleting Keys (2)

P

initial tree
C G M
A B

D E F

J K L

T X
N O

C G M
D E

U V

Y Z

U V

Y Z

F deleted:
case 1
A B

Q R S

J K L

T X
N O

Q R S

Case 1: If the key k is in node x, and x is a leaf,

delete k from x

Deleting Keys (3)

Case 2: If the key k is in node x, and x is not a

leaf, delete k from x
a) If the child y that precedes k in node x has at least t
keys, then find the predecessor k of k in the sub-tree
rooted at y. Recursively delete k, and replace k with
k in x.
b) Symmetrically for successor node z

M deleted:
case 2a
C G L
A B

D E

J K

x
N O

T X
Q R S

U V

Y Z

Deleting Keys (4)

If both y and z have only t 1 keys, merge k with

the contents of z into y, so that x loses both k
and the pointers to z, and y now contains 2t 1
keys. Free z and recursively delete k from y.
P

G deleted:
case 2c
A B

C L

x-k

D E J K

N O

y = y+k + z - k

T X
Q R S

U V

Y Z

Deleting Keys - Distribution

Descending down the tree: if k not found in
current node x, find the sub-tree ci[x] that has to
contain k.
If ci[x] has only t 1 keys take action to ensure
that we descent to a node of size at least t.
We can encounter two cases.

If ci[x] has only t-1 keys, but a sibling with at least t

keys, give ci[x] an extra key by moving a key from x to
ci[x], moving a key from ci[x]s immediate left and right
sibling up into x, and moving the appropriate child from
the sibling into ci[x] - distribution

Deleting Keys Distribution(2)

x
ci[x]

... k ...

ci[x]

...

C L P T X

delete B
ci[x]

A B

... k
A B

... k ...

E J K

N O

Q R S

U V

Y Z

sibling
E L P T X

B deleted:
A C

J K

N O

Q R S

U V

Y Z

Deleting Keys - Merging

If

ci[x] and both of ci[x]s siblings have t 1

keys, merge ci with one sibling, which
involves moving a key from x down into
the new merged node to become the
median key for that node

x
ci[x]

... l k m...
m

... l
A

... l m ...
...l k m ...
A B

Deleting Keys Merging (2)

P

ci[x]

delete D
A B

C L

D E J K

D deleted:
A B

sibling
N O

T X

Q R S

U V

Y Z

U V

Y Z

C L P T X
E J K

tree shrinks in height

N O

Q R S

Deletion: Running Time

Most of the keys are in the leaf, thus deletion
most often occurs there!
In this case deletion happens in one downward
pass to the leaf level of the tree
Deletion from an internal node might require
backing up (case 2)
Disk I/O: O(h), since only O(1) disk operations
are produced during recursive calls
CPU: O(th) = O(t logtn)

Two-pass Operations
Simpler,

practical versions of algorithms

use two passes (down and up the tree):
Down

Find the node where deletion or

insertion should occur
Up If needed, split, merge, or distribute;
propagate splits, merges, or distributes up the
tree
To

avoid reading the same nodes twice,

use a buffer of nodes

Case Study: Searching for Patterns

Problem: find all occurrences of pattern

P of length m inside the text T of
length n.
Exact matching problem

String Matching - Applications

Text

editing
Term rewriting
Lexical analysis
Information retrieval
And, bioinformatics

Exact matching problem

Given a string P (pattern) and a longer string T (text).
Aim is to find all occurrences of pattern P in text T.

The naive method:

T

P Pag Pag Pgaag Pgag Pag Pga

a
g

a
g

a
g

If m is the length of P, and n is the length of T, then

Time complexity = O(m.n),
Space complexity = O(m + n)
3

Can we be more clever ?

When a mismatch is detected, say at position
k in the pattern string, we have already
successfully matched k-1 characters.
We try to take advantage of this to decide
where to restart matching

g
a

Pga Pag

a
g

Pga Pga agg ga

a
g

The Knuth-Morris-Pratt Algorithm

Observation: when a mismatch occurs, we
may not need to restart the comparison all way
back (from the next input position).
What to do:
Constructing an array h, that determines how
many characters to shift the pattern to the right
in case of a mismatch during the patternmatching process.

KMP (2)
The key idea is that if we have
successfully matched the prefix P[1i1] of the pattern with the substring T[ji+1,,j-1] of the input string and P(i)
T(j), then we do not need to reprocess
any of the suffix T[j-i+1,,j-1] since we
know this portion of the text string is
the prefix of the pattern that we have
just matched.
6

The failure function h

For each position i in pattern P, define hi to be
the length of the longest proper suffix of P[1,,i]
that matches a prefix of P.
i = 11
Pattern:

1
a

2
b

3
a

4
a

5
b

6
a

7
b

8
a

9
a

10
b

11
a

12
a

13
b

Hence, h(11) = 6.

If there is no proper suffix of P[1,,i] with the

.property mentioned above, then h(i) = 0
7

The KMP shift rule

The first mismatch in position
k=12 of T and in pos. i+1=12 of P.

T: a

P: a

P:

h11 = 6
Shift P to the right so that P[1,,h(i)] aligns with the suffix
T[k- h(i),,k-1].
They must match because of the definition of h.
In other words, shift P exactly i h(i) places to the right.
If there is no mismatch, then shift P by m h(m) places to
the right.
8

The KMP algorithm finds all occurrences of

P in T.
First mismatch in position
k of T and in pos. i+1 of P.

Suppose not,
T:

P before shift
P after shift
missed occurrence of P

Correctness of KMP.
First mismatch in position
k of T and in pos. i+1 of P.

T(k)
T:

P before shift

P after shift

missed occurrence of P

|| > || = h(i)
It is a contradiction.
10

An Example
Given:
Pattern:
Next funciton:

Array h:

1
a
0
0

Input string:

2
b
1

3
a
0

4
a
2

5
b
1

6
a
0

7
b
4

8
a
0

9
a
2
4

10
b
1
5

11
a
0
6

12
a
7
4

13
b
1
5

abaababaabacabaababaabaab.

i+1 = 12

Scenario 1:

a b a a b a b a a b a a b
a b a a b a b a a b a c a b a a b a b a a b a a b

k = 12

h(11) = 6 i h(i) = 11 6 = 5

What is h(i)= h(11) = ?

Scenario 2:

i = 6 , h(6)= 3

i+1

a b a a b a b a a b a a b
a b a a b a b a a b a c a b a a b a b a a b a a b
k
11

An Example
Scenario 3:

i = 3, h(3) = 1

i+1

a b a a b a b a a b a a b
a b a a b a b a a b a c a b a a b a b a a b a a b
k

Subsequently

i = 2, 1, 0

Finally, a match is found:

i+1

a b a a b a b a a b a a b
a b a a b a b a a b a c a b a a b a b a a b a a b
k

12

Complexity of KMP
In the KMP algorithm, the number of character
comparisons is at most 2n.
After shift, no character will be
compared before this position

no backtrack

T:
P before shift
P after shift

In any shift at most one comparison involves a

character of T that was previously compared.
Hence

#comparisons #shifts + |T| 2|T| = 2n.

13

Computing the failure function

We can compute h(i+1) if we know h(1)..h(i)

To do this we run the KMP algorithm where the text is
the pattern with the first character replaced with a $.
Suppose we have successfully matched a prefix of the
pattern with a suffix of T[1..i]; the length of this match is
h(i).
If the next character of the pattern and text match then
h(i+1)=h(i)+1.
If not, then in the KMP algorithm we would shift the
pattern; the length of the new match is h(h(i)).
If the next character of the pattern and text match then
h(i+1)=h(h(i))+1, else we continue to shift the pattern.
Since the no. of comparisons required by KMP is length
of pattern+text, time taken for computing the failure
function is O(n).
14

Computing h: an example
Given:
Pattern: h:
Failure function

Text:

10 11 12 13

10 11 12 13

Pattern:
Pattern

b
h(11)=6
b

h(6)=3

h(12)= 4 = h(6)+1 = h(h(11))+1

h(13)= 5 = h(12)+1
15

KMP - Analysis

The

KMP algorithm never needs to

backtrack on the text string.
preprocessing

searching

Time complexity = O(m + n)

Space complexity = O(m + n),
where m =|P| and n=|T|.

16

Tries
Standard Tries
Compressed Tries
Suffix Tries

Text Processing
We have seen that preprocessing the pattern
speeds up pattern matching queries
After preprocessing the pattern in time
proportional to the pattern length, the KMP
algorithm searches an arbitrary English text in
time proportional to the text length
If the text is large, immutable and searched for
often (e.g., works by Shakespeare), we may want
to preprocess the text instead of the pattern in
order to perform pattern matching queries in time
proportional to the pattern length.

Standard Tries
The standard trie for a set of strings S is an ordered
tree such that:
each node but the root is labeled with a character
the children of a node are alphabetically ordered
the paths from the external nodes to the root yield
the strings of S
Eg. S = { bear, bell, bid, bull, buy, sell, stock, stop }

Running time for operations

A

standard trie uses O(W) space.

Operations (find, insert, remove) take time
O(dm) each, where:
W

= total size of the strings in S,

m = size of the string involved in the operation
d = alphabet size,

Applications of Tries
A

standard trie supports the following

operations on a preprocessed text in time
O(m), where m = |X|
word matching: find the first occurence
of word X in the text
prefix matching: find the first occurrence
of the longest prefix of word X in the text
Each operation is performed by tracing a
path in the trie starting at the root
5

Compressed Tries
Trie with nodes of degree at least 2
Obtained from standard trie by compressing
chains of redundant nodes.

Standard Trie:

Compressed Trie:

Why Compressed Tries ?

A

tree in which every node has at least 2

children has at most L-1 internal nodes,
where L is the number of leaves.
The number of nodes in a compressed
trie is O(s), where s = |S|.
The label in each node can be stored by
using O(1) space index ranges at the
nodes
8

Insertion/Deletion in Compressed Tries

10

Tries and Web Search Engines

The index of a search engine (collection of all searchable

words) is stored into a compressed trie
Each leaf of the trie is associated with a word and has a
list of pages (URLs) containing that word, called
occurrence list
The trie is kept in internal memory
The occurrence lists are kept in external memory and are
ranked by relevance
Boolean queries for sets of words (e.g., Java and coffee)
correspond to set operations (e.g., intersection) on the
occurrence lists
Additional information retrieval techniques are used, such
as

stopword elimination (e.g., ignore the a is)

stemming (e.g., identify add adding added)
link analysis (recognize authoritative pages)
11

Tries and Internet Routers

Computers on the internet (hosts) are identified by a

unique 32-bit IP (internet protocol) address, usually
written in dotted-quad-decimal notation
E.g., www.google.com is 64.233.189.104
Use nslookup on Unix to find out IP addresses
An organization uses a subset of IP addresses with the
same prefix, e.g., IITD uses 10.*.*.*
Data is sent to a host by fragmenting it into packets.
Each packet carries the IP address of its destination.
A router forwards packets to its neighbors using IP
prefix matching rules.
Routers use tries on the alphabet 0,1 to do prefix
matching.
12

Back to Pattern Matching

Instead of preprocessing the pattern P,
preprocess the text T !
Use a tree structure where all suffixes of the
text are represented;
Search for the pattern by looking for
substrings of the text;
You can easily test whether P is a substring of
T because any substring of T is the prefix of
some suffix of T .

13

Suffix Tree
A suffix tree T for string S is a rooted directed tree whose
edges are labeled with nonempty substrings of S.
Each leaf corresponds to a suffix of S in the sense that
the concatenation of the edge-labels on the unique path
from the root to the leaf spells out this suffix.
Each internal node, other than the root, has at least two
children.
xa b x a c
No two out-edges of
1
c
b
a node can have edge-labels
4
x
with the same first character. a
c

6
2
Suffix tree for string xabxac.

14

Suffix Tree (2)

A suffix tree is essentially a compressed trie for all the

suffixes of a text
The suffix tree for a text X of size n from an alphabet of
size d stores all the n suffixes of X in O(n) space.

15

Compact Representation

16

Note on Suffix Tree

If two suffixes have a same prefix, then their
corresponding paths are the same at their
beginning, and the concatenation of the edge
labels of the mutual part is the prefix.
xa
x

For example, suffix xabxac

and suffix xac have the same
prefix, xa.

b x a c
c
4

6
2
Suffix tree for string xabxac.
17

Note on Suffix Tree

Not all strings guaranteed to have corresponding
suffix trees
For example:
consider xabxa: it does not have a suffix tree,
because here suffix xa is also a prefix of another
suffix, xabxa.
(The path spelling out xa would not end at a
leaf.)
How to fix the problem: add $ - a special
termination character to the alphabet.

18

Building a Suffix Tree

The method starts from a single edge for suffix S[1..n], then
it successively enters the edges for suffix S[i..n] into the
growing tree, for i increasing from 2 to n.

The general step of the algorithm

Enter

the edge for suffix S[i..n] into the tree as follows.

Starting at the root find the longest path from the root
whose label matches a prefix of S[i..n]. At some point, no
further matches are possible.
If this point is at a node, then denote this node by w.
If it is in the middle of an edge, then insert a new
node, called w, at this point.
Create a new edge running from w to a new leaf labeled
S[i..n].
19

Example
x a b x a

Time complexity: O(n2)

1
c
a

4
x

x
a

5
6

Building suffix tree for string xabxac.

20

Suffix Trees in Pattern Matching

Given

a string P (pattern) and a longer string T

(text).Our aim is to find all occurrences of pattern P
in text T.
The idea of the algorithm is that every occurrence
of P in T is a prefix of a suffix of T,
Thus, an occurrence of P in T can be obtained as
the of the labels of edges along the path beginning
concatenation at the root.

21

Suffix Trees in Pattern Matching

Build a suffix tree T for text T.

Match the characters of P along the unique path in
T beginning at the root until

P is exhausted or
no more matches are possible

In case 2, P does not appear anywhere in T.

In case 1, P is a prefix of a suffix
obtained by extending the path until we reach a leaf.
Each extension gives a suffix a prefix of which is P,
thus, each extension provides an occurrence of P in T.
22

Example
a

Root

Find all occurences of P = aw

in T = awyawxawxz.

y
a
w

x
w

awyawxawxz
x

awxz

awxawxz

awyawxawxz
23

Constructing a Suffix Trees

A

suffix tree can be constructed in linear

time
[Weiner73, McCreight76, Ukkonen95]

24

Complexity of Pattern matching

Time complexity
Preprocessing : O(|T| )
Searching : O(|P| + k),
where k is # occurences of P in T

Space complexity
O(|P| + |T| )

25

Data Compression
File

Compression
Huffman Tries
0
0
C

1
0
A

0
D

1
B

ABRACADABRA
01011011010000101001011011010
1

File Compression

Text files are usually stored by representing each

character with an 8-bit ASCII code (type man ascii in a
Unix shell to see the ASCII encoding)
The ASCII encoding is an example of fixed-length
encoding, where each character is represented with
the same number of bits
In order to reduce the space required to store a text
file, we can exploit the fact that some characters are
more likely to occur than others
variable-length encoding uses binary codes of
different lengths for different characters; thus, we can
assign fewer bits to frequently used characters, and
more bits to rarely used characters.
2

File Compression: Example

An Encoding Example
text: java
encoding: a = 0, j = 11, v = 10
encoded text: 110100 (6 bits)

How to decode (problems in ambiguity)?

encoding: a = 0, j = 01, v = 00
encoded text: 010000 (6 bits)
could be "java", or "jvv", or "jaaaa"

Encoding Trie

To prevent ambiguities in decoding, we require

that the encoding satisfies the prefix rule: no
code is a prefix of another.
a = 0, j = 11, v = 10 satisfies the prefix rule
a = 0, j = 01, v= 00 does not satisfy the
prefix rule (the code of 'a' is a prefix of the
codes of 'j' and 'v')

Encoding Trie(2)
We use an encoding trie to satisfy this prefix rule
the characters are stored at the external
nodes.
a left child (edge) means 0
a right child (edge) means 1
0

A = 010
B = 11

0
C

0
D

1
C = 00
B
D = 10
R = 011

Example of Decoding
0

A = 010

B = 11

trie

0
C

encoded text:
text:

0
D

1
B

C = 00

D = 10

R = 011

01011011010000101001011011010

ABRACADABRA
6

Trie this!
10000111110010011000111011110001010100110
100
0
0

R
0
0
S

1
W

1
0
T

1
B

0
E

1
C

0
K

1
N
7

Optimal Compression

An issue with encoding tries is to ensure that the encoded

text is as short as possible:
0

ABRACADABRA
0101101101000010100101101010 0
29 bits
C

1
0

0
A

1
0

ABRACADABRA
001011000100001100101100
24 bits

D
8

Construction algorithm
Given

frequencies of characters we wish

to compute a trie so that the length of the
encoding is minimum possible.
Each character is a leaf of the trie
The number of bits used to encode a
character is its level number (root is 0).
Thus if fi is the frequency of the ith
character and li is the level of the leaf
corresponding to it then we want to find a
tree which minimises i fili
9

Total weighted external path length

The quantity i fili is called the total external
weighted path length of a tree.
We view each leaf as having a weight equal to
the frequency of the corresponding character.
Given weights f1,f2,,fn we wish to find a tree
whose weighted external path length is
minimum.
We denote this by WEPL(f1,f2,, fn)

10

Huffman Encoding Trie

ABRA CAD ABRA
character
frequency
5
A

2
B

2
R

2
C

D 1

A
B

5
A

6
4

11

Huffman Encoding Trie (contd.)

5
A

6
4

11
0

1
6

5
A

4
0
2
B

2
1

D
12

Final Huffman Encoding Trie

11
0

1
6

5
A

0
4
0
2
B

2
1

ABRACADABRA
0 100 101 0 110 0 111 0 100 1010
23 bits
13

Another Huffman Encoding Trie

ABRA CAD ABRA
character

frequency

5
A

2
B

2
R

2
C 1

5
A

2
B

D 1

4
2
R

2
C 1

D 1

14

Another Huffman Encoding Trie

5
A

2
B

4
2
R

2
C 1

D 1

5
A

2
B
2
R

2
C 1

D 1
15

Another Huffman Encoding Trie

6

5
A

2
B
2
R

2
C

D 1

11
6

5
A

2
B
2
R

2
C

D 1

16

Another Huffman Encoding Trie

0

11

5
A

2
B

1
0

2
R

1
0

C 1

1
D 1

ABRACADABRA
0 10 110 0 1100 0 1111 0 10 110 0
23 bits
17

Correctness of the algorithm

Why

does this algorithm compute a tree

with the minimum weighted external path
length?
We prove this by induction on the number
of leaves/characters.
The claim is true when we have only two
leaves.
Suppose claim is true when we have n-1
characters.
18

Correctness of the algorithm(2)

When we have n characters at the first step we
replace the two characters with frequencies
f1,f2 with one character of frequency f1+f2.
Beyond this point the algorithm behaves as if it
had only n-1 characters. Hence it computes a
tree with min. total weighted external path
length i.e. WEPL(f1+f2, f3,fn)
Hence the tree computed has weighted
external path length f1+f2+ WEPL(f1+f2, f3,fn)

19

Correctness of the algorithm(3)

We now argue that
WEPL(f1,f2, f3,fn ) = f1+f2+ WEPL(f1+f2, f3,fn)

This follows from the fact that in the optimum

tree the leaves with the two lowest weights are
siblings

20

Priority Queues

Scheduling example
The priority queue ADT
Implementing a priority queue with a sequence
Binary Heaps
Insertion in a Heaps and Heapify

Scheduling
In a multi-user computer system, multiple users
submit jobs to run on a single processor.
We assume that the time required by each job is
known in advance. Further, jobs can be
preempted (stopped and resumed later)
One policy which minimizes the average waiting
time is SRPT (shortest remaining processing
time).
The processor schedules the job with the
smallest remaining processing time.
If while a job is running a new job arrives with
processing time less than the remaining time of
current job, the current job is preempted.

Data Structure for SRPT

We

need to maintain the remaining

processing time of the unfinished jobs at
any point in time.
We need to find the job with the shortest
remaining processing time.
When a job finishes we should remove it
from our collection.
When a new job arrives we need to add it
to the collection.
3

Priority Queues
A priority queue is an ADT(abstract data type)
for maintaining a set S of elements, each with an
associated value called priority
A PQ supports the following operations

Insert(x) insert element x in set S (SS{x})

Minimum() returns the element of S with smallest
priority.
Delete-min() returns and removes the element of S
with smallest priority.

Priorities and Total Order Relations

A Priority Queue ranks its elements by priority.
Every element has a priority. Priorities are not
necessarily unique and are totally ordered.
Total Order Relation, denoted by
Reflexive: k k
Antisymetric: if k1 k2 and k2 k1, then k1 k2
Transitive: if k1 k2 and k2 k3, then k1 k3

Comparators
The most general and reusable form of a priority
queue makes use of comparator objects.
Comparator objects are external to the keys that
are to be compared and compare two objects.
When the priority queue needs to compare two
keys, it uses the comparator it was given to do the
comparison.
Thus a priority queue can be general enough to
store any object.
The comparator ADT includes:

isLessThan(a, b), isLessThanOrEqualTo(a,b),

isEqualTo(a, b), isGreaterThan(a,b),
isGreaterThanOrEqualTo(a,b), isComparable(a)
6

Implem. with Unsorted Sequence

The items are pairs (priority, element)
We can implement insert() by using insertLast() on
the sequence. This takes O(1) time.

However, because we always insert at the end,

irrespective of the key value, our sequence is not
ordered.
7

Unsorted Sequence (contd.)

Thus, for methods such as minimum(),delete-min()

we need to look at all the elements of S. The worst
case time complexity for these methods is O(n).

Implem. with Sorted Sequence

Another implementation uses a sequence S,
sorted by increasing priorities.
minimum() and delete-min() take O(1) time.

However, to implement insert(), we must now

scan through the entire sequence in the worst
case. Thus insert() runs in O(n) time.

Priority Queues
Applications:
job

scheduling shared computing resources

(Unix)
Event simulation
As a building block for other algorithms
A

Heap can be used to implement a PQ

10

(Binary) Heaps
A binary tree that stores priorities (or priorityelement) pairs at nodes
Structural

property:
All levels except last
are full. Last level is
left-filled.
Heap property:
Priority of node is at
least as large as that 18
of its parent.
43

11
17

13

21
23

26

19
29

17

31
11

Examples of non-Heaps
Heap

property violated
11
19

13

18
43

21
23

26

19
29

17

31
12

Example of non-heap
Last

level not left-filled

11
17

13

18
43

21
26

19
29

17

31
13

Finding the minimum element

The

element with smallest priority always

sits at the root of the heap.
This is because if it was elsewhere, it
would have a parent with larger priority
and this would violate the heap property.
Hence minimum() can be done in O(1)
time.
14

Height of a heap
Suppose

a heap of n nodes has height h.

Recall: complete binary tree of height h
has 2h+1-1 nodes.
Hence 2h-1 < n <= 2h+1-1.
n = log2 h

15

Implementing Heaps
Parent (i)
return i/2

11
17

Left (i)
return 2i

21

18

Right (i)
return 2i+1

43

13

23

19

17

26

1 2 3 4 5 6 7 8 9 10
11 17 13 18 21 19 17 43 23 26

Level: 0

Heap property: A[Parent(i)] <= A[i]

16

Implementing Heaps (2)

Notice

the implicit tree links; children of

node i are 2i and 2i+1
Why is this useful?
In

a binary representation, a
multiplication/division by two is left/right shift
Adding 1 can be done by adding the lowest bit

17

Insertion in a Heap
Insert

12
Insert 8

11
17

13

18
43

21
23

26

17

19
29

31

12

18

Another View of Insertion

Enlarge heap
Consider path from root to inserted node
Find topmost element on this path with higher priority that
that of inserted element.
Insert new element at this location by shifting down the
other elements on the path
12
11
17

13

18
43

21
23

26

19
29

31

17

19

Correctness of Insertion
The only nodes whose contents change are the
ones on the path.
Heap property may be violated only for children
of these nodes.
But new contents of these nodes only have
lower priority.
11
So heap property not violated.

17

13

12
18
43

21
23

26

19
29

31

17

20

Heapify
i

is index into the array A

Binary trees rooted at Left(i) and Right(i)
are heaps
But, A[i] might be smaller than its children,
thus violating the heap property
The method Heapify makes binary tree
rooted at i a heap by moving A[i] down the
heap.
21

Heapify
Heap property violated at node with index 1 but
subtrees rooted at 2, 3 are heaps.
heapify(1)

17
10

11

16
43

21
23

26

13
29

31

12
19
22

Another View of Heapify

Heapify(i) traces a path down the tree.

Last node on path (say j) has both A[left(j)], A[right(j)] are
larger than A[i]
All elements on path have lower priority than their siblings.
All elements on this path are moved up. 17
A[i] goes to location j.
This establishes correctness
10
11

16
43

21
23

26

29

13
31

12
19
23

Running time Analysis

A

heap of n nodes has height O(log n).

While inserting we might have to move the
element all the way to the top.
Hence at most O(log n) steps required.
In Heapify, the element might be moved all
the way to the last level.
Hence Heapify also requires O(log n) time.

24

Binary Heaps
Delete-min
Building

a heap in O(n) time

Heap Sort

Delete-min
The

minimum element is the one at the top

of the heap.
We can delete this and move one of its
children up to fill the space.
Empty location moves down the tree.
Might end up at any position on last level.
Resulting tree would not be left filled.

Delete-min in a Heap
8
10

11

18
43

21
23

26

13
29

31

12
19

17

This is not a heap

3

Delete-min in a Heap (2)

Replace

top element with last element

of heap.
Heapify(1)

8
10

11

16
43

21
23

26

13
29

31

12
19

17

Building a heap
We start from the bottom and move up
All leaves are heaps to begin with

23
43

26

21

10
16

12

13
29

11

31
19

17
5

Building a Heap: Analysis

Correctness:

induction on i, all trees

rooted at m > i are heaps
Running time: n calls to Heapify = n O(lg
n) = O(n lg n)
We can provide a better O(n) bound.
Intuition:

for most of the time Heapify works

on smaller than n element heaps

Building a Heap: Analysis (2)

height

of node: length of longest path from

node to leaf
height of tree: height of root
time for Heapify(i) = O(height of subtree
rooted at i)
assume n = 2k 1 (a complete binary
tree)

Building a heap: Analysis (3)

For the n/2 nodes of height 1, heapify() requires
at most 1 swap each.
For the n/4 nodes of height 2, heapify() requires
at most 2 swaps each.
For the n/2i nodes of height i, heapify() requires
at most i swaps each.
So total number of swaps required is

n +1
n +1 n +1

T ( n) = O
+
2+
3 + ... + 1 k
4
8
2

lg n

i
= O ( n + 1) i
i =1 2

= O ( n)

since

lg n

i =1

i
1/ 2
=
=2
2
i
2 (1 1/ 2 )
8

Building a Heap: Analysis (4)

How? By using the following "trick"

1
if x < 1 //differentiate
1 x
i =0

1
i 1
//multiply by x
i

x
=

2
i =1
(1 x )
i
x
=

i
i

x
=
i =1

(1 x )

//plug in x =

1
2

i 1/ 2
=
=2

i
1/ 4
i =1 2

Therefore Build-Heap time is O(n)

9

Heap Sort
Create

a heap.
Do delete-min
8
repeatedly till
heap becomes
10
empty.
To do an in place
21
12
sort, we move
deleted element
16 23 43 29
to end of heap.

11

17

13
26

19

31

10

Running times of heap operations

Insert:

O(log n)
Heapify: O(log n)
Find minimum: O(1)
Delete-min: O(log n)
Building a heap: O(n)
Heap Sort: O(nlog n)

11

Why Sorting?
When

in doubt, sort one of the

principles of algorithm design. Sorting
used as a subroutine in many of the
algorithms:
Searching

in databases: we can do binary

search on sorted data
A large number of computer graphics and
computational geometry problems
Closest pair, element uniqueness
1

Why Sorting? (2)

A

large number of sorting algorithms are

developed representing different algorithm
design techniques.
A lower bound for sorting (n log n) is
used to prove lower bounds of other
problems

Sorting Algorithms so far

Insertion

sort, selection sort

Worst-case

Heap

running time (n2); in-place

sort

Worst-case

running time (n log n).

Divide and Conquer

Divide-and-conquer method for algorithm

design:
Divide: if the input size is too large to deal with in a
straightforward manner, divide the problem into two or
more disjoint subproblems
Conquer: use divide and conquer recursively to solve
the subproblems
Combine: take the solutions to the subproblems and
merge these solutions into a solution for the original
problem

Merge Sort Algorithm

Divide: If S has at least two elements (nothing
needs to be done if S has zero or one elements),
remove all the elements from S and put them
into two sequences, S1 and S2 , each containing
about half of the elements of S. (i.e. S1 contains
the first n/2 elements and S2 contains the
remaining n/2 elements).
Conquer: Sort sequences S1 and S2 using
Merge Sort.
Combine: Put back the elements into S by
merging the sorted sequences S1 and S2 into
one sorted sequence

Merge Sort: Algorithm

Merge-Sort(A,
Merge-Sort(A, p,
p, r)
r)
if
if pp << rr then
then
q(p+r)/2
q(p+r)/2
Merge-Sort(A,
Merge-Sort(A, p,
p, q)
q)
Merge-Sort(A,
Merge-Sort(A, q+1,
q+1, r)
r)
Merge(A,
Merge(A, p,
p, q,
q, r)
r)

Merge(A,
Merge(A, p,
p, q,
q, r)
r)
Take
Take the
the smallest
smallest of
of the
the two
two topmost
topmost elements
elements of
of
sequences
sequences A[p..q]
A[p..q] and
and A[q+1..r]
A[q+1..r] and
and put
put into
into the
the
resulting
resulting sequence.
sequence. Repeat
Repeat this,
this, until
until both
both sequences
sequences
are
are empty.
empty. Copy
Copy the
the resulting
resulting sequence
sequence into
into A[p..r].
A[p..r].

MergeSort (Example)

MergeSort (Example)

MergeSort (Example)

MergeSort (Example)

10

MergeSort (Example)

11

MergeSort (Example)

12

MergeSort (Example)

13

MergeSort (Example)

14

MergeSort (Example)

15

MergeSort (Example)

16

MergeSort (Example)

17

MergeSort (Example)

18

MergeSort (Example)

19

MergeSort (Example)

20

MergeSort (Example)

21

MergeSort (Example)

22

MergeSort (Example)

23

MergeSort (Example)

24

MergeSort (Example)

25

MergeSort (Example)

26

MergeSort (Example)

27

MergeSort (Example)

28

Merging Two Sequences (cont.)

29

Merging Two Sequences (cont.)

30

Merging Two Sequences (cont.)

31

Merging Two Sequences (cont.)

32

Merging Two Sequences (cont.)

33

Merge Sort Revisited

To sort n numbers
if n=1 done!
recursively sort 2 lists of
numbers n/2 and n/2
elements
merge 2 sorted lists in (n)
time

Strategy
break problem into similar
(smaller) subproblems
recursively solve
subproblems
combine solutions to answer

34

Recurrences
Running times of algorithms with Recursive
calls can be described using recurrences
A recurrence is an equation or inequality that
describes a function in terms of its value on
smaller inputs

solving_trivial_problem
if n = 1

T ( n) =
num_pieces T (n / subproblem_size_factor) + dividing + combining if n > 1

Example: Merge Sort

(1)
if n = 1

T ( n) =
2T (n / 2) + (n) if n > 1

35

Solving Recurrences

Repeated substitution method

Expanding the recurrence by substitution and noticing

patterns

Substitution method
guessing the solutions
verifying the solution by the mathematical induction

Recursion-trees
Master method

templates for different classes of recurrences

36

Repeated Substitution Method

Lets find the running time of merge sort (lets

assume that n=2b, for some b).
1
if n = 1

T (n) =
2T ( n / 2) + n if n > 1
T (n) = 2T ( n / 2 ) + n substitute
= 2 ( 2T ( n / 4 ) + n / 2 ) + n expand
= 22 T (n / 4) + 2n substitute
= 22 (2T (n / 8) + n / 4) + 2n expand
= 23 T (n / 8) + 3n

observe the pattern

T (n) = 2i T (n / 2i ) + in
= 2lg n T (n / n) + n lg n = n + n lg n
37

Repeated Substitution Method

The procedure is straightforward:

Substitute
Expand
Substitute
Expand

Observe a pattern and write how your expression

looks after the i-th substitution
Find out what the value of i (e.g., lgn) should be to get
the base case of the recurrence (say T(1))
Insert the value of T(1) and the expression of i into
your expression

38

Java Implementation of Merge-Sort

39

Java Implementation of MergeSort (cont.)

public class ListMergeSort implements SortObject {
public void sort(Sequence S, Comparator c) {
int n = S.size();
if (n < 2) return; //sequence with 0/1 element is sorted.
// divide
Sequence S1 = (Sequence)S.newContainer();
// put the first half of S into S1
for (int i=1; i <= (n+1)/2; i++) {
S1.insertLast(S.remove(S.first()));
}
Sequence S2 = (Sequence)S.newContainer();
// put the second half of S into S2
for (int i=1; i <= n/2; i++) {
S2.insertLast(S.remove(S.first()));
}
sort(S1,c); // recur
sort(S2,c);
merge(S1,S2,c,S); // conquer
40
}

Java Implementation of MergeSort (cont.)

public void merge(Sequence S1, Sequence S2, Comparator c, Sequence S) {
while(!S1.isEmpty() && !S2.isEmpty()) {
if(c.isLessThanOrEqualTo(S1.first().element(),
S2.first().element())) {
// S1s 1st elt <= S2s 1st elt
S.insertLast(S1.remove(S1.first()));
}else { // S2s 1st elt is the smaller one
S.insertLast(S2.remove(S2.first()));
}
}if(S1.isEmpty()) {
while(!S2.isEmpty()) {
S.insertLast(S2.remove(S2.first()));
}
}if(S2.isEmpty()) {
while(!S1.isEmpty()) {
S.insertLast(S1.remove(S1.first()));
}}}
41

More Sorting
radix sort
bucket sort
in-place sorting
how fast can we sort?

Radix Sort

Unlike other sorting methods, radix sort considers the

structure of the keys
Assume keys are represented in a base M number
system (M = radix), i.e., if M = 2, the keys are
represented in binary

Sorting is done by comparing bits in the same position

Extension to keys that are alphanumeric strings

Radix Exchange Sort

Examine bits from left to right:
1. Sort array with respect to leftmost bit
2. Partition array

3.Recursion
recursively sort top subarray, ignoring leftmost bit
recursively sort bottom subarray, ignoring leftmost bit
Time to sort n b-bit numbers:

O(b n)
3

Radix Exchange Sort

How do we do the sort from the previous page? Same idea as
partition in Quicksort.
repeat
scan top-down to find key starting with 1;
scan bottom-up to find key starting with 0;
exchange keys;
until scan indices cross;

Radix Exchange Sort

Radix Exchange Sort vs. Quicksort

Similarities
both partition array
both recursively sort sub-arrays
Differences
Method of partitioning
radix exchange divides array based on greater than or
less than 2b-1
quicksort partitions based on greater than or less than
some element of the array
Time complexity
Radix exchange
O (bn)
Quicksort average case O (n log n)
6

Straight Radix Sort

Examines bits from right to left
for k := 0 to b-1
sort the array in a stable way,
looking only at bit k

Note order of these bits after sort.

7

What is sort in a stable way!!!

In a stable sort, the initial relative order of equal keys is
unchanged. For example, observe the first step of the sort from
the previous page:

Note that the relative order of those keys ending with 0 is

unchanged, and the same is true for elements ending in 1
8

The Algorithm is Correct (right?)

We show that any two keys are in the correct relative order at the
end of the algorithm
Given two keys, let k be the leftmost bit-position where they
differ

At step k the two keys are put in the correct relative order
Because of stability, the successive steps do not change the
relative order of the two keys
9

For Instance,
Consider a sort on an array with these two keys:

10

Radix sorting applied to decimal numbers

11

Straight Radix Sort Time Complexity

for k = 0 to b - 1
sort the array in a stable way, looking only at bit k

Suppose we can perform the stable sort above in O(n) time.

The total time complexity would be

O(bn)
As you might have guessed, we can perform a stable sort based
on the keys kth digit in O(n) time.
The method, you ask? Why its Bucket Sort, of course.

12

Bucket Sort
BASICS:
n numbers
Each number {1, 2, 3, ... m}
Stable
Time: O (n + m)
For example, m = 3 and our array is:
(note that there are two 2s and two 1s)
First, we create M buckets
13

Bucket Sort
Each element of the array is put in one of the m buckets

14

Bucket Sort

Now, pull the elements from

the buckets into the array

At last, the sorted array (sorted

in a stable way):
15

In-Place Sorting

A sorting algorithm is said to be in-place if

it uses no auxiliary data structures (however, O(1)
auxiliary variables are allowed)
it updates the input sequence only by means of
operations replaceElement and swapElements

Which sorting algorithms seen can be made to

work in place?

16

Lower Bd. for Comparison Based Sorting

internal node: comparison

external node: permutation
algorithm execution: root-to-leaf path

17

How Fast Can We Sort?

How Fast Can We Sort?
Proposition: The running time of any
comparison-based algorithm for sorting an nelement sequence S is (n log n).
Justification: The running time of a
comparison-based sorting algorithm must be
equal to or greater than the depth of the
decision tree T associated with this algorithm.

18

How Fast Can We Sort? (2)

Each internal node of T is associated with a
comparison that establishes the ordering of two
elements of S.
Each external node of T represents a distinct
permutation of the elements of S.
Hence T must have at least n! external nodes which
again implies T has a height of at least log(n!)
Since n! has at least n/2 terms that are greater than
or equal to n/2, we have: log(n!) (n/2) log(n/2)
Total Time Complexity: (n log n).

19

Graphs
Definitions
Examples
The Graph ADT

What is a Graph?

A graph G = (V,E) is composed of:

V: set of vertices
E: set of edges connecting the vertices in V
An edge e = (u,v) is a pair of vertices
Example:

E= {(a,b),(a,c),(a,d),
(b,e),(c,d),(c,e),
(d,e)}

c
d

V= {a,b,c,d,e}

Applications

electronic circuits

CS201

start

find the path of least resistance to CS201

networks (roads, flights, communications)
3

more examples
scheduling (project planning)
A typical student day

wake up

eat

cs201 meditation
work

more cs201

play
cs201 program
sleep
dream of cs201
4

Graph Terminology
adjacent vertices: vertices connected by an edge
degree (of a vertex): # of adjacent vertices

What is the sum of the degrees of all vertices?

Twice the number of edges, since adjacent
vertices each count the adjoining edge, it will be
counted twice
3
2

3
3

3
5

Graph Terminology(2)
path:

sequence of vertices v1,v2,. . .vk

such that consecutive vertices vi and vi+1
are adjacent.
a

a
c

c
e

d
abedce

d
bedc

Graph Terminology (3)

a

simple path: no repeated

vertices
cycle: simple path, except
that the last vertex is the
same as the first vertex
a

bec

c
e

d
b
acda

c
d

e
7

More Terminology
connected graph: any two vertices are
connected by some path

connected

not connected

More Terminology(2)

subgraph: subset of vertices and edges forming

a graph

subgraph of G
9

More Terminology(3)

connected component: maximal connected

subgraph. E.g., the graph below has 3
connected components.

10

Yet Another Terminology Slide!

(free) tree connected graph
without cycles
forest - collection
of trees

tree

tree

for est
tree
tree

11

Connectivity
Let n = #vertices, and m = #edges
Complete graph: one in which all pairs of
vertices are adjacent
How many edges does a complete graph have?
There are n(n-1)/2 pairs of vertices and so
m = n(n -1)/2.
Therefore, if a graph is not complete,
m < n(n -1)/2

n=5
m = (5 4)/2 = 10
12

More Connectivity

n = #vertices
m = #edges
For a tree m = n 1
If m < n - 1, G is not
connected

n=5
m=4

n=5
m=3

13

Spanning Tree

A spanning tree of G is a subgraph which is a tree and

which contains all vertices of G

G
spanning tree of G
Failure on any edge disconnects system (least fault
tolerant)
14

The Bridges of Koenigsberg

Gilligan

s Isle?

Pregal River

A
B

Can one walk across each bridge

exactly once and return at the
starting point?

Suppose you are a postman, and you didnt want

to retrace your steps.
In 1736, Euler proved that this is not possible

15

Graph Model (with parallel edges)

Eulerian Tour: path
that traverses every
edge exactly once
and returns to the first
vertex
Eulers Theorem: A
graph has a Eulerian
Tour if and only if all
vertices have even
degree

C
A

D
B
16

The Graph ADT

The Graph ADT is a positional container whose positions

are the vertices and the edges of the graph.

size()
Return the number of vertices + number of edges of G.
isEmpty()
elements()
positions()
swap()
replaceElement()

Notation: Graph G; Vertices v, w; Edge e; Object o

numVertices()
numEdges()
vertices()
edges()

Return the number of vertices of G.

Return the number of edges of G.
Return an enumeration of the vertices of G.
Return an enumeration of the edges of G.

17

The Graph ADT (contd.)

directedEdges() enumeration of all directed edges in G.

undirectedEdges() enumeration of all undirected edges in G.
incidentEdges(v) enumeration of all edges incident on v.
inIncidentEdges(v) enumeration of all edges entering v.
outIncidentEdges(v) enumeration of all edges leaving v.
opposite(v, e) an endpoint of e distinct from v
degree(v) the degree of v.
inDegree(v) the in-degree of v.
outDegree(v) the out-degree of v.

18

More Methods ...

adjacentVertices(v) enumeration of vertices adjacent to v.

inAdjacentVertices(v) enumeration of vertices adjacent to
v along incoming edges.
outAdjacentVertices(v) enumeration of vertices adjacent to
v along outgoing edges.
areAdjacent(v,w) whether vertices v and w are adjacent.
endVertices(e) the end vertices of e.
origin(e) the end vertex from which e leaves.
destination(e) the end vertex at which e arrives.
isDirected(e) true iff e is directed.

19

Update Methods

makeUndirected(e) Set e to be an undirected edge.

reverseDirection(e) Switch the origin and destination
vertices of e.
setDirectionFrom(e, v) Sets the direction of e away from
v, one of its end vertices.
setDirectionTo(e, v) Sets the direction of e toward v, one
of its end vertices.
insertEdge(v, w, o) Insert and return an undirected edge
between v and w, storing o at this position.
insertDirectedEdge(v, w, o) Insert and return a directed
edge between v and w, storing o at this position.
insertVertex(o) Insert and return a new (isolated) vertex
storing o at this position.
removeEdge(e) Remove edge e.
20

Data Structures for Graphs

Edge list
Adjacency lists
Adjacency matrix

NW 35

DL 247

BOS

AA 49

LAX

DL 335

AA 1387

DFW

E
AA 523

JFK

AA 41 1

UA 120

MIA

AA 903

ORD

UA 877

TW 45

SFO
21

Data Structures for Graphs

A Graph! How can we represent it?

To start with, we store the vertices and the edges into two
containers, and each edge object has references to the
TW 45
BOS
vertices it connects.
ORD
JFK
SFO

DFW
LAX
MIA

Additional structures can be used to perform efficiently the methods

of the Graph ADT
22

Edge List

The edge list structure simply stores the vertices and the
edges into unsorted sequences.
Easy to implement.
Finding the edges incident on a given vertex is inefficient
since it requires examining the entire edge sequence
E
NW 35

DL 247

BOS

AA 49

LAX

DL 335 AA 1387 AA 523

DFW

JFK

AA 411

UA 120

MIA

AA 903

ORD

UA 877

TW 45

SFO
23

Performance of the Edge List Structure

Operation

Time

size, isEmpty, replaceElement, swap

O(1)

numVertices, numEdges

O(1)

vertices

O(n)

edges, directedEdges, undirectedEdges

O(m)

elements, positions

O(n+m)

endVertices, opposite, origin, destination, isDirected

O(1)

incidentEdges, inIncidentEdges, outIncidentEdges, adjacent

Vertices , inAdjacentVertices, outAdjacentVertices,areAdjacent,
degree, inDegree, outDegree

O(m)

insertVertex, insertEdge, insertDirectedEdge,

removeEdge, makeUndirected, reverseDirection,
setDirectionFrom, setDirectionTo

O(1)

removeVertex

O(m)24

Adjacency List (traditional)

adjacency list of a vertex v:

sequence of vertices adjacent to v
represent the graph by the adjacency lists of all the vertices
a

b
c
e

d
b

Space =
(N + deg(v)) = (N + M)

25

Adjacency List (modern)

The adjacency list structure extends the edge list structure by

adding incidence containers to each vertex.
NW

35

DL 247

AA

BOS

in

DL

DL

LAX

out
NW

49

in
35

247

AA
AA

335

41 1

1387

DFW

out
49

AA

UA

in
120

UA

877

AA

523

523

AA

JFK

out

AA1387

AA

in

DL335
AA

49

NW

41 1

120

MIA

out
35

UA

in

AA1387
AA
TW

903

AA

903

UA

ORD

out

in

DL

247

AA523

UA

120

AA

903

AA

DL

335

41 1

877

TW

45

SFO

out
UA

in
877

TW

out
45

45

The space requirement is O(n + m).

26

Performance of the Adjacency List Structure

size, isEmpty, replaceElement, swap

O(1)

numVertices, numEdges

O(1)

vertices

O(n)

edges, directedEdges, undirectedEdges

O(m)

elements, positions

O(n+m)

endVertices, opposite, origin, destination, isDirected, degree,

inDegree, outDegree

O(1)

incidentEdges(v), inIncidentEdges(v), outIncidentEdges(v),

adjacentVertices(v), inAdjacentVertices(v),
outAdjacentVertices(v)
areAdjacent(u, v)

O(deg(v))

insertVertex, insertEdge, insertDirectedEdge, removeEdge,

makeUndirected, reverseDirection, insertVertex, insertEdge,
insertDirectedEdge, removeEdge, makeUndirected,
reverseDirection,
removeVertex(v)

O(1)

O(min(deg(
u),deg(v)))

27
O(deg(v))

Adjacency Matrix (traditional)

a

b
a
b
c
d
e

c
d

F
T
T
T
F

T
F
F
F
T

T
F
F
T
T

T
F
T
F
T

F
T
T
T
F

matrix M with entries for all pairs of vertices

M[i,j] = true means that there is an edge (i,j) in the graph.
M[i,j] = false means that there is no edge (i,j) in the graph.
There is an entry for every possible edge, therefore:
Space = (N2)

28

Adjacency Matrix (modern)

The adjacency matrix structures augments the edge list

structure with a matrix where each row and column
corresponds to a vertex.

29

Performance of the Adjacency

Matrix Structure

30

Data Structures for Graphs

Edge list
Adjacency lists
Adjacency matrix

NW 35

DL 247

BOS

AA 49

LAX

DL 335

AA 1387

DFW

E
AA 523

JFK

AA 41 1

UA 120

MIA

AA 903

ORD

UA 877

TW 45

SFO
1

Data Structures for Graphs

A Graph! How can we represent it?

To start with, we store the vertices and the edges into two
containers, and each edge object has references to the
vertices it connects.

Additional structures can be used to perform efficiently the methods

of the Graph ADT
2

Edge List

The edge list structure simply stores the vertices and

the edges in two unsorted sequences.
Easy to implement.
Finding the edges incident on a given vertex is
inefficient since it requires examining the entire edge
E
sequence
NW 35

DL 247

BOS

AA 49

LAX

DL 335 AA 1387 AA 523

DFW

JFK

AA 411

UA 120

MIA

AA 903

ORD

UA 877

TW 45

SFO
3

Performance of Edge List

Operation

Time

size, isEmpty, replaceElement, swap

O(1)

numVertices, numEdges

O(1)

vertices

O(n)

edges, directedEdges, undirectedEdges

O(m)

elements, positions

O(n+m)

endVertices, opposite, origin, destination, isDirected

O(1)

incidentEdges, inIncidentEdges, outIncidentEdges, adjacent

Vertices , inAdjacentVertices, outAdjacentVertices,areAdjacent,
degree, inDegree, outDegree

O(m)

insertVertex, insertEdge, insertDirectedEdge,

removeEdge, makeUndirected, reverseDirection,
setDirectionFrom, setDirectionTo

O(1)

removeVertex

O(m) 4

Adjacency List (traditional)

adjacency list of a vertex v: sequence of vertices

adjacent to v
represent the graph by the adjacency lists of all the
vertices
a

b
c
e

d
b

Space =
(N + deg(v)) = (N + M)

Adjacency List (modern)

The adjacency list structure extends the edge list

structure by adding adjacency lists to each vertex.
NW

35

DL 247

AA

BOS

in

DL

DL

LAX

out
NW

49

in
35

247

AA
AA

335

41 1

1387

DFW

out
49

AA

UA

in
120

UA

877

AA

523

523

AA

JFK

out

AA1387

AA

in

DL335
AA

49

NW

41 1

120

MIA

out
35

UA

in

AA1387
AA
TW

903

AA

903

UA

ORD

out

in

DL

247

AA523

UA

120

AA

903

AA

DL

335

41 1

877

TW

45

SFO

out
UA

in
877

TW

out
45

45

The space requirement is O(n + m).

6

Performance of the Adjacency List Structure

size, isEmpty, replaceElement, swap

O(1)

numVertices, numEdges

O(1)

vertices

O(n)

edges, directedEdges, undirectedEdges

O(m)

elements, positions

O(n+m)

endVertices, opposite, origin, destination, isDirected, degree,

inDegree, outDegree

O(1)

incidentEdges(v), inIncidentEdges(v), outIncidentEdges(v),

adjacentVertices(v), inAdjacentVertices(v),
outAdjacentVertices(v)

O(deg(v))

areAdjacent(u, v)

O(min(deg(
u),deg(v)))

insertVertex, insertEdge, insertDirectedEdge, removeEdge,

makeUndirected, reverseDirection,

O(1)

removeVertex(v)

O(deg(v))
7

Adjacency Matrix (traditional)

a

b
a
b
c
d
e

c
d

F
T
T
T
F

T
F
F
F
T

T
F
F
T
T

T
F
T
F
T

F
T
T
T
F

matrix M with entries for all pairs of vertices

M[i,j] = true means that there is an edge (i,j) in the graph.
M[i,j] = false means that there is no edge (i,j) in the graph.
There is an entry for every possible edge, therefore:
Space = (N2)
8

Adjacency Matrix (modern)

The adjacency
matrix structures
augments the edge
list structure with a
matrix where each
row and column
corresponds to a
vertex.

Performance of Adjacency Matrix

10

Graph Searching Algorithms

Systematic search of every edge and vertex of
the graph
Graph G = (V,E) is either directed or undirected
Today's algorithms assume an adjacency list
representation
Applications

Compilers
Graphics
Maze-solving
Mapping
Networks: routing, searching, clustering, etc.

11

Breadth First Search

A Breadth-First Search (BFS) traverses a
connected component of a graph, and in doing
so defines a spanning tree with several useful
properties
BFS in an undirected graph G is like wandering
in a labyrinth with a string.
The starting vertex s, it is assigned a distance 0.
In the first round, the string is unrolled the length
of one edge, and all of the vertices that are only
one edge away from the anchor are visited
(discovered), and assigned distances of 1

12

Breadth-First Search (2)

In the second round, all the new vertices that
can be reached by unrolling the string 2 edges
are visited and assigned a distance of 2
This continues until every vertex has been
assigned a level
The label of any vertex v corresponds to the
length of the shortest path (in terms of edges)
from s to v

13

BFS Example
r

Q s

Q r t x

1 2 2

Qw r

1 1

Q t x v

2 2 2

14

BFS Example
r

Q x v u

2 2 3

Q u y

3 3

Q v u y

2 3 3

Q y

15

BFS Example: Result

r

Q -

16

BFS - A Graphical representation

0

a)

0
A

c)

b)

d)

17

More BFS
0

18

BFS Algorithm
BFS(G,s)
01 for each vertex u V[G]-{s}
02
color[u] white
03
d[u]
04
[u] NIL
05 color[s] gray
06 d[s] 0
07 [u] NIL
08 Q {s}
09 while Q do
10
u head[Q]
11
for each v Adj[u] do
12
if color[v] = white then
13
color[v] gray
14
d[v] d[u] + 1
15
[v] u
16
Enqueue(Q,v)
17
Dequeue(Q)
18
color[u] black

Init all
vertices
Init BFS
with s
Handle all us
children
before
handling any
children of
children
19

BFS Running Time

Given a graph G = (V,E)

Vertices are enqueued if there color is white
Assuming that en- and dequeuing takes O(1) time the
total cost of this operation is O(V)
Adjacency list of a vertex is scanned when the vertex
is dequeued (and only then)
The sum of the lengths of all lists is (E).
Consequently, O(E) time is spent on scanning them
Initializing the algorithm takes O(V)

Total running time O(V+E) (linear in the size of

the adjacency list representation of G)
20

BFS Properties
Given an undirected graph G = (V,E), BFS
discovers all vertices reachable from a
source vertex s
For each vertex v at level i, the path of the BFS
tree T between s and v has i edges, and any
other path of G between s and v has at least i
edges.
If (u, v) is an edge then the level numbers of u
and v differ by at most one.
It computes the shortest distance to all
reachable vertices

21

Breadth First Tree

Predecessor subgraph of G
G = (V , E )

V = {v V : [v] NIL} {s}

E = {([v], v) E : v V {s}}

G is a breadth-first tree

V consists of the vertices reachable from s, and

for all v V, there is a unique simple path from s to v
in G that is also a shortest path from s to v in G

The edges in G are called tree edges

For any vertex v reachable from s, the path in the
breadth first tree from s to v, corresponds to a
shortest path in G

22