CSE-243: Algorithm Design and Analysis

 Lecture 08-09: Solving Recurrence Relations

Instructor Teaching Assistants
Md. Sabir Hossain Rakibul Islam
Lecturer Id: 1604060
Dept. of CSE, CUET Email: [email protected]
Mobile: 01737143868/01882826575 Saiful Islam Rimon
Email: [email protected]
Id: 1604061
Email : [email protected]
Acknowledgment: Most of the content of this slide is adopted from the book: Fundamentals of Computer Algorithms by Ellis
Horowitz, Sanguthevar Rajasekaran, Sartaj Sahni, Introduction to Algorithms by Thomas H. Cormen, Charles E. Leiserson,
Ronald L. Rivest, Clifford Stein and
Recap

 BigOh, O, Big Theta, Θ, Big Omega, Ω,

Little o and Little 
 Mathematical Induction: Review
 Example Worked out
Today’s Target

 Review of Recursion
 Solving Techniques
Substitution Method
Recurrence Tree Method
Master Method
What are the Basic Steps of
Mathematical Induction

 Let us denote the proposition in question by P (n), where n is a positive

integer. The proof involves two steps:

 Step 1: Basis (Base Case) We first establish that the proposition P (n) is true for
the lowest possible value of the positive integer n (=0 or 1).

 Step 2: (Inductive Step) We assume that P (k) is true and establish that P (k+1) is
also true
Recurrences and Running Time
 An equation or inequality that describes a function in
terms of its value on smaller inputs.

T(n) = T(n-1) + n

 Recurrences arise when an algorithm contains recursive

calls to itself

 What is the actual running time of the algorithm?

 Need to solve the recurrence

 Find an explicit formula of the expression

5  Bound the recurrence by an expression that involves n

Example Recurrences

 T(n) = T(n-1) + n Θ(n2)

 Recursive algorithm that loops through the input to eliminate one item
 T(n) = T(n/2) + c Θ(lgn)
 Recursive algorithm that halves the input in one step
 T(n) = T(n/2) + n Θ(n)
 Recursive algorithm that halves the input but must examine every item in the input
 T(n) = 2T(n/2) + 1 Θ(n)
 Recursive algorithm that splits the input into 2 halves and does a constant amount
of other work

6
Recurrent Algorithms: BINARY-SEARCH
 for an ordered array A, finds if x is in the array A[lo…hi]

Alg.: BINARY-SEARCH (A, lo, hi, x)

1 2 3 4 5 6 7 8

if (lo > hi) 2 3 5 7 9 10 11 12

return FALSE
mid  (lo+hi)/2 lo
mid
hi

if x = A[mid]
return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x)
if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x) 7
Example

 A[8] = {1, 2, 3, 4, 5, 7, 9, 11}

 lo = 1 hi = 8 x=7

1 2 3 4 5 6 7 8

1 2 3 4 5 7 9 11 mid = 4, lo = 5, hi = 8

5 6 7 8

1 2 3 4 5 7 9 11 mid = 6, A[mid] = x
Found!

8
Another Example

 A[8] = {1, 2, 3, 4, 5, 7, 9, 11}

– lo = 1 hi = 8 x=6
1 2 3 4 5 6 7 8

1 2 3 4 5 7 9 11 mid = 4, lo = 5, hi = 8

low high

1 2 3 4 5 7 9 11 mid = 6, A[6] = 7, lo = 5, hi = 5

low high

1 2 3 4 5 7 9 11 mid = 5, A[5] = 5, lo = 6, hi = 5
NOT FOUND!

1 2 3 4 5 7 9 11
high low

9
Analysis of BINARY-SEARCH
Alg.: BINARY-SEARCH (A, lo, hi, x)
if (lo > hi) constant time: c1
return FALSE
mid  (lo+hi)/2 constant time: c2

if x = A[mid] constant time: c3

return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x) same problem of size n/2

if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x) same problem of size n/2

 T(n) = c + T(n/2)
T(n) – running time for an array of size n
10

Methods of Solving Recurrence Relations
 Substitution
 Iteration Method
 Accounting method
 Draw the recursion tree
 The Master Theorem*
 Guess at an upper bound, prove it

* See Cormen, Leiserson, & Rivest, Introduction to Algorithms

Md. Sabir Hossain, CSE, CUET
Methods of Solving Recurrence Relations
 Substitution
 Iteration Method
 Draw the recursion tree
 The Master Theorem

Md. Sabir Hossain, CSE, CUET

Substitution Method

 The most general method:

 Guess the form of the solution.
 Verify by induction.
 Solve for constants.
Substitution Method: Example 4.1

 Consider the recurrence tn=2tn-1 +1. n>=1, with

initial condition t0=0. Obtain the solution using
substation method.
Solution
 Computing the first values

n 1 2 3 4 5 6
tn 1 3 7 15 31 63

 From above computation we can guess the solution as

tn=2n -1 for all n>=0
 By mathematical induction method we can easily prove that our guess is correct.
 Thus,
if n=0, then t0=0=20-1 also the statement is correct for n=1.

 Now, for some k>=0

tk=2k -1
Solution
 By induction hypothesis,
tk+1=2tk+ 1
=2(2k - 1)+1
=2k+1-1
 Thus we can say that as our guess as the solution is correct.
Self Study

 Example: 4.2-4.10
Methods of Solving Recurrence Relations
 Substitution
 Iteration Method
 Draw the recursion tree
 The Master Theorem

Md. Sabir Hossain, CSE, CUET

 Eg. 1 - Linear Search
• Recursively
• Look at an element (constant work, c),
then search the remaining elements…

• T(n) = T( n-1 ) + c
• “The cost of searching n elements is the
cost of looking at 1 element, plus the cost
of searching n-1 elements”
Md. Sabir Hossain, CSE, CUET
 Linear Seach (cont)
Caveat:
• You need to convince yourself (and others) that the single
step, examining an element, *is* done in constant time.
• Can I get to the ith element in constant time, either
directly, or from the (i-1)th element?
• Look at the code

Md. Sabir Hossain, CSE, CUET

 Linear Search (cont.)
• We’ll “unwind” a few of these
T(n) = T(n-1) + c (1)
But, T(n-1) = T(n-2) + c, from above
Substituting back in:
T(n) = T(n-2) + c + c
Gathering like terms
T(n) = T(n-2) + 2c (2)

Md. Sabir Hossain, CSE, CUET

 Linear Search (cont.)
• Keep going:
T(n) = T(n-2) + 2c
T(n-2) = T(n-3) + c
T(n) = T(n-3) + c + 2c
T(n) = T(n-3) + 3c (3)
• One more:
T(n) = T(n-4) + 4c (4)

Md. Sabir Hossain, CSE, CUET

 Looking for Patterns
• Note, the intermediate results are enumerated
• We need to pull out patterns, to write a general
expression for the kth unwinding
– This requires practise. It is a little bit art. The brain learns
patterns, over time. Practise.
• Be careful while simplifying after substitution

Md. Sabir Hossain, CSE, CUET

 Eg. 1 – list of intermediates
Result at ith unwinding i

T(n) = T(n-1) + 1c 1

T(n) = T(n-2) + 2c 2

T(n) = T(n-3) + 3c 3

T(n) = T(n-4) + 4c 4
Md. Sabir Hossain, CSE, CUET
 Linear Search (cont.)
• An expression for the kth unwinding:
T(n) = T(n-k) + kc
• We have 2 variables, k and n, but we have a relation
• T(d) is constant (can be determined) for some constant d
(we know the algorithm)
• Choose any convenient # to stop.

Md. Sabir Hossain, CSE, CUET

 Linear Search (cont.)
• Let’s decide to stop at T(0). When the list
to search is empty, you’re done…
• 0 is convenient, in this example…
Let n-k = 0 => n=k
• Now, substitute n in everywhere for k:
T(n) = T(n-n) + nc
T(n) = T(0) + nc = nc + c0 = O(n)
( T(0) is some constant, c0 )

Md. Sabir Hossain, CSE, CUET

 Binary Search
• Algorithm – “check middle, then search lower ½ or
upper ½”
• T(n) = T(n/2) + c
where c is some constant, the cost of checking the middle…
• Can we really find the middle in constant time? (Make
sure.)

Md. Sabir Hossain, CSE, CUET

 Binary Search (cont)
Let’s do some quick substitutions:
T(n) = T(n/2) + c (1)
but T(n/2) = T(n/4) + c, so
T(n) = T(n/4) + c + c
T(n) = T(n/4) + 2c (2)
T(n/4) = T(n/8) + c
T(n) = T(n/8) + c + 2c
T(n) = T(n/8) + 3c (3)
Md. Sabir Hossain, CSE, CUET
 Binary Search (cont.)
Result at ith unwinding i

T(n) = T(n/2) + c 1

T(n) = T(n/4) + 2c 2

T(n) = T(n/8) + 3c 3

T(n) = T(n/16) + 4c 4

Md. Sabir Hossain, CSE, CUET

 Binary Search (cont)
• We need to write an expression for the kth unwinding (in n
& k)
– Must find patterns, changes, as i=1, 2, …, k
– This can be the hard part
– Do not get discouraged! Try something else…
– We’ll re-write those equations…
• We will then need to relate n and k

Md. Sabir Hossain, CSE, CUET

 Binary Search (cont)
Result at ith unwinding i

T(n) = T(n/2) + c =T(n/21) + 1c 1

T(n) = T(n/4) + 2c =T(n/22) + 2c 2

T(n) = T(n/8) + 3c =T(n/23) + 3c 3

T(n) = T(n/16) + 4c =T(n/24) + 4c 4

Md. Sabir Hossain, CSE, CUET
 Binary Search (cont)
• After k unwindings:
T(n) = T(n/2k) + kc
• Need a convenient place to stop unwinding – need to
relate k & n
• Let’s pick T(0) = c0 So,
n/2k = 0 =>
n=0
Hmm. Easy, but not real useful…

Md. Sabir Hossain, CSE, CUET

 Binary Search (cont)
• Okay, let’s consider T(1) = c0
• So, let:
n/2k = 1 =>
n = 2k =>
k = log2n = lg n

Md. Sabir Hossain, CSE, CUET

 Binary Search (cont.)
• Substituting back in (getting rid of k):
T(n) = T(1) + c lg(n)
= c lg(n) + c0
= O( lg(n) )

Md. Sabir Hossain, CSE, CUET

Methods of Solving Recurrence Relations
 Substitution
 Iteration Method
 Draw the recursion tree
 The Master Theorem

Md. Sabir Hossain, CSE, CUET

Self Study

 Example: 4.23-4.28
Self Study

 From Coreman Book

 Example of pages: 95-96
Methods of Solving Recurrence Relations
 Substitution
 Iteration Method
 Draw the recursion tree
 The Master Theorem

Md. Sabir Hossain, CSE, CUET

Recursion-tree method

 A recursion tree models the costs (time) of a recursive

execution of an algorithm.
 The recursion-tree method promotes intuition, however.
 The recursion tree method is good for generating guesses
for the substitution method.
Example of recursion tree
Example of recursion tree
Example of recursion tree
Example of recursion tree
Example of recursion tree
Example of recursion tree
Example of recursion tree
Example of recursion tree
Recap: Geometric Series
Self Study

 Example: 4.13-4.16
Next 2/3 Classes

 Divide and Conquer Algorithms

References

 Introduction to Algorithms, Third Edition (International Edition), Thomas

H. Cormen, Charles E. Leiserson, Ronald L. Rivest, Clifford Stein, ISBN-13:
978-0262033848. Page: 41-50
 Fundamentals of Computer Algorithms, Second Edition, Ellis Horowitz,
Sanguthevar Rajasekaran, Sartaj Sahni, ISBN 10: 8173716129 / ISBN 13:
9788173716126, Published by Universities Press/Orient BlackSwan, 2008.
Page: 29-49
 Allah Hafez