Design and Analysis of Algorithms

CSE 5311
Lecture 4 Master Theorem

Junzhou Huang, Ph.D.

Department of Computer Science and Engineering

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 1

 Reviewing: Solving Recurrences
• Recurrence
– The analysis of integer multiplication from last lecture required us to solve a
recurrence
– Recurrences are a major tool for analysis of algorithms
– Divide and Conquer algorithms which are analyzable by recurrences.

• Three steps at each level of the recursion:

– Divide the problem into a number of subproblems that are smaller
instances of the same problem.
– Conquer the subproblems by solving them recursively. If the subproblem
sizes are small enough, however, just solve the subproblems in a
straightforward manner.
– Combine the solutions to the subproblems into the solution for the original
problem.

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 2

 Recall: Integer Multiplication

• Let X = A B and Y = C D where A,B,C and D are n/2

bit integers
• Simple Method: XY = (2n/2A+B)(2n/2C+D)
• Running Time Recurrence
T(n) < 4T(n/2) + Θ(n)

How do we solve it?

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 3

 Reviewing: Substitution Method

The most general method:

1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.

Example: T(n) = 4T(n/2) + Θ(n)

• [Assume that T(1) = Θ(1).]
• Guess O(n3) . (Prove O and Ω separately.)
• Assume that T(k) ≤ ck3 for k < n .
• Prove T(n) ≤ cn3 by induction.

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 4

 The Master Method
The master method applies to recurrences of the form
T(n) = a T(n/b) + f (n) ,
where a ≥ 1, b > 1, and f is asymptotically positive.

1. f (n) = O(nlogba – ε) for some constant ε > 0. Then, T(n) = Θ(nlogba)

2. f (n) = Θ(nlogba ) for k ≥ 0. Then, T(n) = Θ(nlogba lgn) .

3. f (n) = Ω(nlogba + ε) for some constant ε > 0 and f (n) satisfies

the regularity condition that a f (n/b) ≤ c f (n) for some
constant c < 1. Then, T(n) = Θ( f (n) )

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 5

 Application of Master Theorem

• T(n) = 9T(n/3)+n;
– a=9,b=3, f(n) =n
– nlogba = nlog39 = Θ (n2)
– f(n)=O(nlog39-ε) for ε=1
– By case 1, T(n) =Θ (n2).
• T(n) = T(2n/3)+1
– a=1,b=3/2, f(n) =1
– nlogba = nlog3/21 = Θ (n0) = Θ (1)
– By case 2, T(n)= Θ(lg n).

6
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 6
 Application of Master Theorem

• T(n) = 3T(n/4)+nlg n;
– a=3,b=4, f(n) =nlg n
– nlogba = nlog43 = Θ (n0.793)
– f(n)= Ω(nlog43+ε) for ε≈0.2
– Moreover, for large n, the “regularity” holds for c=3/4.

af(n/b) =3(n/4)lg (n/4) ≤ (3/4)nlg n = cf(n)
– By case 3, T(n) =Θ (f(n))=Θ (nlg n).

7
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 7
 Exception to Master Theorem

• T(n) = 2T(n/2)+nlg n;
– a=2,b=2, f(n) =nlg n
– nlogba = nlog22 = Θ (n)
– f(n) is asymptotically larger than nlogba , but not polynomially
larger because
– f(n)/nlogba = lg n, which is asymptotically less than nε for any
ε>0.
– Therefore, this is a gap between 2 and 3.

8
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 8
 Where Are the Gaps

f(n), case 3, at least polynomially larger

nε Gap between case 3 and 2
c1
nlogba f(n), case 2: within constant distances
c2
nε Gap between case 1 and 2
f(n), case 1, at least polynomially smaller

Note: 1. for case 3, the regularity also must hold.

2. if f(n) is lg n smaller, then fall in gap in 1 and 2
3. if f(n) is lg n larger, then fall in gap in 3 and 2
4. if f(n)=Θ(nlogbalgkn), then T(n)=Θ(nlogbalgk+1n) (as exercise)
9
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 9
 Master Theorem
The master method applies to recurrences of the form
T(n) = a T(n/b) + f (n) ,
where constants a ≥ 1, b > 1, and f is asymptotically positive function

1. f (n) = O(nlogba – ε) for some constant ε > 0, then T(n) = Θ(nlogba)

2. f (n) = O(nlogba ) for some constant ε > 0, then T(n) = Θ(nlogba lgn)
3. f (n) = O(nlogba + ε) for some constant ε > 0, and if a f (n/b) ≤ c f (n)
for some constant c < 1, then T(n) = Θ( f (n) ) .

How to theoretically prove it?

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 10

 Proof for Exact Powers
• Suppose n=bk for k≥1.
• Lemma 4.2
– for T(n) = Θ(1) if n=1
– aT(n/b)+f(n) if n=bk for k≥1
– where a ≥ 1, b>1, f(n) be a nonnegative function defined on
exact powers of b, then
logbn-1
– T(n) = Θ(nlogba) + ∑ ajf(n/bj)
j=0

• Proof:
– By iterating the recurrence
– By recursion tree (See figure 4.3)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 11

 Recursion Tree for T(n)=aT(n/b)+f(n)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 12

 Proof for Exact Powers (cont.)

• Lemma 4.3:
– Let constants a ≥ 1, b>1, f(n) be a nonnegative function
defined on exact power of b, then
logbn-1
– g(n)= ∑ ajf(n/bj) can be bounded asymptotically for exact
j=0
power of b as follows:

1. If f(n)=O(nlogba-ε) for some ε>0, then g(n)= O(nlogba).

2. If f(n)= Θ(nlogba), then g(n)= Θ(nlogba lg n).
3. If f(n)= Ω(nlogba+ε) for some ε>0 and if af(n/b) ≤cf(n) for
some c<1 and all sufficiently large n ≥b, then g(n)= Θ(f(n)).
13
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 13
 Proof of Lemma 4.3

• For case 1: f(n)=O(nlogba-ε) implies f(n/bj)=O((n /bj)logba-ε), so

logbn-1 logbn-1
• g(n)= ∑ ajf(n/bj) =O( ∑ aj(n /bj)logba-ε )
j=0 j=0
logb n-1 logbn-1
• = O(nlogba-ε ∑ aj/(blogba-ε)j ) = O(nlogba-ε ∑ aj/(aj(b-ε)j))
j=0 j=0
logbn-1
• = O(nlogba-ε ∑ (bε)j ) = O(nlogba-ε (((bε ) logbn-1)/(bε-1) )
j=0

• = O(nlogba-ε (((blogbn)ε -1)/(bε-1)))

• = O(nlogba n-ε (nε -1)/(bε-1))
• = O(nlogba )
14
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 14
 Proof of Lemma 4.3(cont.)

• For case 2: f(n)= Θ(nlogba) implies f(n/bj)= Θ((n /bj)logba), so

logbn-1 logbn-1
• g(n)= ∑ ajf(n/bj) = Θ( ∑ aj(n /bj)logba)
j=0 j=0
logbn-1 logbn-1
• = Θ(nlogba ∑ aj/(blogba)j ) = Θ(nlogba ∑1)
j=0 j=0

• = Θ(nlogba logbn) = Θ(nlogbalg n)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 15

 Proof of Lemma 4.3(cont.)
• For case 3:
– Since g(n) contains f(n), g(n) = Ω(f(n))
– Since a f(n/b) ≤ c f(n), so f(n/b) ≤ (c/a) f(n),
– Iterating j times, f(n/bj) ≤ (c/a)j f(n), thus aj f(n/bj) ≤ cj f(n)
logbn-1 logbn-1 ∞
– g(n)= ∑ aj f(n/bj) ≤ ∑ cj f(n) ≤ f(n) ∑ cj = f(n) (1/(1-c))
j=0 j=0 j=0

=O(f(n))

– Thus, g(n)=Θ(f(n))

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 16

 Proof for Exact Powers (cont.)
• Lemma 4.4:
– for T(n) = Θ(1) if n=1
aT(n/b)+f(n) if n=bk for k≥1
– where a ≥ 1, b>1, f(n) be a nonnegative function,
1. If f(n)=O(nlogba-ε) for some ε>0, then T(n)= Θ(nlogba).
2. If f(n)= Θ(nlogba), then T(n)= Θ(nlogba lg n).
3. If f(n)=Ω(nlogba+ε) for some ε>0, and if af(n/b) ≤cf(n) for some
c<1 and all sufficiently large n, then T(n)= Θ(f(n)).

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 17

 Proof of Lemma 4.4 (cont.)
• Combine Lemma 4.2 and 4.3,
– For case 1:

T(n)= Θ(nlogba)+O(nlogba)=Θ(nlogba).
– For case 2:

T(n)= Θ(nlogba)+Θ(nlogba lg n)=Θ(nlogba lg n).
– For case 3:

T(n)= Θ(nlogba)+Θ(f(n))=Θ(f(n)) because f(n)= Ω(nlogba+ε).

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 18

 Floors and Ceilings (n≠bk for k≥1)
• T(n)=a T( n/b )+f(n) and T(n)=a T( n/b )+f(n)
• Want to prove both equal to T(n)=a T(n/b)+f(n)
• Two results:
– Master theorem applied to all integers n.
– Floors and ceilings do not change the result.

(Note: we proved this by domain transformation too).
• Since n/b≤n/b, and n/b≥ n/b, upper bound for
floors and lower bound for ceiling is held.
• So prove upper bound for ceilings (similar for lower
bound for floors).

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 19

 Upper bound of proof for T(n)=aT(n/b)+f(n)
• consider sequence n, n/b,  n/b/b,   n/b /b/b, …
• Let us define nj as follows:
• nj = n if j = 0
• = nj-1/b if j > 0
• The sequence will be n0, n1, …, nlogbn
Let j=logb n , then
n0 <= n
n1 <= n/b+1 nlogbn < n / b logbn + b/(b-1)
n2 <= n/b2+n/b+1
≤ n / b logbn -1+ b/(b-1)
... j-1
nj <= n/bj + ∑ 1/bi = n/(n/b) + b/(b-1) = b + b/(b-1) = O(1)
i=0
j
< n/b + b/(b-1)
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 20
 Recursion Tree

Recursion Tree of T(n)=a T( n/b )+f(n)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 21

 The Proof of Upper Bound for Ceiling

logbn -1
– T(n) =
j=0
∑
Θ(nlogba)+ ajf(nj)

– Thus similar to Lemma 4.3 and 4.4, the upper bound is

proven.

logbn -1
g(n) = ∑
j=0
aj f(nj)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 22

 The Simple Format of Master Theorem

• T(n)=aT(n/b)+cnk, with a, b, c, k are positive constants,

and a≥1 and b≥2,
O(nlogba), if a>bk.
• T(n) = O(nklogn), if a=bk.
O(nk), if a<bk.

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 23

 Exercise (1)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 24

 Exercise (2)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 25

 Exercise (3)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 26

 Exercise (4)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 27

 Exercise (5)

The easy way to do this is with a change of variables.

Let m = lg n and S(m) = T (2m )

T (2m ) = T (2m/2 )+ 1, So S(m)= S(m/2)+1,

Using the master theorem, a=1, b=2. nlogba = 1 and f (n) = 1.

Case 2 applies and S(m) = Θ(lg m).

Therefore, T (n) = Θ (lg lg n).

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 28