CS483-05 Analysis of Recursive Algorithms and Brute
Force
Instructor: Fei Li
Room 443 ST II
Office hours: Tue. & Thur. 4:30pm - 5:30pm or by appointments
http://www.cs.gmu.edu/∼ lifei/teaching/cs483_fall07/
This lecture note is based on Introduction to The Design and Analysis of Algorithms by Anany Levitin.
CS483 Design and Analysis of Algorithms 1 Lecture 05, September 11, 2007
Example 4: Counting Binary Bits
Input: A positive decimal integer n.
Output: The number of binary digits in n’s binary representation.
Algorithm 0.1: C OUNT B INARY B ITS(n)
count = 1
while n>1
count = count + 1
do
n = ⌊n/2⌋
return (count)
CS483 Design and Analysis of Algorithms 2 Lecture 05, September 11, 2007
Outline
➣ Analysis of Recursive Algorithms
➣ Brute Force
Ideas
Examples: Selection Sort & Bubble Sort
Examples: String Matching
Examples: Exhaustive Search
CS483 Design and Analysis of Algorithms 3 Lecture 05, September 11, 2007
Analysis of Recursive Algorithms
➣ The iteration method
Expand (iterate) the recurrence and express it as a summation of terms
depending only on n and the initial conditions.
➣ The substitution method
➣ Master Theorem
(To be introduced in Chapter 4.)
CS483 Design and Analysis of Algorithms 4 Lecture 05, September 11, 2007
Iteration Method: Examples
n!
T (n) = T (n − 1) + 1
Tower of Hanoi
T (n) = 2T (n − 1) + 1
CS483 Design and Analysis of Algorithms 5 Lecture 05, September 11, 2007
Iteration: Example
n! (T (n) = T (n − 1) + 1)
T (n) = T (n − 1) + 1
= (T (n − 2) + 1) + 1
= T (n − 2) + 2
··· ···
= T (n − i) + i
··· ···
= T (0) + n = n
Tower of Hanoi (T (n) = 2T (n − 1) + 1) ???
CS483 Design and Analysis of Algorithms 6 Lecture 05, September 11, 2007
Tower of Hanoi (T (n) = 2T (n − 1) + 1)
T (n) = 2T (n − 1) + 1
= 2(2T (n − 2) + 1) + 1
= 22 T (n − 2) + 2 + 1
··· ···
= 2i T (n − i) + 2i−1 + · · · + 1
··· ···
= 2n−1 T (1) + 2n−1 + 2n−1 + · · · + 1
n−2
X
= 2n−1 T (1) + 2i
i=0
= 2n−1 + 2n−1 − 1
= 2n − 1
CS483 Design and Analysis of Algorithms 7 Lecture 05, September 11, 2007
Analysis of Recursive Algorithms
➣ The iteration method
Expand (iterate) the recurrence and express it as a summation of terms
depending only on n and the initial conditions.
➣ The substitution method
1. Guess the form of the solution
2. Use mathematical induction to find the constants
➣ Master Theorem
CS483 Design and Analysis of Algorithms 8 Lecture 05, September 11, 2007
Substitution Method: Example 1
Count number of bits (T (n) = T (⌊n/2⌋) + 1)
CS483 Design and Analysis of Algorithms 9 Lecture 05, September 11, 2007
Substitution Method: Example 1
Count number of bits (T (n) = T (⌊n/2⌋) + 1)
– Guess T (n) ≤ log n.
T (n) = T (⌊n/2⌋) + 1
≤ log(⌊n/2⌋) + 1
≤ log(n/2) + 1
≤ (log n − log 2) + 1
≤ log n − 1 + 1
= log n
CS483 Design and Analysis of Algorithms 10 Lecture 05, September 11, 2007
Substitution Method: Example 2
Tower of Hanoi (T (n) = 2T (n − 1) + 1)
CS483 Design and Analysis of Algorithms 11 Lecture 05, September 11, 2007
Substitution Method: Example 2
Tower of Hanoi (T (n) = 2T (n − 1) + 1)
– Guess T (n) ≤ 2n .
T (n) = 2T (n − 1) + 1
≤ 2 · 2n−1 + 1
≤ 2n + 1, wrong!
CS483 Design and Analysis of Algorithms 12 Lecture 05, September 11, 2007
Substitution Method: Extension Fn
Tower of Hanoi (T (n) = 2T (n − 1) + 1)
– Guess T (n) ≤ 2n .
T (n) = 2T (n − 1) + 1
≤ 2 · 2n−1 + 1
≤ 2n + 1, wrong!
– Guess T (n) ≤ 2n − 1.
CS483 Design and Analysis of Algorithms 13 Lecture 05, September 11, 2007
T (n) = 2T (n − 1) + 1
≤ 2(2n−1 − 1) + 1
= 2n − 2 + 1
= 2n − 1, correct!
CS483 Design and Analysis of Algorithms 14 Lecture 05, September 11, 2007
Substitution Method: Extension Fn
Fibonacci Numbers (F0 = 0, F1 = 1, Fn = Fn−1 + Fn−2 )
CS483 Design and Analysis of Algorithms 15 Lecture 05, September 11, 2007
Substitution Method: Extension Fn
Fibonacci Numbers (F0 = 0, F1 = 1, Fn = Fn−1 + Fn−2 )
Fn−2 < Fn−1 < Fn , ∀n ≥ 1
CS483 Design and Analysis of Algorithms 16 Lecture 05, September 11, 2007
Substitution Method: Extension Fn
Fibonacci Numbers (F0 = 0, F1 = 1, Fn = Fn−1 + Fn−2 )
Fn−2 < Fn−1 < Fn , ∀n ≥ 1
Assume 2n−1 < Fn < 2n
Guess Fn = c · φn , 1 < φ < 2.
CS483 Design and Analysis of Algorithms 17 Lecture 05, September 11, 2007
Substitution Method: Extension Fn
Fibonacci Numbers (F0 = 0, F1 = 1, Fn = Fn−1 + Fn−2 )
Fn−2 < Fn−1 < Fn , ∀n ≥ 1
Assume 2n−1 < Fn < 2n
Guess Fn = c · φn , 1 < φ < 2.
c · φn = c · φn−1 + c · φn−2
φ2 = φ+1
√
1± 5
φ =
2
General solution: Fn = c1 · φn1 + c2 · φn2
F1 = 0, F2 = 1
CS483 Design and Analysis of Algorithms 18 Lecture 05, September 11, 2007
General solution: Fn = c1 · φn1 + c2 · φn2
F1 = 0, F2 = 1
√ √
1 1+ 5 n 1 1− 5 n
Fn = √ ( ) − √ ( )
5 2 5 2
CS483 Design and Analysis of Algorithms 19 Lecture 05, September 11, 2007
Summary: Algorithm Analysis
➣ Order of growth of functions
➣ Analyze algorithms’ order of growth (using asymptotic notations).
Non-recursive algorithms
Recursive algorithms
a. The iteration method
b. The substitution method
c. Master Theorem (to be introduced) (T (n) = aT (n/b) + f (n).)
CS483 Design and Analysis of Algorithms 20 Lecture 05, September 11, 2007
Outline
➣ Analysis of Recursive Algorithms
➣ Brute Force
Ideas
Examples: Selection Sort & Bubble Sort
Examples: String Matching
Examples: Exhaustive Search
CS483 Design and Analysis of Algorithms 21 Lecture 05, September 11, 2007
Brute Force — Ideas
➣ Brute force is a straightforward approach to solve a problem, usually directly
based on the problem statement and definitions of the concepts involved.
CS483 Design and Analysis of Algorithms 22 Lecture 05, September 11, 2007
Outline
➣ Analysis of Recursive Algorithms
➣ Brute Force
Ideas
Examples: Selection Sort & Bubble Sort
Examples: String Matching
Examples: Exhaustive Search
CS483 Design and Analysis of Algorithms 23 Lecture 05, September 11, 2007
Selection Sort & Bubble Sort
➣ Given n orderable items, sort them in non-decreasing order.
CS483 Design and Analysis of Algorithms 24 Lecture 05, September 11, 2007
Selection Sort
➣ Given n orderable items, sort them in non-decreasing order.
➣ Input: An array A[0, . . . , n − 1] of orderable elements.
➣ Output: An array A[0, . . . , n − 1] sorted in non-decreasing order.
Algorithm 0.2: S ELECTION S ORT(A[0, · · · n − 1])
for i= 0 to n − 2
min = i
for j = i + 1 to n − 1
do if A[j] < A[min]
do
then min = j
Swap A[i] and A[min]
CS483 Design and Analysis of Algorithms 25 Lecture 05, September 11, 2007
Selection Sort
Analysis
Input size: n.
Basic operation: A[j] < A[min]
Running time:
X n−1
n−2 X
C(n) = 1
i=0 j=i+1
n−2
X n−2
X
= [(n − 1) − (i − 1) + 1] = (n − 1 − i)
i=0 i=0
(n − 1)n
=
2
= Θ(n2 )
CS483 Design and Analysis of Algorithms 26 Lecture 05, September 11, 2007
Bubble Sort
➣ Given n orderable items, sort them in non-decreasing order.
➣ Input: An array A[0, . . . , n − 1] of orderable elements.
➣ Output: An array A[0, . . . , n − 1] sorted in non-decreasing order.
Algorithm 0.3: B UBBLE S ORT(A[0, · · · n − 1])
for i= 0 to n − 2
for j = 0 to n − 2 − i
do if A[j + 1] < A[j]
do
then Swap A[j] and A[j + 1]
CS483 Design and Analysis of Algorithms 27 Lecture 05, September 11, 2007
Bubble Sort
Analysis
Input size: n.
Basic operation: A[j + 1] < A[j]
Running time:
X n−2−i
n−2 X
C(n) = 1
i=0 j=0
n−2
X n−2
X
= [(n − 2 − i) − 0 + 1] = (n − 1 − i)
i=0 i=0
(n − 1)n
=
2
= Θ(n2 )
CS483 Design and Analysis of Algorithms 28 Lecture 05, September 11, 2007
Outline
➣ Analysis of Recursive Algorithms
➣ Brute Force
Ideas
Examples: Selection Sort & Bubble Sort
Examples: String Matching
Examples: Exhaustive Search
CS483 Design and Analysis of Algorithms 29 Lecture 05, September 11, 2007
String Matching
➣ Given a string of n characters called the text; and a string of m characters
called the pattern, find a substring of the text that matches the pattern.
➣ Input: An array T [0, . . . , n − 1] of n characters representing a text
An array P [0, . . . , m] characters representing a pattern
➣ Output: The index of the first character in the text that starts a matching
substring or −1 if the search is unsuccessful
➣ Example: Pattern: 001011 Text: 10010101101001100101111010
Pattern: happy Text: It is never too late to have a happy childhood.
CS483 Design and Analysis of Algorithms 30 Lecture 05, September 11, 2007
String Matching
Algorithm 0.4: S TRING M ATCHING(T [0, · · · n − 1], P [0, . . . , m − 1])
for i= 0 to n − m
j=0
while j < m and P [j] = T [i + j]
do do j = j + 1
if j = m
then return (i)
return (−1)
CS483 Design and Analysis of Algorithms 31 Lecture 05, September 11, 2007
String Matching
Analysis
Input size: n + m.
Basic operation: P [j] = T [i + j]
Running time (worst-case):
C(n + m) = (n − m + 1) · m = Θ(nm)
CS483 Design and Analysis of Algorithms 32 Lecture 05, September 11, 2007
Outline
➣ Analysis of Recursive Algorithms
➣ Brute Force
Ideas
Examples: Selection Sort & Bubble Sort
Examples: String Matching
Examples: Exhaustive Search
CS483 Design and Analysis of Algorithms 33 Lecture 05, September 11, 2007
Traveling Salesman Problem
➣ TSP: Find the shortest tour through a given set of n cities that visits each city
exactly once before returning to the city where it starts.
2
a b
5 7
8 3
c d
1
CS483 Design and Analysis of Algorithms 34 Lecture 05, September 11, 2007
a 2
b
5 7 3
8
c d
1
Tour Cost
a→b→c→d→a 2 + 3 + 7 + 5 = 17
a→b→d→c→a 2 + 4 + 7 + 8 = 21
a→c→b→d→a 8 + 3 + 4 + 5 = 20
a→c→d→b→a 8 + 7 + 4 + 2 = 21
a→d→b→c→a 5 + 4 + 3 + 8 = 20
a→d→c→b→a 5 + 7 + 3 + 2 = 17
CS483 Design and Analysis of Algorithms 35 Lecture 05, September 11, 2007
Traveling Salesman Problem
Analysis
Input size: n · (n − 1).
Running time:
C(n) = (n − 1)!/2.
CS483 Design and Analysis of Algorithms 36 Lecture 05, September 11, 2007
Knapsack Problem
➣ Knapsack Problem: Given n objects, each object i has weight wi and value
vi , and a knapsack of capacity W , find most valuable items that fit into the
knapsack
Items are not splittable
CS483 Design and Analysis of Algorithms 37 Lecture 05, September 11, 2007
Example: Knapsack capacity W = 16
Item Weight Value
1 2 $20
2 5 $30
3 10 $50
4 5 $10
CS483 Design and Analysis of Algorithms 38 Lecture 05, September 11, 2007
Subset Total weight Total value
{1} 2 $20
{2} 5 $30
{3} 10 $50
{4} 5 $10
{1, 2} 7 $50
{1, 3} 12 $70
{1, 4} 7 $30
{2, 3} 15 $80
{2, 4} 10 $40
{3, 4} 15 $60
{1, 2, 3} 17 not feasible
{1, 2, 4} 12 $60
{1, 3, 4} 17 not feasible
{2, 3, 4} 20 not feasible
{1, 2, 3, 4} 22 not feasible
CS483 Design and Analysis of Algorithms 39 Lecture 05, September 11, 2007
Knapsack Problem
Analysis
Input size: n (items).
Running time:
The number of subsets of an n-element set is 2n .
C(n) = Ω(2n ).
CS483 Design and Analysis of Algorithms 40 Lecture 05, September 11, 2007
Assignment Problem
➣ Assignment Problem: There are n people to execute n jobs, one person per
job. If ith person is assigned the j th job, the cost is C[i, j], i, j = 1, . . . , n.
Find the assignment with the minimum total cost.
Job 1 Job 2 Job 3 Job 4
Person 1 9 2 7 8
Person 2 6 4 3 7
Person 3 5 8 1 8
Person 4 7 6 9 4
CS483 Design and Analysis of Algorithms 41 Lecture 05, September 11, 2007
Assignment Problem
Analysis
Input size: n.
Running time:
C(n) = n!.
CS483 Design and Analysis of Algorithms 42 Lecture 05, September 11, 2007
Summary for Brute Force
➣ Strengths
1. Wide applicability
2. Simplicity
3. Yields reasonable algorithms for some important problems (e.g., matrix
multiplication, sorting, searching, string matching)
➣ Weaknesses
1. Rarely yields efficient algorithms
2. Some brute-force algorithms are unacceptably slow
3. Not as constructive as some other design techniques
CS483 Design and Analysis of Algorithms 43 Lecture 05, September 11, 2007
Summary for Brute Force
➣ Exhaustive-search algorithms run in a realistic amount of time only on very
small instances
➣ In some cases, there are much better alternatives
Shortest paths (greedy)
Minimum spanning tree (greedy)
Assignment problem (iterative improvement)
➣ In many cases, exhaustive search or its variation is the only known way to get
exact solution
CS483 Design and Analysis of Algorithms 44 Lecture 05, September 11, 2007
Summary
➣ Read Chap. 3.
➣ Next class: Chap. 4 and Master Theorem.
CS483 Design and Analysis of Algorithms 45 Lecture 05, September 11, 2007
