Algorithm
Analysis
Prof.Dr.Eng.Ir Taufik Djatna
1
Algorithm Analysis - Class Goals
Analysis of Algorithms from both practical
(programming) and theoretical (mathematical) points of
view
Efficiency of Algorithms - Time and Space
Explore Fundamental Algorithmic Paradigms
– Divide and Conquer
– Graph Algorithms
– Greedy Algorithms
– Dynamic Programming
– Linear Programming
– Local Search
– Intelligent Search
– Probabilistic Algorithms
2
Algorithms
Is how to solve the model mission with a
sequence of measured activities
The key to efficient problem solving
The foundation of the power of computing
The better we become at creating good
algorithms, the more efficient we become at
computing
3
Additional Goals
Develop your general problem solving and programming
skills
Use Python
Prep for Job Interviews
Become a better person - Always remember the big picture
4
Before we ask what is an Algorithm,
Let's ask:
What is Computation?
5
What is Computation?
6
What is Computation?
Deterministic mappings
What Features would we like for such computational machines
Humans?, Machine vs Human Intelligence?, come to CS 472
7
What is an Algorithm?
What is an Algorithm?
No single formal definition
A typical informal definition: “A finite sequence of
well defined steps for solving a problem.”
This definition has been influenced by the types of
computational machines we have used.
Is all computation done by algorithms so defined?
– Sequence of operations?
– Termination?
3 Basic Questions about any Algorithm
Start with a problem and then propose an algorithmic solution:
1. ?
10
3 Basic Questions about any Algorithm
Start with a problem and then propose an algorithmic solution:
1. Is it Correct (not always as obvious as it sounds)?
11
3 Basic Questions about any Algorithm
Start with a problem and then propose an algorithmic solution:
1. Is it Correct (not as obvious as it sounds)?
2. How long does it take as a function of n, the problem size?
– Can also ask how much space (memory) does it require
– Worst, best, and average case
12
3 Basic Questions about any Algorithm
Start with a problem and then propose an algorithmic solution:
1. Is it Correct (not as obvious as it sounds)?
2. How long does it take as a function of n, the problem size?
– Can also ask how much space (memory) does it require
– Worst, best, and average case
3. Can we do Better?
– Efficiency - Time and Space
– Correctness/Accuracy
– Simplicity - Intelligibility
– Etc.
13
3 Basic Questions about any Algorithm
Start with a problem and then propose an algorithmic solution:
1. Is it Correct (not as obvious as it sounds)?
2. How long does it take as a function of n, the problem size?
– Can also ask how much space (memory) does it require
– Worst, best, and average case
3. Can we do Better?
– Efficiency - Time and Space
– Correctness/Accuracy
– Simplicity - Intelligibility
– Etc.
Example - Is a certain name in a list of names of length n?
14
A Bit of History
Abu Jafar Muhammad ibn Musa, c. 780 - 850
– “Al Khwarizmi” (of Kwarizm) his place of origin (Uzbekistan)
– Persian scholar in Baghdad
– Used decimal (positional system as contrasted with Roman
numerals) and proposed numerous arithmetic algorithms which
could not be done the the Roman numeral system
– Latin version of this name is Algoritmi
– From this came the word Algorithm
Leonardo of Pisa (Fibonacci) 1170-1250
– Pushed the use of decimal system in Europe
– Most remembered for the Fibonacci series of numbers
15
Fibonacci Series
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, …
What is a simple algorithm to find F(n)?
16
An Algorithm - fib1
Function for computing the n-th Fibonacci number Fn
function fib1(n)
if n = 0: return 0
if n = 1: return 1
return fib1(n-1) + fib1(n-2)
Is it correct?
How much time does it take, as a function of n?
Can we do better?
17
How long does it take
Note that with each call there are two more calls
(doubling each time)
What is the depth of the calling tree?
18
How long does it take
19
How long does it take
Note that with each call there are two more calls
(doubling each time)
What is the depth of the calling tree?
Thus, exponential time!
Since tree is binary, but not full it is somewhat less than,
but close to O(2n)
How bad is that?
20
How long does it take
Note that with each call there are two more calls
(doubling each time)
What is the depth of the calling tree?
Thus exponential time
Since tree is binary, but not full it is somewhat less than,
but close to O(2n)
How bad is that? Calculating fib1(200) takes longer on
current fastest computer than the life of the sun.
Moore’s Law?
Parallelism?
Exponential is Intractable!
21
How long does it take - Bit more Careful
Assume T(n) is the time necessary to compute F(n) for
fib1. What is the recurrence relation?
T(n) = ?
function fib1(n)
if n = 0: return 0
if n = 1: return 1
return fib1(n-1) + fib1(n-2)
22
How long does it take - Bit more Careful
Assume T(n) is the time necessary to compute F(n) for
fib1. What is the recurrence relation?
T(n) = T(n-1) + T(n-2) + 3 when n > 1
function fib1(n)
if n = 0: return 0
if n = 1: return 1
return fib1(n-1) + fib1(n-2)
23
How long does it take - a Bit more Careful
Assume T(n) is the time necessary to compute F(n) for
fib1. What is the recurrence relation?
T(n) = T(n-1) + T(n-2) + 3 when n > 1
Note that T(n) > F(n)
F(n) ≈ 20.694n ≈ 1.6n, where .694 are the first 3 digits of an
irrational number
T(n) ≈ 1.6n
24
Can we do better?
25
Approximation
Sometimes we can find a closed-form solution
Fib(n) ≈ 20.694n ≈ 1.6n, where .694 are the first 3 digits of an irrational
number
This direct calculation Fib(n) ≈ 20.694n trades off correctness for speed
Approximate solution – very common for many problems as we shall
see later
26
Can we do better?
27
Dynamic Programming Approach
Store the intermediate results and avoid duplicate work
function fib2(n)
if n=0: return 0
create an array f[0..n]
f[0] = 0, f[1] = 1
for i = 2 to n:
f[i] = f[i-1] + f[i-2]
return f[n]
Dynamic Programming approach
Space complexity? - Tradeoffs
Ask the questions
28
Orders of Growth – Some Complexity Classes
n log2n n nlog2n n2 n3 2n n!
10 3.3 10 3.3*10 102 103 103 3.6*106
100 6.6 102 6.6*102 104 106 1.3*1030 9*10157
1000 10 103 1.0*104 106 109 1.1*10301 4*102567
10000 13 104 1.3*105 108 1012 2.0*103010 2.8*1035659
105 17 105 1.7*106 1010 1015 1030102 2.9*10456573
106 20 106 2.0*107 1012 1018 10301029 8.3*105565708
Efficient Not Efficient
29
Orders of Growth – Some Complexity Classes
n log2n n nlog2n n2 n3 2n n!
10 3.3 10 3.3*10 102 103 103 3.6*106
100 6.6 102 6.6*102 104 106 1.3*1030 9*10157
1000 10 103 1.0*104 106 109 1.1*10301 4*102567
10000 13 104 1.3*105 108 1012 2.0*103010 2.8*1035659
105 17 105 1.7*106 1010 1015 1030102 2.9*10456573
106 20 106 2.0*107 1012 1018 10301029 8.3*105565708
To calibrate, there are about 1057 atoms in the solar Efficient Not Efficient
system and 1080 atoms in the current known universe 30
Asymptotic Complexity - Big-O analysis
We want to consider significant algorithm differences
Independent of the particular machine
Consider time complexity in terms of number of elementary
operations (we will define more carefully) and as a function of
input size n.
– Space complexity in terms of number of bits required
(g(n)) is set of all functions in the same complexity class
– Forms an equivalence class of functions within a constant factor of
each other
(n) includes: 2n, 4n+1, etc., (n2) includes 4n2/3, etc.
Are these constants important in real applications?
We will focus in this class on paradigms which allow you to
jump all the way into more efficient complexity classes
31
Asymptotic Notation
Definition: given asymptotically nonnegative g(n),
An asymptotically nonnegative function f(n) belongs to the set Q(g(n)) if
there exist positive constants c1 and c2 such that it can be “sandwiched”
between c1g(n) and c2g(n) for sufficiently large n.
We say
32
Examples
How could you prove that 4n = (n)?
How could you prove that n/2 - 5 = (n)?
Note that represents an equivalence class
– if f(n) = (g(n)) then g(n) = (f(n))
Could we represent (n) as (3+n/2)
– Why don't we?
– (n) includes all functions which differ by only a constant factor * n
33
Asymptotic Notation
Definition: given asymptotically nonnegative g(n),
34
Asymptotic Notation
c2g(n) c g(n) f(n)
f(n)
f(n)
c1g(n) c g(n)
n0 n0 n0
Functions in same class differ only by a constant factor times f(n)
35
Asymptotic Analysis
O and Ω are sets but not equivalence classes like
O(n) includes: 2n, log(n), 4, 1/n, etc. Thus, f = O(g) is like
saying f ≤ g in terms of Big-O complexity.
– Thus log(n) O(n), but n O(log(n))
f = Ω(g) iff g = O(f), i.e. f ≥ g (Duality Rule)
f = (g) means that g = O(f) and f = O(g), i.e. f = g
Examples and some guidelines
– na dominates nb for a > b: na is a higher complexity class than nb
– Any exponential dominates any polynomial (the great divide)
– Which means that any polynomial dominates any logarithm
Usage warning - Many say big O when they really are
thinking big . However, sometimes actually correct
– Sorting in general vs a particular sorting algorithm
36
Max Rule (do first!):
O( f(n) + g(n) ) = O( max( f(n), g(n) ) )
The Limit Rule
Some Examples
f(n) = 4n g(n) = 2n Try both ways
f(n) = 3n-6 g(n) = 2n2 + 4n Use max rule first
f(n) = logn2 g(n) = logn3
37
** Challenge Question **
For each f and g below answer if f is big O, , or Ω compared
to g and be ready to share a "proof" of your answer
1. f(n) = 2n g(n) = 2n+1
2. f(n) = log2n g(n) = log3n hint: logan = logbn/logba
3. f(n) = 2n g(n) = 3n Click ready when done with this.
38
Complexity Analysis
Best case, average case, worst case
Usually consider worst case, unless very rare
– If worst case rare, then average case becomes more interesting
39
