Data Structures
6. Complexity Analysis
6-Complexity 1
Comparing Algorithms
• Given two or more algorithms to solve the same problem, how do
we select the best one?
• Some criteria for selecting an algorithm
– Is it easy to implement, understand, modify?
– How long does it take to run it to completion?
– How much of computer memory does it use?
• Software engineering is primarily concerned with the first criteria
• In this course we are interested in the second and third criteria
6-Complexity 2
Comparing Algorithms
• Time complexity
– The amount of time that an algorithm needs to run to completion
– Better algorithm is the one which runs faster
Has smaller time complexity
• Space complexity
– The amount of memory an algorithm needs to run
• In this lecture, we will focus on analysis of time complexity
6-Complexity 3
How To Calculate Running Time
• Most algorithms transform input objects into output objects
sorting
5 3 1 2 1 2 3 5
algorithm
input object output object
• The running time of an algorithm typically grows with input size
– Idea: analyze running time as a function of input size
6-Complexity 4
How To Calculate Running Time
• Most important factor affecting running time is usually the size of
the input
int find_max( int *array, int n ) {
int max = array[0];
for ( int i = 1; i < n; ++i ) {
if ( array[i] > max ) {
max = array[i];
}
}
return max;
}
• Regardless of the size n of an array the cost will always be same
– Every element in the array is checked one time
6-Complexity 5
How To Calculate Running Time
• Even on inputs of the same size, running time can be very different
int search(int arr[], int n, int x) {
int i;
for (i = 0; i < n; i++)
if (arr[i] == x)
return i;
return -1;
}
• Idea: Analyze running time for different cases
– Best case
– Worst case
– Average case
6-Complexity 6
How To Calculate Running Time
• Best case running time is usually best case
not very useful average case
worst case
120
• Average case time is very useful 100
but often hard to determine
Running Time
80
60
• Worst case running time is easier
40
to analyze
– Crucial for real-time applications 20
such as games, finance and 0
robotics 1000 2000 3000 4000
Input Size
6-Complexity 7
Experimental Evaluations of Running Times
• Write a program implementing
the algorithm 9000
8000
• Run the program with inputs of 7000
varying size 6000
Time (ms)
5000
• Use clock methods to get an 4000
accurate measure of the actual 3000
running time
2000
1000
• Plot the results 0
0 50 100
Input Size
6-Complexity 8
Limitations Of Experiments
Experimental evaluation of running time is very useful but
• It is necessary to implement the algorithm, which may be difficult
• Results may not be indicative of the running time on other inputs
not included in the experiment
• In order to compare two algorithms, the same hardware and
software environments must be used
6-Complexity 9
Theoretical Analysis of Running Time
• Uses a pseudo-code description of the algorithm instead of an
implementation
• Characterizes running time as a function of the input size n
• Takes into account all possible inputs
• Allows us to evaluate the speed of an algorithm independent of the
hardware/software environment
6-Complexity 10
Analyzing an Algorithm – Operations
• Each machine instruction is executed in a fixed number of cycles
– We may assume each operation requires a fixed number of cycles
• Idea: Use abstract machine that uses steps of time instead of secs
– Each elementary operation takes 1 steps
• Example of operations
– Retrieving/storing variables from memory
– Variable assignment =
– Integer operations + - * / % ++ --
– Logical operations && || !
– Bitwise operations &|^~
– Relational operations == != < <= => >
– Memory allocation and deallocation new delete
6-Complexity 11
Analyzing an Algorithm – Blocks of Operations
• Each operation runs in a step of 1 time unit
• Therefore any fixed number of operations also run in 1 time step
– s1; s2; …. ; sk
– As long as number of operations k is constant
Tree_node *lrl = left->right->left; // Swap variables a and b
Tree_node *lrr = left->right->right; int tmp = a;
parent = left->right; a = b;
parent->left = left; b = tmp;
parent->right = this;
left->right = lrl;
left = lrr;
6-Complexity 12
Analyzing an Algorithm
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A
int SumArray(int A[], int n){
int s=0; 1
for (int i=0; i< n; i++)
2 3 4
s = s + A[i];
5
return s;
6 7
} 8
• Operations 1,2,8 are executed once
• Operations 4,5,6,7: Once per each iteration of for loop n iteration
• Operation 3 is executed n+1 times
• The complexity function of the algorithm is : T(n) = 5n +4
6-Complexity 13
Analyzing an Algorithm – Growth Rate
• Estimated running time for different values of n:
– n = 10 => 54 steps
– n = 100 => 504 steps
– n = 1,000 => 5004 steps
– n = 1,000,000 => 5,000,004 steps
• As n grows, number of steps T(n) grow in linear proportion to n
6-Complexity 14
Growth Rate
6-Complexity 15
Growth Rate
6-Complexity 16
Growth Rate
• Changing the hardware/software environment
– Affects T(n) by a constant factor, but
– Does not alter the growth rate of T(n)
• Thus we focus on the big-picture which is the growth rate of an
algorithm
• The linear growth rate of the running time T(n) is an intrinsic
property of algorithm sumArray
6-Complexity 17
Constant Factors
• The growth rate is not affected by
– Constant factors or
– Lower-order terms
• Examples
– 102n + 105 is a linear function
– 105n2 + 108n is a quadratic function
6-Complexity 18
Growth Rate – Example
• Consider the two functions
– f(n) = n2
– g(n) = n2 – 3n + 2
• Around n = 0, they look very different
6-Complexity 19
Growth Rate – Example
• Yet on the range n = [0, 1000], f(n) and g(n) are (relatively)
indistinguishable
6-Complexity 20
Growth Rate – Example
• The absolute difference is large, for example,
– f(1000) = 1 000 000
– g(1000) = 997 002
• But the relative difference is very small
f(1000) g(1000)
0.002998 0.3%
f(1000)
– The difference goes to zero as n → ∞
6-Complexity 21
Constant Factors
• The growth rate is not affected by
– Constant factors or
– Lower-order terms
• Examples
– 102n + 105 is a linear function
– 105n2 + 108n is a quadratic function
• How do we get rid of the constant factors to focus on the essential
part of the running time?
– Asymptotic Analysis
6-Complexity 22
Upper Bound – Big-Oh Notation
• Indicates the upper or highest growth rate that the algorithm can
have
– Ignore constant factors and lower order terms
– Focus on main components of a function which affect its growth
Definition: Given functions f(n) and g(n)
• We say that f(n) is O(g(n))
• If there are positive constants c and n0 such that
– f(n) cg(n) for n n0
6-Complexity 23
Big-Oh Notation – Examples
• 7n-2 is O(n)
– Need c > 0 and n0 1 such that 7n-2 c.n for n n0
– True for c = 7 and n0 = 1
• 3n3 + 20n2 + 5 is O(n3)
– Need c > 0 & n0 1 such that 3n3 + 20n2 + 5 c.n3 for n n0
– True for c = 4 and n0 = 21
• 3 log n + 5 is O(log n)
– Need c > 0 & n0 1 such that 3 log n + 5 clog n for n n0
– True for c = 8 and n0 = 2
6-Complexity 24
Analyzing an Algorithm
• Simple Assignment
– a = b
– O(1)
• Simple loops
– for(i=0; i<n; i++) { s; }
– O(n)
• Nested loops
– for(i=0; i<n; i++)
for(j=0; j<n; j++) { s; }
– O(n2)
6-Complexity 25
Analyzing an Algorithm
• Loop index doesn’t vary linearly
– h = 1;
while ( h <= n ) {
s;
h = 2 * h;
}
– h takes values 1, 2, 4, … until it exceeds n
– There are 1 + log2n iterations
– O(log2 n)
6-Complexity 26
Analyzing an Algorithm
• Loop index depends on outer loop index
– for(j=0;j<=n;j++)
for(k=0;k<j;k++){
s;
}
– Inner loop executed 0, 1, 2, 3, …., n times
n n(n+1)
S i =
i=1 2
– O(n2)
6-Complexity 27
Relatives of Big-Oh
• Big-Omega
– f(n) is (g(n))
– If there is a constant c > 0 and an integer constant n0 1
– Such that f(n) c.g(n) for n n0
• Big-Theta
– f(n) is (g(n))
– if there are constants c1 > 0 and c2 > 0 and an integer constant n0
1
– such that c1.g(n) f(n) c2.g(n) for n n0
6-Complexity 28
Intuition for Asymptotic Notation
Big-Oh
• f(n) is O(g(n)) if f(n) is asymptotically less than or equal to
g(n)
Big-Omega
• f(n) is (g(n)) if f(n) is asymptotically greater than or equal to
g(n)
• Note: f(n) is (g(n)) if and only if g(n) is O(f(n))
Big-Theta
• f(n) is (g(n)) if f(n) is asymptotically equal to g(n)
• Note: f(n) is (g(n)) if and only if
– g(n) is O(f(n)) and
– f(n) is O(g(n))
6-Complexity 29
Final Notes
• Even though in this course we focus on the Running time
asymptotic growth using big-Oh notation,
practitioners do care about constant factors A
occasionally
• Suppose we have 2 algorithms
– Algorithm A has running time 30000n B
– Algorithm B has running time 3n2
• Asymptotically, algorithm A is better than 10000
algorithm B
problem size
• However, if the problem size you deal with
is always less than 10000, then the
quadratic one is faster
6-Complexity 30
Any Question So Far?
6-Complexity 31
