CS214 – Data Structures

Lecture 03: Complexity

Slides by
Ahmed Kamal, PhD
Mohamed El-Ramly, PhD
Basheer Youssef, PhD
Lamia Abo Zaid, PhD
 Lecture 3 Outline
1.Math Revision
2. Algorithm Complexity
I. Math Revision
 Mathematical Background*
Set concepts and notation
Logarithms
Summations
Recursion
Induction Proofs

4
 1. Set
A set is a collection of distinguishable members or elements.
 2. Logarithm
• Definition:
3. Summation
 4. Recursion

• An algorithm is recursive if it calls itself

to do part of its work.

• Example:
• 1. Compute n!
• fn = n* fn-1
• 2. Fibonacci n
• fn = fn-1 + fn-2
 5. Mathematical Proof

• Three templates for mathematical proof

1. Proof by Mathematical Induction

2. Proof by Counterexample
3. Proof by Contradiction

• Read Weiss, 1.2.5

 1. Proof by Induction
• A proof by induction has two standard
parts.
1. Proving a base case, that is, establishing
that a theorem is true for a small value
2. An inductive hypothesis is assumed. This
means that the theorem is assumed to be
true for all cases up to some limit k.
• Using this assumption, the theorem is
shown to be true for the next value, k + 1.
• This proves the theorem for finite k.
 Example 1 of Proof by Induction
• Prove that Fibonacci numbers
F0 = 1, F1 = 1, F2 = 2, …… Fi = Fi-1 + Fi-2

• Satisfy Fi < (5/3)i for i ≥ 1

 Example1 of Proof by Induction
Proof that Fi < (5/3)i
• Base case F1 = 1 < (5/3)1
• Base case F2 = 2 < (5/3)2 < (25/9)

• Inductive hypothesis: assume the theorem

is true for i = 1, 2, .., k
• To prove this we need to prove that
• Fk < (5/3)k
 Example 1 of Proof by Induction
Proof that Fi < (5/3)i
• Fk = Fk-1 + Fk-2
• Using the inductive hypothesis:
• Fk < (5/3)k-1 + (5/3)k-2
• Fk < (5/3)k-2 (5/3 + 1)
• Fk < (5/3)k-2 (8/3)
• But 8/3 < 25/9
• Fk < (5/3)k-2 (25/9)
• Fk < (5/3)k-2(5/3)2
• Fk < (5/3)k
 2. Proof by Counterexample
• Proof by Counterexample is a way of
showing that a given statement cannot
possibly be correct by showing an
instance that contradicts a with that
statement.
• Example 2
• Fk ≤ k2 is false
• F11 ≤ (11)2 is false because F11 = 144
 3. Proof by Contradiction
• Proof by contradiction proceeds by
assuming that the theorem is false.
• Then it shows that this assumption implies
that some known property is false, and
hence the original assumption was
erroneous.
 Example 3 Proof by Contradiction
• Theorem: There is no largest integer.
• Step 1:
 Contrary Assumption: Assume this is false.
 Assume that there is a largest integer, B.
• Step 2:
 Show this assumption leads to a contradiction
 Consider C = B + 1. C is an integer because
it is the sum of two integers.
 Also C > B, which means that B is not the
largest integer.
 Contradiction! hence the theorem is true.
 Introduction to
Algorithm Complexity
 Lecture Objectives
• To precisely define the term algorithm
• To specify the main characteristics of algorithms
• To estimate the time and space complexity of algorithms
• To get familiar with asymptotic rate growth of functions
and algorithms
• To introduce the big-oh, Theta, and Omega operators to
measure the worst, average and best time complexity of
algorithms
• To classify algorithms based on their time complexities
• To learn how to solve recurrence relations and estimate
the complexity of recursive algorithms
 Algorithms
• An algorithm is a finite sequence of instructions that, if
followed, accomplishes a particular task
• Algorithms must satisfy the following criteria:
1) Input – Zero or more quantities are externally supplied.
2) Output – At least one quantity is produced.
3) Definiteness – Each instruction is clear and
unambiguous.
4) Finiteness – If we trace out the instructions of an
algorithm, then for all cases, the algorithm terminates after
a finite number of steps.
5) Efficiency – The algorithm must consume a reasonable
amount of resources and takes a reasonable of time. It
must be feasible to run the algorithm.
 Solving Problems by Computers
• So, we use algorithms to solve problems
• A problem is defined in principle as a task that is
amenable to being solved by a computer, e.g.
 Sorting a list of numbers
 Searching for a particular key value in a collection of items
 Finding the shortest distance between two cities in a map
• Not all problems are solved by computers, computer
scientists discovered that
 Some problems are easy to solve
 Other problems are hard to solve since they take a lot of time
 A group of problems cannot be solved because we cannot
design algorithms for them
 Problem vs. Problem Instance
• We should differentiate between a problem
(general description of a task) and its instances
(a particular case with specific set of input data)
• Example #1:
 Problem: Sort a collection
 Problem instance: sort the list [5,2,1,7,3] in
ascending order
• Example #2:
 Problem: Search for a key value in a collection
 Problem instance: given key=3, find this key in the
sorted list [1,2,6,9,10,45,78]
 Algorithm Complexity
• The term complexity refers to the amount
of resources required by an algorithm to
solve a problem instance
 The term time complexity refers to the
amount of time needed to solve a problem
instance
 The term space complexity refers to the
amount of memory needed to solve a problem
instance
 Estimation of Time Complexity
• Since an algorithm is a sequence of
instructions written by a pencil on a piece
of paper and cannot be executed directly
to measure its performance, we are faced
with the following challenges:
 How to measure an algorithm performance?
 What are the units of measurement, is it
seconds, or any other units?
 Estimation of Time Complexity
Approach #1
• Implement a program for this algorithm and run it
for different instances of the problem under study

• Analysis
 Very expensive approach
 Depends on the machine architecture, current
technology, type of programming language used, the
programmer skills, etc.
• All of the above mentioned reasons lead to the
conclusion: This approach is not practical!!
.
 Estimation of Time Complexity
Approach #2
• Select the most fundamental operation done by
the algorithm, then count the number of times
this operation takes place to solve a problem
instance of size n
 Not all operations are equal. E.g. the CPU time to do
a multiplication is longer than the addition operation
 Nevertheless, if the addition operation is more
frequent than the multiplication operation, the total
time taken to execute all additions adds up and
dominates the total time taken to execute the
multiplication operations, so fundamental means
dominant !
 Approach #2
• In this approach, we use the number of times
the fundamental operation is executed as a
measure of time complexity. It is not measured
in seconds (or any other time units)
• Example
int Sum(int A[], int N) {
int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s; }
 Approach #2
• In this approach, we use the number of times
the fundamental operation is executed as a
measure of time complexity. It is not measured
in seconds (or any other time units)
• Example
int Sum(int A[], int N) {
int s=0;
for (int i=0; i< N; i++) The complexity function of the
algorithm is : f(N) = 5N +3
s = s + A[i];
return s; }
 Approach #2
• Analysis:
 People may choose different fundamental operations for the
same algorithm, so you may get more than one time complexity
function for the same algorithm!
 Again, this approach depends on the architecture and the
technology used, if for example we design a machine that
executes * operation faster than + operation, our analysis will not
be same!
• Does it make any difference if someone tell you that the
time complexity of an algorithm A is T(n)=3n+2 and
somebody else insisted that it is T(n)=2n?
• Do you have to know the fundamental operation that the
analysis is based on?
 Approach #2
• Example
int Sum(int A[], int N) {
int s=0;
The complexity function of the
for (int i=0; i< N; i++)
s = s + A[i];
algorithm is : f(N) = 5N +3
return s; }

• Estimated running time for different values of N:

• N = 10 => 53 steps
• N = 100 => 503 steps
• N = 1,000 => 5003 steps
• N = 1,000,000 => 5,000,003 steps
 As N grows, the number of steps grow in linear
proportion to N
 Let’s continue approximating…
• Easy is Good and Fast! Since we are
satisfied with a rough estimate, how about
simplifying the time complexity function
further by ignoring all the coefficients!!!
 So, if an algorithm has time complexity
T(n)=4n2+100, we simply say the time
complexity is approximated to T(n)≈n2
• To do that, we need some mathematical
justification or reasoning!!!
 Term Contribution
• Assume the actual time complexity of an
algorithm is:
T(n) = 3n2+8n+10,
what is the approximate time complexity of that
algorithm?

 Since T(n) is getting bigger (i.e. monotonically

increasing) by increasing the problem size n,
 we can study the contribution of each term;
3n2, 8n, and 10, on the increase of T(n)
 n 3*n^2 8*n 10 T(n) C(3n^2) C(8n) C(10)

Experiment#1 1
5
3
75
8
40
10
10
21
126
14.29%
59.52%
38.10%
31.75%
47.62%
8.73%
T(n) = 3n2+8n+10 10 300 80 10 392 76.53% 20.41% 3.06%
15 675 120 10 808 83.54% 14.85% 1.61%
100.00% 20 1200 160 10 1374 87.34% 11.64% 1.02%
90.00% 25 1875 200 10 2090 89.71% 9.57% 0.72%
80.00%
30 2700 240 10 2956 91.34% 8.12% 0.54%
70.00%
35 3675 280 10 3972 92.52% 7.05% 0.43%
60.00%
C(3n^2) 40 4800 320 10 5138 93.42% 6.23% 0.35%
50.00%
C(8n) 45 6075 360 10 6454 94.13% 5.58% 0.29%
40.00%
30.00% C(10) 50 7500 400 10 7920 94.70% 5.05% 0.25%

20.00% 55 9075 440 10 9536 95.17% 4.61% 0.22%

10.00% 60 10800 480 10 11302 95.56% 4.25% 0.19%
0.00% 65 12675 520 10 13218 95.89% 3.93% 0.17%
0 20 40 60 80 100 120 70 14700 560 10 15284 96.18% 3.66% 0.16%
75 16875 600 10 17500 96.43% 3.43% 0.14%
80 19200 640 10 19866 96.65% 3.22% 0.13%
As problem size n
85 21675 680 10 22382 96.84% 3.04% 0.12%
increases, the contribution
90 24300 720 10 25048 97.01% 2.87% 0.11%
of 3n2 term increases and 95 27075 760 10 27864 97.17% 2.73% 0.10%
other terms decrease! 100 30000 800 10 30830 97.31% 2.59% 0.10%
 Experiment #1
• Observation:
 As n∞ the term 3n2 dominates (i.e.
approaches 100%) while the other terms
decease (i.e. approaches 0%)
• Conclusion:
 We can ignore the lower degree terms from
the complexity function as n∞
 This leads to the first approximation of the
previous complexity function to T(n)≈3n2
• Now how about the coefficient 3?
 n C(n^2) C(8n) C(10)
Experiment #2 1 5.26% 42.11% 52.63%
5 32.89% 52.63% 14.47%
100.00% 10 52.08% 41.67% 6.25%
90.00% 15 62.85% 33.52% 3.63%
80.00%
20 69.69% 27.87% 2.44%
70.00%
60.00% 25 74.40% 23.81% 1.79%
C(n^2)
50.00% 30 77.85% 20.76% 1.38%
C(8n)
40.00%
C(10) 35 80.49% 18.40% 1.12%
30.00%
20.00% 40 82.56% 16.51% 0.93%
10.00% 45 84.23% 14.98% 0.79%
0.00%
50 85.62% 13.70% 0.68%
0 20 40 60 80 100 120
55 86.78% 12.62% 0.60%
60 87.76% 11.70% 0.54%
65 88.61% 10.91% 0.48%
If we ignore the coefficient of
70 89.35% 10.21% 0.44%
the highest degree term, it still 75 90.00% 9.60% 0.40%
dominates the other two 80 90.57% 9.06% 0.37%
terms as n is getting bigger 85 91.09% 8.57% 0.34%
90 91.55% 8.14% 0.32%
95 91.96% 7.74% 0.30%
100 92.34% 7.39% 0.28%
 Experiment #2
• Observation:
 Ignoring the coefficient of the highest degree
term does not affect the contribution of that
term on the growth of the complexity function
T(n), i.e. it still dominates the other two terms
as long as n∞
• Conclusion:
 As n∞ we can simply drop the coefficient of
the highest degree term since it is still
dominating the other terms in the complexity
function and therefore T(n)≈n2
 The Effect of the Problem Size (n)

• So as the problem size n increases, we

can approximate the complexity function,
but what is the minimum problem size
beyond which we can approximate????

• For example, in the previous example, we

know that the exact time complexity is T(n)
= 3n2+8n+10 and the approximate one is
T(n)≈n2, what is the minimum problem size
n that satisfies this approximation?
• Draw Approximated T(n) Experiment #3
over Exact T(n) as n
increases Ratio between the Approximate T(n)
and Actual T(n)
• As you can see, when n is
beyond a certain value,

Ratio of Approx T(n)/Exact T(n)

0.35
0.3
the ratio=0.33 starts to
0.25
stabilize 0.2

• We conclude from this 0.15

0.1
experiment that the
0.05
approximated T(n) is 0
growing with the same 0 50 100 150
problem Size n
rate as the actual T(n)
when the problem size is
around 25 or more
 Rate of Function Growth
• Generally speaking, mathematicians when
they studied functions, they decided to group
these functions according to their rate of
growth
• This field of science is known as asymptotic
growth of functions
• We will focus on three important asymptotic
notations:
 Big-oh Notation: O(f(n))  Worst Case complexity
 Theta Notation: θ(f(n))  Average Case complexity
 Omega Notation: Ω(f(n))  Best Case complexity
 Big-oh Notation
• Let g(n) be a function
• The set O(g(n)) is defined as

• In other words, f (n)O(g(n)) if and only if

there exist positive constants c, and n0,
such that for all n≥n0, the inequality
0≤f(n)≤ cg(n) is satisfied
• We say that f (n) is Big-Oh of g(n), or that
g(n) is an asymptotic upper bound for f (n)
 Big-Oh Notation

• We use the big-Oh notation Starting from n0,

to study the worst-case time the f(n) ≤ cg(n)
complexity of an algorithm:
• When we say that the worst-
case time complexity of an
algorithm A is O(n2) we
mean: “We don’t know the
exact time complexity of the
algorithm A but we can
assure that when the
problem instance size f (n)O(g(n)) if and only if
exceed n0, it is not higher
than cn2 “
0≤f(n)≤ cg(n)
 Example 1
3n+2 = O(n)
• Prove that f(n)=3n+2 is
an element of O(n) 10

• Proof:

Time Complexity
8
 We need to find two real 6
numbers n0>0 and c>0
where the inequality 4 3n+2
0≤f(n)≤cn is fulfilled
2 5n
 Take n0=1 and c=5 
0≤3n+2≤5n 0
 Since the inequality is 0 1 2 3
fulfilled with n0=1 and Problem size n
c=5, therefore f(n)ЄO(n)
 Example 2
400
350
• Show that f(n)=3n2+20

Time Complexity
300
has O(n2) 3n^2+20
250 4n^2
 We need to find two real 200
numbers n0>0 and c>0
150
where the inequality 0≤
100
3n2+20 ≤cn2 is fulfilled
50
 Let n0=5 and c=4
0
 0≤ 3n2+20 ≤4n2 0 2 4 6 8 10 12
Problem size n
 3n2+20  O(n2)
What is the time complexity of multiplying two
arrays of size n?

Algorithm multiply (x[], y[], n)

sum  0;
for (i=0; i<n; i++)
sum  sum +x[i]*y[i];
return sum;
What is the big-oh of multiplying
two arrays of size n?

Algorithm multiply (x[], y[], n)

sum  0;
for (i=0; i<n; i++)
sum  sum +x[i]*y[i];
return sum;
 Example 3 T(n)=8 n +3
 O(n)= n
• What is the big-oh of
multiplying two arrays of size n?

Algorithm multiply (x[], y[], n)

sum  0;
for (i=0; i<n; i++)
sum  sum +x[i]*y[i];
return sum;
 Big-Oh vs. Actual RunTime
• Example 4: let algorithms A and B have
running times TA(n) =20n and TB(n) =0.1 n
log2n
• In the “Big-Oh” sense, B is better than A…
• But: on which data volume can A
outperform B?
TA(n) < TB(n) if 20n < 0.1n log2n, or
log2n > 200, that is, when n >2200 ≈ 1060 !
• Thus, in all practical cases B is better than46
A…
 Big-Oh vs. Actual Running
Time
• Example 5: let algorithms A and B have
running times TA(n) =20n and TB(n) =0.1n2
• In the “Big-Oh”sense, A is better than B…
• But: on which data volumes A outperforms
B?
TA(n) < TB(n) if 20n < 0.1n2, or n >200
• Thus A is better than B in most practical
cases except for n < 200 when B becomes
faster…
47
 Some Big-Oh Rules
• Rule#1:
O(f(n))+O(g(n)) = O(f(n)+g(n))

• The above rule simply says that if you

have two algorithms that are executed one
after the other and one of them has O(f(n))
and the other one has O(g(n)) then the
overall complexity of these two algorithms
is the big-oh of the sum of f(n) and g(n)
 Example 6: Reading then Sorting an Array

• Algorithm readArray (x[], n) The time

complexity of
for(i=0; i<n; i++) read x[i]; readArray
return; algorithm =
O(n)
• Algorithm Sort(x[], n)
The time complexity of sort
for (i=0; i<n; i++) algorithm = O(n2)
for (j=0; j<i; j++)
if (x[j]>x[i]) The time
complexity of
swap(x[i],x[j]);
reading then
return; sorting the array
is O(n+n2)=O(n2)
 Some Big-Oh Rules
• Rule #2:
O(f(n)) * O(g(n)) = O(f(n) * g(n))

• This rule is applied when one algorithm

with complexity O(f(n)) is calling another
algorithm with complexity O(g(n))
 Example 7
• Algorithm
Read_sort_write (n, m)
for (array1; array <= m; array++)
readArray (x, n);
sortArray( x,n);
return;
T(readArray) = O(n)
T(sortArray)=O(n2)
T(read+sort)=O(n+n2)=O(n2)
The above algorithms are
executed m times
Therefore the overall time complexity of R
algorithm is O(m*n2)
 Example 8
• Algorithm
Read_sort_write (n, m)
for (array1; array <= m; array++)
readArray (x, n);
sortArray( x,n);
printArray(x, n);
return; T(readArray) = O(n)
T(sortArray)=O(n2)
T(printArray)=O(n)
T(read+sort+print)=O(n+n2+n)=O(n2)
The above three algorithms are
executed m times
Therefore the overall time complexity of
Read_sort_write algorithm is O(m*n2)
 Example 9
What is the time complexity of:
sum = 0;
for (k=1; k<=n; k*=2)
for (j=1; j<=n; j++)
sum++;

A. O(n^2)
B. O(n)
C. O(log n)
D. O(n log n)
 Example 10
What is the time complexity of:
sum = 0;
for (k=1; k<=n; k*=2) // Do log n
times
for (j=1; j<=n; j++) // Do n times
sum++;
A. O(n^2)
B. O(n)
C. O(log n)
D. O(n log n)
7 functions used in analysis of algorithms
1. The exponential function
f(n) = 2n

2. The logarithmic function

f(n) = logb n,
The most fundamental
logarithmic function is
g(n) = log2 n

3. The Constant Function

f(n) = c, c is a constant
The most fundamental
constant function is: g(n) = 1
55
7 functions used in analysis of algorithms

Comparing the growth of the running time as the input grows to the growth of known
functions.

Input (1) log n n n log n n² n³ 2ⁿ

Size:
n
5 1 3 5 15 25 125 32

10 1 4 10 33 100 10³ 10³

100 1 7 100 664 104 106 1030

1000 1 10 1000 104 106 109 10300

10000 1 13 10000 105 108 1012 103000

7 functions used in analysis of algorithms
2ⁿ

57
Example 11: Comparing Algorithm Efficiency

• Consider the following 3 Algorithms for

computing 1+2+…+n , n > 0

Algorithm A
sum = 0
for i = 1 to n
sum = sum +i

58
Example 11: Comparing Algorithm Efficiency

• Consider the following 3 Algorithms for

computing 1+2+…+n , n > 0

Algorithm A Algorithm B
sum = 0 sum = 0
for i =1 to n for i = 1 to n
sum = sum +i {for j =n to i
sum = sum +1
}

59
Example 11: Comparing Algorithm Efficiency

• Consider the following 3 Algorithms for

computing 1+2+…+n , n > 0

Algorithm A Algorithm B Algorithm C

sum = 0 sum = 0 sum = n * ( n +1)/ 2
for i =1 to n for i = 1 to n
sum = sum +i {for j =n to i
sum = sum +1
}

60
Example 11: Comparing Algorithm Efficiency
Algorithm A Algorithm B Algorithm C
sum = 0 sum = 0 sum = n * ( n +1)/ 2
for i =1 to n for i = 1 to n
sum = sum +i {for j =n to i
sum = sum +1 }

The number of operations required in each algorithm is

Alg. A Alg. B Alg. C
Assignments n +1 1
Additions n 1
Multiplications 1
Divisions 1
Total 2 n +1 4

61
 Big-Omega
• The function g(n) is W(f(n)) iff there exist a
real positive constant c > 0 and a positive
integer n0 such that g(n)  cf(n) for all n  n0
 Big Omega is just opposite to Big Oh
 It generalises the concept of “lower bound” ()
in the same way as Big Oh generalises the
concept of “upper bound” (≤)
If f(n) is O(g(n)) then g(n) is W(f(n))

62
 Big-Theta
• The function g(n) is Q(f(n)) iff there exist
two real positive constants c1 > 0 and c2 > 0
and a positive integer n0 such that:
c1f(n)  g(n)  c2f(n) for all n  n0
 Whenever two functions, f and g, are of the
same order, g(n) is Q(f(n)), they are each Big-
Oh of the other:
g(n) is O(f(n)) AND f(n) is O(g(n))

63
 W and Q Notations
• f(n) is W (g(n)) iff g(n) is O(f(n))

• f(n) is Q (g(n)) if it is W (g(n)) and O(g(n))

64
 To DO
• Read chapter 2 in book