ECS 36C - Algorithm analysis

Prof. Joël Porquet-Lupine

UC Davis - Spring Quarter 2020

Copyright © 2018-2020 Joël Porquet-Lupine - CC BY-NC-SA 4.0 International License 1 / 34

Introduction
Predict efficiency?
Consider a program for classroom
scheduling
Preliminary testing shows it can
schedule 10 classrooms in 1.42 seconds!

Is the program efficient? Should UCD

buy it?

How will it perform in real conditions? 240?

Running time (in seconds)
With UCD's 130 classrooms (and
thousands of courses)...

18.5?
1.42
3?
10 Number of classrooms 130

2 / 34
Introduction
Compare algorithms?
Search for a certain number in a sorted list of numbers

Linear search: Binary search:

6 12 46 47 52 73 84 91 6 12 46 47 52 73 84 91

Best-case: First of list! Best-case: Middle of list!

Worst-case: Scan the entire list Worst-case: Scan half of remaining sub-
list at each step

Is there a better way to express each algorithm's characteristics?

3 / 34
Introduction
Algorithm analysis
1. Predict performance
What should be the running time of my program if the amount of input data change?
What about the memory occupation?
2. Compare algorithms
Before starting a costly implementation, can we ensure a specific algorithm is better
than an other?
Can we define "better"?
3. Characterize
What is the worst-case efficiency?
Average-case? Best-case?

This class: Informal approach

More formal approaches in advanced classes (e.g. 122A/122B)

4 / 34
Experimental approach
3-SUM problem
Definition Example
In a set of N integers, how many (unique) $ cat 8ints.txt
triplets sum to zero? 30 -40 -20 -10 40 0 10 5

$ ./3sum < 8ints.txt

Number of triplets: 4

a[i] a[j] a[k] sum

30 -40 10 0

30 -20 -10 0

-40 40 0 0

-10 0 10 0

Intersection of 3 lines in a plane

5 / 34
Experimental approach
Brute-force algorithm
int Count3Sum(const std::vector<int> &a) {
Read integers from standard
int N = a.size(); input into a vector
int count = 0;
Examine all possible triplets!
for (int i = 0; i < N; i++)
for (int j = i + 1; j < N; j++)
for (int k = j + 1; k < N; k++)
if (a[i] + a[j] + a[k] == 0)
count++;

return count;
}

int main(void) {
int n;
std::vector<int> a;

while (std::cin >> n)

a.push_back(n);

std::cout << "Number of triplets: "

<< Count3Sum(a) << std::endl;

return 0;
}
3sum.cc

6 / 34
 Experimental approach
Timing measurement *
Multiple runs, each with a different amount of data:

$ time ./3sum < 1Kints.txt

Number of triplets: 644996
./3sum < 1Kints.txt 0.98s user 0.00s system 99% cpu 0.983 total

$ time ./3sum < 2Kints.txt

Number of triplets: 5344063
./3sum < 2Kints.txt 7.82s user 0.00s system 99% cpu 7.828 total

$ time ./3sum < 4Kints.txt

Number of triplets: 41964881
./3sum < 4Kints.txt 62.66s user 0.00s system 99% cpu 1:02.68 total

$ time ./3sum < 8Kints.txt

Number of triplets: 342566644
./3sum < 8Kints.txt 500.52s user 0.00s system 100% cpu 8:20.50 total

Somewhat difficult to comprehend...

Let's plot for better visualization!

*CSIF computer: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz 7 / 34

Experimental approach
Plot (standard)
Running time T (N ) vs input size N
600
?

500

400
Running time T(N)

300

200

100

0
1K 2K 4K 8K
Problem size N

So, not a linear progression...

Can we predict the running time for even larger input (e.g. 16K)?

8 / 34
 Experimental approach
Plot (log-log)
Running time T (N ) vs input size N on a log-log plot
1000
Typical linear curve:
f (x) = m ∗ x + b
Line equation on a log-log scale:
100 lg(T (N )) = m ∗ lg(N ) + b

Find the slope and intercept:

lg(y2 /y1 )
m= ≈ 2.9978
Running time lg(T(N))

​ ​

lg(x2 /x1 )
​

​ ​

10 b = lg(y) − 2.9978 ∗ lg(x) ≈ −30.00

Now, can translate back into equation on

standard scale:
1
T (N ) = N m ∗ 2b
T (N ) = N 2.9978 ∗ 2−30.00

1K 2K 4K 8K
Problem size lg(N)

9 / 34
Experimental approach
Back to standard plot
Running time T (N ) vs input size N
600
?

500

400
Running time T(N)

300

200

100

0
1K 2K 4K 8K
Problem size N

T (N ) = N m ∗ 2b
T (N ) = N 2.9978 ∗ 2−30.00

10 / 34
Experimental approach
Validation of prediction?
Hypothesis:

T (N ) = N 2.9978 ∗ 2−30.00

Predictions:
For N = 8K, running time should be about 502 seconds (08m:22s)
For N = 16K, running time should be about 4122 seconds (01h:08m:42s)

Experiments:

$ time ./3sum < 8Kints.txt

...
./3sum < 8Kints.txt 500.52s user 0.00s system 100% cpu 8:20.50 total

$ time ./3sum < 16Kints.txt

...
./3sum < 16Kints 4132.14s user 0.00s system 99% cpu 1:08:52.42 total

Hypothesis is validated!

11 / 34
Experimental approach
Conclusion
Issues
Need to actually implement the algorithm in order to characterize it experimentally!
Difficult to get precise measurements as it heavily depends on:
The input data
The hardware (CPU, memory, caches, etc.)
The software (compiler, garbage collector, etc.)
The system (OS, network, etc.)
Require to run a significant (potentially huge) number of experiments to gather data
points

Next time
Need for a better model...

12 / 34
 ECS 36C - Algorithm analysis (part 2)
Prof. Joël Porquet-Lupine

UC Davis - Spring Quarter 2020

Copyright © 2018-2020 Joël Porquet-Lupine - CC BY-NC-SA 4.0 International License 13 / 34

Recap
Experimental approach
Algorithm implementation Measurements
int Count3Sum(const std::vector<int> &a) { $ time ./3sum < 1Kints.txt
int N = a.size(); Number of triplets: 644996
int count = 0; ./3sum < 1Kints.txt 0.98s user 0.00s system 99% cpu 0.983 total

for (int i = 0; i < N; i++)

$ time ./3sum < 2Kints.txt
for (int j = i + 1; j < N; j++)
Number of triplets: 5344063
for (int k = j + 1; k < N; k++)
./3sum < 2Kints.txt 7.82s user 0.00s system 99% cpu 7.828 total
if (a[i] + a[j] + a[k] == 0)
count++;
$ time ./3sum < 4Kints.txt
return count; Number of triplets: 41964881
} ./3sum < 4Kints.txt 62.66s user 0.00s system 99% cpu 1:02.68 total
3sum.cc

$ time ./3sum < 8Kints.txt

Number of triplets: 342566644
./3sum < 8Kints.txt 500.52s user 0.00s system 100% cpu 8:20.50 total
Model
Prediction
600
?

500
For N = 8K, running time should be about
502 seconds (08m:22s)
400
Running time T(N)

For N = 16K, running time should be about

300

200
4122 seconds (01h:08m:42s)
100

0
1K 2K 4K 8K
Problem size N

Running time (T(N)) vs input size (N)

14 / 34
Mathematical approach
Total running time is sum of (cost x frequency) for all operations:

∑ cost(op) ∗ f req(op)
​

op

Might be very complicated:

Need to analyse entire program to determine all the actual operations
Actual cost depends on computer, compiler
Actual frequency depends on algorithm, input data

Instead, let's simplify:

Count number of statements
As a function of input size N

15 / 34
Mathematical approach
1-SUM
int count = 0; Statement Frequency

for (int i = 0; i < N; i++) int count = 0; 1

if (a[i] == 0)
count++; int i = 0; 1
return count;
1sum.cc i < N; N +1

i++; N

if (a[i] == 0) N

count++; 0 to N

return count; 1

Estimated running time (worst-case):

T (N ) = 4N + 4

16 / 34
Mathematical approach
2-SUM
int count = 0; Statement Frequency

for (int i = 0; i < N; i++) int count = 0; 1

for (int j = i + 1; j < N; j++)
int i = 0; 1
if (a[i] + a[j] == 0)
count++; i < N; N +1
return count; i++; N
2sum.cc

How to compute the frequency of the inner loop?

int j = i + 1; N
Sum of series (Gauss): M + ... + 2 + 1 =
1 1
M (M + 1) j < N; 2
​N (N + 1)
2
​

1
j++; 2
​(N − 1)N
1
if (a[i] + a[j] == 0) 2
​(N − 1)N

count++; 0 to 12 (N − 1)N
​

return count; 1

Estimated running time (worst-case):

T (N ) = 2N 2 + N + 4
Was difficult to compute...

17 / 34
Mathematical approach
Simplify some more
Identify the operation that represents
for (int i = 0; i < N; i++)
the overall running time the best for (int j = i + 1; j < N; j++)
Usually the body of the inner loop if (a[i] + a[j] == 0)
1
− 1) = 12 N 2 − 12 N count++;
2 N (N
​ ​ ​

Could we ignore the lower order terms? 5x10

7

2
1/2x
1 2 1
2N − 2N
2
​ ​
1/2x - 1/2x

∼ 12 N 2
​
4x10
7

7
3x10
Tilde notation: Hint: both curves are
indistinguishable!
f (N )
f (N ) ∼ g(N ) means lim =1
g(N )
​ ​

7
2x10
N →∞

7
1x10

0
0 2000 4000 6000 8000 10000

18 / 34
Mathematical approach
Tilde approximation and order of growth

Function Tilde approximation Order of growth

1 3
6N
​ − 12 N 2 + 13 N
​ ∼ 16 N 3
​ N3
1 2
2N
​ − 12 N
​ ∼ 12 N 2
​ N2

lg N + 1 ∼ lg N lg N

3 ∼3 1

Conclusion
1. In principle, accurate mathematical models are available
2. In practice, approximate models do a good job: T (N ) ∼ cN x
3. For the mass, order of growth is usually enough...

19 / 34
Order-of-growth classification
Typical set of functions describing common algorithms
Constant (1)
a = b + c;
Simple statements

Logarithmic (log N ) while (N > 1) { N = N / 2; ... }

Divide in half

Linear (N )
for (int i = 0; i < N; i++)
Simple loop

Linearithmic (N log N ) [see mergesort lecture]

Divide and conquer

Quadratic (N 2 )
for (int i = 0; i < N; i++)
Double loop for (int j = 0; j < N; j++)

Cubic (N 3 )
for (int i = 0; i < N; i++)
Triple loop for (int j = 0; j < N; j++)
for (int k = 0; k < N; k++)

Exponential (2N )/Factorial (N !)/etc.

f(k) = f(k-1) + f(k-2) // k > 1
Exhaustive search

20 / 34
Order-of-growth classification
Visual representation
100

exponential
ar

ic
quadratic

rithm
e
lin

linea
80
Running time

60

40

20

logarithmic
constant
0
0 20 40 60 80 100

Input size

21 / 34
Order-of-growth classification
Practical implications
Assume a computer with a 1 GHz processor (and IPC of 1),
And an input size N of 100,000.

How long would it take to process the input according to different algorithm's growth
rates?

Growth rate Running time

Logarithmic 1.7e−8 seconds

Linear 0.0001 seconds

Linearithmic 0.00166 seconds

Quadratic 10 seconds

Cubic 11 days

Exponential 3.17 ∗ 1030086 years!

22 / 34
 ECS 36C - Algorithm analysis (part 3)
Prof. Joël Porquet-Lupine

UC Davis - Spring Quarter 2020

Copyright © 2018-2020 Joël Porquet-Lupine - CC BY-NC-SA 4.0 International License 23 / 34

 Recap
Mathematical approach
Counting statements Estimate running time
1 2 1
for (int i = 0; i < N; i++) 2N − 2N
​ ​

for (int j = i + 1; j < N; j++)

if (a[i] + a[j] == 0) ∼ 12 N 2
​

count++;
Order of growth: N 2

Order of growth Implications

100
1GHz processor, input size N=100,000
exponential

r
ic

ea Growth rate Running time

quadratic

rithm

lin
linea

80
Logarithmic 1.7e−8 seconds

Linear 0.0001 seconds

Running time

60

Linearithmic 0.00166 seconds

40
Quadratic 10 seconds

Cubic 11 days
20

logarithmic Exponential 3.17 ∗ 1030086 years!

constant
0
0 20 40 60 80 100

Input size

24 / 34
Big O
Definition
Indicates upper asymptotic bound, using order-of-growth classification
Most usual notation for describing computational complexity of algorithms

Formal definition
T (N ) is O(F (N )) if there exist
c > 0 and n0 > 0 such as
​

T (N ) ≤ cF (N ) when N ≥ n0 ​

90
x2
3x 2 +2x+4
Example
80
9x 2

3N 2 + 2N + 4 = O(N 2 )
70

60
2 2
Because 3N + 2N + 4 ≤ 9N when 50
N ≥1
40
c=9
30
n0 = 1
​

20

10

0
0 0.5 1 1.5 2 2.5 3

25 / 34
Big O
Big-Ω (omega)
Same idea as Big-O but indicates lower bound
T (N ) is Ω(F (N )) if there exist c > 0 and n0 > 0 such as T (N ) ≥ cF (N ) when
​

N ≥ n0 ​

Big-Θ (theta)
Asymptotic tight bound
T (N ) is Θ(F (N )) if T (N ) is both
O(F (N )) and Ω(F (N ))
There exist c1 > 0 and c2 > 0 and n0 > 0 such as
​ ​ ​

T (N ) ≥ c1 F (N ) and T (N ) ≤ c2 F (N ) when N ≥
​ ​

n0 ​

26 / 34
Big O
For the math nerds!
Big-O is only an upper bound
3N 2 + 2N + 4 = O(N 2 )
And also O(N 3 ), and O(N 4 ), and O(N !)

Equality vs inclusion
O(N ) = O(N 2 ) but O(N 2 )  = O(N )
= is assumed unidirectional
∈ is considered a better and non-ambiguous notation
O(N ) ∈ O(N 2 )

Why not always Big-Θ?

Can't all functions be in Big-Θ of something?
f (N ) = N (1 + sin(N ))
∈ O(N ) and ∈ Ω(0)
∈
/ Θ(N ) and ∈
/ Θ(0)

27 / 34
Big O
Algorithm complexity, in practice...
Be aware of the mathematical background and notations
But using Big-O is usually good enough
Ignore lower order terms and drop the leading term's coefficient!
E.g. T (N ) = 4N 2 + N + 3 is considered to be O(N 2 )

Examples
1-SUM: T (N ) = O(N ) for (int i = 0; i < N; i++)
if (a[i] == 0)
count++;
1sum.cc

2-SUM: T (N ) = O(N 2 ) for (int i = 0; i < N; i++)

for (int j = i + 1; j < N; j++)
if (a[i] + a[j] == 0)
count++;
2sum.cc

3-SUM: T (N ) = O(N 3 ) for (int i = 0; i < N; i++)

for (int j = i + 1; j < N; j++)
for (int k = j + 1; k < N; k++)
if (a[i] + a[j] + a[k] == 0)
count++;
3sum.cc

28 / 34
Big O
Some pitfalls...
O(N 2 )?
No, O(N )
for (int i = 0; i < N; i++) {
for (int j = 0; j < 5; j++) { Innermost loop is
... constant!
}
}

O(N 2 )?
No, O(N )
for (int i = 0; i < N; i++) {
if (i == (N - 1)) { Innermost loop is
for (int j = 0; j < N; j++) { executed only once
...
}
}
}

29 / 34
Big O
Function calls (1)
double sum_squares(int count) {
double sum = 0.0;

for (int i = 0; i < count; i++)

sum += std::sqrt(i);

return sum;
}

We know that the for loop is O(N )

What about the function call to std::sqrt()?

Substitute Big O for that function
E.g. std::sqrt() is usually considered as O(1)

Result: sum_squares() is O(N )

30 / 34
Big O
Function calls (2)
bool str_contains(std::string str, std::string pattern)
{
for (int i = 0; i < str.length() - pattern.length(); i++)
if (!std::strncmp(str + i, pattern, pattern.length()))
return true;
return false;
}

We know that the loop is O(N )

What about the function call to std::strncmp()?

By examining the parameters, complexity depends of input size (pattern,
pattern.length())

Result: str_contains() is O(N 2 )

31 / 34
Big O
Log
int BSearch(const std::vector<int> &array, Observations:
int key) {
int lo = 0, hi = array.size() - 1; Divides the input data by 2 at each
while (lo <= hi) { iteration
int mid = lo + (hi - lo) / 2;
Until it finds the searched item (or
if (array[mid] == key) doesn't)
return mid;

if (array[mid] < key) Do the math:

lo = mid + 1;
else
N is the size of the input data
hi = mid - 1;
}
x is the number of iterations
N
return -1; 2x = 1
​

}
binary_search.cc ⇒ x = log2 (N )
​

Result: bsearch() is O(log2 N )

​

Info: exists as a standard function!

std::binary_search(array.begin(), array.end(), key);

32 / 34
Space complexity
Intro
Estimate how much memory an algorithm uses
Input space: space taken by the algorithm's input values
Auxiliary space: temporary extra space used during the execution
Count number of variables and their size
Is total amount of memory related to input size N?

Primitive types (recall from ECS 36A/B)

Type name Size on CSIF computers (GNU/Linux/x86_64)

char 1 byte (8 bits)

short int 2 bytes (16 bits)

int 4 bytes (32 bits)

long int 8 bytes (64 bits)

long long int 8 bytes (64 bits)

float 4 bytes (32 bits)

double 8 bytes (64 bits)

void* 8 bytes (64 bits)

33 / 34
Space complexity
Examples (1)
Input size is N characters int CountLetter(const std::string &str,
const char letter) {
Fixed amount of additional variables int count = 0;
regardless of input's size for (auto c : str)
Overall memory complexity is O(N ) + if (c == letter)
count++;
O(1) = O(N )
return count;
}
count_letter.cc

Examples (2)
Input size is N std::string StrUppercase(const std::string &input) {
std::string output = input;
characters
for (auto &c : output)
Extra memory allocation c = std::toupper(static_cast<unsigned char>(c));
proportional to input's
return output;
size }
uppercase.cc

Memory complexity is O(N ) + O(N ) = O(N )

34 / 34