Algorithm Analysis

Insertion Sort
Sorting
• Arrange an unordered list of elements in some
order.
• Some common algorithms
• Bubble Sort
• Insertion Sort
• Merge Sort
• Quick Sort
Sorting
• Important primitive
• For today, we’ll pretend all elements are distinct.

6 4 3 8 1 5 2 7

1 2 3 4 5 6 7 8

Length of the list is n

Insertion Sort
Insertion Sort 6 4 3 8 5
example

Insertion-Sort(A, n) Start by moving A[1] toward

for i = 1 to n – 1 the beginning of the list until
key = A[i]
j = i – 1 you find something smaller
while j >= 0 and A[j] > key (or can’t go any further):
A[j + 1] = A[j]
j = j – 1
A[j + 1] = key

6 4 3 8 5
4 6 3 8 5
Insertion Sort 6 4 3 8 5
example

Insertion-Sort(A, n)
for i = 1 to n – 1
Then move A[2]:
key = A[i] key = 3
j = i – 1
while j >= 0 and A[j] > key
A[j + 1] = A[j] 4 6 3 8 5
j = j – 1
A[j + 1] = key
4 6 6 8 5
4 4 6 8 5
3 4 6 8 5
Insertion Sort 6 4 3 8 5
example

Insertion-Sort(A, n)
for i = 1 to n – 1
Then move A[3]:
key = A[i] key = 8
j = i – 1
while j >= 0 and A[j] > key
A[j + 1] = A[j]
j = j – 1
A[j + 1] = key 3 4 6 8 5
3 4 6 8 5
Insertion Sort 6 4 3 8 5
example

Insertion-Sort(A, n)
for i = 1 to n – 1
Then move A[4]:
key = A[i] key = 5
j = i – 1
while j >= 0 and A[j] > key
A[j + 1] = A[j]
j = j – 1 3 4 6 8 5
A[j + 1] = key

3 4 6 8 8
3 4 6 6 8
3 4 5 6 8
Insertion Sort 6 4 3 8 5
example
Start by moving A[1] toward
the beginning of the list until
you find something smaller
(or can’t go any further): Then move A[3]:

6 4 3 8 5 3 4 6 8 5
4 6 3 8 5 3 4 6 8 5
Then move A[2]: Then move A[4]:
4 6 3 8 5 3 4 6 8 5
3 4 6 8 5 3 4 5 6 8
Then we are done!
Why does this work?

• Say you have a sorted list, 3 4 6 8 , and

another element 5 .

• Insert 5 right after the largest thing that’s still

smaller than 5 . (Aka, right after 4 ).

• Then you get a sorted list: 3 4 5 6 8

This sounds like a job for…

Proof By
Induction!
Outline of a proof by induction
Let A be a list of length n
• Base case:
• A[:1] is sorted at the end of the 0’th iteration. ✓
• Inductive Hypothesis:
• A[:i+1] is sorted at the end of the ith iteration (of the outer loop).
• Inductive step:
• For any 0 < k < n, if the inductive hypothesis holds for i=k-1, then it holds
for i=k.
• Aka, if A[:k] is sorted at step k-1, then A[:k+1] is sorted at step k
(previous slide)
• Conclusion:
• The inductive hypothesis holds for i = 0, 1, …, n-1.
• In particular, it holds for i=n-1.
• At the end of the n-1’st iteration (aka, at the end of the algorithm), A[:n] =
A is sorted.
• That’s what we wanted! ✓
Algorithm Analysis
• Estimate the resources required by an algorithm
• Memory
• Communication Bandwidth
• Energy Consumption
• Computational Time

• Helps identify the most efficient one

Algorithm Analysis (Insertion Sort)
• Timing the run of insertion sort on our computer
• We will get runtime estimates for,
• A particular computer
• A particular input
• A particular implementation
• A particular compiler/interpreter
• Particular libraries and background tasks

• What about others?

Algorithm Analysis
• Assumptions
• One-processor
• Random-access machine (RAM) model of computation
Random-access Machine
• Random-access machine (RAM)
• Instructions execute one after another
• No concurrent operations
• Each instructions takes the same amount of time
• Each data access takes the same amount of time
• Contains commonly found instructions
• Arithmetic (add subtract, multiply, divide, remainder, floor,
ceiling)
• Data movement (load, store, copy) and
• Control (branching, call and return)
• Includes common data types
• Doesn’t model memory hierarchy
Algorithm Analysis (Insertion Sort)
• Runtime depends on inputs
• Sort an array of 10000 items vs Sort and array of 3 items

• Input Size
• Problem specific
• For sorting, the number of items in the input
• For multiplication, the total number of bits needed for
representation
• For graph, the number of nodes and edges
Algorithm Analysis (Insertion Sort)
• Running time of an algorithm
• For a given input
• The number of instructions and data access executed

• Assumption
• Constant time taken by each line of the pseudocode
Algorithm Analysis (Insertion Sort)

𝑡𝑖 denotes the number of times the while loop test

in line 5 is executed for that value of i
Algorithm Analysis (Insertion Sort)
• T(n) denote the running time of an algorithm on an
input of size of n
Algorithm Analysis (Insertion Sort)
Algorithm Analysis (Insertion Sort)

• Best Case
• The array is sorted
• 𝑡𝑖 = 1 for all 𝑖 = 2, 3, … , 𝑛
Algorithm Analysis (Insertion Sort)

• Best Case
• The array is sorted A linear function of n
• 𝑡𝑖 = 1 for all 𝑖 = 2, 3, … , 𝑛
𝑇 𝑛 = 𝑎𝑛 + 𝑏
where a = c1 + c2 + c4 + c5 + c8
and b = −(c2 + c4 + c5 + c8 )
Algorithm Analysis (Insertion Sort)

• Worst Case
• The array is sorted in reverse order
• 𝑡𝑖 = 𝑖 for all 𝑖 = 2, 3, … , 𝑛
Algorithm Analysis (Insertion Sort)

• Best Case
• The array is sorted
A quadratic function of n
• 𝑡𝑖 = 𝑖 for all 𝑖 = 2, 3, … , 𝑛

𝑇 𝑛 = 𝑎𝑛2 + 𝑏𝑛 + 𝑐
where a, b and 𝑐 are constants
Algorithm Analysis (Insertion Sort)
• Instead of exact function, we estimate the rate of
growth or the order of growth

• Best Case
• Most significant term 𝑎𝑛
• Linear

• Worst Case
• Most significant term 𝑎𝑛2
• Quadratic
More on this later