Introduction to Algorithms: 6.

006
Massachusetts Institute of Technology February 9, 2018
Instructors: Srini Devadas, Jason Ku, and Nir Shavit Recitation 2

Recitation 2

Model of Computation
In order to precisely calculate the resources used by an algorithm, we will need a model for how
long a computer takes to perform basic operations. Specifying such a set of operations provides a
model of computation upon which we can base our analysis.

Word-RAM
In the last recitation, we made an assumption that the length of a student’s name is at most a
constant number of characters so that we could read and write names in constant time. However,
if we have n students and want to allow each student to have a different name, we would need to
store each name using at least log n bits, or else there would be more students than possible names.
If we want to allow representations of names or numbers containing a logarithmic number of bits,
does that mean it will take Ω(log n) time to read a name or number that is Ω(log n) bits long?
In actuality, modern computers perform computation on words, not individual bits. A word is
a fixed-length sequence of w bits: words on a 32-bit machine are 32-bits long, and 64-bits long on a
64-bit machine. The Word-RAM model of computation models a computer’s memory as an array
of words, with each word accessible in constant time given a memory address, the word’s index
(i.e. pointer) into the array. The Word-RAM also has a number of w-bit length word registers
upon which it can perform simple operations in constant time: read, write, add, compare, etc.
In order to perform operations requiring an address (like a read or copy), registers need to be
able to contain a memory address within a word. If a word is w bits long, the memory address space
will be limited to 2w values, bounding the possible size of your computer’s accessible memory1 .
Thus, when solving a problem on an input with size n, the Word-RAM model assumes word size
w is at least Ω(log n) bits, or else you would not be able to access all of the input in memory. To
put this limitation in perspective, a Word-RAM model of a byte-addressable 64-bit machine allows
inputs storable in up to ∼ 1010 GB.
So in a Word-RAM model, we assume that access (read, write, copy, etc.) and bitwise (add,
subtract, multiply, divide, shift, mod, compare, etc.) operations can be performed on numbers
stored using a constant number of words in constant time, a reasonable approximation of existing
CPU architectures. So, if student names are at most O(log n) characters, each name can fit into
a constant number of words and can be operated on in constant time. Most of the time, we will
ignore this subtlety and simply assume that numbers or identifying strings appearing in a problem
input fit inside a constant number of words, but how we store numbers will be relevant later in the
term, when we talk about linear sorting and hashing.
1
For example, on a typical 32-bit machine, each byte (8-bits) is addressable (for historical reasons), so the size of
the machine’s random-access memory (RAM) will be limited to (8-bits)×(232 ) ≈ 4 GB.
 Recitation 2 2

Python
In this class, we will use Python 3 to implement algorithms, so it will be useful to have a model
of computation defined in terms of Python data structures rather than the Word-RAM (though the
Python model can be implemented by a Word-RAM). In a Python model of computation, constant
time operations include basic operations on reasonably sized numbers: arithmetic, bit manipula-
tions, comparisons, etc. Additionally, Python uses two primary data structures, lists and dictio-
naries, supporting different operations whose running times can depend on the data structure’s
size n at the time of the operation.2 Below are tables specifying operation running times. Right
now, don’t worry about understanding how these data structures achieve the listed running times;
we will learn how later in this class. Python lists are implemented using dynamic arrays, while
dictionaries are implemented using hash tables.
List len get/set append/pop insert/pop in/index/remove
Operations (at index) (last index) (at index) (by item)
Running Time O(1) O(1) O(1)† O(n) O(n)

Dictionary len get/in set/del

Operations (by key) (by key)
Running Time O(1) O(1)∗ O(1)†∗
† Amortized, ∗ Expected

Peak Finding
Let’s use Python to find a peak in a 1D array of n integers. By peak, we mean a (weak) local
maximum, i.e., an element that is at least as large as its neighbors in the array. There’s an obvious
linear time, brute force algorithm: scan through each element, and check whether it is a peak.
1 def peak_find_brute(A):
2 ’’’Find index of a peak from an input list of integers.’’’
3 n = len(A) # O(1)
4 for i in range(n): # O(n) Loop through array
5 if ((i != 0 and A[i] < A[i - 1]) or # O(1) Check larger left
6 (i != n - 1 and A[i] < A[i + 1])): # O(1) Check larger right
7 continue # O(1) Not a peak
8 return i # O(1) Peak found!
9 return None # O(1) No peak found

Alternatively, a (weak) global maximum is an element that is at least as large as every element
of the array, so a global maximum is also a peak. A global maximum always exists, so the last line
of the code will never be reached, assuming a nonempty array. A global maximum requires Ω(n)
time to find; if it didn’t, at least one array index cannot have been observed by the algorithm, and
an adversary could place the largest integer there. By contrast, we will be able to find a local peak
in sub-linear time.
2
Python sets and object attributes are essentially specialized versions of Python dictionaries.
 Recitation 2 3

Our approach will be to repeatedly reduce the peak search range by a constant factor. Since we
know the array contains a peak, if we divide the array in half, a peak must exist in at least one of the
halves (possibly both). If we can quickly identify one half that contains a peak, we can continue
the search in that half recursively. If we can guarantee that the search space contains a peak each
time we reduce it, when we finally reduce the search space to a single element of the array, that
element must be a peak. This type of argument is called an inductive argument. The guarantee
is called an invariant, which we must prove holds throughout the computation. The invariant we
would like to maintain is that each sub-array in our search contains a peak.
How can we quickly identify a half of the array that contains a peak? Consider the sub-array
extending from index i to index j. If A[i] is at least as large as its left neighbor (A[i − 1] ≤ A[i])
and A[j] is at least as large as its right neighbor (A[j] ≥ A[j + 1]), then a global maximum of
that sub-array must also be a peak.3 Therefore, we can quickly guarantee the existence of a peak
by looking only at the boundary of a range, without looking at all of its elements. This property
suggests a stronger invariant: that the elements stored in each endpoint of the search range are at
least as large as their outer neighbors. Then to maintain the invariant, we simply look at the middle
two elements, and recursively search on the side containing the larger of the two.

1 def peak_find(A, r = None):

2 ’’’
3 Find index of a peak from range r of an input list of integers.
4 Assumes that the range specied by r contains a peak.
5 ’’’
6 if r is None: # O(1)
7 r = (0, len(A) - 1) # O(1)
8 i, j = r # O(1)
9 if i == j: # O(1)
10 return i # O(1) Base case, one element range
11 c = (i + j + 1) // 2 # O(1) Compute center
12 if A[c - 1] < A[c]: # O(1)
13 r = (c, j) # O(1) Right half has peak
14 else: # O(1)
15 r = (i, c - 1) # O(1) Left half has peak
16 return peak_find(A, r) # O(?) Recursive call

What is the running time of this recursive algorithm? We can compute it by evaluating a
recurrence relation, where T (k) represents the running time of the function on an input (i.e., a
remaining search range) of size k. This function does a constant amount of work, then makes at
most one recursive call on a problem of roughly half the size. A recurrence relation representing
our algorithm is T (k) = T (k/2) + O(1), where solving the base case takes constant time T (1) =
O(1). One way to solve a recurrence is to guess a solution, and substitute into the recurrence to
show that the solution is correct. Guessing T (n) = O(log n) shows that this algorithm runs in
logarithmic time relative to the input.
3
Proving this might be a nice way to brush up on your proof writing from 6.042. Consider the cases where the max
lies on the boundary or not.
Recitation 2 4

Exercises
There are three primary methods for solving recurrences:

• Substitution: Guess a solution and substitute to show the recurrence holds.

• Recursion Tree: Draw a tree representing the recurrence and sum computation at nodes.

• Master Theorem: A general formula to solve a large class of recurrences.

We will discuss these methods in more detail, including introducing the Master Theorem next
week, but try to apply the first two methods to solve some of the following recurrences, assuming
T (1) = O(1).

1. T (n) = T (n − 1) + O(1)
Solution: T (n) = O(n), length n chain, O(1) work per node.

2. T (n) = T (n − 1) + O(n)
Solution: T (n) = O(n2 ), length n chain, O(k) work per node at height k.

3. T (n) = 2T (n − 1) + O(1)
Solution: T (n) = O(2n ), height n binary tree, O(1) work per node.

4. T (n) = T (2n/3) + O(1)

Solution: T (n) = O(log n), length log3/2 (n) chain, O(1) work per node.

5. T (n) = 2T (n/2) + O(1)

Solution: T (n) = O(n), height log2 n binary tree, O(1) work per node.

6. T (n) = T (n/2) + O(n)

Solution: T (n) = O(n), length log2 n chain, O(2k ) work per node at height k.

7. T (n) = 2T (n/2) + O(n)

Solution: T (n) = O(n log n), height log2 n binary tree, O(2k ) work per node at height k.

8. T (n) = 4T (n/2) + O(n)

Solution: T (n) = O(n2 ), height log2 n degree-4 tree, O(2k ) work per node at height k.