Dynamic Programming:

The Matrix Chain Algorithm

Andreas Klappenecker

!
[partially based on slides by Prof. Welch]
 Matrix Chain Problem

Suppose that we want to multiply a sequence of rectangular

matrices. In which order should we multiply?

A x (BxC) or (AxB) x C
 Matrices
An n x m matrix A over the real numbers is a rectangular array of
nm real numbers that are arranged in n rows and m columns.

For example, a 3 x 2 matrix A has 6 entries

!
a11 a12

A =
a21 a22

a31 a32
!
!

where each of the entries aij is e.g. a real number.

 Matrix Multiplication

Let A be an n x m matrix

B an m x p matrix

The product of A and B is n x p matrix AB whose (i,j)-th entry is

∑k=1m aik bkj

In other words, we multiply the entries of the i-th row of A

with the entries of the j-th column of B and add them up.
Matrix Multiplication
Complexity of Matrix Multiplication
Let A be an n x m matrix, B an m x p matrix. Thus,

AB is an n x p matrix. Computing the product AB takes

nmp scalar multiplications

n(m-1)p scalar additions

for the standard matrix multiplication algorithm.

 Matrix Chain Order Problem

Matrix multiplication is associative, meaning that (AB)C = A(BC).

Therefore, we have a choice in forming the product of several
matrices.

What is the least expensive way to form the product of several

matrices if the naïve matrix multiplication algorithm is used?

[We use the number of scalar multiplications as cost.]

 Why Order Matters
Suppose we have 4 matrices:

A: 30 x 1

B: 1 x 40

C: 40 x 10

D: 10 x 25

((AB)(CD)) : requires 41,200 scalar multiplications

(A((BC)D)) : requires 1400 scalar multiplications

 Matrix Chain Order Problem

Given matrices A1, A2, …, An,

where Ai is a di-1 x di matrix.

[1] What is minimum number of scalar multiplications required to

compute the product A1· A2 ·… · An?

[2] What order of matrix multiplications achieves this minimum?

We focus on question [1], and sketch an answer to [2].

 A Possible Solution

Try all possibilities and choose the best one.

Drawback: There are too many of them (exponential in the number

of matrices to be multiplied)

We need to be smarter: Let’s try dynamic programming!

 Step 1: Develop a Recursive
Solution
• Define M(i,j) to be the minimum number of multiplications needed to
compute Ai· Ai+1 ·… · Aj

• Goal: Find M(1,n).

• Basis: M(i,i) = 0.

• Recursion: How can one define M(i,j) recursively?

 Defining M(i,j) Recursively
• Consider all possible ways to split Ai through Aj into two
pieces.

• Compare the costs of all these splits:

• best case cost for computing the product of the two pieces

• plus the cost of multiplying the two products

• Take the best one

• M(i,j) = mink(M(i,k) + M(k+1,j) + di-1dkdj)

 Defining M(i,j) Recursively

(Ai ·…· Ak)·(Ak+1 ·… · Aj)

P1 P2
•minimum cost to compute P1 is M(i,k)
•minimum cost to compute P2 is M(k+1,j)
•cost to compute P1· P2 is di-1dkdj
 Step 2: Find Dependencies
Among Subproblems

M: 1 2 3 4 5
1 0 GOAL!
2 n/a 0
computing the pink
3 n/a n/a 0 square requires the
4 n/a n/a n/a 0 purple ones: to the
left and below.
5 n/a n/a n/a n/a 0
 Defining the Dependencies

Computing M(i,j) uses

everything in same row to the left:

M(i,i), M(i,i+1), …, M(i,j-1)

and everything in same column below:

M(i,j), M(i+1,j),…,M(j,j)
 Step 3: Identify Order for
Solving Subproblems
Recall the dependencies between subproblems just found

Solve the subproblems (i.e., fill in the table entries) this way:

- go along the diagonal

- start just above the main diagonal

- end in the upper right corner (goal)

 Order for Solving Subproblems

M: 1 2 3 4 5
1 0 4
3
2
2 n/a 0 1
3 n/a n/a 0
4 n/a n/a n/a 0
5 n/a n/a n/a n/a 0
 Pseudocode
for i := 1 to n do M[i,i] := 0

for d := 1 to n-1 do // diagonals

for i := 1 to n-d to // rows w/ an entry on d-th diagonal

j := i + d // column corresp. to row i on d-th diagonal

pay attention here
M[i,j] := infinity
to remember actual
for k := i to j-1 to
sequence of mults.
M[i,j] := min(M[i,j], M[i,k]+M[k+1,j]+di-1dkdj)

endfor

endfor
running time O(n3)
endfor
 Example
M: 1 2 3 4
1: A is 30x1
1 0 1200 700 1400 2: B is 1x40
3: C is 40x10
2 n/a 0 400 650 4: D is 10x25
3 n/a n/a 0 10,000 BxC: 1x40x10

(BxC)xD:

4 n/a n/a n/a 0 400 + 1x10x25

Bx(CxD):

... + 10,000
 Keeping Track of the Order

• It's fine to know the cost of the cheapest order, but what is that
cheapest order?

• Keep another array S and update it when computing the minimum

cost in the inner loop

• After M and S have been filled in, then call a recursive algorithm
on S to print out the actual order
 Modified Pseudocode
for i := 1 to n do M[i,i] := 0

for d := 1 to n-1 do // diagonals

for i := 1 to n-d to // rows w/ an entry on d-th diagonal

j := i + d // column corresponding to row i on d-th diagonal

M[i,j] := infinity

for k := i to j-1 to

M[i,j] := min(M[i,j], M[i,k]+M[k+1,j]+di-1dkdj)

if previous line changed value of M[i,j] then S[i,j] := k

endfor
keep track of cheapest split point
endfor
found so far: between Ak and Ak+1
endfor
 Example
M: 1 2 3 4
1: A is 30x1
S: 1 0 1200 700 1400 2: B is 1x40
1 1 1
3: C is 40x10
2 n/a 0 400 2 650 4: D is 10x25
3
3 n/a n/a 0 10,000
3 A x (BCD)

4 n/a n/a n/a 0 A x ((BC) x D)

A x ((BxC) x D)
 Using S to Print Best Ordering

Call Print(S,1,n) to get the entire ordering.

Print(S,i,j):

if i = j then output "A" + i //+ is string concat

else

k := S[i,j]

output "(" + Print(S,i,k) + Print(S,k+1,j) + ")"