Dynamic Programming (DP)

Based on: Chapter 18 of WLW*

*Wayne L. Winston, Operations Research, Duxbury Press, 4th Edition. ISBN: 978-0534380588

1
 Description
 Dynamic programming is a technique that can be used to
solve many optimization problems.
 In most applications, dynamic programming obtains
solutions by working backward from the end of the
problem toward the beginning, thus breaking up a large,
unwieldy problem into a series of smaller, more tractable
problems

2
Match Puzzle Example
 We show how working backward can make a seemingly
difficult problem almost trivial to solve.
 Suppose there are 30 matches on a table. I begin by
picking up 1, 2, or 3 matches. Then my opponent must
pick up 1, 2, or 3 matches. We continue in this fashion
until the last match is picked up. The player who picks up
the last match is the loser. How can I (the first player) be
sure of winning the game?

3
Match Puzzle Example
 If I can ensure that it will be opponent’s turn when 1
match remains, I will certainly win.
 Working backward one step, if I can ensure that it will be
my opponent's turn when 5 matches remain, I will win.
 If I can force my opponent to play when 5, 9, 13, 17, 21,
25, or 29 matches remain, I am sure of victory.
 Thus I cannot lose if I pick up 1 match on my first turn.

4
 Milk Example:
• I have a 9-oz cup and a 4-oz cup. My mother has
ordered me to bring home exactly 6oz of milk.
How can I accomplish this goal?
By starting near the end of the problem,
I realize that the problem will be solved
if I can get 1 oz of milk into the 4-oz
cup.
Then I can fill the 9-0z cup and empty 3
oz from the 9oz cup into the partially
filled 4-oz cup.
At this point, I will be left with 6 oz of
milk.
At Table 1 the initial ituation is written
last and the final situation is written
first.
A Network Problem
 Many applications of dynamic programming reduce to
finding the shortest (or longest) path that joins two
points in a given network.
 For larger networks dynamic programming is much more
efficient for determining a shortest path than the explicit
enumeration of all paths.

6
Shortest Path Problem
 Joe Cougar lives in New York City, but he plans to drive to
Los Angeles to seek fame and fortune. Joe’s funds are
limited, so he has decided to spend each night on his trip
at a friend’s house. Joe has friends in Columbus,
Nashville, Louisville, Kansas City, Omaha, Dallas, San
Antonio, and Denver. Joe knows that after one day’s
drive he can reach Columbus, Nashville, or Louisville.
After two days of driving, he can reach Kansas City,
Omaha, or Dallas. After three days of driving, he can
reach San Antonio or Denver. Finally, after four days of
driving, he can reach Los Angeles. To minimize the
number of miles traveled, where should Joe spend each
night of the trip? The actual road mileages between
cities are given in Figure 1 (next slide). 7
Shortest Path Problem

8
Shortest Path Problem
 We define to be the road mileage between city and city .
For example, is the road mileage between Nashville and
Kansas City.
 We let be the length of the shortest path from city to Los
Angeles, given that city is a stage city.
– The length of the shortest path (sp) from NYC to LA is given by .
Stage 4 Computations
 the length of sp from Denver to LA
 the length of sp from San Ant. to LA

9
Shortest Path Problem
Stage 3 Computations
 We now work backward one stage (to stage 3 cities) and find the
shortest path to LA from each stage 3 city.
 For example, to determine , we note that the shortest path from city 5
to LA must be one of the following:
– Path 1 Go from city 5 to city 8 and then take the shortest path from city 8 to
city 10.
– Path 2 Go from city 5 to city 9 and then take the shortest path from city 9 to
city 10.
 The length of path 1 may be written as , and
 the length of path 2 may be written as .
 Hence, the shortest distance from city 5 to city 10 may be written as

 10
Shortest Path Problem
Stage 3 Computations
 Similarly, we find

 The shortest path from city 6 to city 10 is the path 6–8–10.

 We find

 The shortest path from city 7 to city 10 is the path 7–9–10.

11
Shortest Path Problem
Stage 2 Computations
 Given our knowledge of , , and , it is now easy to work
backward one more stage and compute , , and .
 Let’s first find the shortest path (and it length) from city 2
to city 10.

 , and the shortest path from city 2 to city 10 is to go from

city 2 to city 5 and then follow the shortest path from city
5 to city 10 (5–8–10).

12
Shortest Path Problem
Stage 2 Computations


 , and the sp from city 3 to city 10 consists of arc 3–5 and

the sp from city 5 to city 10 (5–8–10).


 , and the sp from city 4 to city 10 consists of arc 4–5 and

the sp from city 5 to city 10 (5–8–10).

13
Shortest Path Problem
Stage 1 Computations
 We can now use our knowledge of , , and to work backward
one more stage to find .

Determination of the optimal path

 , and the shortest path from city 1 to city 10 goes from city 1 to
city 2 and then follows the shortest path from city 2 to city 10.
Checking back to the f2(2) calculations, we see that the
shortest path from city 2 to city 10 is 2–5–8–10. Translating the
numerical labels into real cities, we see that the shortest path
from New York to LA passes through New York, Columbus,
Kansas City, Denver, and LA. This path has a length of miles. 14
Characteristics of Dynamic Programming Applications
 Characteristic 1
– The problem can be divided into stages with a decision
required at each stage.
 Characteristic 2
– Each stage has a number of states associated with it.
– By a state, we mean the information that is needed at any
stage to make an optimal decision.
 Characteristic 3
– The decision chosen at any stage describes how the state at
the current stage is transformed into the state at the next
stage.

15
Characteristics of Dynamic Programming Applications
 Characteristic 4
– Given the current state, the optimal decision for each of the
remaining stages must not depend on previously reached
states or previously chosen decisions.
– This idea is known as the principle of optimality.
 Characteristic 5
– If the states for the problem have been classified into on of T
stages, there must be a recursion that related the cost (or
reward earned) during stages to the cost (or reward earned)
from stages .
– In shortest path example, our recursion could have been
written as

16
An Inventory Problem
 Dynamic programming can be used to solve an inventory
problem with the following characteristics:
1. Time is broken up into periods, the present period being period 1,
the next period 2, and the final period T. At the beginning of period
1, the demand during each period is known.
2. At the beginning of each period, the firm must determine how
many units should be produced. Production capacity during each
period is limited.
3. Each period’s demand must be met on time from inventory or
current production. During any period in which production takes
place, a fixed cost of production as well as a variable per-unit cost
is incurred.
4. The firm has limited storage capacity. This is reflected by a limit on
end-of-period inventory. A per-unit holding cost is incurred on each
period’s ending inventory.
5. The firms goal is to minimize the total cost of meeting on time the
17
An Inventory Problem
 In this model, the firm’s inventory position is reviewed at
the end (or beginning) of each period, and then the
production decision is made.
 Such a model is called a periodic review model.
 This model is in contrast to the continuous review model
in which the firm knows its inventory position at all times
and may place an order or begin production at any time.

18
Inventory Problem
 A company knows that the demand for its product during each of
the next four months will be as follows:
– month 1: 1 unit month 2: 3 units month 3: 2 units month 4: 4 units.
 At the beginning of each month, the company must determine how
many units should be produced during the current month. During a
month in which any units are produced, a setup cost of $3 is
incurred. In addition, there is a variable cost of $1 for every unit
produced.
 At the end of each month, a holding cost of 50¢ per unit on hand is
incurred. Capacity limitations allow a maximum of 5 units to be
produced during each month. The size of the company’s warehouse
restricts the ending inventory for each month to 4 units at most.
Assume that 0 units are on hand at the beginning of the first month.
 The company wants to determine a production schedule that will
meet all demands on time and will minimize the sum of production
and holding costs during the four months. 19
Inventory Problem
 We must first define to be the minimum cost of meeting
demands for months if units are on hand at the beginning of
month t.
–  minimum cost of the problem
 We define to be the cost of producing units during a period.
– then
– and for , .
 Because of the limited storage capacity and the fact that all
demand must be met on time, the possible states during
each period are 0, 1, 2, 3, and 4.
 Thus, we begin by determining , , , , and
 We define to be a production level during month that
minimizes the total cost during months if units are on hand
at the beginning of month .
20
Inventory Problem- Month 4 Computations

21
Inventory Problem - Month 3 Computations

22
Inventory Problem - Month 2 Computations

23
Inventory Problem- Month 1 Computations

24
Inventory Problem
Determination of the Optimal Production Schedule
 Let’s determine a production schedule that minimizes the total
cost of meeting the demand for all four months on time.
 Since our initial inventory is 0 units, the minimum cost for the
four months will be . To attain , we must produce unit during
month 1.
 Then the inventory at the beginning of month 2 will be . Thus,
in month 2, we should produce units.
 Then at the beginning of month 3, our beginning inventory will
be . Hence, during month 3, we need to produce units.
 Then month 4 will begin with units on hand. Thus, units should
be produced during month 4.
 In summary, the optimal production schedule incurs a total cost
of $20 and produces 1 unit during month 1, 5 units during
25
18.6 Formulating Dynamic Programming Recursions

• In many dynamic programming problems, a

given stage simply consists of all the possible
states that the system can occupy at that stage.
• If this is the case, then the dynamic
programming recursion can often be written in
the following form:
ft(i) = min{(cost during stage t) + ft+1 (new state at stage t +1)}
(11)
where the minimum in the above equation is
over all decisions that are allowable, or feasible,
 Dynamic Lot-Size Models

 We let period production.

 Period production can be used to meet period demand.

IE318: Applications in OR 29
 Dynamic Lot-Size Models

 With the possible exception of the first period, production will

occur only during periods in which beginning inventory is zero.
 During each period in which beginning inventory is zero (and ),
production must occur.

IE318: Applications in OR 30
 A Dynamic Lot-Size Problem
 We now determine an optimal production schedule for a
five-period dynamic lot-size model with , , , , , , , and .
 We assume that the initial inventory level is zero.
 The solution to this example will be discussed but before
starting with the solution we need to provide the
recursion relation (next slide).

IE318: Applications in OR 31
 A Dynamic Lot-Size Problem
 Using dynamic programming we can develop a recursion relation
to determine an optimal production policy.
 We know that the initial inventory level is zero.
 Define as the minimum cost incurred during periods , given that
at the beginning of period , the inventory level is zero. Then must
satisfy

 where and is the total cost incurred during periods if production

during period is exactly sufficient to meet demands for periods .
Thus,

setup cost variable production cost inventory holding cost

IE318: Applications in OR 32
 A Dynamic Lot-Size Problem
 To find an optimal production schedule by DP, use the
recursion relation in the previous slide and compute , , ,
…, . Once has been determined, an optimal production
schedule can be easily obtained.
 Now let’s apply DP for this example.
 Stage 6 and 5 Computations

 If we begin period 5 with zero inventory, we should

produce enough during period 5 to meet period 5
demand.
IE318: Applications in OR 33
 A Dynamic Lot-Size Problem
 Stage 4 Computations

 If we begin period 4 with zero inventory, we should

produce enough during period 4 to meet the demand for
period 4.

IE318: Applications in OR 34
 A Dynamic Lot-Size Problem
 Stage 3 Computations

 If we begin period 3 with zero inventory, we should

produce enough during period 3 to meet the demand for
periods 3 and 4.

IE318: Applications in OR 35
 A Dynamic Lot-Size Problem
 Stage 2 Computations

 If we begin period 2 with zero inventory, we should

produce enough during period 2 to meet the demand for
period 2.

IE318: Applications in OR 36
 A Dynamic Lot-Size Problem
Stage 1 Computations

 If we begin period 1 with zero inventory, we should

produce enough during period 1 to meet the demand for
period 1.
IE318: Applications in OR 37
 A Dynamic Lot-Size Problem
Determination of the Optimal Production Schedule
 Since is attained by producing period 1’s demand, it is
optimal to produce units during period 1; then we begin
period 2 with zero inventory.
 Since is attained by producing period 2’s demand, we
should produce units during period 2; then we enter
period 3 with zero inventory.
 Since is attained by meeting the demands for periods 3
and 4, we produce units during period 3; then we enter
period 5 with zero inventory.
 Finally, we should produce units during period 5.
 The optimal production schedule will incur at total cost
of .
IE318: Applications in OR 38
 Generalized Resource Allocation Problem
 Resource-allocation problems, in which limited resources
must be allocated among several activities, are often
solved by dynamic programming.
 The problem of determining the allocation of resources
that maximizes total benefit subject to the limited
resource availability may be written as
t T
max  rt ( xt )
t 1
t T
s.t.  g ( x ) w
t 1
t t

where must be a member of .

IE318: Applications in OR 39
 Generalized Resource Allocation Problem
 To solve this problem by dynamic programming, define
to be the maximum benefit that can be obtained from
activities if unites of the resource may be allocated to
activities .
 We may generalize the recursions to this situation by
writing

 where must be a non-negative integer satisfying .

 Let be any value of that attains .

IE318: Applications in OR 40
 Knapsack Problem
 Suppose a 10-lb knapsack is to be filled with the items
listed in Table 9. To maximize total benefit, how should
the knapsack be filled?

IE318: Applications in OR 41
 Knapsack Problem
 We have , , , , ,
 Define to be the maximum benefit that can be earned from a
pound knapsack that is filled with items of Type .
 Stage 3 Computations

IE318: Applications in OR 42
 Knapsack Problem
 Stage 2 Computations

IE318: Applications in OR 43
 Knapsack Problem
 Stage 2 Computations (continued)

IE318: Applications in OR 44
 Knapsack Problem
 Stage 2 Computations (continued)

IE318: Applications in OR 45
 Knapsack Problem
 Stage 1 Computations

 Determination of the Optimal Solution to Knapsack Problem

We have and . Hence, we should include one Type 1 item in the
knapsack. Then we have lb left for Type 2 and Type 3 items, so we
should include Type 2 items. Finally, we have lb left for Type 3 items,
and we include Type 3 items. In summary, the maximum benefit that
can be gained from a 10-lb knapsack is . To obtain a benefit of 25,
one Type 1 and two Type 2 items should be included.

IE318: Applications in OR 46