Dynamic programming in continuous time

Jesús Fernández-Villaverde1 and Galo Nuño2

October 15, 2021
1
University of Pennsylvania
2
Banco de España
Basic ideas
Dynamic optimization

• Many (most?) macroeconomic models of interest require the solution of dynamic optimization
problems, both in deterministic and stochastic environments.

• Two time frameworks:

1. Discrete time.

2. Continuous time.

• Three approaches:

1. Calculus of Variations and Lagrangian multipliers on Banach spaces.

2. Hamiltonians.

3. Dynamic Programming.

• We will study dynamic programming in continuous time.

1
Why dynamic programming in continuous time?

• Continuous time methods transform optimal control problems into partial differential equations
(PDEs):

1. The Hamilton-Jacobi-Bellman equation, the Kolmogorov Forward equation, the Black-Scholes

equation,... they are all PDEs.

2. Solving these PDEs turns out to be much simpler than solving the Bellman or the Chapman-Kolmogorov
equations in discrete time. Also, much knowledge of PDEs in natural sciences and applied math.

3. Key role of typical sets in the “curse of dimensionality.”

• Dynamic programming is a convenient framework:

1. It can do everything economists could get from calculus of variations.

2. It is better than Hamiltonians for the stochastic case.

2
The development of “continuous-time methods”

• Differential calculus introduced in the 17th century by Isaac Newton and Gottfried Wilhelm Leibniz.

• In the late 19th century and early 20th century, it was extended to accommodate stochastic processes
(“stochastic calculus”).
• Thorvald N. Thiele (1880): Introduces the idea of Brownian motion.

• Louis Bachelier (1900): Formalizes the Brownian motion and applies to the stock market.

• Albert Einstein (1905): A model of the motion of small particles suspended in a liquid.

• Norbert Wiener (1923): Uses the ideas of measure theory to construct a measure on the path space of
continuous functions.

• Andrey Kolmogorov (1931): Diffusions depend on drift and volatility, Kolmogorov equations.

• Wolfgang Döblin (1938-1940): Modern treatment of diffusions with a change of time.

• Kiyosi Itô (1944): Itô’s Lemma.

3
• Paul Malliavin (1978): Malliavin calculus.
4
The development of “dynamic programming”

• Calculus of variations: Issac Newton (1687), Johann Bernoulli (1696), Leonhard Euler (1733),
Joseph-Louis Lagrange (1755).

• 1930s and 1940s: many problems in aerospace engineering are hard to tackle with calculus of
variations. Example: minimum time interception problems for fighter aircraft.

• Closely related to the Cold War.

• Lev S. Pontryagin, Vladimir G. Boltyanskii, and Revaz V. Gamkrelidze (1956): Maximum principle.

• Magnus R. Hestenes, Rufus P. Isaacs, and Richard E. Bellman at RAND (1950s):

1. Distinction between controls and states.

2. Principle of optimality.

3. Dynamic programming.

5
Figure 2. Figure
The1: mathematicians at Steklov:
Lev S. Pontryagin, Vladimir G. Boltyanskii,Lev Semyonovich
and Revaz V. GamkrelidzePontryagin,
Vladimir Grigor’evich Boltyanskii, and Revaz Valerianovich Gamkrelidze
6
Figure 1. TheFigure
mathematicians at RAND:
2: Magnus R. Hestenes, Rufus P.Magnus
Isaacs, andR. Hestenes,
Richard Rufus P. Isaacs,
E. Bellman
and Richard E. Bellman
7
8
9
10
11
12
Optimal control

• An agent maximizes: Z ∞
max e −ρt u (αt , xt ) dt,
{αt }t≥0 0

subject to:
dxt
= µt (αt , xt ) , x0 = x.
dt

• Here, xt ∈ X ⊂ RN is the state, αt ∈ A ⊂ RM is the control, ρ > 0 is the discount factor,

µ (·) : A × X → RN the drift, and u (·) : A × X → R the instantaneous reward (utility).

13
Hamilton-Jacobi-Bellman

William Hamilton (1805-1865) Carl Jacobi (1804-1851) Richard Bellman (1920-1984)

14
The Hamilton-Jacobi-Bellman equation

• If we define the value function:

Z ∞
V (t, x) = max e −ρ(s−t) u (αs , xs ) ds,
{αs }s≥t t

then, under technical conditions, it satisfies the Hamilton-Jacobi-Bellman (HJB) equation:

( N
)
∂V X ∂V
ρVt (x) = + max u (α, x) + µt,n (x, α) ,
∂t α
n=1
∂xn

with a transversality condition limT →∞ e −ρT VT (x) = 0.

15
HJB: proof

1. Apply the Bellman optimality principle (and limT →∞ e −ρT VT (x) = 0):
Z t  h i
−ρ(s−t0 )
V (t0 , x) ≡ max e u (αs , xs ) ds + e −ρ(t−t0 ) V (t, xt )
{αs }t t0
0 ≤s≤t

2. Take the derivative with respect to t with the Leibniz integral rule and limt→t0 :
"
 #
d e −ρ(t−t0 ) V (t, xt )

−ρ(t−t0 )
0 = lim max e u (αt , x) +
t→t0 αt dt

16
Example: consumption-savings problem

• A household solves: Z ∞
max e −ρt log (ct ) dt,
{ct }t≥0 0

subject to:
dat
= rat + y − ct , a0 = ā,
dt
where r and y are constants.

• The HJB is:  

dV
ρV (a) = max log (c) + (ra + y − c)
c da

• Intuitive interpretation.

17
Example: solution
 
• We guess V (a) = 1 1 r 1 y


ρ log ρ + ρ ρ −1 + ρ log a + r .

• The first-order condition is:

1 dV 1
= =
,
c da ρ a + yr
and hence:  
Human wealth
Financial wealth z}|{
 z}|{ y 
c = ρ a + 
 r 

• Then, we verify the HJB:

  y    y  dV
ρV (a) = log ρ a + + ra + y − ρ a +
r r da

18
The Hamiltonian

• Assume µn (x, α) = µt,n (xn , α) (to simplify matters).

• Define the costates λnt ≡ ∂V

∂xn (xt ) in the HJB.

• Then, the optimal policies are those that maximize the Hamiltonian H (α, x, λ):
H(α,x,λ)
 

 z }| { 

 

 XN 

max u (α, x) + µn (x, α) λnt
α  

 n=1 


 


∂2V ∂ 2 V dxt
• Notice: dλnt
dt = ∂t∂xn + ∂xn2 dt .

19
Pontryagin maximum principle

• Recall the Hamilton-Jacobi-Bellman (HJB) equation:

( N
)
∂V X ∂V
ρVt (x) = + max u (α, x) + µn (x, α)
∂t α
n=1
∂xn

• If we take derivatives with respect to xn in the HJB, we obtain:

∂V ∂2V ∂u ∂µn ∂V ∂2V
ρ = + + + µn (x, α) ,
∂xn ∂t∂xn ∂xn ∂xn ∂xn ∂xn2
dλnt ∂2V ∂ 2 V dxt
which combined with dt = ∂t∂xn + ∂xn2 dt yields:

dλnt ∂H
= ρλnt −
dt ∂xn

• Plus the transversality conditions, limT →∞ e −ρT λnT = 0.

20
Example: now with the maximum principle

• The Hamiltonian H (c, a, λ) = log (c) + λ (ra + y − c).

• The first order condition ∂H

∂c = 0:
1
=λ
c

• The dynamics of the costate dλt

dt = ρλt − ∂H
∂a = (ρ − r ) λt .

• Then, by basic ODE theory:

λt = λ0 e (ρ−r )t ,
and ct = c0 e −(ρ−r )t .

• You need to determine the initial value c0 = ρ a0 + y

r using the budget constraint.

• But how do you take care of the filtration in the stochastic case?
21
Stochastic calculus
Brownian motion

• Large class of stochastic processes.

• But stochastic calculus starts with the Brownian motion.

• A stochastic process W is a Brownian motion (a.k.a. Wiener process) if:

1. W (0) = 0.

2. If r < s < t < u : W (u) − W (t) and W (s) − W (r ) are independent random variables.

3. For s < t : W (t) − W (s) ∼ N (0, t − s).

4. W has continuous trajectories.

22
Simulated paths

• Notice how E [W (t)] = 0 and Var [W (t)] = t.

60

40

20

-20

-40

-60
0 100 200 300 400 500 600 700 800 900 1000 23
Why do we need a new concept of integral?

• We will deal with objects such as the expected value function.

• But the value function is now a stochastic function because it depends on stochastic processes.

• How do we think about that expectation?

• More importantly, we need to deal with diffusions, which will include an integral.

• We cannot apply standard rules of calculus: Almost surely, a Brownian motion is nowhere
differentiable (even though it is everywhere continuous!).

• Brownian motion exhibits self-similarity (if you know what this means, the Hurst parameter of a
Brownian motion is H = 12 > 0).

• We need an appropriate concept of integral: Itô stochastic integral.

24
 Chebyshev polynomial of order 11 Chebyshev polynomial of order 11
1 0

-0.1

-0.2
0.5
-0.3

-0.4
0
-0.5

-0.6
-0.5
-0.7

-0.8

-1 -0.9
0 0.2 0.4 0.6 0.8 1 0 0.02 0.04 0.06 0.08 0.1

Chebyshev polynomial of order 11 Chebyshev polynomial of order 11

0 0

-0.02 -0.002

-0.04 -0.004

-0.06 -0.006

-0.08 -0.008

-0.1 -0.01

-0.12 -0.012
0 0.002 0.004 0.006 0.008 0.01 0 0.2 0.4 0.6 0.8 1
10 -3 25
 Brownian motion Brownian motion
1.014 1.0005

1.012
1
1.01

1.008 0.9995

1.006
0.999
1.004

1.002 0.9985

1
0.998
0.998

0.996 0.9975
0 0.2 0.4 0.6 0.8 1 0 0.02 0.04 0.06 0.08 0.1

Brownian motion Brownian motion

1.0006 1.00005

1.0004 1

1.0002 0.99995

1 0.9999

0.9998 0.99985

0.9996 0.9998

0.9994 0.99975

0.9992 0.9997
0 0.002 0.004 0.006 0.008 0.01 0 0.2 0.4 0.6 0.8 1
10 -3 26
The stochastic integral

• Recall that the Riemann-Stieltjes integral of a (deterministic) function g (t) with respect to the
(deterministic) function w (t) is:
Z t n−1
X
g (s)dw (s) = lim g (tk ) [w (tk+1 ) − w (tk )] ,
0 n→∞
k=0
where t0 = 0 and tn = t.

• We want to generalize the Riemann-Stieltjes integral to an stochastic environment.

• Given a stochastic process g (t), the stochastic integral with respect to the Brownian motion W (t) is:
Z t n−1
X
g (s)dW (s) = lim g (tk ) [W (tk+1 ) − W (tk )] ,
0 n→∞
k=0
where t0 = 0 and tn = t and the limit converges in probability.

• Notice: both the integrand and the integrator are stochastic processes and that the integral is a
random variable.
27
Mean of the stochastic integral

" n−1
#
Z t  X
E g (s)dW (s) = E lim g (tk ) [W (tk+1 ) − W (tk )]
0 n→∞
k=0
n−1
X
= lim g (tk )E [W (tk+1 ) − W (tk )]
n→∞
k=0
n−1
X
= lim g (tk ) · 0 = 0
n→∞
k=0

28
Variance of the stochastic integral

"Z
t 2 # " n−1
X
#
E g (s)dW (s) = Var lim g (tk ) [W (tk+1 ) − W (tk )]
0 n→∞
k=0
n−1
X
= lim g 2 (tk )Var [W (tk+1 ) − W (tk )]
n→∞
k=0
n−1
X Z t
= lim g 2 (tk ) (tk+1 − tk ) = g 2 (s)ds
n→∞ 0
k=0

29
Differentiation in stochastic calculus

• In an analogous way that we can define a stochastic integral, we can define a new idea derivative
with respect to Brownian motion.

• Malliavin derivative.

• Applications in finance.

• However, in this course, we will not need to use it.

30
31
Stochastic differential equations (SDEs)

• We define a stochastic differential equation (diffusion) as

dX (t) = µ (t, X (t)) dt + σ (t, X (t)) dW (t), X (0) = x,

as a short-cut to express:
Z t Z t
X (t) = x + µ (t, X (s)) ds + σ (s, X (t)) dW (s)
0 0

• µ (·) is the drift and σ (t, X (t)) the volatility.

• Any stochastic process (without jumps) can be approximated by a diffusion.

32
Example I: Brownian motion with drift

• Simplest example (random walk with drift in discrete time)

dX (t) = µdt + σdW (t), X (0) = x0 ,

where:
Z t Z t
X (t) = x0 + µds + σdW (s)
0 0
= x0 + µt + σW (t)

• Then X (t) ∼ N x + µt, σ 2 t . This is not stationary.

• Equivalent to a random walk with drift in discrete time.

33
Example II: Ornstein-Uhlenbeck process

• Continuous-time counterpart of an AR(1) in discrete time:

dX (t) = θ X − X (t) dt + σdW (t), X (0) = x0

• Named after Leonard Ornstein and George Eugene Uhlenbeck, although in economics and finance is
a.k.a. the Vašı́ček model of interest rates (Vašı́ček, 1977).

• Stationary process with mean reversion:

E [X (t)] = x0 e −θt + X 1 − e −θt

and
σ2
1 − e −2θt


Var [X (t)] =
2θ

• Take the limits as t → ∞!

34
Euler–Maruyama method

• Except in a few cases (such as the ones before), we do not know how to get an analytic solution for a
SDE.

• How do we get an approximate numerical solution of a SDE?

• Euler–Maruyama method: Extension of the Euler method for ODEs.

• Given a SDE:
dX (t) = µ (t, X (t)) dt + σ (t, X (t)) dW (t), X (0) = x,
it can be approximated by:

X (t + ∆t) − X (t) = µ (t, X (t)) ∆t + σ (t, X (t)) ∆W (t),

iid
where ∆W (t) ∼ N (0, ∆t).

35
Euler–Maruyama method (Proof)

• If we integrate the SDE:

Z t+∆t Z t+∆t
X (t + ∆t) − X (t) = µ (t, X (s)) ds + σ (s, X (t)) dW (s)
t t
≈ µ (t, X (t)) ∆t + σ (t, X (t)) (W (t + ∆t) − W (t))
iid
where W (t + ∆t) − W (t) = ∆W (t) ∼ N (0, ∆t).

• The smaller the ∆t, the better the method will work.

• Let us look at some code.

36
Stochastic calculus

• Now, we need to learn how to manipulate SDEs.

• Stochastic calculus = “normal” calculus + simple rules:

2 2
(dt) = 0, dt · dW = 0, (dW ) = dt

• The last rule is the key. It comes from:

E W (t)2 = Var [W (t)] = t,

 

and:
2
Var W (t)2 = E W (t)4 − E W (t)2 = 2t 2  t
    
| {z }
3t 2

37
Functions of stochastic processes: Itô’s formula

• Chain rule in standard calculus. Given f (t, x) and x(t):

df ∂f ∂f dx ∂f ∂f
= + =⇒ df = dt + dx
dt ∂t ∂x dt ∂t ∂x

• Chain rule in stochastic calculus (Itô’s lemma). Given f (t, X ) and:

dX (t) = µ (t, X (t)) dt + σ (t, X (t)) dW (t),

we get:
1 ∂2f 2
 
∂f ∂f ∂f
df = + µ+ 2
σ dt+ σdW
∂t ∂x 2 ∂x ∂x

38
Itô’s formula: proof

• Taylor expansion of f (t, X ):

∂f ∂f 1 ∂2f 2 1 ∂2f 2 ∂2f
df = dt + dX + (dX ) + (dt) + dt · dX
∂t ∂x 2 ∂x 2 2 ∂t 2 ∂t∂X

• Given the rules:

dX = µdt + σdW ,
2 2 2
(dX ) = µ2 (dt) + σ 2 (dW ) + 2µσdtdW = σ 2 dt,
2
(dt) = 0,
2
dt · dX = µ (dt) + σdtdW = 0.

• Then:
1 ∂2f 2
 
∂f ∂f ∂f
df = + µ+ σ dt + σdW
∂t ∂x 2 ∂x 2 ∂x
39
Multidimensional Itô’s formula

• Given f (t, x1 , x2 , ..., xn ), and

dXi (t) = µi (t, X1 (t), ..., Xn (t)) dt + σi (t, X1 (t), ..., Xn (t)) dWi (t),
i = 1, ..., n,

then
n n n
∂f X ∂f 1 X X ∂2f
df = dt + dXi + dXi dXj
∂t ∂xi 2 ∂xi ∂xj
i=1 i=1 j=1

40
Application: Geometric Brownian motion (GBM)

• Basic model for asset prices (non-negative):

dX (t) = µX (t)dt + σX (t)dW (t), X (0) = x0 ,

where Z t Z t
X (t) = x0 + µX (s)ds + σX (s)dW (s)
0 0

• How can we solve it?

41
GBM solution using Itôs formula

• Define Z (t) = ln (X (t)) , the Itô’s formula gives us:

1 ∂ 2 ln (x) 2 2
 
∂ ln (x) ∂ ln (x)
dZ (t) = + µx + σ x dt
∂t ∂x 2 ∂x 2
∂ ln (x)
+ σxdW
 ∂x 
1 1 1 2 2 1
= 0 + µx − 2
σ x dt + σxdW
x 2x x
 
1 2
= µ − σ dt + σdW
2
 
1 2
=⇒ Z (t) = ln (x0 ) + µ − σ t + σW (t)
2

• Therefore:
1 2
X (t) = x0 e (µ− 2 σ )t+σW (t)

42
Dynamic programming with
stochastic processes
The problem

• An agent maximizes: Z ∞
V0 (x) = max E0 e −ρt u (αt , Xt ) dt,
{αt }t≥0 0

subject to:
dXt = µt (Xt , αt ) dt + σt (Xt , αt ) dWt , X0 = x.

• σ (·) : A × X → RN .

• We consider feedback control laws αt = αt (Xt ) (no improvement possible if they depend on the
filtration Ft ).

43
HJB equation with SDEs

N N
∂2V
 
∂V X ∂V 1 X
σt2 (x, α) n1 ,n2


ρVt (x) = + max u (α, x) + µt,n (x, α) + ,
∂t α
n=1
∂xn 2 n ,n =1 ∂xn1 ∂xn2
1 2

where σt2 (x, α) = σt (x, α) σt> (x, α) ∈ RN×N is the variance-covariance matrix.

44
HJB with SDEs: proof

1. Apply the Bellman optimality principle:

Z t  h i
Vt0 (x) = max Et0 e −ρ(s−t0 ) u (αs , Xs ) ds + Et0 e −ρ(t−t0 ) Vt (Xt )
{αs }t t0
0 ≤s≤t

2. Take the derivative with respect to t, apply Itô’s formula and take limt→t0 :
"
 #
Et0 d e −ρ(t−t0 ) V (t, Xt )

−ρ(t−t0 )
0 = lim max e u (αt , x) +
t→t0 αt dt

Notice:
σ2 ∂ 2 V
   
h  i ∂V ∂V
Et0 d e −ρ(t−t0 ) V (t, Xt ) = Et0 e −ρ(t−t0 ) −ρV + +µ + dt
∂t ∂x 2 ∂x 2
 
∂V
+e −ρ(t−t0 ) Et0 σ dWt
∂x
| {z }
0

45
The infinitesimal generator

• The HJB can be compactly written as:

∂V
ρV = + max {u (α, x) + AV } ,
∂t α

where A is the infinitesimal generator of the stochastic process Xt , defined as:

E0 [f (Xt )] − f (x)
Af = lim
t↓0 t
N N
X ∂f 1 X ∂2f
σ 2 n1 ,n2


= µn +
n=1
∂xt,n 2 n ,n =1 ∂xn1 ∂xn2
1 2

• Intuitively: the infinitesimal generator describes the movement of the process in an infinitesimal time
interval.

46
Boundary conditions

• The boundary conditions of the HJB equation are not free to be chosen, they are imposed by the
dynamics of the state at the boundary ∂X.

• Only three possibilities:

dV
1. Reflection barrier: The process is reflected at the boundary: dx
= 0.
∂X

2. Absorbing barrier: The state jumps at a different point y when the barrier is reached: V (x) = V (y ).
∂X

3. State constraint: The policy αt guarantees that the process does not abandon the boundary.

47
Example: Merton portfolio model

• An agent maximize its discounted utility:

Z ∞
V (x) = max E e −ρt log (ct ) dt,
{ct ,∆t }t≥0 0

by investing in ∆t shares of a stock (GBM) and saving the rest in a bond with return r :
dSt = µSt dt + σSt dWt
dBt = rBt dt

• The value of the portfolio evolves according to:

dXt = ∆t dSt + r (Xt − ∆t St ) dt − ct dt
= ∆t (µSt dt + σSt dWt ) + r (Xt − ∆t St ) dt − ct dt
= [rXt + ∆t St (µ − r )] dt + ∆t σSt dWt − ct dt

48
Merton model: The HJB equation

• We redefine one of the controls:

∆t St
ωt =
Xt

• The HJB results in:

σ2 ω2 x 2 ∂ 2 V
 
∂V
ρV (x) = max log (c) + [rx + ωx (µ − r ) − c] +
c,ω ∂x 2 ∂x 2

• The FOC are:

1 ∂V
− = 0,
c ∂x
∂V ∂2V
x (µ − r ) + ωσ 2 x 2 2 = 0
∂x ∂x

49
Solution to Merton portfolio model

• Guess and verify:

1
V (x) = log (x) + κ2 ,
ρ
∂V 1
= ,
∂x ρx
∂2V 1
= − 2
∂x 2 ρx
with κ1 and κ2 constants.

• The FOC are:

1 1
− = 0 =⇒ c = ρx,
c ρx
κ1 κ1 (µ − r )
x (µ − r ) − ωσ 2 x 2 2 = 0 =⇒ ω =
x x σ2

50
The case with Poisson processes

• The HJB can also be solved for the case of Poisson shocks.

• The state is now:

dXt = µ (Xt , αt , Zt ) dt, X0 = x, Z0 = z0 .

• Zt is a two-state continuous-time Markov chain Zt ∈ {z1 , z2 }. The process jumps from state 1 to
state 2 with intensity λ1 and vice-versa with intensity λ2 .

• The HJB in this case is

 
∂Vi ∂Vi
ρVti (x) = + max u (α, x) + µ (x, α, zi ) + λi (Vj − Vi ),
∂t α ∂x
i, j = 1, 2, i 6= j, where Vi (x) ≡ V (x, zi ).

• We can have jump-diffusion processes (Lévy processes): HJB includes the two terms (volatility and
jumps).
51
Viscosity solutions

• Relevant notion of “solutions” to HJB introduced by Pierre-Louis Lions and Michael G. Crandall in
1983 in the context of PDEs.

• Classical solution of a PDE (to be defined below) are too restrictive.

• We want a weaker class of solutions than classical solutions.

• More concretely, we want to allow for points of non-differentiability of the value function.

• Similarly, we want to allow for convex kinks in the value function.

• Different classes of “weaker solutions.”

52
What is a viscosity solution?

• There are different concepts of what a “solution” to a PDE F x, Dw (x), D 2 w (x) = 0, x ∈ X is:

1. “Classical” (Strong) solutions. There is a smooth function u ∈ C 2 (X ) ∩ C (X̄ ) such that

F x, Du(x), D 2 u(x) = 0, x ∈ X .

• Hard to find for HJBs.

2. Weak solutions. There is a function u ∈ H 1 (X ) (Sobolev space) such that for any function φ ∈ H 1 (X ),
then X F x, Du(x), D 2 u(x) φ(x)dx = 0, x ∈ X .
R

• Problem with uniqueness in HJBs.

3. Viscosity solutions. There is a locally bounded u that is both a subsolution and a supersolution of
F x, Dw (x), D 2 w (x) = 0, x ∈ X .

• If it exists, it is unique.

53
Subsolutions and supersolutions

• An upper semicontinuous function u in X is a “subsolution” if for any point x0 ∈ X and any C 2

function φ ∈ C 2 (X ) such that φ(x0 ) = u(x0 ) and φ≥u in a neighborhood of x0 , we have:

F (x0 , φ(x0 ), Dφ(x0 ), D 2 φ(x0 ))≤0

• An upper semicontinuous function u in X is a “supersolution” if for any point x0 ∈ X and any C 2

function φ ∈ C 2 (X ) such that φ(x0 ) = u(x0 ) and φ≤u in a neighborhood of x0 , we have:

F (x0 , φ(x0 ), Dφ(x0 ), D 2 φ(x0 ))≥0

54
More on viscosity solutions

• Viscosity solution is unique.

• A baby example: consider the boundary value problem F (u 0 ) = |u 0 | − 1 = 0, on (−1, 1) with

boundary conditions u(−1) = u(1) = 0. The unique viscosity solution is the function u(x) = 1 − |x|.

• Coincides with solution to sequence problem of optimization.

• Numerical methods designed to find viscosity solutions.

• Check, for more background, User’s Guide to Viscosity Solutions of Second Order Partial Differential
Equations by Michael G. Crandall, Hitoshi Ishii, and Pierre-louis Lions.

• Also, Controlled Markov Processes and Viscosity Solutions by Wendell H. Fleming and Halil Mete
Soner.
55
Finite difference method
Solving dynamic programming problems = solving PDEs

• We want to numerically solve the Hamilton-Jacobi-Bellman (HJB) equation:

 N
∂Vi X ∂Vi
ρVti (x) = + max u (α, x) + µnt (x, α, zi )
∂t α
n=1
∂xn
+ λi (Vj − Vi )
N
∂ 2 Vi

1 X 2


+ σ (x, α) n ,n ,
2 n ,n =1 t 1 2 ∂x ∂x
1 n2
1 2

with a transversality condition limT →∞ e −ρT VT (x) = 0, and some boundary conditions defined by
the dynamics of Xt .

56
Overview of methods to solve PDEs

1. Perturbation: consider a Taylor expansion of order n to solve the PDEs around the deterministic
steady state (not covered here, similar to discrete time).

2. Finite difference: approximate derivatives by differences.

3. Projection (Galerkin): project the value function over a subspace of functions (non-linear version
covered later in the course).

4. Semi-Lagragian. Transform it into a discrete-time problem (not covered here, well known to
economists)

57
A (limited) comparison from Parra-Álvarez (2018)

58
Numerical advantages of continuous-time methods: Preview

1. “Static” first order conditions. Optimal policies only depend on the current value function:
N N
∂u X ∂µn ∂V 1 X ∂ ∂2V
σt2 (x, α) n1 ,n2


+ + =0
∂α n=1 ∂α ∂xn 2 n ,n =1 ∂α ∂xn1 ∂xn2
1 2

2. Borrowing constraints only show up in boundary conditions as state constraints.

• FOCs always hold with equality.

3. No need to compute expectations numerically.

• Thanks to Itô’s formula.

4. Convenient way to deal with optimal stopping and impulse control problems (more on this later
today).

5. Sparsity (with finite differences).

59
Our benchmark: consumption-savings with incomplete markets

• An agent maximizes: Z ∞ 
−ρt
max E0 e u(ct )dt ,
{ct }t≥0 0

subject to:
dat = (zt + rat − ct ) dt, a0 = ā
where zt ∈ {z1 , z2 } is a Markov chain with intensities z1 → z2 : λ1 and z2 → z1 : λ2 .

• Exogenous borrowing limit:

at ≥ −φ

60
The Hamilton-Jacobi-Bellman equation

• The value function in this problem:

Z ∞ 
−ρt
vi (a) = max E0 e u(ct )ds|a0 = a, z0 = zi
{ct }t≥0 0

must satisfy the HJB equation:

ρvi (a) = max {u(c) + si (a) vi0 (a)} + λi (vj (a) − vi (a)) ,
c

where si (a) is the drift,

si (a) = zi + ra − c (a) , i = 1, 2

• The first-order condition is:

u 0 (ci (a)) = vi0 (a)

61
How can we solve it?

• The model proposed above does not yield an analytical solution.

• Therefore we resort to numerical techniques in order to find a solution.

• TIn particular, we employ an upwind finite difference scheme (Achdou et al., 2017).

• This scheme converges to the viscosity solution of the problem.

62
Grid

• We approximate the value function v (a) on a finite grid with step ∆a : a ∈ {a1 , ..., aJ }, where

aj = aj−1 + ∆a = a1 + (j − 1) ∆a

for 2 ≤ j ≤ J. The bounds are a1 = −φ and aJ = a∗ .

• We use the notation vj ≡ v (aj ), j = 1, ..., J.

63
Finite differences

• v 0 (aj ) can be approximated with a forward (F ) or a backward (B) approximation,

vi,j+1 − vi,j
vi0 (aj ) ≈ ∂F vi,j ≡
∆a
vi,j − vi,j−1
vi0 (aj ) ≈ ∂B vi,j ≡
∆a

64
Forward and backward approximations

65
Upwind scheme

−1
• The choice of ∂F vi,j or ∂B vi,j depends on the sign of the drift function si (a) = zi + ra − (u 0 ) (vi0 (a)):
−1 −1
1. If siF (aj ) ≡ zi + raj − (u 0 ) (∂F vi,j ) > 0 −→ ci,j = (u 0 ) (∂F vi,j ).
−1 −1
2. Else, if siB (aj ) ≡ zi + raj − (u 0 ) (∂B vi,j ) < 0 −→ ci,j = (u 0 ) (∂B vi,j ).

3. Otherwise, si (a) = 0 −→ ci,j = zi + raj .

• Why? Key for stability.

66
HJB approximation, I

• Let superscript n denote the iteration counter.

• The HJB equation is approximated by:

n+1 n
vi,j − vi,j n+1 n n n+1
+ ρvi,j = u(ci,j ) + si,j,F 1si,j,F
n >0 ∂F vi,j
∆
n n+1
+si,j,B 1si,j,B
n <0 ∂B vi,j
 
n+1 n+1
+λi v−i,j − vi,j ,

for j = 1, ..., J, where 1 (·) is the indicator function and:

−1
n
(zi + raj ) − (u 0 ) n


si,j,F = ∂F vi.j
−1
n
(zi + raj ) − (u 0 ) n


si,j,B = ∂B vi.j

67
HJB approximation, II

• Collecting terms, we obtain:

n+1 n
vi,j − vi,j n+1 n n+1 n n+1 n n+1 n n+1
+ ρvi,j = u(ci,j ) + vi,j−1 xi,j + vi,j yi,j + vi,j+1 zi,j + v−i,j λi ,
∆
where:
n
n
si,j,B 1si,j,B
n <0
xi,j ≡ − ,
∆a
n n
n
si,j,F 1si,j,F
n >0 si,j,B 1si,j,B
n <0
yi,j ≡ − + − λi ,
∆a ∆a
n
n
si,j,F 1si,j,F
n >0
zi,j ≡
∆a

68
Boundary conditions

• State constraint a ≥ 0 −→ si,1,B

n n
= 0 −→ xi,1 = 0.

• State constraint a ≤ a∗ −→ si,J,F

n n
= 0 −→ zi,J = 0.

69
Matrix notation

• The HJB is a system of 2J linear equations which can be written in matrix notation as:
1
v n+1 − v n + ρv n+1 = u n + An v n+1


∆

• This is equivalent to a discrete-time, discrete-space dynamic programming problem 1

∆ =0 :

v = u + βΠv ,
1
where Π = I + (1−ρ) A and β = (1 − ρ) .

70
Matrix A

• Matrix A is the discrete-space approximation of the infinitesimal generator A.

• Advantage: this is a sparse matrix.

n n
 
y1,1 z1,1 0 ··· λ1 0 0 ··· 0
n n n

 x1,2 y1,2 z1,2 ··· 0 λ1 0 ··· 0 

n n n
0 x1,3 y1,3 z1,3 ··· 0 λ1 ··· 0
 
 
 .. .. .. .. .. .. .. .. .. 
. . . . . . . . .
 
 
..
 
. n n
 
n
A =
 0 x1,J y1,J 0 0 0 0 λ1  
n n

 λ2 0 0 ··· y2,1 z2,1 0 ··· 0  
n n n
0 λ2 0 ··· x2,2 y2,2 z2,2 0 ··· 
 

n n n
λ2 ···
 
 0 0 0 x2,3 y2,3 z2,3 0 
.. .. ..
 
 .. .. .. .. .. .. 
 . . . . . . . . . 
n n
0 0 0 λ2 0 ··· 0 x2,J y2,J
71
How to solve it

 n
  n+1 
u(c1,1 ) v1,1
 ..   .. 

 . 


 . 
 u(c n )  n+1 
 v1,J 
• Given u n =  1,J
 , v n+1 =  n+1 ,
  
n
 u(c2,1 )   v2,1 
.. .. 
   
  
 .   . 
n n+1
u(c2,J ) v2,J
the system can in turn be written as:
 
1 1 n
B n v n+1 = b n , Bn = + ρ I − An , bn = u n + v
∆ ∆

72
The algorithm

0 u(zi +raj )
1. Begin with an initial guess vi,j = ρ .

2. Set n = 0.

3. Then:
n n n
3.1 Policy update: Compute ∂F vi.j , ∂B vi.j , and ci,j .
n+1
3.2 Value update: Compute vi,j solving the linear system of equations.
n+1 n
3.3 Check: If vi,j is close enough to vi,j , stop. If not, set n := n + 1 and go to 1.

73
Results

4 1.15

3
1.1

2
1.05

0.95
-1

0.9
-2

-3 0.85
-2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4

74
The case with diffusions

• Assume now that labor productivity evolves according to a Ornstein–Uhlenbeck process:

dzt = θ(ẑ − zt )dt + σdBt ,

on a bounded interval [z, z̄] with z≥ 0, where Bt is a Brownian motion.

¯ ¯
• The HJB is now:
∂V ∂V σ2 ∂ 2 V
ρV (a, z) = max u(c) + s (a, z, c) + θ(ẑ − z) +
c≥0 ∂a ∂z 2 ∂z 2

75
The new grid

• We approximate the value function V (a, z) on a finite grid with steps ∆a and ∆z : a ∈ {a1 , ..., aI },
z ∈ {z1 , ..., zJ }.

• We use now the notation Vi,j := V (ai , zj ), i = 1, ..., I ; j = 1, ..., J.

• It does not matter if we consider forward or backward for the first derivative with respect to the
exogenous state.

• Use central for the second derivative:

∂V (ai , zj ) Vi,j+1 − Vi,j
≈ ∂z Vi,j := ,
∂z ∆z
∂ 2 V (ai , zj ) Vi,j+1 + Vi,j−1 − 2Vi,j
≈ ∂zz Vi,j := 2
∂z 2 (∆z)

76
HJB approximation

n+1 n
Vi,j − Vi,j n+1 n n+1 n+1 n+1 n+1 n+1
+ ρVi,j = u(ci,j ) + Vi−1,j %i,j + Vi,j βi,j + Vi+1,j χi,j + Vi,j−1 ξ + Vi,j+1 ςj ,
∆

n
si,j,B 1si,j,B
n <0
%i,j = − ,
∆a
n n
si,j,F 1si,j,F
n >0 si,j,B 1si,j,B
n <0 θ(ẑ − zj ) σ2
βi,j = − + − − 2,
∆a ∆a ∆z (∆z)
n
si,j,F 1si,j,F
n >0
χi,j = ,
∆a
2
σ
ξ = 2,
2 (∆z)
σ2 θ(ẑ − zj )
ςj = 2 +
2 (∆z) ∆z
77
Boundary conditions

• The boundary conditions with respect to z are:

∂V (a, z) ∂V (a, z̄)

¯ = = 0,
∂z ∂z
as the process is reflected.

• At the boundaries in the j dimension, the HJB becomes:

n+1 n
Vi,j − Vi,j n+1 n n+1 n+1 n+1 n+1
+ ρVi,j = u(ci,1 ) + Vi−1,j %i,1 + Vi,1 (βi,1 + ξ) + Vi+1,1 χi,1 + Vi,2 ς1 ,
∆
n+1 n
Vi,j − Vi,j n+1 n n+1 n+1 n+1 n+1
+ ρVi,j = u(ci,J ) + Vi−1,J %i,J + Vi,J (βi,J + ςJ ) + Vi+1,J χi,J + Vi,J−1 ξJ
∆

78
The problem

• In matrix notation as:

V n+1 − V n
+ ρV n+1 = u n + An V n+1 ,
∆
where (sparsity again):
β1,1 + ξ χ1,1 ··· ς1 ···
 
0 0 0 0 0
 %2,1
 β 2,1 + ξ χ2,1 0 ··· 0 ς1 0 ··· 0 

0 %3,1 β3,1 + ξ χ3,1 0 ··· 0 ς1 ··· 0
 
 
 .. .. .. .. ..

. . . . . 0 0 0 ··· 0
 
 
 
0 0 0 %I ,1 βI ,1 + ξ χI ,1 0 0 ··· 0
An = 
 


 ξ 0 ··· 0 %1,2 β1,2 χ1,2 0 ··· 0 

0 ξ ··· 0 0 %2,2 β2,2 χ2,2 ··· 0
 
 
 .. .. .. .. .. .. .. .. ..

. . . . . . . . . 0
 
 
 
 0 0 0 0 ··· 0 0 %I −1,J βI −1,J + ςJ χI −1,J 
0 0 0 0 ··· 0 0 0 %I ,J βI ,I + ςJ
79
Results

80
Why does the finite difference method work?

• The finite difference method converges to the viscosity solution of the HJB as long as it satisfies
three properties:

1. Monotonicity.

2. Stability.

3. Consistency.

• The proposed method does satisfy them (proof too long, check Fleming and Soner, 2006).

81