10.

34: Numerical Methods

Applied to
Chemical Engineering
Lecture 11:
Unconstrained Optimization
Newton-Raphson and trust region methods

1
 Recap

• Optimization

• Steepest descent

2
 Recap
• Method of steepest decent: xi+1 = xi ↵i g(xi )
• Estimating an optimal ↵i with a Taylor expansion:

1 2
f (xi+1 ) = f (xi ) ↵i g(xi ) g(xi ) + ↵i g(xi )T H(xi )g(xi ) + . . .
T
2

• This is quadratic in ↵i , so find the critical point:

1

0.8

0.6

0.4

0.2
g(xi )T g(xi )
x2 0 ↵i =
−0.2
g(xi )T H(xi )g(xi )
−0.4

−0.6

−0.8

−1
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
3
x1
 Recap

mẍ = ẋ + F

mẍ = ẋ rU

4
 Recap
• Method of steepest decent: xi+1 = xi ↵i g(xi )
3

-3

-2

2
1
-1
2

1 -2 -1

-3

x2 0 -5
x2

-2
-3

-2

2
1
-4
-4
-3

0
-2
-1
-1

-2
0
-1
-2
-3

-3 2
-3 -2 -1 0 1 2 3
x1
x1 5
 Recap
• Method of steepest decent: xi+1 = xi ↵i g(xi )
3

-3

-2

2
1
-1
2

1 -2 -1

-3

x2 0 -5
x2

-2
-3

-2

2
1
-4
-4
-3

0
-2
-1
-1

-2
0
-1
-2
-3

-3 2
-3 -2 -1 0 1 2 3
x1
x1 6
Unconstrained Optimization
• Conjugate gradient method:
1 T
• Consider the minimization of: f (x) = x Ax
2
T
b x
g(x) = rf (x) = Ax b
H(x) = A

• This has a minimum when?

• Ax = b
• the Hessian, A, is symmetric, positive definite

• Iterative method: xi+1 = xi + ↵i pi

• pi is a descent dir. but not necessarily the steepest

• Let’s determine the optimal ↵i for a given pi

7
 Unconstrained Optimization
• Conjugate gradient method
1 2 T
T
f (xi+1 ) = f (xi ) + ↵i g(xi ) pi + ↵i pi Ap
2
• f (xi+1 ) is quadratic in ↵i . g(xi )T pi
• f (xi+1 ) is minimized when ↵i = pTi Api

• For a given direction pi there is an optimal step size ↵i

• How can we choose the optimal direction?

• f (xi+1 ) is already minimized along pi : g(xi+1 ) T

pi = 0
• Can this hold for f (xi+2 ) also?
• Let g(xi+2 ) pi = 0 , then:
T

T
[A (xi+1 + ↵i+1 pi+1 ) b] pi = 0 ) pTi+1 Api = 0 8
Unconstrained Optimization
• Conjugate gradient method:
1 2 T
T
f (xi+1 ) = f (xi ) + ↵i g(xi ) pi + ↵i pi Ap
2
• f (xi+1 ) is quadratic in ↵i . g(xi )T pi
• f (xi+1 ) is minimized when ↵i = pTi Api

• For a given direction pi there is an optimal step size ↵i

• How can we choose the optimal direction?

• f (xi+1 ) is already minimized along pi : g(xi+1 ) pi = 0

T

• Can this hold for f (xi+2 ) also?

• Let g(xi+2 ) pi = 0 , then: g(xi+1 )T Api
T

pi+1 = g(xi+1 ) + i+1 pi , i+1 =

pTi Api 9
 Unconstrained Optimization
• Method of steepest decent/conjugate gradient:
✓ ◆
• Example: f (x) = x21 + 10x22 A=
1
0
0
10
,b = 0

• Contours for the function: ↵i = 0.015 in SD

10

x2 0

−2

−4

−6

−8

−10
−10 −8 −6 −4 −2 0 2 4 6 8 10
10
x1
Unconstrained Optimization

• Conjugate gradient method:

• Used to solve linear equations with O(N ) iterations

• Requires only the ability to compute the product:

• The actual matrix is never needed. We only need

to compute its action on different vectors, Ay !

• Only for symmetric, positive definite matrices.

• More sophisticated minimization methods exist for

arbitrary matrices.

• Optimization applied to linear equations is the state-of-

the-art for solutions of linear equations.

11
 Newton-Raphson
• Finding local minima in unconstrained optimization
problems involve solutions of the equation:
g(x) = rf (x) = 0
• at minima in f (x)

• If we begin close enough to a minimum, can we expect

the NR method to converge to that minimum?

• Yes! NR is locally convergent.

• Accuracy of the iterates will improve quadratically!

• Newton-Raphson iteration:

• What is the Jacobian of g(x)?

12
 Unconstrained Optimization
• Method of steepest decent/Newton-Raphson:

• Example: f (x) = x21 + 10x22

• Contours for the function: ↵i = 0.015
10

x2 0

−2

−4

−6

−8

−10
−10 −8 −6 −4 −2 0 2 4 6 8 10

13
x1
 Unconstrained Optimization
• Method of steepest decent/Newton-Raphson:

• Example: log f (x) = 2

x1 + 2
10x2
g(xi ) g(xi ) T
• Contours for the function: ↵i =
g(xi )T H(xi )g(xi )
1

0.8

0.6

0.4

0.2

x2 0

−0.2

−0.4

−0.6

−0.8

−1
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1

14
x1
 Newton-Raphson
• Compare:

• Optimized steepest decent:

g(xi )T g(xi )
• xi+1 = xi ↵i g(xi ) with ↵i =
g(xi )T H(xi )g(xi )
• Newton-Raphson:

• xi+1 = xi H(xi ) 1
g(xi )
• What is the difference?

• What are the strengths of Newton-Raphson?

• What are the weaknesses of Newton-Raphson?

• What are the strengths of steepest descent?

• What are the weaknesses of steepest decent?

15
 Trust-Region Methods
• Both Newton-Raphson and the optimized steepest
descent methods assume the objective function can be
described locally by a quadratic function.

xi

xi+1

• That quadratic approximation may be good or bad

16
 Trust-Region Methods
• Both Newton-Raphson and the optimized steepest
descent methods assume the objective function can be
described locally by a quadratic function.

xi

xi+1

• That quadratic approximation may be good or bad

17
 Trust-Region Methods
• Trust region methods choose between the Newton-
Raphson direction when the quadratic approximation is
good and the steepest decent direction when it is not.
Newton-Raphson
Steepest decent Ri

• This choice is based on whether the Newton-Raphson

step is too large.

18
 Trust-Region Methods
Newton-Raphson
• Newton step: diNR 1
= H(xi ) g(xi ) Steepest decent
• SD
Steepest decent: di = ↵i g(xi )
Ri
• NR NR
If kdi k2 < Ri and f (xi + di ) < f (xi )

• Take the Newton-Raphson step

• Else
• Take a step in the steepest descent direction
• If kdi k2 < Ri and f (xi + di ) < f (xi ) with
SD SD

optimal step size

Ri
• Take the optimal steepest descent

• Else step to the trust boundary using:

↵i = Ri /kg(xi )k2
19
 Trust-Region Methods
• The size of the trust region can be set arbitrarily initially.

• The trust region grows or shrinks depending on which of

the two steps we choose.

• If the Newton-Raphson step was chosen:

• The quadratic approximation has minimum value:

1 T
T
= f (xi ) + g(xi ) di + di H(xi )di
2
• GROW the trust-radius when > f (xi + di ),
because the function was smaller than predicted

• otherwise, SHRINK the trust-radius.

• If the steepest descent step was chosen, keep the trust

radius the same.
20
 Trust-Region Methods
• What is a good value of the trust-region radius?

• MATLAB uses one initially!

• Variations on the trust-region method exist as well.

• MATLAB uses the dog-leg step instead of the optimal

steepest descent step:
Newton-Raphson
Optimal steepest decent
Dog-leg step

Ri

21
 Unconstrained Optimization
• Method of steepest decent/Newton-Raphson/Trust-Region:

• Example: log f (x) = 2

(x1 + 2 2
10x2 )
g(xi ) g(xi ) T
• Contours for the function: ↵i =
g(xi )T H(xi )g(xi )
1

0.8

0.6

0.4

0.2

x2 0

−0.2

−0.4

−0.6

−0.8

−1
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1

22
x1
 Unconstrained Optimization
• Method of steepest decent/Newton-Raphson/Trust-Region:

• Example: log f (x) = 2

(x1 + 2 2
10x2 )
g(xi ) g(xi ) T
• Contours for the function: ↵i =
g(xi )T H(xi )g(xi )
1.1

1.08

1.06

1.04

1.02

x2 1

0.98

0.96

0.94

0.92

0.9
0.9 0.92 0.94 0.96 0.98 1 1.02 1.04 1.06 1.08 1.1

23
x1
MIT OpenCourseWare
httpV://ocw.mit.edu

10.34 Numerical Methods Applied to Chemical Engineering

Fall 2015

For information about citing these materials or our Terms of Use, visit: httpV://ocw.mit.edu/terms.