Univariate optimization

Prof. Sujit S. Jogwar

Department of Chemical Engineering

Indian Institute Technology, Bombay
[email protected]
 Learning Objectives
At the end of this lecture, you will be able to
• Derive optimality conditions for unconstrained univariate
optimization
• Solve univariate optimization problem using direct search
(gradient-free) and gradient-based methods

CL 603 Optimization 2
 Motivating example
• Reactor design for consecutive reactions
• is the desired product
• Design variable: residence time
• Governing equations:

represents composition
• Simplifying assumptions/process specifications
• No in feed 
• First order reaction kinetics 

What is the corresponding optimization problem?

CL 603 Optimization 3
 Motivating example
•

• Optimization problem:

0.35

0.3

0.25

0.2

0.15

0.1

or 0.05

0
0 5 10 15 20 25 30

Unimodal function

Univariate unconstrained optimization

CL 603 Optimization 4
 Univariate unconstrained optimization

• Most elementary optimization problem

• Commonly encountered in engineering applications
• Arises as a subproblem in solving multivariable optimization
problems

CL 603 Optimization 5
 Numerical optimization
• Static question: How to determine if a point is the
optimal solution?
• Dynamic question: If is not optimal, how do I travel from
this point to the optimal point ?

CL 603 Optimization 6
 Static question: Optimality conditions
• Necessary conditions: For a twice differentiable function ,
is local minimum if
•
•
• Sufficient conditions: and the first non-zero
higher order derivative is of order n.
• If n is even and this derivative is positive, is local
minimum
• If n is odd, is an inflection point

Proof is based on Taylor series expansion

CL 603 Optimization 7
 Dynamic question: Solution methods
• Direct search or Gradient-free methods
• Region elimination methods
• Point estimation methods
• Gradient-based methods

CL 603 Optimization 8
 Region elimination methods
• Idea: Using function evaluations at certain points,
successively remove part of the interval where optimum
does not lie.
• Theorem (Elimination property): Suppose is strictly
unimodal function over [ ] with as the minimum.
Consider two points and such that .
• If , then the minimum of does not lie in ( ).
• If , then the minimum of does not lie in ( ).
• If , then the minimum of does not lie in ( , ) and
( ).

CL 603 Optimization 9
 Region elimination methods
Steps:
1. Bounding phase: Starting at some guest point, obtain
an interval around the optimum point using elimination
property via coarse search.
2. Interval refinement phase: A finite sequence of
interval reductions to arrive at the optimum point.

CL 603 Optimization 10
 Bounding phase
• Typically based on heuristics
• Commonly used algorithm is Swan’s method1
1. Choose a starting point and a step parameter
2. Evaluate and
3. If
, . Go to step 4.
, . Go to step 4.
, optimum lies in [ , ]
, function is not unimodal

Large leads to poor (large) bracket

small requires large number of function evaluations
1. W.H. Swan, Central Instr. Res. Lab, Res. Note, 64/3, London, 1964

CL 603 Optimization 11
 Bounding phase
Application to motivating example

Data: min-1, min-1, M

1. Initial guess: 𝑥 = 1min, Δ = 0.5min
2. 𝑓 0.5 = −0.0465, 𝑓 1 = −0.0866, 𝑓 1.5 = −0.1213
3. 𝑓 0.5 ≥ 𝑓 1 ≥ 𝑓(1.5) ⇒ Δ > 0
4. 𝑥 = 𝑥 + Δ = 1.5, 𝑥 = 𝑥 + 2Δ = 2.5
5. 𝑓 𝑥 = −0.1778 ⇒ 𝑥 ⋆ > 1.5. 𝑥 = 𝑥 + 4Δ = 4.5
6. 𝑓 𝑥 = −0.2533 ⇒ 𝑥 ⋆ > 2.5. 𝑥 = 𝑥 + 8Δ = 8.5
7. 𝑓 𝑥 = −0.3224 ⇒ 𝑥 ⋆ > 4.5. 𝑥 = 𝑥 + 16Δ = 16.5
8. 𝑓 𝑥 = −0.3412 ⇒ 𝑥 ⋆ > 8.5. 𝑥 = 𝑥 + 32Δ = 32.5
9. 𝑓 𝑥 = −0.2913 ⇒ 𝑥 ⋆ < 32.5. 𝑥 ⋆ ∈ (8.5,32.5)

CL 603 Optimization 12
 Interval refinement
• Different algorithms differ in the number and placement of
internal points.
• It is desirable to eliminate the largest possible fraction of the
interval.
• It is desirable to eliminate the same fraction of interval each
time.

CL 603 Optimization 13
 Interval refinement
1. Interval halving method
• Select 3 points inside the interval at 25%, 50% and 75% length.
• Based on function values at these three points, half of the interval
can be eliminated.
• Every subsequent iteration, one of the previous points is carried
forward, so only two new function evaluations are required.
• Terminate when length of remaining interval is smaller than set
threshold.

If , interval ( can be eliminated. The 50% point

of the new interval is .
CL 603 Optimization 14
 Interval refinement
2. Golden section method
• Select 2 points at fraction of length from each side.
• Based on function values at these two points, ( ) fraction of the
interval can be eliminated.
• Every subsequent iteration, one of the previous points is carried forward,
so only one new function evaluation is required. This requires

• Terminate when length of remaining interval is smaller than set threshold.

If , interval ( can be eliminated. The point is now

fraction from 1.
CL 603 Optimization 15
 Interval refinement
Application to motivating example:
We have deduced that the optima lies in (8.5,32.5).
Interval halving Golden section
Iteration
# Resulting
Points evaluated Points evaluated Resulting interval
interval
1 14.5, 20.5, 26.5 (8.5,20.5) 17.6672, 23.3328 (8.5, 23.3328)
2 11.5, 14.5, 17.5 (11.5,17.5) 14.1656, 17.6672 (8.5, 17.6672)
3 13, 14.5, 16 (13,16) 12.0016, 14.1656 (12.0016, 17.6672)
4 13.75, 14.5, 15.25 (13.75,15.25) 14.1656, 15.5031 (12.0016, 15.5031)
5 14.125, 14.5, 14.875 (13.75,14.5) 13.339, 14.1656 (13.339, 15.5031)
13.9375, 14.125, (13.9375,
6 14.1656, 14.6765 (13.339, 14.6765)
14.3125 14.3125)
14.0312, 14.125, (14.0312,
7 13.8499, 14.1656 (13.8499, 14.6765)
14.2188 14.2188)
14.0781, 14.125, (14.0781,
8 14.1656, 14.3608 (13.8499, 14.3608)
14.1719 14.1719)

CL 603 Optimization 16
 Point estimation methods
• Idea: If is smooth, it can be approximated by a
polynomial. This polynomial can be used to predict the
location of optimum point.
• For example, if three points ( ) are used, a quadratic
function ( ) can be used to estimate the minimum point.

CL 603 Optimization 17
 Point estimation methods
Powell’s successive quadratic estimation method
1. Choose starting point and step size . Compute .
2. Evaluate and .
3. If . If .
4. Evaluate and determine
being corresponding to
⋆
5. Use and to estimate using quadratic estimation formula
⋆ ⋆
6. If and , terminate.
Else go to step 7.
⋆
7. Save the current best point ( or ) and the two points bracketing
it or closest to it. Relabel them and go to step 4.

CL 603 Optimization 18
 Point estimation methods
Powell’s successive quadratic estimation method applied to
motivating example
1. Initial guess: 𝑥 = 1min, Δ = 0.5min, 𝑥 = 1.5min
2. 𝑓 𝑥 = −0.0866, 𝑓 𝑥 = −0.1213
3. 𝑓 𝑥 > 𝑓 𝑥 , 𝑥 = 𝑥 + 2Δ = 2
4. 𝑓 𝑥 = −0.1515, 𝑓 = −0.1515, 𝑥 =2
5. 𝑥 ⋆ = 5.049, 𝑓 𝑥 ⋆ = −0.2679
6. 𝑓 𝑥⋆ − 𝑓 = 0.1164, 𝑥 ⋆ − 𝑥 = 3.049
7. Save 𝑥 ⋆ , 𝑥 and 𝑥 . Next iteration gives 𝑥 ⋆ = 6.5755, 𝑓 𝑥 ⋆ = −0.2985
8. 𝑓 𝑥⋆ − 𝑓 = 0.0307, 𝑥 ⋆ − 𝑥 = 1.5265
9. Save 𝑥 ⋆ , 𝑥 ⋆ and 𝑥 . Next iteration gives 𝑥 ⋆ = 8.3552, 𝑓 𝑥 ⋆ = −0.3211
10. Save 𝑥 ⋆ , 𝑥 ⋆ and 𝑥 ⋆ . Next iteration gives 𝑥 ⋆ = 10.2787, 𝑓 𝑥 ⋆ = −0.3348
11. Save 𝑥 ⋆ , 𝑥 ⋆ and 𝑥 ⋆ . Next iteration gives 𝑥 ⋆ = 11.716, 𝑓 𝑥 ⋆ = −0.3402
12. Save 𝑥 ⋆ , 𝑥 ⋆ and 𝑥 ⋆ . Next iteration gives 𝑥 ⋆ = 12.869, 𝑓 𝑥 ⋆ = −0.3424
13. Save 𝑥 ⋆ , 𝑥 ⋆ and 𝑥 ⋆ . Next iteration gives 𝑥 ⋆ = 13.6155, 𝑓 𝑥 ⋆ = −0.343
CL 603 Optimization 19
 Gradient-based methods
• Idea: Optimality condition is . Find a solution to
this equation using any nonlinear solvers.
• Example methods
• Newton Raphson (requires second derivative)
• Secant (requires bracketing)
• Cubic search (requires bracketing)

CL 603 Optimization 20
 Gradient-based methods
• Newton Raphson:

• Secant: (preferably )

• Cubic search: (preferably )

• Approximate function by a cubic polynomial using
.
• Iterate similar to successive quadratic estimation method

CL 603 Optimization 21
 Gradient-based methods
Application to motivating example

Iteration Newton Raphson Secant Cubic search

#
1 1 -0.0746 1 -0.0746 1 -0.0746
2 4.483 -0.0286 15.349 9 11.376 -0.003
3 8.192 -0.0101 14.128 -1
4 11.48 -0.003
5 13.507 -6

Secant and Cubic search used bracketing method similar to Swan’s

method (using derivative check) to obtain initial bracket of
(8.5,16.5)

CL 603 Optimization 22
 Comparison of methods
• All methods (except Newton Raphson require unimodality assumption).
• Region elimination methods are applicable even for discontinuous
functions.
• Point estimation methods require continuity assumption.
• Gradient-based methods require differentiability assumption.
• Golden section method gives linear rate of convergence.
• Successive quadratic estimation gives superlinear rate of convergence.
• Newton’s method gives quadratic rate of convergence
• For high accuracy, polynomial approximations are better.
• For high reliability, golden section method is preferred.

CL 603 Optimization 23
 Rate of convergence
A sequence of iterates that approximate solution is
convergent if

An algorithm has rate of convergence of order if

• Linear:
• Superlinear:
• Quadratic:

CL 603 Optimization 24
 Conclusions
• Univariate optimization is the simplest optimization problem
• It also arises as a subproblem during multivariable
optimization
• There exists trade-off between computational effort (number
of function evaluations), speed of convergence and
reliability
• Preferred choice is Powell’s successive quadratic
estimation with Golden section as a fallback option.

CL 603 Optimization 25
 Learning Objectives (revisited)
At the end of this lecture, you will be able to
• Derive optimality conditions for unconstrained univariate
optimization
• Solve univariate optimization problem using direct (gradient-
free) and gradient-based methods

CL 603 Optimization 26