Optimization Techniques
Single-variable Optimization Algorithms :
Topic: Bisection or Interval halving method
Bisection Method
• The method is known as the Bolzano method and can be
called interval halving technique.
• The method is simple and straight-forward.
• Given a bracketed root, the method repeatedly halves
the interval while continuing to bracket the root and it
will converge on the solution.
• We can use this to find minimum of g(x) as well.
• First find f(x) = dg/dx and then locate root of f(x).
• If at root the f’(x) is positive the root value is minimum
for g(x).
Step 1 for Root finding:
Choose xl and xu as
f(x)
two guesses for the
root such that f(xl)
f(xu) < 0, or in other
words, f(x) changes
sign between xl and
x u.
x
x
xu
Step 2
Estimate the root, xm of
f(x)
the equation f (x) = 0 as
the mid-point between xl
and xu as
x xu
xm =
2 x
x
xu
Step 3
Now check the following
f(x)
If f(xl) f(xm) < 0, then the root lies
between xl and xm; then xl = xl ;
xu = xm.
If f(xl ) f(xm) > 0, then the root lies
between xm and xu; then xl =
xm; xu = xu.
x xm
x
xu
If f(xl) f(xm) = 0; then the root is xm.
Stop the algorithm if this is
true.
Step 4
x xu
New estimate xm =
2
Absolute Relative Approximate Error
x new x old
m
a m
new
100
x m
xmold previous estimate of root
xmnew current estimate of root
Step 5
Check if absolute
relative approximate
error is less
than pre-specified Yes Stop
tolerance or if
maximum number
of iterations is
reached. Using the new
No
upper and lower
guesses from Step
3, go to Step 2.
Example
• You are working for ‘DOWN THE TOILET COMPANY’ that makes
floats for ABC commodes. The ball has a specific gravity of 0.6 and
has a radius of 5.5 cm. You are asked to find the distance to which
the ball will get submerged when floating in water.
Solution
The equation that gives the depth ‘x’ to which the ball is submerged
under water is given by
1 4
g ( x) x -0.055 x 3+3.993x10- 4 x
4
f x dg / dx x 3-0.165 x 2+3.993x10- 4
Use the Bisection method of finding
roots of equations to find the depth
‘x’ to which the ball is submerged
under water. Conduct three
iterations to estimate the root of
the above equation.
Graph of function f(x)
3 2
f x x -0.165 x +3.993x10 -4
Checking if the bracket is valid
Choose the bracket
x 0.00
xu 0.11
f 0.0 3.993x10 4
f 0.11 2.662x10 4
Iteration #1
x 0, xu 0.11
0 0.11
xm 0.055
2
f 0 3.993x10 4
f 0.11 2.662x10 4
f 0.055 6.655x10 5
x 0.055
xu 0.11
Iteration #2
x 0.055, xu 0.11
0.055 0.11
xm 0.0825
2
a 33.33%
f 0.055 6.655x10 5
f 0.11 2.662 x10 4
f 0.0825 1.62216 x10 4
x 0.055, xu 0.0825
Iteration #3
x 0.055, xu 0.0825
0.055 0.0825
xm 0.06875
2
a 20%
f 0.055 6.655x10 5
f 0.0825 1.62216 x10 4
f 0.06875 5.5632 x10 5
Convergence
Table 1: Root of f(x)=0 as function of number of iterations for bisection method.
Iteration x xu xm a % f(xm)
1 0.00000 0.11 0.055 ---------- 6.655x10-5
2 0.055 0.11 0.0825 33.33 -1.6222x10-4
3 0.055 0.0825 0.06875 20.00 -5.5632x10-5
4 0.055 0.06875 0.06188 11.11 4.4843x10-6
5 0.06188 0.06875 0.06531 5.263 -2.5939x10-5
6 0.06188 0.06531 0.06359 2.702 -1.0804x10-5
7 0.06188 0.06359 0.06273 1.369 -3.1768x10-6
8 0.06188 0.06273 0.0623 0.6896 6.4973x10-7
9 0.0623 0.06273 0.06252 0.3436 -1.2646x10-6
10 0.0623 0.06252 0.06241 0.1721 -3.0767x10-7
Bisection Method
A basic loop that is used to find root between two points for a
function f(x) is shown here.
DO while 0.5*|x1 - x2| >= tolerance_value
Set xmid =(x1 + x2)/2
IF f(xmid) of opposite sign of f(x 1);
Set x2 = xmid;
ELSE
Set x1 = xxmid;
ENDIF
END loop
Bisection Method
DemoBisect: the example program does a simple bisection for a cubic
polynomial equation.
Computer Program for Bisection method
% A program in matlab
% bisection method to find the roots of x^5+x^3+4*x^2-3*x-2=0
xleft = -2.0 ; xright = -1.0 ; n = 100
% Input: xleft,xright = left and right brackets of the root
% n = (optional) number of iterations; default: n =15
% Output: x = estimate of the root
if nargin<3, n=15; end % Default number of iterations
a = xleft; b =xright; % Copy orig bracket to local variables
fa = a^5 + a^3 +4*a^2 - 3*a - 2; % Initial values
fb = b^5 + b^3 +4*b^2 - 3*b - 2; % Initial values
fprintf(' k a xmid b f(xmid)\n');
for k=1:n
xm = a + 0.5*(b-a); % Minimize roundoff in the midpoint
fm = xm^5 + xm^3 +4*xm^2 - 3*xm - 2; % f(x) at midpoint
fprintf('%3d %12.8f %12.8f %12.8f %12.3e\n',k,a,xm,b,fm);
if sign(fm) == sign(fa)
a = xm; fa = fm;
else
b = xm; fb = fm;
end
end
Advantages & Disadvantages
Advatages:
• Always convergent
• The root bracket gets halved with each iteration - guaranteed.
Disadvatanges:
• Slow convergence
• If one of the initial guesses is close to the root, the convergence is
slower
Drawbacks (continued)
• If a function f(x) is such that it just touches the x-axis it will be
unable to find the lower and upper guesses.
f x x 2
Drawbacks (continued)
Function changes sign but root does not exist
1
f x
x
2.3.2 Fibonacci search method
• In this method, the search interval is reduced according to
Fibonacci numbers.
• The property of the Fibonacci numbers is that, given two
consecutive numbers Fn - 2 and Fn - 1 , the third number is calculated
as follows: Fn = Fn – 1 +Fn-2 , where n = 2,3,4, ...
• The first few Fibonacci numbers are
Fo = 1, F1 = 1, F2 = 2,
F3 = 3, F4 = 5, F5 = 8,
F6 = 13, and so on.
• The property of the Fibonacci numbers can be used to create a
search algorithm that requires only one function evaluation at each
iteration.
2.3.2 Fibonacci search method
• The principle of Fibonacci search is that out of two points required
for the use of the region-elimination rule, one is always the
previous point and the other point is new.
• Thus, only one function evaluation is required at each iteration.
• At iteration k, two intermediate points, each L away from either
end of the search space (L = b - a) are chosen.
• When the region-elimination rule eliminates a portion of the
search space depending on the function values at
• these two points, the remaining search space is Lk.
Fibonacci search method
• By defining L*k = (Fn-k+ 1/Fn+1)L and Lk
= (Fn-k+2/Fn+1)L it can be shown that
L*k - Lk = L*k+1 which means that one
of the two points used in iteration k
remains as one point in iteration (k
+1).
• This can be seen from Figure 2.8. If
the region (a, x2) is eliminated in
the k-th iteration, the point x1 is at
a distance (Lk – L*k) or L*k+1 from the
point x2 in the (k + 1)-th iteration.
• Since, the first two Fibonacci
numbers are the same, the
algorithm usually starts with k = 2.
Algorithm for Fibonacci search method
• Step 1: Choose a lower bound a and an upper bound b. Set L
= b - a. Assume the desired number of function evaluations to
be n. Set k = 2.
• Step 2: Compute L*k . Set x1 = a +L*k and x2 = b – L*k.
• Step 3: Compute one of f(x1) or f(x2), which was not
evaluated earlier. Use the fundamental region-elimination
rule to eliminate a region. Set new a and b.
• Step 4: Is k = n? If no, set k =k +1 and go to Step 2;
• Else Terminate.
Remarks
• In this algorithm, the interval reduces to (2/Fn+1)L after n function
evaluations.
• Thus, for a desired accuracy ε, the number of required function
evaluations n can be calculated using the following equation:
• As is clear from the algorithm, only one function evaluation is required at
each iteration.
• At iteration k, a proportion of Fn-k / Fn-k+2 of the search space at the
previous iteration is eliminated.
• For large values of n, this quantity is close to 38.2 per cent, which is
better than that in the interval halving method. (Recall that in the interval
halving method this quantity is 25 per cent.)
• However, one difficulty with this algorithm is that the Fibonacci numbers
must be calculated in each iteration.
EXERCISE 2.3.2
• Minimize the function f(x) = x2 + 54/x .
• Step 1: We choose a = 0 and b = 5.
• Thus, the initial interval is L= 5. Let us also choose the
desired number of function evaluations to be three (n =
3).
• In practice, a large value of n is usually chosen.
• We set k = 2.
• Step 2: We compute L*2:
•
EXERCISE 2.3.2
• Step 2: We compute L*2:
• Thus, we calculate x1 = 0 + 2 = 2 and x2 = 5 - 2 = 3
• Step 3: We compute the function values: f(x1) = 31 and
f(x2) = 27.
• Since f(x1) > f(x2), we eliminate the region (0,x1) or (0,2).
In other words, we set a = 2 and b = 5.
Figure 2.9 shows the function values at these two points and the
resulting region after Step 3.
The exact minimum of the function is also shown.
EXERCISE 2.3.2
EXERCISE 2.3.2
• Step 4: Since k = 2 ≠ n = 3, we set k = 3 and go to Step 2. This
completes one iteration of the Fibonacci search method.
• Step 2: We compute L: = (F1/F4 )L = (1/5)*5 = 1,
x1 = 2 +1 = 3, and x2 = 5 - 1 = 4.
• Step 3: we observe that function was computed at x1 = 3 previously,
so f(x2) = 29.5 By comparing function values at x1= 3 and x2 = 4, we
observe that f(x1) < f(X2) . Therefore, we set a = 2 and b = x2 = 4,
since the fundamental rule suggests that the minimum cannot lie
beyond X2 = 4.
• Step 4: At this iteration, k = n = 3 and we terminate the
• algorithm. The final interval is (2,4).
Drawbacks of Fibonacci search method
• One difficulty of the Fibonacci search method is that
the Fibonacci numbers have to be calculated and
stored.
• Another problem is that at every iteration the
proportion of the eliminated region is not the same.
• In order to overcome these two problems and yet
calculate one new function evaluation per iteration,
the golden section search method is used.
• In this algorithm, the search space (a, b) is first linearly
mapped to a unit interval search space (0,1).
