0% found this document useful (0 votes)

18 views

Numerical Analysis Chapter 2

The document outlines numerical methods for finding the roots of a nonlinear function, including bisection, fixed point, Newton's, and secant methods. It introduces the problem of solving f(x)=0 and discusses ensuring a unique solution exists and is contained within a specified domain. The bisection method is described as iteratively bisecting the interval and identifying which half contains the root to define the new interval.

Uploaded by

Anthony Michael

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

18 views

Numerical Analysis Chapter 2

Uploaded by

Anthony Michael

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 213

Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Numerical Analysis
Chapter 2: Numerical Solutions of Nonlinear Equations of One
Variable

Xiaoming He
Department of Mathematics & Statistics
Missouri University of Science & Technology

Textbook: Classical and Modern Numerical Analysis:

Theory, Methods and Practice
A. S. Ackleh, E. J. Allen, R. B. Kearfott and P. Seshaiyer
CRC Press, Boca Raton, FL, 2010

1 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Outline

1 Introduction

2 Bisection method

3 The fixed point method

4 Newton’s method

5 Secant method

6 Other topics

2 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Outline

1 Introduction

2 Bisection method

3 The fixed point method

4 Newton’s method

5 Secant method

6 Other topics

3 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

What’s the target problem?

Solve f (x) = 0 for a given function f (x). That is, find the
roots of f (x).

4 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

What’s the target problem?

Solve f (x) = 0 for a given function f (x). That is, find the
roots of f (x).
Why do we start from this problem?

4 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

What’s the target problem?

Solve f (x) = 0 for a given function f (x). That is, find the
roots of f (x).
Why do we start from this problem?
How to guarantee that the given problem has a unique
solution on a specified domain?

4 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

What’s the target problem?

Solve f (x) = 0 for a given function f (x). That is, find the
roots of f (x).
Why do we start from this problem?
How to guarantee that the given problem has a unique
solution on a specified domain?
How to construct a convergent numerical method?

4 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

What’s the target problem?

4 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

What’s the target problem?

4 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Reasons to begin with f (x) = 0

This problem occurs frequently and the methods are are

useful.
The algorithm and the code are easy to develop. The idea and
theory are easy to understand and derive. Hence this problem
provides us a quick but complete look at different numerical
methods for a problem.
This problem illustrates the iteration method, which is a
common numerical technique. It will well prepare us for more
complicated iterative methods, not only in our understanding
and the theory, but also in coding.
This problem illustrates that, when many methods are
available, selection of the most suitable method depends on
the particular problem and the computation equipment
available.
5 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Existence of the solution

Theorem (Intermediate value theorem)

If f ∈ C [a, b] and k is a number between m = min f (x) and
a≤x≤b
M = max f (x), then there exists a number c ∈ [a, b] such that
a≤x≤b
f (c) = k.

Figure???

6 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Existence of the solution

Corollary (I)
If f ∈ C [a, b] and f (a)f (b) < 0, then there is a z ∈ [a, b] such that
f (z) = 0.

7 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Existence of the solution

Corollary (I)
If f ∈ C [a, b] and f (a)f (b) < 0, then there is a z ∈ [a, b] such that
f (z) = 0.

Proof.
Consider to apply the intermediate value theorem.

f (a)f (b) < 0 ⇒ m = min f (x) < 0 and M = max f (x) > 0.
a≤x≤b a≤x≤b

Hence 0 ∈ [m, M]. Then set the k = 0 in the intermediate value

theorem, we finish the proof.

Figure???

7 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Uniqueness of the solution

Corollary (II)
If f ∈ C [a, b], f is monotonic on [a, b] and f (a)f (b) < 0, then
there is a unique z ∈ [a, b] such that f (z) = 0.

Figure???
Remark
f 0 (x) can provide the monotonicity of f (x) on certain
intervals.
If there are more than one solutions in [a, b], then the
numerical methods may or may not converge!

8 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Outline

1 Introduction

2 Bisection method

3 The fixed point method

4 Newton’s method

5 Secant method

6 Other topics

9 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the bisection method

Suppose there is only one solution z ∈ [a, b].

10 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the bisection method

Suppose there is only one solution z ∈ [a, b].

Then z must be in either the left half or the right half of the
interval [a, b] .

10 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the bisection method

Suppose there is only one solution z ∈ [a, b].

10 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the bisection method

Suppose there is only one solution z ∈ [a, b].

10 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the bisection method

Suppose there is only one solution z ∈ [a, b].

Then z must be in either the left half or the right half of the
interval [a, b] .
Once we identify the half interval which contains z, we define
it to be the new interval and its length is half of that of the
original interval [a, b].
Then we repeat this procedure on the new interval again and
again to obtain smaller and smaller intervals which contains
the solution z.
When the interval is small enough, any point in the interval is
a good approximation to the solution z. Usually we pick up
the middle point of the last interval to be the final numerical
solution.

10 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the bisection method

A geometric explanation of the fixed point method:

11 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the bisection method

1 Find a and b such that f (a)f (b) < 0. Otherwise, stop.

12 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the bisection method

12 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the bisection method

1 Find a and b such that f (a)f (b) < 0. Otherwise, stop.
2 Initial preparation:
(1) Define a0 = a and b0 = b.
(2) Set the iteration counter k = 0.
(3) Specify a tolerance ε to stop the following loop.
bk −ak
3 While 2 ≥ ε, repeat the following procedure in a loop:
(1) xk = ak +b2 .
k

(2) If f (xk ) = 0, then

stop the loop;
elseif f (ak )f (xk ) < 0, then
ak+1 = ak and bk+1 = xk ;
elseIf f (xk )f (bk ) < 0, then
ak+1 = ak and bk+1 = xk .
end
(3) k ← k + 1.

12 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the bisection method

(2) If f (xk ) = 0, then

stop the loop;
elseif f (ak )f (xk ) < 0, then
ak+1 = ak and bk+1 = xk ;
elseIf f (xk )f (bk ) < 0, then
ak+1 = ak and bk+1 = xk .
end
(3) k ← k + 1.
4 Output the final xk from the above loop as the numerical
solution. 12 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the bisection method

Several implementation issues:

No need to store all of ak and bk . We may just define two
variables a and b, then store the current ak and bk into them.

13 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the bisection method

Several implementation issues:

No need to store all of ak and bk . We may just define two
variables a and b, then store the current ak and bk into them.
We have two ways to code for a loop: “for” and “while”.
Example:
for k = 0 : 10 while (a < 10)
..... .....
end end

13 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the bisection method

Several implementation issues:

13 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the bisection method

Example 1: f (x) = e x + x, f (0) = 1, and f (−1) = −0.632.

Choose ε = 0.01.

f (x) is monotonically increasing on any interval since

f 0 (x) = e x + 1 > 0 for all x.

14 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the bisection method

Example 1: f (x) = e x + x, f (0) = 1, and f (−1) = −0.632.

Choose ε = 0.01.

f (x) is monotonically increasing on any interval since

f 0 (x) = e x + 1 > 0 for all x.

Let’s code for this simple algorithm together!

Open your Matlab!

14 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the bisection method

Based on the code we developed together just now, we obtain

Numerical results:
k ak bk xk
0 −1 0 −0.5
1 −1 −0.5 −0.75
2 −0.75 −0.5 −0.625
3 −0.625 −0.5 −0.5625
4 −0.625 −0.5625 −0.59375
5 −0.59375 −0.5625 −0.578125
6 −0.578125 −0.5625 −0.5703125

Verification: f (−0.5703125) ≈ 4.963760 × 10−3 . A smaller

tolerance ε will give a smaller error.

15 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the bisection method

Several more implementation issues:

Call a separate subroutine to find a and b such that
f (a)f (b) < 0. (Independent study problem)
A tolerance εf for f (xk ) can be also defined such that we can
stop the loop when |f (xk )| < εf instead of f (xk ) = 0.
A maximum number M of iteration steps can be also set up
to stop the loop in case the iteration does not converge.

16 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Enriched algorithm of the bisection method

1 Call a subroutine to obtain the a and b such that
f (a)f (b) < 0. Otherwise, stop.

17 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Enriched algorithm of the bisection method

17 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Enriched algorithm of the bisection method

1 Call a subroutine to obtain the a and b such that
f (a)f (b) < 0. Otherwise, stop.
2 Initial preparation:
(1) Define a0 = a and b0 = b.
(2) Set the iteration counter k = 0.
(3) Specify ε, εf and M.
bk −ak
3 While 2 ≥ ε and k ≤ M, repeat the following procedure:
(1) xk = ak +b2 .
k

(2) If |f (xk )| < εf , then

stop the loop;
elseif f (ak )f (xk ) < 0, then
ak+1 = ak and bk+1 = xk ;
elseIf f (xk )f (bk ) < 0, then
ak+1 = ak and bk+1 = xk .
end
(3) k ← k + 1.
17 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Enriched algorithm of the bisection method

(2) If |f (xk )| < εf , then

stop the loop;
elseif f (ak )f (xk ) < 0, then
ak+1 = ak and bk+1 = xk ;
elseIf f (xk )f (bk ) < 0, then
ak+1 = ak and bk+1 = xk .
end
(3) k ← k + 1.
4 Output the final xk from the above loop. 17 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the bisection method

Theorem
Assume z is the only solution of f (x) = 0 on [a, b]. If f is a
continuous function on [a, b] and f (a)f (b) < 0, then

b−a
|xk − z| ≤ . (1)
2k+1

18 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the bisection method

Theorem
Assume z is the only solution of f (x) = 0 on [a, b]. If f is a
continuous function on [a, b] and f (a)f (b) < 0, then

b−a
|xk − z| ≤ . (1)
2k+1

Proof.
bk − ak
z ∈ [ak , bk ] and ⇒ |xk − z| ≤ .
2
b−a
Then combine this with bk − ak = 2k
, we finish the proof.

18 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the bisection method

Theorem
Assume z is the only solution of f (x) = 0 on [a, b]. If f is a
continuous function on [a, b] and f (a)f (b) < 0, then

b−a
|xk − z| ≤ . (1)
2k+1

Proof.
bk − ak
z ∈ [ak , bk ] and ⇒ |xk − z| ≤ .
2
b−a
Then combine this with bk − ak = 2k
, we finish the proof.

Remark
If f (x) ≥ 0 and f (z) = 0, then the bisection method is not valid
since f (a)f (b) < 0 requires either f (a) < 0 or f (b) < 0.
18 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the bisection method

Example 2: Assume the solution z of f (x) = 0 is in [0, 2].

How many iteration steps are required to reduce the error to
less than 10−4 ?

19 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the bisection method

Example 2: Assume the solution z of f (x) = 0 is in [0, 2].

How many iteration steps are required to reduce the error to
less than 10−4 ?

Solution: |xk − z| ≤ b−a

2k+1
≤ 10−4 ⇒ 2−0
2k+1
≤ 10−4 .
4
Then 2k ≥ ⇒ k ≥ lnln102 ≈ 13.2877.
104
Hence 14 iteration steps are required.

19 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Outline

1 Introduction

2 Bisection method

3 The fixed point method

4 Newton’s method

5 Secant method

6 Other topics

20 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the fixed point method

Still consider f (x) = 0.

21 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the fixed point method

Still consider f (x) = 0.

What if f (x) = g (x) − x or f (x) = x − g (x)?

21 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the fixed point method

Still consider f (x) = 0.

What if f (x) = g (x) − x or f (x) = x − g (x)?

Then f (x) = 0 ⇒ x = g (x).

Define z to be a fixed point if z = g (z).

21 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the fixed point method

Still consider f (x) = 0.

What if f (x) = g (x) − x or f (x) = x − g (x)?

Then f (x) = 0 ⇒ x = g (x).

Define z to be a fixed point if z = g (z).

How about this iteration: xk+1 = g (xk )?

21 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea of the fixed point method

A geometric explanation of the fixed point method:

22 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Implementation issues

A tolerance ε should be used to stop the loop when

|xk+1 − xk | < ε.

A tolerance εf for f (xk ) can be used to stop the loop when

|f (xk )| < εf instead of f (xk ) = 0.

A maximum number M of iteration steps can be used to stop

the loop in case the iteration does not converge.

23 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the fixed point method

1 Initial preparation:

24 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the fixed point method

1 Initial preparation:
(1) Pick up the initial point x0 .
(2) Set the iteration counter k = 0.
(3) Specify the tolerances ε, εf , and the maximum number M
of iteration steps.

24 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the fixed point method

24 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the fixed point method

1 Initial preparation:
(1) Pick up the initial point x0 .
(2) Set the iteration counter k = 0.
(3) Specify the tolerances ε, εf , and the maximum number M
of iteration steps.
2 First step: x1 = g (x0 ).
3 While |xk+1 − xk | ≥ ε and k + 1 ≤ M, repeat the following
procedure in a loop:
(1) k ← k + 1.
(2) xk+1 = g (xk ).
(3) If |g (xk+1 ) − xk+1 | < εf , then stop the loop.

24 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Algorithm of the fixed point method

24 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 1: f (x) = −x+12 and x0 = 0. Then g (x) = x+1

2 .
Choose ε = 0.01, εf = 10−6 , and M = 100. It’s
straightforward to verify that the fixed point is 1.

25 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 1: f (x) = −x+12 and x0 = 0. Then g (x) = x+1

2 .
Choose ε = 0.01, εf = 10−6 , and M = 100. It’s
straightforward to verify that the fixed point is 1.

Do you feel familiar with the structure of the above

algorithm? If so, can you code for it quickly?

Open your Matlab! Just modify the code of the bisection

method a little bit!

25 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

26 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Based on the code we developed together just now, we obtain
Numerical results:
k xk
0 0
1 0.5
2 0.75
3 0.875
4 0.9375
5 0.96875
6 0.984375
7 0.9921875
Verification:
f (0.9921875) = g (0.9921875) − 0.9921875 ≈ 0.00390625. A
smaller tolerance ε will give a smaller error.
Try other initial values?
26 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Based on the code we developed together just now, we obtain
Numerical results:
k xk
0 0
1 0.5
2 0.75
3 0.875
4 0.9375
5 0.96875
6 0.984375
7 0.9921875
Verification:
f (0.9921875) = g (0.9921875) − 0.9921875 ≈ 0.00390625. A
smaller tolerance ε will give a smaller error.
Try other initial values? Similar performance.
26 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 2: f (x) = 4 + 13 sin 2x − x and x0 = 0 . Then

g (x) = 4 + 31 sin 2x.

27 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 2: f (x) = 4 + 13 sin 2x − x and x0 = 0 . Then

g (x) = 4 + 31 sin 2x.
Numerical results:
k xk
0 4
1 4.329786
2 4.230895
3 4.273634
4 4.273634
5 4.256383
Verification:
f (4.256383) = g (4.256383) − 4.256383 ≈ 0.00719599.

27 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 2: f (x) = 4 + 13 sin 2x − x and x0 = 0 . Then

27 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 2: f (x) = 4 + 13 sin 2x − x and x0 = 0 . Then

27 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 2: f (x) = 4 + 13 sin 2x − x and x0 = 0 . Then

g (x) = 4 + 31 sin 2x.
Numerical results:
k xk
0 4
1 4.329786
2 4.230895
3 4.273634
4 4.273634
5 4.256383
Verification:
f (4.256383) = g (4.256383) − 4.256383 ≈ 0.00719599.
Try other initial values? Similar performance.
Why it’s always convergent?
27 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 3: f (x) = − x2 + x1 . Then g (x) = x2 + x1 . It’s
√
straightforward
√ to verify that there are two fixed points: 2
and − 2. Choose x0 = 10 and x0 = −5 separately.
Numerical results:
k xk from x0 = 10 xk from x0 = −5
0 10 −5
1 5.1 −2.7
2 2.746078 −1.720370
3 1.737195 −1.441455
4 1.444238 −1.414471
5 1.414526 −1.414214
6 1.414214
Verification: f (1.414214) ≈ −4.376268 × 10−7 and
f (−1.414214) ≈ 4.376268 × 10−7 .

28 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 4: f (x) = x 2 − x. Then g (x) = x 2 . Choose x0 = 2

and x0 = 0.5 separately.

29 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 4: f (x) = x 2 − x. Then g (x) = x 2 . Choose x0 = 2

and x0 = 0.5 separately.
Numerical results:
k xk from x0 = 2 xk from x0 = 0.5
0 2 0.5
1 4 0.25
2 16 0.0625
3 256 3.90625 × 10−3
4 6.5536 × 104 1.5259 × 10−5
5 4.2950 × 109

Diverge sometimes!

29 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 5: f (x) = e x + x and x0 = 0. Then g (x) = e x + 2x
or g (x) = −e x .

30 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Example 5: f (x) = e x + x and x0 = 0. Then g (x) = e x + 2x
or g (x) = −e x .
Numerical results:
k xk for g (x) = e x + 2x xk for g (x) = −e x
0 0 0
1 1 −1
2 4.718282 −0.367879
3 1.214123 × 102 −0.692201
4 5.3539 × 1052 −0.500474
5 −0.606244
6 −0.545396
7 −0.579612
8 −0.560115
9 −0.571143
10 −0.564879
30 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Verification for the case with g (x) = −e x :

f (−0.564879) ≈ 3.549922 × 10−3 .

31 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Verification for the case with g (x) = −e x :

f (−0.564879) ≈ 3.549922 × 10−3 .

Try other initial values for the case with g (x) = e x + 2x?

31 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Verification for the case with g (x) = −e x :

f (−0.564879) ≈ 3.549922 × 10−3 .

Try other initial values for the case with g (x) = e x + 2x?

Diverge, diverge, diverge for g (x) = e x + 2x! What’s wrong?

31 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Question 1: When does there exist a solution of the original

problem?
Question 2: When is the solution unique?
Question 3: When do the numerical solutions {xk } converge
to that solution?
Question 4: How fast do the numerical solutions {xk }
converge to that solution?

32 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Practice in Matlab for the fixed point method

Question 1: When does there exist a solution of the original

Answer: Theory of the conditions for the convergence!

32 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Definition (Lipschitz condition)

A function g satisfies a Lipschitz condition on a set G if there
exists a Lipschitz constant L ≥ 0 such that

|g (x) − g (y )| ≤ L |x − y | , for all x, y ∈ G . (2)

If g satisfies (2) with 0 ≤ L < 1, g is said to be a contraction on

the set G .

33 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Definition (Lipschitz condition)

A function g satisfies a Lipschitz condition on a set G if there
exists a Lipschitz constant L ≥ 0 such that

|g (x) − g (y )| ≤ L |x − y | , for all x, y ∈ G . (2)

If g satisfies (2) with 0 ≤ L < 1, g is said to be a contraction on

the set G .

Lemma (I)
If g satisfies a Lipschitz condition on G , then g is continuous on G .

Proof.
Inequality (2) directly gives the continuity based on its definition in
the calculus book.
33 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Definition (Cauchy sequence)

A sequence {xk }∞k=1 is called a Cauchy sequence if, given any
ε > 0, there exists a N dependent on ε such that |xk − xn | < ε for
every k and n greater than N.

34 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Definition (Cauchy sequence)

A sequence {xk }∞k=1 is called a Cauchy sequence if, given any
ε > 0, there exists a N dependent on ε such that |xk − xn | < ε for
every k and n greater than N.

Lemma (II)
Cauchy sequences in R and C must converge to a point in R or C.

34 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Definition (Cauchy sequence)

A sequence {xk }∞k=1 is called a Cauchy sequence if, given any
ε > 0, there exists a N dependent on ε such that |xk − xn | < ε for
every k and n greater than N.

Lemma (II)
Cauchy sequences in R and C must converge to a point in R or C.

Definition (Complete spaces)

Number systems in which Cauchy sequences must converge are
called complete spaces.

34 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Theorem (I: Contraction Mapping Theorem in one variable)

Suppose that g maps G into itself (i.e., if x ∈ G , then g (x) ∈ G )
and g satisfies a Lipschitz condition with with 0 ≤ L < 1(i.e., g is
a contraction on G ). Then there exists a unique fixed point z ∈ G
(i.e., z = g (z)), and the sequence {xk } determined by x0 ∈ G and
xk+1 = g (xk ) (k = 0, 1, 2, · · · ) converges to z with error estimates

Lk
|xk − z| ≤ |x1 − x0 | (3)
1−L
L
|xk − z| ≤ |xk − xk−1 | (4)
1−L

35 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Theorem (I: Contraction Mapping Theorem in one variable)

Lk
|xk − z| ≤ |x1 − x0 | (3)
1−L
L
|xk − z| ≤ |xk − xk−1 | (4)
1−L

Remark
This theorem answers Questions 1, 2, and 3.

35 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Proof: We first consider the convergence of {xk } and the existence

of the fixed point z.

36 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Proof: We first consider the convergence of {xk } and the existence

of the fixed point z.
Since g maps G into itself and the sequence is determined by
x0 ∈ G and xk+1 = g (xk ) (k = 0, 1, 2, · · · ), then
xk ∈ G (k = 0, 1, 2, · · · ).

36 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Proof: We first consider the convergence of {xk } and the existence

of the fixed point z.
Since g maps G into itself and the sequence is determined by
x0 ∈ G and xk+1 = g (xk ) (k = 0, 1, 2, · · · ), then
xk ∈ G (k = 0, 1, 2, · · · ).
Since g satisfies a Lipschitz condition on G , then

|xk+1 − xk | = |g (xk ) − g (xk−1 )| ≤ L |xk − xk−1 | . (5)

36 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Proof: We first consider the convergence of {xk } and the existence

|xk+1 − xk | = |g (xk ) − g (xk−1 )| ≤ L |xk − xk−1 | . (5)

By reduction, we can get

|xk+1 − xk | ≤ Lk |x1 − x0 | . (6)

36 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

37 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

37 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
So
L
|xn − xk | ≤ |xk − xk−1 |
1−L
Lk
≤ |x1 − x0 | . (8)
1−L

38 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
So
L
|xn − xk | ≤ |xk − xk−1 |
1−L
Lk
≤ |x1 − x0 | . (8)
1−L

Any idea from the above result?

38 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
So
L
|xn − xk | ≤ |xk − xk−1 |
1−L
Lk
≤ |x1 − x0 | . (8)
1−L

Any idea from the above result?

{xk } is a Cauchy sequence!
Given any ε, there exists an N sufficiently large such that if
k, n > Nm |xk − xn | < ε.

38 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
Actually, we can obtain the N as follows.

Lk ε(1 − L)
1−L |x1 − x0 | ≤ ε ⇒ Lk ≤
|x1 − x0 |
ε(1−L) ε(1 − L)
⇒ ln(Lk ) ≤ ln( |x1 −x0 |
) ⇒ k ln(L) ≤ ln( )
|x1 − x0 |
ε(1−L)
ln( )
|x1 −x0 |
⇒ k≥ ln(L) . (9)

Here ln(L) < 0 if 0 < L < 1. As for L = 0, |xk − xn | ≤ 0 < ε

for any k and n.

39 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
" ε(1−L)
#
ln( )
|x1 −x0 |
Then we can pick N = ln(L) + 1 where [P] is the

minimum integer greater than P.

40 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
" ε(1−L)
#
ln( )
|x1 −x0 |
Then we can pick N = ln(L) + 1 where [P] is the

minimum integer greater than P.

By Lemma (II), the sequence {xk } must converge. Assume
lim xk = z.
k→∞

40 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
" ε(1−L)
#
ln( )
|x1 −x0 |
Then we can pick N = ln(L) + 1 where [P] is the

minimum integer greater than P.

By Lemma (II), the sequence {xk } must converge. Assume
lim xk = z.
k→∞

By Lemma (I), g is continuous since g satisfies a Lipschitz

condition.

40 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
" ε(1−L)
#
ln( )
|x1 −x0 |
Then we can pick N = ln(L) + 1 where [P] is the

minimum integer greater than P.

By Lemma (II), the sequence {xk } must converge. Assume
lim xk = z.
k→∞

By Lemma (I), g is continuous since g satisfies a Lipschitz

condition.
Then taking the limit on both sides of xk+1 = g (xk ), we get
lim xk+1 = lim g (xk ).
k→∞ k→∞

40 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
" ε(1−L)
#
ln( )
|x1 −x0 |
Then we can pick N = ln(L) + 1 where [P] is the

minimum integer greater than P.

By Lemma (II), the sequence {xk } must converge. Assume
lim xk = z.
k→∞

By Lemma (I), g is continuous since g satisfies a Lipschitz

condition.
Then taking the limit on both sides of xk+1 = g (xk ), we get
lim xk+1 = lim g (xk ).
k→∞ k→∞

And then z = g (z). Hence z is a fixed point and the sequence

{xk } converges to z.
40 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
Now let’s turn to the uniqueness of the fixed point z.

41 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
Now let’s turn to the uniqueness of the fixed point z.
Assume that there are two different fixed points z1 and z2 .
Hence z1 = g (z1 ), z2 = g (z2 ), and z1 − z2 6= 0.

41 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
Now let’s turn to the uniqueness of the fixed point z.
Assume that there are two different fixed points z1 and z2 .
Hence z1 = g (z1 ), z2 = g (z2 ), and z1 − z2 6= 0.
Then

|z1 − z2 | = |g (z1 ) − g (z2 )| ≤ L |z1 − z2 | . (10)

41 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
Now let’s turn to the uniqueness of the fixed point z.
Assume that there are two different fixed points z1 and z2 .
Hence z1 = g (z1 ), z2 = g (z2 ), and z1 − z2 6= 0.
Then

|z1 − z2 | = |g (z1 ) − g (z2 )| ≤ L |z1 − z2 | . (10)

Since g is a contraction on G , then 0 ≤ L < 1. Recall that

z1 − z2 6= 0. Then

|z1 − z2 | ≤ |z1 − z2 | , (11)

which is a contradiction. Hence there exists a unique fixed

point z ∈ G .
41 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Remark
It may be difficult to find G such that g maps G to itself.

It may be difficult to show that g is a contraction on G .

42 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Remark
It may be difficult to find G such that g maps G to itself.

It may be difficult to show that g is a contraction on G .

Question: How to obtain the two conditions required by the

Contraction Mapping Theorem: g maps G to itself and g is a
contraction on G ?

42 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Theorem (II: Mean Value Theorem)

Suppose f ∈ C 1 [a, b], x ∈ [a, b], and y ∈ [a, b]. Then there exists
a ξ ∈ [x, y ] ⊆ [a, b] such that

f (y ) − f (x) = f 0 (ξ)(y − x). (12)

Here C 1 [a, b] denotes the space of the functions which are first
differentiable.

43 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Lemma (III)
Suppose that g is continuously differentiable and that |g 0 (x)| ≤ L
for x ∈ G . Then g satisfied a Lipschitz condition with Lipschitz
constant L for x ∈ G .

44 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Lemma (III)
Suppose that g is continuously differentiable and that |g 0 (x)| ≤ L
for x ∈ G . Then g satisfied a Lipschitz condition with Lipschitz
constant L for x ∈ G .

Proof.
By the Mean Value Theorem, we have

g (y ) − g (x) = g 0 (ξ)(y − x). (13)

Also, by |g 0 (x)| ≤ L for x ∈ G , we have |g 0 (ξ)| ≤ L. Combining

these two conclusions, we obtain

g (y ) − g (x) ≤ L(y − x). (14)

Hence we finish the proof.

44 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

The above lemma provides a technique to find the Lipschitz

constant L.

45 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

The above lemma provides a technique to find the Lipschitz

constant L.
3 x5
Example 6: Suppose g (x) = − x6 + 120 . Find the Lipschitz
constant L for g on G = [− 21 , 21 ].

45 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

The above lemma provides a technique to find the Lipschitz

constant L.
3 x5
Example 6: Suppose g (x) = − x6 + 120 . Find the Lipschitz
constant L for g on G = [− 21 , 21 ].

2 x4
Solution: First, g 0 (x) = − x2 + 24 .

45 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

The above lemma provides a technique to find the Lipschitz

constant L.
3 x5
Example 6: Suppose g (x) = − x6 + 120 . Find the Lipschitz
constant L for g on G = [− 21 , 21 ].

2 4
Solution: First, g 0 (x) = − x2 + 24
x
.
2 1 4
Since − 21 ≤ −x 2 ≤ 0 and 0 ≤ x 4 ≤ on G = [− 12 , 12 ],

2
2 x4
we get − 81 ≤ − x2 + 24 1
≤ 384 .

45 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

The above lemma provides a technique to find the Lipschitz

constant L.
3 x5
Example 6: Suppose g (x) = − x6 + 120 . Find the Lipschitz
constant L for g on G = [− 21 , 21 ].

2 4
Solution: First, g 0 (x) = − x2 + 24
x
.
2 1 4
Since − 21 ≤ −x 2 ≤ 0 and 0 ≤ x 4 ≤ on G = [− 12 , 12 ],

2
2 x4
we get − 81 ≤ − x2 + 24 1
≤ 384 .
2 x4
Then |g 0 (x)| = − x2 + 24 ≤ 18 . Thus L = 81 .

45 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Lemma (IV)
Suppose that g is continuously differentiable and that
|g 0 (x)| ≤ L < 1 for x ∈ G . Then g is a contraction on G .

46 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Lemma (IV)
Suppose that g is continuously differentiable and that
|g 0 (x)| ≤ L < 1 for x ∈ G . Then g is a contraction on G .

Proof.
Consider x ∈ G and x̃ ∈ G . Without loss of generality, assume
that x < x̃. Then
Z x̃ Z x̃
0
|g (x) − g (x̃)| = g (s) ds ≤ g 0 (s) ds
x x
≤ L |x − x̃| . (15)

Since |g 0 (x)| ≤ L < 1 ⇒ 0 ≤ L < 1, then g is a contraction on

46 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Lemma (V)
Let ρ > 0 and G = [c − ρ, c + ρ]. Suppose that g is a contraction
on G with Lipschitz constant 0 ≤ L < 1 and
|g (c) − c| ≤ (1 − L)ρ. Then g maps G into itself.

47 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Lemma (V)
Let ρ > 0 and G = [c − ρ, c + ρ]. Suppose that g is a contraction
on G with Lipschitz constant 0 ≤ L < 1 and
|g (c) − c| ≤ (1 − L)ρ. Then g maps G into itself.

Proof.
Consider x ∈ G . Then |x − c| ≤ ρ. Hence

|g (x) − c| = |g (x) − g (c) + g (c) − c|

≤ |g (x) − g (c)| + |g (c) − c|
≤ L |x − c| + (1 − L)ρ
≤ Lρ + (1 − L)ρ
= ρ. (16)

So g (c) ∈ [c − ρ, c + ρ] = G , which completes the proof.

47 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Theorem (III)
Assume that z is a solution of x = g (x), g 0 (x) is continuous in an
interval about z, and |g 0 (z)| < 1. Then g is a contraction in a
sufficiently small interval about z, and g maps this interval into
itself. Thus, provided x0 is picked sufficiently close to z, the
iterations will converge.

48 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Remark
The above lemmas and theorem provide more details about
the conditions for the convergence than the Contraction
Mapping Theorem.

48 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Remark
The above lemmas and theorem provide more details about
the conditions for the convergence than the Contraction
Mapping Theorem.
But they are sufficient conditions, not necessary conditions!

48 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Remark
The above lemmas and theorem provide more details about
the conditions for the convergence than the Contraction
Mapping Theorem.
But they are sufficient conditions, not necessary conditions!
Any more details about “sufficiently small” and “sufficiently
close”? (Independent study problem)
48 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Proof.
Since |g 0 (z)| < 1, there exists a L such that |g 0 (z)| < L < 1.

49 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Proof.
Since |g 0 (z)| < 1, there exists a L such that |g 0 (z)| < L < 1.
Since g 0 (x) is continuous in an interval about z, there exists a
sufficiently small interval I = [z − ε, z + ε] such that
maxx∈I |g 0 (x)| ≤ L < 1.

49 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

49 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

|g (x) − z| = |g (x) − g (z)| ≤ L |x − z| ≤ ε. (17)

Hence g maps I into I .

49 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

|g (x) − z| = |g (x) − g (z)| ≤ L |x − z| ≤ ε. (17)

Hence g maps I into I .

Then the Contraction Mapping Theorem can be used with
G = I to obtain the convergence of the iterations.

49 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 2: g (x) = 4 + 31 sin 2x. What initial values

can give convergent numerical solutions of the fixed point
method?

50 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 2: g (x) = 4 + 31 sin 2x. What initial values

can give convergent numerical solutions of the fixed point
method?
Solution: First, g is continuous everywhere.

50 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 2: g (x) = 4 + 31 sin 2x. What initial values

can give convergent numerical solutions of the fixed point
method?
Solution: First, g is continuous everywhere.
Second,g 0 (x) = 2
3 cos 2x ⇒ |g 0 (x)| ≤ 2
3 < 1 for any x ∈ R.

50 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 2: g (x) = 4 + 31 sin 2x. What initial values

50 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 2: g (x) = 4 + 31 sin 2x. What initial values

50 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 2: g (x) = 4 + 31 sin 2x. What initial values

can give convergent numerical solutions of the fixed point
method?
Solution: First, g is continuous everywhere.
Second,g 0 (x) = 2
3 cos 2x ⇒ |g 0 (x)| ≤ 2
3 < 1 for any x ∈ R.
Then by Lemma (IV), g is a contraction on G = R.
Obviously, g maps G = R into itself.
Thus, by the the Contraction Mapping Theorem, the fixed
point method is convergent for any initial value x0 ∈ G = R.

50 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 3: g (x) = x2 + x1 . Can we prove that the fixed
point iteration converges for any initial value x0 ∈ G = [1, 2]?

51 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 3: g (x) = x2 + x1 . Can we prove that the fixed
point iteration converges for any initial value x0 ∈ G = [1, 2]?
Solution: First g is continuous except x = 0.

51 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

51 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

51 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

51 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Example 3: g (x) = x2 + x1 . Can we prove that the fixed
point iteration converges for any initial value x0 ∈ G = [1, 2]?
Solution: First g is continuous except x = 0.
Second, g 0 (x) = 12 − x12 . Since 14 ≤ x12 ≤ 1 on G = [1, 2], then
− 12 x12 ≤ 14 on G = [1, 2]. Hence |g 0 (x)| ≤ 21 < 1 on
G = [1, 2].
Then by Lemma (IV), g is a contraction on G = [1, 2] and
L = 12 .
Pick up ρ = 12 and c = 32 in Lemma (V). Then
3 3 1 1
|g (c) − c| = g ( ) − = ≤ = (1 − L)ρ. (18)
2 2 12 4
Then by Lemma (V), g maps G = [1, 2] into itself.

51 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Question 4: How fast do the numerical solutions {xk }
converge to that solution?

52 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Question 4: How fast do the numerical solutions {xk }
converge to that solution?

52 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Question 4: How fast do the numerical solutions {xk }
converge to that solution?

Definition (Convergence rate/order)

Let {xk } be a sequence convergent to x ∗ . If there exist constants
C and α and an integer N such that |xk+1 − x ∗ | ≤ C |xk − x ∗ |α
for all k ≥ N, then we say that the rate of convergence is of order
at least α. The rate is said to be linear if α = 1 with C < 1 and
quadratic if α = 2.

52 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Recall Question 4: How fast do the numerical solutions {xk }
converge to that solution?

Definition (Convergence rate/order)

Remark
Since we can get |xk+1 − z| = |g (xk ) − g (z)| ≤ L |xk − z| and
L < 1 from the Contract Mapping Theory, the fixed point method
has at least linear rate of convergence. But is it possible to have a
higher rate of convergence?
52 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Theorem (IV: Taylor’s Expansion)

Suppose that f ∈ C n+1 [a, b] and x0 ∈ [a, b]. Then for any
x ∈ [a, b], we have the following Taylor’s expansion of f (x) at x0 :

f (x) = Pn (x) + Rn (x), (19)

where
n
X 1 (k)
Pn (x) = f (x0 )(x − x0 )k
k!
k=0
1
= f (x0 ) + f 0 (x0 )(x − x0 ) + f 00 (x0 )(x − x0 )2 + · · ·
2
1 (n)
+ f (x0 )(x − x0 )n , (20)
n!

53 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Theorem (IV: Taylor’s Expansion(Continued))

1
Rn = f (n+1) (ξ)(x − x0 )n+1 (21)
(n + 1)!
for some ξ ∈ [x0 , x] (Lagrange form of the remainder) ,

or
1 x (n+1)
Z
Rn = f (s)(x − s)n ds (22)
n! x0
for some ξ ∈ [x0 , x] (Integral form of the remainder) .

54 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Theorem (V)
Assume that the iterations xk+1 = g (xk ) converge to a fixed point
z and g ∈ C q (G ) where G contains z. Furthermore, assume that q
is the first positive integer for which g (q) (z) 6= 0 and if q = 1 then
g 0 (z) < 1. Then the sequence {xk } converges to z with order q.

55 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Remark
Usually g 0 (z) 6= 0, hence q = 1.

It’s not easy to find g (q) (z) (q = 1, 2, 3, · · · ) since the

solution z is unknown.

55 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Proof:
Using xk+1 = g (xk ) and the Taylor’s expansion of g (xk ) at z,
for some ξk between xk and z, we have

|xk+1 − z| = |g (xk ) − z|
q
X 1 (k)
= g (z)(x − z)k − z
k!
k=0
= g (z) + g 0 (z)(xk − z) + · · ·
1
+ g (q−1) (ξk )(xk − z)(q−1)
(q − 1)!
1
+ g (q) (ξk )(xk − z)q − z
q!
1 (q)
= g (ξk ) |xk − z|q . (23)
q!
56 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
Hence

1 (q)
|xk+1 − z| ≤ max g (ξk ) |xk − z|q
ξk ∈G q!
= C |xk − z|q . (24)

Thus the sequence {xk } converges to z with order q.

57 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Convergence theory of the fixed point method

Continued proof:
Hence

1 (q)
|xk+1 − z| ≤ max g (ξk ) |xk − z|q
ξk ∈G q!
= C |xk − z|q . (24)

Thus the sequence {xk } converges to z with order q.

For linear convergence, we need C < 1. And the assumption

“if q = 1 then g 0 (z) < 1” guarantees C < 1 when q = 1.

57 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Outline

1 Introduction

2 Bisection method

3 The fixed point method

4 Newton’s method

5 Secant method

6 Other topics

58 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea and derivation of the Newton’s method

Still consider f (x) = 0. Why is it hard?

59 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea and derivation of the Newton’s method

Still consider f (x) = 0. Why is it hard?

Because f (x) could be complicated and nonlinear.

59 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea and derivation of the Newton’s method

Still consider f (x) = 0. Why is it hard?

Because f (x) could be complicated and nonlinear.

Any idea to approximate f (x) by using a simpler function?

59 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics

Basic idea and derivation of the Newton’s method

Still consider f (x) = 0. Why is it hard?

Because f (x) could be complicated and nonlinear.

Any idea to approximate f (x) by using a simpler function?

Linearization by using Taylor’s expansion!

59 / 85
Introduction Bisection method The fixed point method Newton’s method Secant method Other topics