Scribd
Upload
94 views

Direct and Iterative Methods For Solving Linear Systems of Equations

This document discusses methods for solving linear systems of equations of the form Ax=b. It covers both direct and iterative methods. Direct methods include LU decomposition using Gaussian elimination, which decomposes the matrix A into lower and upper triangular matrices L and U. This allows the system to be solved by solving two simpler triangular systems. The document also discusses matrix norms and condition numbers, which provide measures of sensitivity and error bounds when solving linear systems. Iterative methods covered include Jacobi, Gauss-Seidel, and other techniques. Applications to overdetermined, underdetermined and homogeneous systems are also mentioned.

Uploaded by

rohit2425
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
94 views

Direct and Iterative Methods For Solving Linear Systems of Equations

This document discusses methods for solving linear systems of equations of the form Ax=b. It covers both direct and iterative methods. Direct methods include LU decomposition using Gaussian elimination, which decomposes the matrix A into lower and upper triangular matrices L and U. This allows the system to be solved by solving two simpler triangular systems. The document also discusses matrix norms and condition numbers, which provide measures of sensitivity and error bounds when solving linear systems. Iterative methods covered include Jacobi, Gauss-Seidel, and other techniques. Applications to overdetermined, underdetermined and homogeneous systems are also mentioned.

Uploaded by

rohit2425
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

Direct and Iterative Methods for Solving Linear

Systems of Equations
Many practical problems could be reduced to solving a linear system of equations
formulated as Ax = b. This chapter discusses the computational issues about solving
Ax = b.
A Linear System of Equations
Vector and Matrix Norms
Matrix Condition Number (Cond(A) = A A1 )
Sensitity and Error Bounds
Direct Solution for Ax = b
LU-decomposition by Gaussian Elimination
Gaussian Elimination with Partial Pivoting
Crout Algorithm for A = LU
Cholesky Algorithm for A = LLt (A is positive denite)
Iterative Solutions
Jacobi method
Gauss-Seidel method
Other methods
Applications

Overdetermined, Underdetermined, Homogeneous


Systems
a11 x1 +

a12 x2 + +

a1n xn = b1

a21 x1 +

a22 x2 + +

a2n xn = b2

am1 x1 + am2 x2 + + amn xn = bm


Ax = b

Definition: A linear system is said to be overdetermined if there are more equations


than unknowns (m > n), underdetermined if m < n, homogeneous if bi = 0, 1 i m.

(A)

x +

y =

y =

x + y = 3
(B)

x + 2y = 2

x y = 1

x +
(C)

2x + y = 5

y =

2x + 2y =

y = 2

(A) has no solution, (B) has unique solution, (C) has innitely many solutions

x + 2y +

z = 1

x + 2y + z = 5

(D)

(E)
2x + 4y + 2z =

2x

y + z = 3

(D) has no solution, (E) has innitely many solutions

Vector and Matrix Norms


Definition: A vector norm on Rn is a function
: Rn R+ = {x 0| x R}
that satises
(1) (x) > 0 x = 0, (0) = 0
(2) (cx) = |c| (x) c R, x Rn
(3) (x + y) (x) + (y) x, y Rn
n

H
older norm (p-norm) xp = (
(p=1) x1 =

n

(p=2) x2 = (

i=1

n

i=1

|xi |p )1/p f or p 1.

|xi | (Mahattan or City-block distance)

i=1 |xi |

2 1/2

(Euclidean distance)

(p=) x = max1in {|xi |} (-norm)

Definition: A matrix norm on Rmn is a function


: Rmn R+ = {x 0| x R}
that satises
(1) (A) > 0 A = O, (O) = 0
(2) (cA) = |c| (A) c R, A Rmn
(3) (A + B) (A) + (B) A, B Rmn
Consistency Property: (AB) (A) (B) A, B
(a) (A) = max{|aij | | 1 i m, 1 j n}
(b) AF =


m n
i=1

j=1

a2ij

1/2

(Fr
obenius norm)

Subordinate Matrix Norm: A = maxx=0 {Ax/x}


(1) If A Rmn , then A1 = max1jn (
(2) If A Rmn , then A = max1im

m

i=1


n

|aij |)

j=1 |aij |

(3) Let A Rnn be real symmetric, then A2 = max1in |i |, where i (A)

Matrix Condition Number


Cond(A) = A A1 
For the matrix

A=

2 1 1
1

3 1 4

A1 =

0.5

1.5

0.5

0.5

2.5

0.5

0.5 0.5

Cond1 (A) = A1 A1 1 = 6 4.5 = 27


Cond (A) = A A1  = 8 3.5 = 28
Cond2 (A) = A2 A1 2 = 5.7229 3.0566 = 17.4930

0.5

Sensitivity and Error Bounds


respectively, where x
be the solutions to Ax = b and A
= x + x,
Let x and x
x = b,

b = b + b. Then
b = Ax A x or

x

b
A

x = A1 b A1  b


Thus

x
b
A
A1  b
Cond(A)
x
b
b

Similarly, suppose that (A + E)


x = b and Ax = b, we have
E
x
Cond(A)

x
A
Further derivations to get

x
b E
Cond(A)
+
x
b
A

A Direct Solution of Linear Systems


A linear system
2x +

y +

z =

4x 6y

= 2

2x + 7y + 2z =

A matrix representation

Ax = b, or

2
4
2

6 0
7

Solution using MATLAB

A = [2, 1, 2; 4, 6, 0; 2, 7, 2] ;

b = [5; 2; 9];

x = A\b (x = [1; 1; 2])

x
y
z

5
2
9

Elementary Row Operations


(1) Interchange two rows
(2) Multiply a row by a nonzero real number
(3) Replace a row by its sum with a multiple of another row

E1 =

0 1 0
1 0 0

E2 =

0 0 1

1 0

0 1

0 0 2

E3 =

1 0 0

0 1 0
1 0 1

Example

A=

6 0

4
2

E1 A =

6 0

4
2

E2 A =

4 14 4

E3 A =

L3 =

1 0 0
0 1 0
0 1 1

L2 =

1 0 0
0 1 0

L1 =

1 0 1

Then

L3 L2 L1 A =

0 0

2 1 0
0

0 8 2
0

A=

0 1

= U (Upper )

2
4
2

6 0
7

4 6 0

Let

LU-Decomposition
If a matrix A can be decomposed into the product of a unit lower- matrix L and
an upper- matrix U, then the linear system Ax = b can be written as LUx = b. The
problem is reduced to solving two simpler triangular systems Ly = b and Ux = y by
forward and back substitutions.

A=

6 0

4
2

L3 =

L3 L2 L1 A =

A=

0 1 0

L2 =

0 1 1

Then

where

1 0 0

2
4

6 0

0 1 0

L1 =

1 1
A = L1
1 L2 L3 U = LU

=U

1 1 1

0 8 2
0

= LU

B=

b=

1
0
5

x=

1
2
1

0 0

2 1 0

1 0 1

3 If b = [5, 2, 9]t , then y = [5, 12, 2]t and x = [1, 1, 2]t

0 8 2
0

1 0 0

0 1

Analysis of Gaussian Elimination


Algorithm
for i = 1, 2, , n 1
for k = i + 1, i + 2, , n
mki aki /aii if aii = 0
for j = i + 1, i + 2, , n
akj akj mki aij
endfor
endfor
endfor
The Worst Computational Complexity is O( 23 n3 )
1. # of divisions are (n 1) + (n 2) + + 1 =

n(n1)
2

2. # of multiplications are (n 1)2 + (n 2)2 + + 12 =


3. # of subtractions are (n 1)2 + (n 2)2 + + 12 =

10

n(n1)(2n1)
6

n(n1)(2n1)
6

Gaussian Elimination with partial Pivoting


Algorithm
for i = 1, 2, , n
p(i)=i
endfor
for i = 1, 2, , n 1
(a) select a pivotal element ap(j),i such that |ap(j),i | = max1kn |ap(k),i |
(b) p(i) p(j)
(c) for k = i + 1, i + 2, , n
mp(k),i = ap(k),i /ap(i),i
for j = i + 1, i + 2, , n
ap(k),j = ap(k),j mp(k),i ap(i),j
endfor
endfor
endfor
An example

A=

2 5

1 0

P23 P13 A = LU =

11

1 0 0

1
2

2 5

9
2

Iterative Methods for Solving Linear Systems


1. Iterative methods are msot useful in solving large sparse system.
2. One advantage is that the iterative methods may not require any extra storage and
hence are more practical.
3. One disadvantage is that after solving Ax = b1 , one must start over again from
the beginning in order to solve Ax = b2 .
Jacobi Method
Given Ax = b, write A = C M, where C is nonsingular and easily invertible. Then
Ax = b (C M)x = b Cx = Mx + b
x = C 1 Mx + C 1 b x = Bx + c, where
B = C 1 M, c = C 1 b
Suppose we start with an inital x(0) , then
x(1) = Bx(0) + c and x(k+1) = Bx(k) + c

12

Jacobi Iterative Method for Solving Linear Systems


Suppose x is a solution to Ax = b, then
x(1) x = (Bx(0) + c) (Bx + c) = B(x(0) x)
x(k) x = B k (x(0) x)
x(k) x B k  x(0) x Bk x(0) x
x(k) x 0 as k if B = C 1 M < 1.
For simplest computations, 1-norm or -norm is used.
The simplest choice of C, M with A = C M is C = Diag(A), M = (A C).
Theorem: Let x(0) Rn be arbitrary and dene x(k+1) = Bx(k) + c for k = 0, 1, If
x is a solution to Ax = b, then the necessary and sucient condition for x(k) x is
B < 1.


Theorem: If A is diagonally dominant, i.e., |aii | >


A = C M and B = C 1 M, then

n


B = max1in

Example:

A=

10

C=

Then
B=C

M=

(0)

= Bx

+c=

= max1in

b=

10

M =

0.1

0
0.2
1.1
1.2

11
12

10


(1)

|bij |

10

j=1

2
Let

|aij | for 1 i n. Let

j=i

j=i


(0)

, x

0
0

|aij |
< 1
|aii |

c=C

b=


(2)

, x

13

0.98
0.98

1.1
1.2


(3)

, x

1.002
1.004

Gauss-Seidel Method
Given Ax = b, write A = C M, where C is nonsingular and easily invertible.
Jacobi: C = diag(a11 , a22 , , ann ), M = (A C)
Gauss-Seidel: A = (DL)U = C M, where C = DL, M = U for Jacobi

Let x(0) Rn be nonzero, then Cx(k+1) = Mx(k) + b implies that


(D L)x(k+1) = Ux(k) + b
Dx(k+1) = Lx(k+1) + Ux(k) + b
(k+1)

x1

(k+1)

xi

1
a11

1
aii




n

j=2

i1

(k)

a1j xj + b1

(k+1)
j=1 aij xj

n

j=i+1

(k)

aij xj + bi , f or i = 2, 3, , n

The dierence between Jacobi and Gauss-Seidel iteration is that in the latter case, one
is using the coordinates of x(k+1) as soon as they are calculated rather than in the next
iteration. The program for Gauss-Seidel is much simpler.

14

Convergence of Gauss-Seidel Iterations




Theorem: If A is diagonally dominant, i.e., |aii | > j=i |aij | for 1 i n. Then
Gauss-Seidel iteration converges to a solution of Ax = b.
Proof: Denote A = (D L) U and let
j =

j1

i=1

|aji|, j =

n


|aji|, and Rj =

i=j+1

j
|ajj | j

Since A is diagonally dominant, then |ajj | > j + j , thus


Rj =

j
|ajj | j
<
=1 1jn
|ajj | j
|ajj | j

Therefore, R = Max1jn Rj < 1.


The remaining problem is to show that B = Maxx =1 Bx R < 1, where
B = C 1 M = (D L)1 U.
Let x = 1 and y = Bx, then y = Max1in |yi | = |yk | for some k.
Then
y = Bx = (D L)1 Ux
(D L)y = Ux Dy = Ly + Ux
y = D 1 (Ly + Ux)
Then
yk =

1
akk

k1
i=1

y = |yk |

aki yi

1
|akk |

n

i=k+1 aki xi

(k y + k x )

which implies that for all x with x = 1,


Bx = y

k
|akk |k

Thus, B = Maxx =1 Bx R < 1

15

= Rk < 1

Example for Gauss-Seidel Iterations


Example:

10

A=
2

10

10

b=

11
12

2 0

, x

0 1
0


(0)

0
0

(0)
1
(a12 x2
10

+ b1 ) = 1.1

(1)

(1)
1
(a21 x1
10

0 + b2 ) = 0.98

x2 =
(2)

(1)
1
(a12 x2
10

+ b1 ) = 1.002

(2)

(2)
1
(a21 x1
10

0 + b2 ) = 0.9996

x1 =
x2 =
Thus

(1)

x1 =

Moreover,

10


Then

x(1) = [1.1, 0.98]t


x(2) = [1.002, 0.9996]t
x(3) = [1.00004, 0.99992]t

16

= DLU

You might also like