Matrix Inversion and Solution of General Linear

Equations
Gaussian Elimination and the LU Decompostion

Dr. M. Bikdash

Department of Computational Science and Engineering

North Carolina A&T State University

September 20, 2020

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 1 / 43

By the end of this Unit, students will be able to:

De…ne the matrix inverse and state its basic properties

Understand the limitations of the matrix inverse (in existence and
uniqueness),
Understand/Avoid cancellations, divisors of zero, multiplying by a singular
matrix, and computing the inverse
Use various tests for matrix existence
Become familiar with Gaussian Elimination
Perform it manually, in Matlab, and code it
keep track of operations and counting their numbers
apply it to several RHS
Place the parentheses correctly!! which ordering of the operations is best?

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 2 / 43

Introduction

We cannot divide two matrices like B/A if A is a matrix and we must use
BA 1 where A 1 is the inverse (if it exists)
The matrix inverse is a complicated concept because A can behave like a zero
even when it is not.
Understanding the existence and uniqueness of the inverse is crucial and it
can be tested using many major concepts in matrix theory
Like: determinant, rank, linear independence, eigenvalues, singular values, etc.
The matrix inverse is a useful mathematical notation but it is not useful from
a computational point of view.
Instead, use Gaussian elimination to solve the system Ax = B directly

Basic Idea: Manipulate the matrix zero pattern until

you reach an upper triangular matrix or better

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 3 / 43

Additional Resources

The inverse of a 2x2 matrix can be computed easily with this trick:
https://www.khanacademy.org/math/precalculus/x9e81a4f98389efdf:matrices/
x9e81a4f98389efdf:…nding-inverse-matrix-with-determinant/v/inverse-of-a-
2x2-matrix
Wikipedia has an extensive article on the matrix inverse
https://en.wikipedia.org/wiki/Invertible_matrix#Blockwise_inversion
In particular, read this mega-theorem which relates the inverse to almost
every other concept in matrix theory and linear algebra:
https://en.wikipedia.org/wiki/Invertible_matrix#The_invertible_matrix_theor
Very quick and useful properties are here:
https://en.wikipedia.org/wiki/Invertible_matrix#Other_properties

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 4 / 43

Overview of Solving Equations and Matrix Inversion

In the scalar case In the square matrix case

the solution of ax = b is the solution of Ax = b can be
x = b/a or even better written x = A 1 b if the inverse
x = a 1b A 1 exists
The inverse a 1 exists if a 6= 0 The inverse of A need not exist
even if A 6= 0

This leads to some puzzling facts of matrix algebra

Typically, computing the inverse is wasteful and should be avoided
Essentially, we solve Ax = b directly using Gaussian Elimination
Will consider tricks of "using" the inverse matrix without computing it
We will recall the de…nition of a FLoating-point OPeration (FLOP)1
We count FLOPs for many algorithms

1 The plural of FLOP must be FLOPs, which is often confused with FLOPS (Floating-Point

Opearation per Second), a FLOP rate.

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 5 / 43
Identity Matrices
The n n identity matrix In , or simply I , is a square matrix with ones on the
main diagonal and zeros everywhere else.
2 3
2 3 1 0 0 0
1 0 0 6 7
1 0 4 0 1 0 5 , 6 0 1 0 0 7,
1 , , 4
0 1 0 0 1 0 5
| {z } 0 0 1
| {z } 0 0 0 1
the 2 2 identity
the 3 3 identity
I2
I3

It is a collective name for matrices matri(ces) is denoted e 2 :

of di¤erent dimensions 2 3
Sometimes denoted as E , or E, or 2 3 0
0 6 1 7
1, etc. 0
, 4 1 5, 6 7
4 0 5,
1
Its n columns will be denoted as | {z } 0
| {z } 0
e1, , en e 2 of I2 | {z }
e 2 of I
3
The second column of the identity e 2 of I4

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 6 / 43

Matrix Inverse–De…nition

De…nition
A square n n matrix A has an n n inverse B if

AB = BA = I .

In this case, A is said to be nonsingular or invertible.

A square matrix that does not possess an inverse is called singular.

A matrix A that does not have an inverse is noninvertible (whether square
or not)
Must satisfy both conditions AB = I and BA = I
Counter Example: Assume that C 1 = D and hence CD = DC = I . Then

D
C 0 = CD + 0 0=I Identity matrix. Good!
0
D DC 0 I 0
C 0 = = not an identity matrix!!!
0 0 0 0 0

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 7 / 43

Theorem of inverse of transpose and product
Theorem
Let A, B, be all n n invertible matrices. Then
1 1
(a) A = A.
(b) The inverse matrix, if it exists, is unique.
1 T
(c) AT = A 1 or the inverse of the transpose is the transpose of the
inverse.
(d) If A is symmetric (or Hermitian) then A 1 is symmetric (or Hermitian)
(e) (AB ) 1 = B 1 A 1 or the inverse of the product is the product of the
inverses read backwards
Proof of a. If B is the inverse of A then AB = BA = I which, by de…nition,
means that A is the inverse of B.
Proof of b If A has two distinct inverses X and Y , then AX = XA = I and
AY = YA = I .
Subtracting the two sets of equations yields
A (Y X ) = (Y X ) A = 0,
and since A is invertible, then X = Y .
M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 8 / 43
Proof of (c): Inverse of Transpose
The proof is messy but simple. It requires mastering the EIN.
Let X = A 1 and Y = B 1 . Now if B = AT then we need to show that
Y = X T.
In EIN bjk = akj () B = AT . Now

yij bjk = bjk yki = δki (EIN form of YB = BY = I )

yij akj = akj yki = δki (substitute bjk = akj )
akj yij = yki akj = δki (rearrange)
akj zji = zik akj = δki (de…ne yij = zji () Y T = Z )

Switching back to matrix notation: AZ = ZA = I =)

T
A 1 = Z = YT = B 1
(use de…nitions) (1)
T T T 1
A 1 = B 1 = B 1 = AT (2)

(d) follows from (c) immediately.

(e) can be shown using a development like that of (c)–Left as exercise!
M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 9 / 43
Divisors of zero
Also note that when it comes to matrices one can have the divisors of zero;
in other words, matrices that are not zero, but whose multiplication leads to
a zero matrix. For Example:

1
1 1 = 0
|{z} (3)
| {z } 1
nonzero E
| {z } zero matrix
nonzero F

and E and F are divisors of zero.

A matrix such as E can also be called a pseudo-zero matrix, or simply, a
pseudozero,
because it is not zero but it can behave like a zero sometimes.
Pseudozeros are apparently paradoxical because we are used to scalar algebra:

scalars: ab = 0 implies that either a or b is zero (4)

matrices: AB = 0 does not imply that either A or B is zero (5)

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 10 / 43

Other (apparently paradoxical) inverse matrix relations

For instance, cancellation of matrix factors on both sides of an equation

scalars : ab = ac AND a 6= 0 implies that b = c (6)

matrices : AB = AC AND A 6= 0 does not imply that B = C (7)

Another apparent paradox starts with a system of two equations in one

unknown, which does not have a solution. For instance

1 2
Fx = x= with x 2 R (8)
1 2.1

Premultiplying the equation by the pseudo-zero G = 1/2 1/2 leads to

the scalar equation x = 1.05 2.05
x = 1.05 is a very poor approximate solution
This is because G (and also F) is a pseudo-zero.
Recall multiplying both sides of a false equation (such as 2 = 3) with zero
turns it into a true equation (0 = 0).

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 11 / 43

It is safe to multiply with a nonsingular matrix
It is NOT safe to multiply with a singular or noninvertible matrix

Rule of Thumb
It is safe to premultiply (or postmultiply) by an invertible matrix,

If A 1 exists, it is safe to premultiply AB = ACD by A 1 to get

A 1 AB = A 1 ACD or B = CD
Cancelling an invertible matrix is OK

Rule of Thumb
Premultiplying by a singular matrix is dangerous!!

is true but
1 1 1 1 1 2 1 2
= does not imply =
1 1 1 1 1 2 1 2
| {z } that
singular

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 12 / 43

Pitfalls to Remember

In general, only invertible matrices can be used reliably to conduct typical

algebraic operations.
Noninvertible matrices should be used with care.
Non-square matrices and some square (singular) matrices can behave like a
zero matrix, even though these matrices are not zero.
A pseudozero matrix should not be canceled in an equation
Premultiplying both sides of an equation with such a matrix can create
solutions when no solution exists.
Pseudozeros can have pseudoinverses. These are matrices that behave like
inverses, but only sometimes.

Pseudo-inverses can have some of the properties of the inverse

but not all

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 13 / 43

Nilpotent matrices *

Nonzero divisors of a zero matrix =) there are also nonzero roots of the zero
matrix.
De…nition. An n n matrix that satis…es the conditions M h = 0 and
M h 1 6= 0, h n is called a nilpotent matrix M of index h.
If M 3 = 0 but M 2 6= 0 then M is a cubic root of the zero matrix.
These matrices are crucial to understanding the eigenvalue problem and
singular systems of di¤erential equations.
Theorem. A k–upper triangular matrix A is nilpotent of index n + 1 k.
Proof. (sketch). Every time we premultiply by A we lose one superdiagonal:
2 3 2 3
0 a b 0 0 ac
A = 4 0 0 c 5 ) A2 = 4 0 0 0 5 ) A3 = 0. (9)
0 0 0 0 0 0

If a = c = 1 above, A2 6= 0, and A3 = 0. Therefore this matrix has index 3. Now

if a = 0, and b = 1 then A2 = 0 and hence A will have index 2.

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 14 / 43

Nilpotent Matrices can look full –Example
Exercise. Show that M is a nilpotent matrix.
2 3
9 2 1 3
6 100 2 48 66 7
M=6 4 102
7
4 2 34 5
27 6 3 9
What is the order of nilpotency?

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 15 / 43

Mega-Theorem on Existence of the Inverse
Theorem
Given an n n matrix A, the following statements are equivalent
(a) A is invertible;
(b) det A 6= 0.
(c) A has n linearly independent rows and n linearly independent columns.
(d) All the rows of the matrix U in the LU decomposition of A are nonzero.
(e) All the rows of the matrix R in the QR decomposition of A are nonzero.
(f) A does not have a zero eigenvalue.
(g) A does not have a zero singular value.
(h) The null-space of A is trivial.
(i) The system of equations Ax = b has a unique solution.

Some concepts have not been covered yet

The proofs are intricate and deep.
The inverse touches all aspects of matrix theory.

Self-study: You need to fully understand this theorem, eventually

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 16 / 43

Important: Do Not Compute the Inverse Explicitly
(unless it is clearly advantageous or unavoidable)

Before computing the inverse or pseudo-inverse of a matrix one should always

ask whether such computation is really necessary.
Typically it is not necessary, and the extra e¤ort needed to compute and store
the inverse is not justi…ed.
The solution of the matrix equation Ax = b, if it exists and is unique, can be
formally written as x = A 1 b,
but computing the solution directly, and without computing the inverse is
much preferred.

Do Not compute the matrix inverse explicitly

unless it is unavoidable or clearly advantageous
When in doubt use Gaussian Elimination

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 17 / 43

Gaussian Elimination–Lower Triangular Case
Custom tailored solution (like it should be)

2 3 1 2 3
2 0 0 1/2 0 0 Inverse of lower
4 4 3 0 5 = 4 2/3 1/3 0 5 triangular is (quasi) lower
2 2 1 1/3 2/3 1 triangular. Why??

Put A and I or B side by side zero the elements below a11

2 0 0 1 0 0 R2 R2 -(4/3)*R1
4 3 0 0 1 0 R3 R3 -(2) *R1
2 2 1 0 0 1 1 0 0 1/2 0 0
Scale every pivot !unit matrix A 0 1 0 2/3 1/3 0
(i.e. ones on diagonal) 0 2 1 1 0 1
Keep Track of the operations zero the elements below a22
R1 (1/2)*R1 R3 R3 -(2) *R2
R2 (1/3)*R2 1 0 0 1/2 0 0
1 0 0 1/2 0 0 0 1 0 2/3 1/3 0
4/3 1 0 0 1/3 0 0 0 1 1/3 2/3 1
2 2 1 0 0 1
M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 18 / 43
FLOPs for inverse of lower triangular matrix)
(A=Add, M=Multiply, D=Divide, MA=Multiply-Add)

Scale every pivot To solve Ax = y , one must

A side: 1 D in …rst row, 2 D in compute x = A 1 y = By
second row, etc. (Typically n2 MAs)
I side: 1 1/pivot. n
Divisions. Since B is lower triangular
Zero elements under k th scaled Computing xi = ∑ik =1 bik yk
pivot akk = 1 for k = 1, , requires i MAs
n 1 Total is
A side: overwrite. No FLOPs 1+ + n = n (n + 1) /2 MAs
I side: replace
0 = bik aik bkk for Grand total for A 1 y = n D +
i = k + 1, , n. Needs n k n (n 1) /2 M + n (n + 1) /2 M
Ms for every k. + n (n + 1) /2 A
but (n 1) + (n 2) + +1 =
or n D + n 2 M + n (n + 1) /2 A
n (n 1) /2
Computing B requires n Ds and or 3n 2 /2 + 3n/2 FLOPs
n (n 1) /2 Ms
M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 19 / 43
Forward Substitution=Lower Triangular Case
Custom tailored (like it should be)

2 32 3 2 3
2 0 0 x1 1 Solve the system
4 4 3 0 5 4 x2 5 = 4 2 5 directly. No need to
2 2 1 x3 3 compute the inverse!!

Write the equations x3 = (3 2x1 2x2 ) /2

2x1 = 1 Eq. 1 = (3 2 1/2 0) /2 = 1
4x1 + 3x2 = 2 Eq. 2 In General, for k = 1, ,n
2x1 + 2x2 + x3 = 3 Eq. 3 !
k 1
First equation has 1 unknown.
Solve: x1 = 1/2
xk = yk ∑ akj xj /akk
j =1
With x1 now known, solve Eq. 2 for x2 :
x2 = (2 4x1 ) /3 Needs n Ds, and
= (2 4 1/2) /3 = 0 1+ + (n 1) = n (n 1) /2 MAs
Given x1 , x2 , solve Eq. 3 for x3 :
FLOP Count: n2 /2 + n/2 FLOPs

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 20 / 43

Was computing the inverse necessary?

For solving Ax = b where A is lower triangular:

Computing the solution using Forward Substitution requires n2 /2 + n/2 FLOPs
via GE
Computing the inverse of a lower triangular matrix requires 3n2 /2 + 3n/2 FLOPs
via A 1

Computing the inverse then the solution was totally unnecessary!

It costs n2 more FLOPs

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 21 / 43

Gaussian Elimination–General Case
The key is to keep track of operations

0x1 + 2x1 + 3x2 = 5 Zero the element below A(1,1)

4x1 x2 + 2x3 = 5 R3 R3+( A(3,1)/A(1,1))*R1
2x1 + 0x2 + 2x3 = 4 | {z }
multiplier= 1 /2
Write matrix equivalent 4 1 2 5
0 2 3 5 0 2 3 5
4 1 2 5 0 1/2 1 3/2
2 0 2 4
The (1,1) pivot is zero. Zero the element below A(2,2)
Need pivoting: swap rows 1 and 2 R3 R3 A(3,2)/A(2,2)*R2
| {z }
R1 $ R2 (Keeping track) multiplier= -1 /4
4 1 2 5 4 1 2 5
0 2 3 5 0 2 3 5
2 0 2 4 0 0 1/4 1/4

Yields an Upper-Triangular System: Need Backward Substitution to get x

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 22 / 43

Gaussian Elimination – The Main Algorithm
The matrix A undergoes a series of transformations: A = A(0 ) ! A(1 )
! A(2 ) ! and ends up as U,an upper triangular matrix
A denotes the latest value of A. After transformation i = 4, A denotes A(4 ) .

The operations have the form: When working with pivot A(k,k),
R1 $ R_i1 (if needed) j>k
R2 R2-A(2,1)/A(2,1)*R1
R3 R3-A(3,1)/A(1,1)*R1 A(k:end, 1:k-1) are already zero
.. Rk really denotes A(k,k+1:end)
.
R2 $ R_i2 (if needed) Rj really denotes A(j,k+1:end)
R3 R3-A(3,2)/A(2,2)*R2
The multiplier -A(j,k)/A(k,k)
R4 R4-A(4,2)/A(2,2)*R2
.. can be stored in A(k,j) which is
. supposedly being zeroed.
R3 $ R_i3 (if needed)
Need to store the swaps: start with
R4 R4-A(4,3)/A(3,3)*R3
.. p=[1,2,...,n] and swap as
. needed p(1)$ p(i1), ...

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 23 / 43

Pseudo-Code for Naive Gaussian Elimination

Algorithm
Naive pseudo-Code for Gauss Elimination

initialize U=A and bn=b

for k = 1 : n-1 % main loop
select kp to maximize jU (kp,k)j
swap row k with row kp: U(k,:)$U(kp,:) and bn(k)$bn(kp)
for i = k+1:n % for rows below the pivot
compute the multiplier m = -U(i,k)/U(k,k)
update U(i,:) U(i,:)+m*U(k,:)
update bn(i) bn(i)+m*bn(k)

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 24 / 43

The PLU Decomposition – Matlab Code
For an nxn nonsingular matrix – Pivoting

Algorithm
Input is an n n nonsingular matrix A.
Output is a permutation vector p, and a matrix W=U+(L-I) which can overwrite A
The upper triangular matrix U can be found as triu(W)
The unit lower triangular matrix L can be found as
L=(tril(W)-diag(diag(W))+eye(n)). It stores the multipliers.
The permutation matrix P can be found as In=eye(n); P=In(p,:).

for k = 1 : n-1 % main loop

[pv, dkp] = max(abs(W(k:end,k))); kp = k+dkp-1 ; % the row of new pivot in W
pt = p(k); p(k) = p(kp) ; p(kp) = pt ; % update the permutation vector
ut = W(k,k:end); W(k,k:end)=W(kp,k:end) ; W(kp,k:end)= ut ; % swap rows of U
lt = W(k,1:k-1); W(k,1:k-1)=W(kp,1:k-1) ; W(kp,1:k-1)= lt ; % swap rows of L
for i = k+1:n % loop over rows below the kth pivot
W(i,k) = W(i,k)/W(k,k); % -1* kth multiplier
W(i,k+1:n) = W(i,k+1:n)-W(i,k)*W(k,k+1:n); % update elements of U
end
end
M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 25 / 43
Gaussian Elimination – FLOP count

Finding the largest element in A(k:n,k) requires sorting n k numbers but

no FLOPs
Swapping 2 rows of length n k + 1. No FLOPs.
Zeroing n k elements under the k th pivot A(k,k).
To zero one element, need 1 division to compute the multiplier
and (n k ) MA to add the row A(k,k+1:n) to a row below it.
To zero all elements below k th pivot, need (n k ) divisions and (n k )2 MAs
To complete the triangularization, sum the above for k = 1 : n 1

(2n 1 ) n (n 1) n3 n2 n
(n 1 )2 + + 12 = = + MAs
6 3 2 3

2n 3
Reduction to Upper-Triangular Form requires O 3 FLOPs

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 26 / 43

Back Substitution
Same as Forward Substitution but upside down

Very similar to forward substitution

Given an upper triangular matrix, solve last equation for xn (1 eq. in 1
unknown)
With xn known, solve the equation above for xn 1 ,etc...

function x = backsubstitution(A, b)
[n,m] = size(A) ; % m rows, n columns
% ... diagonstics skipped ...
x(n) = b(n) / A(n,n) ; % start at the bottom
for i = ( n-1 : -1 :1) % loop over the rows up
c = 1 / A(n,n) ;
x(i) = ( b(i) - A(i,i+1:n)*x(i+1:n) )* c ;
% use c because multiplication is more efficient than division
end

This algorithm needs only n 1 divisions, plus

1+2+ + (n 1) = (n 1) n/2 = O n2 multiply-adds.

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 27 / 43

Matlab Commands for Gaussian Elimination (LU and n)
mldivide or backslash n solves the system using GE with partial pivoting
and returns the answer
A = [0 2 3; 4 -1 2; 2 0 2] % specify A by rows
y = [5, 5, 4]’ % make into a column
x = A n y % returns column [1; 1; 1]

The lu command provides more information; Namely: PA = LU where

P is a permutation matrix, satisfying P 1 = P T .
L is unit lower triangular (ones on the main diagonal)
U is upper triangular and is the result of the GE
2 3 2 3 2 3
1 0 0 3 2 2 0 0 1
L=4 0 1 0 5, U = 4 0 4 2 5, P = 4 1 0 0 5
2/3 7/12 1 0 0 1/6 0 1 0
The PLU decomposition means that the GE of
2 32 3 2 3
0 0 1 0 2 3 2 0 2
PA = 4 1 0 0 5 4 4 1 2 5=4 0 2 3 5
0 1 0 2 0 2 4 1 2
proceeds without pivoting (row swapping) to lead to U
M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 28 / 43
The inverse as notation
Computational vs. mathematical equivalence

Above examples showed the essence of Gaussian Elimination in primitive form

For lower triangular: No need to pivot
Pivoting is swapping rows to ensure that the pivot/diagonal elements are
nonzero or large
If a pivot was zero, then lower-triangular A would be singular (det A = 0.
Why?)
The trend is the same: Because matrix multiplication is very expensive, there
is no need to compute the inverse.

Computational POV: x = A 1 y means "Solve Ax = y for x" (10)

Keep this in mind when you see:

A 1 (I + cvv T ) 1 (ABCy ) or B A 1 + B 1 (C + A 1 )DAy
1
or C 1 (A 1 + B )(A + B 1 )Cy or A5 y
Conclusion: keep manipulating the expression until you get the most
computationally e¢ cient form
M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 29 / 43
Computational vs. mathematical equivalence
The art of placing parentheses

The expressions ABCx and (AB ) (Cx ) and (A (BC )) x and A (B (Cx )) are
mathematically equivalent but not computationally equivalent
Assume A, B , C , are n n matrices and x , y , are n 1 vectors
Matrix-vector multiplication is O n 2
Matrix-matrix multiplication is O (n 3 )

(AB ) (Cx ) (A (BC )) x A (B (Cx ))

(1 MM + 1 MV) + 1 MV 1 MM + 1 MM + 1 MV 1 MV + 1 MV +1MV
O n3 + 2n2 O 2n3 + n2 O 3n2

Let us look in detail at (AB )(Cx )

The parentheses must be computed …rst.
(AB ) needs 1 MM for O n 3
(Cx ) needs 1 MV for O n 2
Now Multiply the matrix (AB ) with the vector (Cx ) for 1MV which is O n2

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 30 / 43

Placing the parentheses and Gaussian Elimination

Gaussian Elimination solution is O n3 /3 + n2 /2

Matrix inversion then MV multiplication is O (2n3 /3)

A(B 1 ) (Cx ) A B 1C x A B 1 (Cx )

(1 MI +1 MM) + 1MV (1 MI +1 MM)+1 1 MV + 1 GE + 1 MV
+1MV MM+1 MV O 2n3 /3 + 2n2
O 2n3 /3 + n3 + 2n2 = O 2n3 /3 + 2n3 + n2 =
O (5n3 /3 + 2n2 ) O (8n3 /3 + n2 )

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 31 / 43

Practice grouping of matrix equations
Exercise. Find the most-and least- e¢ cient computational implementations of
the following expressions, and discuss their computational costs. Capital letters
refer to n n matrices and small letters refer to n 1 columns. Assume that
matrices will be needed later and hence should not be destroyed.
(a) A 1 (I + cvv T ) 1 (ABCy )
(b) A 1 + B 1 (C + A 1 )DAy
(c) C 1 (A 1 + B )(A + B 1 )Cy
1
(d) A5 y

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 32 / 43

The LU Decompostion is very useful

Once the LU decomposition 5 …nd a basis for the columns or

PA = LU is computed in rows of A
O n3 /3 FLOPs, one can 6 …nd a basis for the null-space of
1 compute the product Ax , A
x 2 Rn 7 …nd (solvability) conditions on
2 compute the solution x of the vectors y for which Ax = b
Ax = y has a solution
3 compute the solution x of 8 understand which columns of A
AT x = y and the product x T A are linearly independent on other
4 …nd the determinant of A columns and how

Self-study: You need to fully understand this list, eventually

When you get stuck in solving a problem in linear algebra,

think about the LU Decomposition

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 33 / 43

Solving Equations using the PLU

To solve AT x = b for any RHS b

To solve Ax = b for any RHS b using PA = LU

Since P 1 = P T and is invertible,

multiplying by P is safe Rewrite it as AT P T P x = b, since
Form PAx = Pb, PTP = I
Hence LUx = Pb. Let z = Ux. Rearrange it into AT P T (Px ) = b
Then Lz = Pb. Solve for z using
forward substitution, in O n2 /2 then use AT P T = (PA)T = U T LT
operations Obtain U T LT Px = b.
With z known, solve Ux = z using De…ne z = LT Px and solve the lower
backsubstitution in O n2 /2 triangular system U T z = b
Therefore, Ax = b can be solved for Solve LT (Px ) = z for Px using
x using O n2 operations backward substitution. Then
Only O (n ) additional storage is x = P T (Px )
needed for z and x O n2 /2 + n2 /2 = O n2 .

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 34 / 43

Compute the Determinant
subt

To …nd det A note that det (P ) = 1 if P is an odd permutation and +1 if it

is even. Then
T
det A = det
| {zP } det
|{z}L det
| {zU}
=det P =1 =∏ u ii

L is unit lower triangular =) det L = 1 1 = 1 (product of diagonal

elements)
U is upper triangular =) det U = u11 unn using n multiplications
Total FLOPs to compute the determinant = O n 3 /3 to compute the LU +
n multiplications
If U has a zero diagonal element =) det A = 0 and A is singular

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 35 / 43

The LU Decomposition–Finding a null vector

If the last row of U is zero =) there is a vector v such that Av = 0

Ū u v1
PTL = 0 =) Ūv1 + βu = 0
0 0 β

The null vector v has the form

Ū 1 u
β for any β 6= 0
1

To compute k th column y of A 1 solve Ay = e k where e k is the k th column

of the identity matrix.
It requires O n 2 /2 operations given the LU Decomposition

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 36 / 43

Pivoting is tricky *

The choice of an alternative pivot is tricky.

Note for instance that what may …rst appear as an excellent may actually be
poor. 2 3
0.1 2 200
A=4 5 9 12 5
10 90 3000
The element a11 = 0.1 is obviously a poor choice.
The largest element in the …rst column is a31 = 10.
The best choice however is actually a21 = 5.
To see this, we recall that rows of A correspond to equations, and scaling rows
corresponds to scaling equations.

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 37 / 43

Scaling the rows *

We scale the rows of A so that the elements in the …rst column are all ones:
2 32 3 2 3
10 0 0 0.1 2 200 1 20 2000
4 0 2/10 0 54 5 9 12 5 = 4 1 1.8 2.4 5 = Ā
0 0 1/10 10 90 3000 1 9 300
| {z }| {z }
S 1 (0.1 )S 2 (0.2 )S 3 (0.1 ) A
(11)
Recall that since the pivot’s row will have to be subtracted from the other
row, thus "disturbing" them.
It is obvious that the …rst row is a poor choice, because the element
ā13 = 2000 will overpower ā23 = 2.4 thus endangering the accuracy of the
solution.
Similarly, the third row is a better choice because ā33 = 300 still overpowers
ā23 = 2.4. The best choice for a pivot row is the second row, as it will disturb
the other rows slightly.

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 38 / 43

Pivoting can add signi…cant computational cost **

In general, the best pivot alternative in the submatrix In the interest of

accuracy, one must use as a pivot the entry with the highest absolute value
relative to its row.
For partial pivoting (where column exchanges are not allowed), one chooses
as the k th pivot the element in Ā (k : n, k ) that is largest relative to its row.
This is achieved in practice by computing the ratios Ā (j, k ) / jjĀ (j, k : n )jj
for some acceptable norm jj jj , where Ā (j, k : n ) denotes the j th row. The
simplest norm to compute in this context and is quite satisfactory is the
1-norm jjx jj1 = ∑ jxi j , which is easily computed using O (n ) additions.
To …nd the best pivot at the k th iteration one must therefore compute
O (n k ) ratios, and the overall cost of partial pivoting is O n2 /2 divisions
and O n3 /3 additions.

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 39 / 43

Row and Column Pivoting
Permuting rows of A is equivalent to permuting equations.
Permuting columns is equivalent to reordering the unknown variables.
This is achieved by scanning  (k, k : n ) to the right of the desired pivot
location.
Once a suitable number  (k, j ) is found, the k th and the j th columns are
exchanged, and elimination can proceed further as usual ÂQ kj where Q kj is a
permutation matrix obtained by exchanging the k th and j th columns of the
identity matrix. This yields

LU = AQ, with Q = Q 1j1 Q n 1,jn 1

If the desired row and/or column permutations are known in advance, they
can be all collected in two permutation matrices P and Q and applied to A
at the onset.
This result in the full pivoting LU decomposition

LU = PAQ, where P and Q are permutation matrices. (12)

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 40 / 43

Full Pivoting**
For best accuracy, full pivoting can be used, in which the element with the
largest absolute value in the submatrix Acurrent(k:end,k:end) is chosen
as the new k th pivot.
This element is typically expensive to locate and full pivoting and is rarely
implemented.
Mathematically, the k th full pivoting operation can be written as Pk  Qk
where Pk and Qk are permutation matrices.
For total pivoting, one chooses as the k th pivot the element in Ā (k : n, k : n )
that is largest relative to its row. This will require a total of O n3 /3
divisions which adds considerably to the overall cost of the LU decomposition.
In general, preconditioning of A can be thought as …nding a matrix S whose
inverse S 1 can be computed fast (say in O (n ) operations) such that S 1
approximates A 1 .
In other words, assume that A is dominated by S meaning that one can write
A as A = S + T where jjS jj jjT jj . Then S 1 A = I + S 1 T and hence
S T 1 1 = jjI jj . For sparse matrix applications, it is also required
that S 1 T remains sparse.
M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 41 / 43
Use Gaussian Elimination to Compute the Inverse

Solution. The solution (using Gaussian elimination) of Ax = b can be computed

TM
in MATLAB using the n command. In other words, use Anb or Anb. For
instance, consider the 4 4 nonsingular matrix.
A = [ 9 8 8 9; 2 7 4 7; 4 0 7 4; 6 4 6 1]

if b T = [1, 3, 2, 4] then the solution of Ax = b is given by

b = [1 ; 3; 2; 4] , x=Anb

will give the solution

[-0.8573; 0.6995; 1.1737; -0.6967].

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 42 / 43

Matlab’s Gaussian Elimination– the n command

Let A be an n n nonsingular matrix. The i th column of A 1 is the solution

of Ax = ei where ei is the i th column of the n n identity matrix.
In MATLAB: An[0;1;0;0] is the solution of Ax = e2 .
This x should be equal to the second column of A 1 .
To verify: B=inv(A) ; then B(:,2)-An[0;1;0;0] is a very small vector
of the order of 10 16 .
Note that the MATLAB command n works when the matrix A is not square.
For instance, if A = [1 1 1 1; 2 1 3 2] ;
representing 2 equations in 4 unknowns with b =[-1;2] ;
then Anb sometimes returns the "least squares solution" (which will be
explained later) [0; -2.5; 1.5; 0]. Use with care!!
Such solutions may not be unique and/or exact, and they should be
interpreted carefully.

M Bikdash–copyright 2019 (CSE in NCA&T) Gaussian Elimination September 20, 2020 43 / 43