Department of Mathematics

UNIT - II
LINEAR ALGEBRA - II

Topic Learning Objectives

Upon Completion of this unit, students will be able to:
• Study the orthogonal and orthonormal properties of vectors.
• Use Gram-Schmidt process to factorize a given matrix as a product of an orthogonal
matrix(Q) and an upper triangular invertible matrix(R).
• Diagonalize symmetric matrices using eigenvalues and eigenvectors.
• Decompose a given matrix into product of an orthogonal matrix (U ), a diagonal matrix
(Σ) and an orthogonal matrix (V T ).

Introduction
This section deals with the study of orthogonal and orthonormal vectors which forms the
basis for the construction of an orthogonal basis for a vector space. The Gram-Schmidt
process is applied to construct an orthogonal basis for the column space of a given matrix
and further to decompose a given matrix to the form A = QR, where Q has orthonormal
column vectors and R is an upper triangular invertible matrix with positive entries along
the diagonal. This section also deals with finding the Eigenvalues and Eigenvectors of a
square matrix, which is applied to diagonalize a square matrix A as D = P −1 AP . Further
the singular value decomposition is studied wherein, a given matrix is resolved as a product
of an orthogonal matrix (U), a diagonal matrix(Σ ) and an orthogonal matrix V T .

Definition. Two vectors u and v in Rn are orthogonal to each other if u · v = 0.

Example. In R2 , the vectors u = (1, 2) and v = (6, −3) are orthogonal as u · v = (1, 2) ·
(6, −3) = 0.
Definition. A set of vectors {u1 , u2 , . . . , up } in Rn is said to be an orthogonal set if each
pair of distinct vectors from the set is orthogonal, that is, if ui · uj = 0 whenever i 6= j
Example. Let u1 = (3, 1, 1), u2 = (−1, 2, 1), u3 = (− 21 , −2, 72 ). Then u1 · u2 = (3, 1, 1) ·
(−1, 2, 1) = −3 + 2 + 1 = 0, u1 · u3 = (3, 1, 1) · (− 21 , −2, 72 ) = − 32 − 2 + 27 = 0, u2 · u3 =
(−1, 2, 1) · (− 12 , −2, 72 ) = 21 − 4 + 72 = 0. Each pair of distinct vectors is orthogonal and so
{u1 , u2 , u3 } is an orthogonal set.
Definition. A set {u1 , u2 , . . . , up } is an orthonormal set if it is an orthogonal set of unit
vectors.
Example. The standard basis {e1 , e2 , . . . , en } for Rn , is an orthonormal set. Any non-empty
subset of {e1 , e2 , . . . , en } is an orthonormal set.
Definition. An orthogonal basis for a subspace W of Rn is a basis for W that is also an
orthogonal set.
Example. The set S = {(3, 1, 1), (−1, 2, 1), (− 21 , −2, 27 ) is an orthogonal basis for R3 as (i)
S is an orthogonal set and (ii) S forms a basis for R3 .
3 1 1
7 1
−1 2 1 = 3(7 + 2) − 1(− + ) + 1(2 + 1) = 27 + 3 + 3 = 33 6= 0
2 2
− 12 −2 27

Third Semester Linear Algebra and Probability Theory (MA231TC) 1

 Department of Mathematics

Definition. An orthonormal basis for a subspace W of Rn is a basis for W that is also

an orthonormal set.

Example. Show that {( √311 , √111 , √111 ), (− √16 , √26 , √16 ), (− √166 , − √466 , √766 )} is an orthonormal
basis for R3 , where

Solution: Let v1 = ( √311 , √111 , √111 ), v2 = (− √16 , √26 , √1 ), v3

6
= (− √166 , − √466 , √766 ).

3 2 1
v1 · v2 = − √ + √ + √ = 0
66 66 66
3 4 7
v1 · v3 = − √ −√ +√ =0
726 726 726
1 8 7
v2 · v3 = √ −√ +√ =0
396 396 396
Thus {v1 , v2 , v3 } is an orthogonal set.
9 1 1
v1 · v1 = + + =1
11 11 11
2 4 1
v2 · v2 =
+ + =1
6 6 6
1 16 49
v3 · v3 = + + =1
66 66 66
which shows that v1 , v2 , v3 are unit vectors. Thus {v1 , v2 , v3 } is an orthonormal set. Since
the set is linearly independent and consists of three vectors form a basis for R3 .

Definition. A square matrix P with real entries and satisfying the condition P −1 = P T is
called an orthogonal matrix. That is P P T = I.
     
cos θ − sin θ −1 cos θ sin θ T cos θ sin θ
Example. Let P = . Then P = and P = .
sin θ cos θ − sin θ cos θ − sin θ cos θ
Clearly P −1 = P T . Therefore P is an orthogonal matrix.
1 2 2

3
− 3 3
Example. The matrix A =  23 − 13 − 23  is an orthogonal matrix, since
2 2 1
3 3 3
1 2 2
  1 2 2

3 3 3 3
− 3 3
1 0 0
AT A = − 23 − 13 23   32 − 13 − 23  = 0 1 0. The row vector of A, namely ( 31 , − 23 , 32 ),
2
3
− 23 13 2
3
2
3
1
3
0 0 1
2 1 2 2 2 1
( 3 , − 3 , − 3 ) and ( 3 , 3 , 3 ) are orthonormal. So are the column vectors of A.

Note:
Let P be an n × n matrix with real entries. Then P is orthogonal if and only if the row
(column) vectors of A form an orthonormal basis for Rn .

Third Semester Linear Algebra and Probability Theory (MA231TC) 2

 Department of Mathematics

Orthogonal Projections
Given a non-zero vector ~u in Rn , consider the problem of decomposing a vector ~y in Rn into
the sum of two vectors, one a multiple of ~u and the other orthogonal to ~u. We wish to write

~y = ŷ + ~z (1)

where ŷ = α~u, for some scalar α and ~z is some vector orthogonal to ~u.
Given any scalar α, let ~z = ~y − α~u, so that 1 is satisfied. Then ~y − ŷ is orthogonal to ~u iff

0 = (~y − α~u).~u = ~y · ~u − (α~u) · ~u

= ~y · ~u − α(~u · ~u)
That is, 1 is satisfied with ~z orthogonal to ~u iff
~y · ~u ~y · ~u
α= and ŷ = ~u
~u · ~u ~u · ~u
The vector ŷ is called the orthogonal projection of ~y onto ~u, and the vector ~z is called the
component of ~y orthogonal to ~u.

Example. Let ~y = (7, 6) and ~u = (4, 2). The orthogonal projection of ~y onto ~u is given by,

~y · ~u 40
ŷ = ~u = ~u = 2~u = 2(4, 2) = (8, 4)
~u · ~u 20

Note:
1. The orthogonal projection of ~y onto a subspace W spanned by orthogonal vectors
~y · u~1 ~y · u~2
{u1 , u2 } is given by ŷ = u~1 + u~2 .
u~1 · u~1 u~2 · u~2
2. The distance from a point ~y in Rn to a subspace W is defined as the distance from ~y
to the nearest point in W .

Example. The distance from ~y to W = Span{u1 , u2 }, where ~y = (−1, −5, 10), u1 =

(5, −2, 1), u2 = (1, 2, −1). is given by

(−1, −5, 10) · (5, −2, 1) (−1, −5, 10) · (1, 2, −1)
ŷ = (5, −2, 1) + (1, 2, −1)
(5, −2, 1) · (5, −2, 1) (1, 2, −1) · (1, 2, −1)
= (−1, −8, 4)
~y − ŷ = (−1, −5, 10) − (−1, −8, 4) = (0, 3, 6)
√ √ √
The distance from ~y to W is 0 + 32 + 62 = 45 = 3 5.

Exercise:
1. Determine which set of vectors are orthogonal.

(a) u1 = (−1, 4, −3), u2 = (5, 2, 1), u3 = (3, −4, −7) ,

(b) u1 = (5, −4, 0, 3), u2 = (−4, 1, −3, 8), u3 = (3, 3, 5, −1).

Third Semester Linear Algebra and Probability Theory (MA231TC) 3

 Department of Mathematics

2. Show that {(2, −3), (6, 4)} forms an orthogonal basis for R2 .

3. Show that {(1, 0, 1), (−1, 4, 1), (2, 1, −2)} forms an orthogonal basis for R3 .
 3 
√ − √1 − √1
 11 6 66
4. Show that the matrix U =  √111 √26 − √466  is an orthogonal matrix.

√1 √1 − √766
11 6

5. Find the orthogonal projection of y = (2, 6) onto u = (7, 1).

6. Let u1 = (2, 5, −1), u2 = (−2, 1, 1) and y = (1, 2, 3). Find the orthogonal projection of
y onto Span{u1 , u2 }.

Gram-Schmidt Orthogonalization
The Gram-Schmidt process is a simple algorithm for producing an orthogonal or orthonormal
basis for any non-zero subspace of Rn . The construction converts a skewed set of axes into
a perpendicular set.

Gram-Schmidt process
Given a basis {x1 , x2 , . . . , xp } for a subspace W of Rn . Define,

v1 = x1
x2 · v1
v2 = x2 − v1
v1 · v1
x3 · v1 x3 · v2
v3 = x3 − v1 − v2
v1 · v1 v2 · v2
..
.
xp · v 1 xp · v2 xp · vp−1
vp = vp − v1 − v2 − · · · − vp−1
v1 · v1 v2 · v2 vp−1 · vp−1

Then {v1 , v2 , . . . , vp } is an orthogonal basis for W . In addition Span{v1 , v2 , . . . , vk } =

Span{x1 , x2 , . . . , xk } for 1 ≤ k ≤ p.

Example. Let W = Span{x1 , x2 } where x1 = (3, 6, 0) and x2 = (1, 2, 2). Construct an

orthogonal basis {v1 , v2 } for W .
x2 · x1 (1, 2, 2) · (3, 6, 0)
Solution: Let v1 = x1 and v2 = x2 − x1 = (1, 2, 2) − (3, 6, 0) =
x1 · x1 (3, 6, 0) · (3, 6, 0)
(0, 0, 2). Then {v1 , v2 } is an orthogonal set of non-zero vectors in W . Since dim(W ) = 2,
the set {v1 , v2 } is a basis in W .

Example. Let W = Span{v1 , v2 , v3 }, where v1 = (0, 1, 2), v2 = (1, 1, 2), v3 = (1, 0, 1). Con-
struct an orthogonal basis {u1 , u2 , u3 } for W .

Solution: Let u1 = v1 .

v2 · u1 (1, 1, 2) · (0, 1, 2)
u2 = v2 − u1 = (1, 1, 2) − (0, 1, 2) = (1, 0, 0)
u1 · u1 (0, 1, 2) · (0, 1, 2)

Third Semester Linear Algebra and Probability Theory (MA231TC) 4

 Department of Mathematics

v3 · u1 v3 · u2
u3 = v3 − u1 − u2
u1 · u1 u2 , u2
(1, 0, 1) · (0, 1, 2) (1, 0, 1) · (1, 0, 0)
= (1, 0, 1) − (0, 1, 2) − (1, 0, 0)
(0, 1, 2) · (0, 1, 2) (1, 0, 0) · (1, 0, 0)
 
2 2 1
= (1, 0, 1) − (0, 1, 2) − (1, 0, 0) = 0, − , .
5 5 5

QR Factorization
If A is an m × n matrix with linearly independent columns, then A can be factored as
A = QR, where Q is an m × n matrix whose columns form an orthonormal basis for Col(A)
and R is an n × n upper triangular invertible matrix with positive entries on its diagonal.

 
1 0 0
1 1 0
Example. Find a QR factorization of A = 
1

1 1
1 1 1

Solution: The columns of A are the vectors {x1 , x2 , x3 }. Let v1 = x1 = (1, 1, 1, 1)

x2 · v1 (0, 1, 1, 1) · (1, 1, 1, 1)
v2 = x2 − v1 = (0, 1, 1, 1) − (1, 1, 1, 1)
v1 · v1 (1, 1, 1, 1) · (1, 1, 1, 1)
 
3 3 1 1 1
= (0, 1, 1, 1) − (1, 1, 1, 1) = − , , ,
4 4 4 4 4
x3 · v1 x3 · v2
v3 = x3 − v1 − v2
v1 · v1 v2 · v2
 
3 1 1 1
(0, 0, 1, 1) · − , , ,  
(0, 0, 1, 1) · (1, 1, 1, 1) 4 4 4 4 3 1 1 1
= (0, 0, 1, 1) − (1, 1, 1, 1) −     − , , ,
(1, 1, 1, 1) · (1, 1, 1, 1) 3 1 1 1 3 1 1 1 4 4 4 4
− , , , · − , , ,
4 4 4 4 4 4 4 4
   
2 2 3 1 1 1 2 1 2
= (0, 0, 1, 1) − (1, 1, 1, 1) − − , , , = 0, − , ,
4 3 4 4 4 4 3 3 3
3 1 1 1 2 1 2



Therefore
n (1, 1, 1, 1), − , ,
4 4 4 4
, , 0, − ,
3 3 3
, forms
o an orthogonal basis for Col(A) and
1 1 1 1 3 1 1 1 2 1 1
( 2 , 2 , 2 , 2 ), (− √12 , √12 , √12 , √12 ), (0, − √6 , √6 , √6 ) forms an orthonormal basis for Col(A).
Therefore
 √ 
1/2 −3/√ 12 0√
1/2 1/ 12 −2/ 6
Q= √ √ 
1/2 1/ 12 1/
√ √6

1/2 1/ 12 1/ 6
We have A = QR =⇒ QT A = QT QR =⇒ Q T
 A = IR
T
 =⇒ Q A = R i.e., R = Q A.
T
  1 0 0  
1/2√ 1/2
√ 1/2
√ 1/2
√ 1 1 0 2 3/2 √ 1
√
∴ R = −3/ 12 1/ √ 12 1/ √12 1/ √12  1 1 1 = 0 3/ 12 2/ √12 .
  
0 −2/ 6 1/ 6 1/ 6 0 0 2/ 6
1 1 1

Third Semester Linear Algebra and Probability Theory (MA231TC) 5

 Department of Mathematics

 
1 2 5
−1 1 −4
 
−1 4 −3.
Example. Find a QR factorization of A =  
 1 −4 7 
1 2 1

Solution: Let {x1 , x2 , x3 } be the columns of the matrix A. Let v1 = x1 = (1, −1, −1, 1, 1)

x2 · v1 (2, 1, 4, −4, 2) · (1, −1, −1, 1, 1)

v2 = x2 − v1 = (2, 1, 4, −4, 2) − (1, −1, −1, 1, 1)
v1 · v1 (1, −1, −1, 1, 1) · (1, −1, −1, 1, 1)
−5
= (2, 1, 4, −4, 2) − (1, −1, −1, 1, 1) = (3, 0, 3, −3, 3)
5
x3 · v 1 x3 · v2
v3 = x3 − v1 − v2
v1 · v1 v2 · v2
(5, −4, −3, 7, 1) · (1, −1, −1, 1, 1)
= (5, −4, −3, 7, 1) − (1, −1, −1, 1, 1)
(1, −1, −1, 1, 1) · (1, −1, −1, 1, 1)
(5, −4, −3, 7, 1) · (3, 0, 3, −3, 3)
− (3, 0, 3, −3, 3)
(3, 0, 3, −3, 3) · (3, 0, 3, −3, 3)
20 −12
= (5, −4, −3, 7, 1) − (1, −1, −1, 1, 1) − (3, 0, 3, −3, 3) = (2, 0, 2, 2, −2)
5 36
Therefore√{(1, −1,√−1, 1, 1),
√ (3, 0,√3, −3,√3), (2, 0, 2, 2, −2)} forms an orthogonal basis for Col(A)
and {(1/ 5, −1/ 5, −1/ 5, 1/ 5, 1/ 5), (1/2, 0, 1/2, −1/2, 1/2), (1/2, 0, 1/2, 1/2, −1/2)} forms
an orthonormal basis for Col(A). Therefore
 √ 
1/ √5 1/2 1/2
−1/ 5 0 0 
 √ 
Q= −1/
 √ 5 1/2 1/2 

 1/ 5 −1/2 1/2 
√
1/ 5 1/2 −1/2
 
 √ √ √ √ √  1 2 5
1/ 5 −1/ 5 −1/ 5 1/ 5 1/ 5  −1 1 −4

T
R = Q A =  1/2 0 1/2 −1/2 1/2  −1 4 −3


1/2 0 1/2 1/2 −1/2  1 −4 7 
1 2 1
√ √ √ 
5 − 5 4 5
Therefore R = 0  6 −2 
0 0 4
 
3 −5 1
1 1 1
Example. Find the orthogonal basis for the column space of the matrix  −1 5 −2.


3 −7 8

Solution: Let {x1 , x2 , x3 } be the columns of A, where x1 = (3, 1, −1, 3), x2 = (−5, 1, 5, −7), x3 =
(1, 1, −2, 8).

Third Semester Linear Algebra and Probability Theory (MA231TC) 6

 Department of Mathematics

Let v1 = (3, 1, −1, 3),

x2 · v1 (−5, 1, 5, −7) · (3, 1, −1, 3)
v2 = x2 − v1 = (−5, 1, 5, −7) − (3, 1, −1, 3)
v1 · v1 (3, 1, −1, 3) · (3, 1, −1, 3)
−40
= (−5, 1, 5, −7) − (3, 1, −1, 3) = (1, 3, 3, −1)
20
x3 · v1 x3 · v 2 (1, 1, −2, 8) · (3, 1, −1, 3)
v3 = v1 − v2 = (1, 1, −2, 8) − (3, 1, −1, 3)
v1 · v1 v2 · v2 (3, 1, −1, 3) · (3, 1, −1, 3)
(1, 1, −2, 8) · (1, 3, 3, −1)
− (1, 3, 3, −1)
(1, 3, 3, −1) · (1, 3, 3, −1)
30 −10
= (1, 1, −2, 8) − (3, 1, −1, 3) − (1, 3, 3, −1) = (−3, 1, 1, 3)
20 20
Therefore {(3, 1, −1, 3), (1, 3, 3, −1), (−3, 1, 1, 3)} is an orthogonal basis for the column space
of the given matrix.
 
−1 6 6
 3 −8 3 
Example. Find the orthogonal basis for the column space of the matrix   1 −2 6 .


1 −4 −3
Solution: Let {x1 , x2 , x3 } be the columns of A, where x1 = (−1, 3, 1, 1), x2 = (6, −8, −2, −4), x3 =
(6, 3, 6, −3). Let v1 = (−1, 3, 1, 1).
x2 · v 1 (6, −8, −2, −4) · (−1, 3, 1, 1)
v2 = x2 − v1 = (6, −8, −2, −4) − (−1, 3, 1, 1)
v1 · v1 (−1, 3, 1, 1) · (−1, 3, 1, 1)
−36
= (6, −8, −2, −4) − (−1, 3, 1, 1) = (3, 1, 1, −1)
12
x3 · v1 x3 · v2 (6, 3, 6, −3) · (−1, 3, 1, 1)
v3 = v1 − v2 = (6, 3, 6, −3) − (−1, 3, 1, 1)
v1 · v1 v2 · v2 (−1, 3, 1, 1) · (−1, 3, 1, 1)
(6, 3, 6, −3) · (3, 1, 1, −1)
− (3, 1, 1, −1)
(3, 1, 1, −1) · (3, 1, 1, −1)
6 30
= (6, 3, 6, −3) − (−1, 3, 1, 1) − (3, 1, 1, −1) = (−1, −1, 3, −1)
12 12

Therefore {(−1, 3, 1, 1), (3, 1, 1, −1), (−1, −1, 3, −1)} is an orthogonal basis for the column
space of the given matrix.

Exercise
1. Let W = Span{v1 , v2 }, where v1 = (1, 1) and v2 = (2, −1). Construct an orthogonal
basis {u1 , u2 } for W .
2. Find the orthonormal basis for the subspace spanned by the vectors u1 = (1, −4, 0, 1),
u2 = (7, −7, −4, 1).
 
1 3 5
−1 −3 1
 
3. Find the QR factorization of the matrix A = 0 2 3
1 5 2
1 5 8

Third Semester Linear Algebra and Probability Theory (MA231TC) 7

 Department of Mathematics

Answer
3 3
1. v1 = (1, 1), v2 = ( , − )
2 2
2. v1 = (1, −4, 0, 1), v2 = (5, 1, −4, −1)

Eigenvalues and Eigenvectors

If A is a square matrix of order n, we can find the matrix A − λI, where I is unit matrix of
order n. The determinant of this matrix equated to zero, i.e,
a11 − λ a12 ... a1n
a21 a22 − λ . . . a2n
|A − λI| = .. .. ... .. is called the characteristic equation of A.
. . .
an1 an2 . . . ann − λ
On expanding the determinant, the characteristic equation takes the form
(−1)n λn + k1 λn−1 + k2 λn−2 + · · · + kn = 0,
where ki s are expressible in terms of the elements aij . The roots of this equation are called
the characteristic
  rootsor latent roots or eigenvalues
 of the matrix A.
x1 a11 a12 . . . a1n
 x2   a21 a22 . . . a2n 
If x =  ..  and A =  .. .. , then the linear transformation
   
.. ...
.  . . . 
xn an1 an2 . . . ann

Y = AX (2)

carries the column vector X into the column vector Y by means of the square matrix A.
In practice it is often required to find such vectors which transform into themselves or to a
scalar multiple of themselves. Let X be such a vector which transforms into λX by means
of the transformation 2. Then, λX = AX or AX − λIX = 0 or

[A − λI]X = 0 (3)

The equation (3) represents a system of n homogeneous linear equations,

(a11 − λ)x1 + a12 x2 + · · · + a1n xn = 0

a21 x1 + (a22 − λ)x2 + · · · + a2n xn = 0 (4)
..
.
an1 x1 + an2 x2 + . . . + (ann − λ)xn = 0

which will have a non-trivial solution only if the coefficient matrix is singular. i.e, |A − λI| =
0. This is called the characteristic equation of the transformation and is same as the char-
acteristic equation of the matrix A. It has n roots and corresponding   to each root, the
x1
 x2 
equation (3)( or equation (4)) will have a non-zero solution, x =  .. , which is known as
 
.
xn
the Eigenvector or latent vector.
Observation 1: Corresponding to n distinct Eigenvalues, we get n independent Eigenvec-
tors. But when two or more Eigenvalues are equal, it may or may not be possible to get

Third Semester Linear Algebra and Probability Theory (MA231TC) 8

 Department of Mathematics

linearly independent Eigenvectors corresponding to the repeated roots.

Observation 2: If xi is a solution for a Eigenvalues λi then it follows from (3) that cxi is
also a solution, where c ia an arbitrary constant. Thus the Eigenvector corresponding to an
Eigenvalues is not unique, but may be any one of the vectors cxi .

 
1/2 1/2
Example. Find the Eigenvalues and Eigenvectors of the matrix A = .
1/2 1/2
1 1
−λ
Solution: |A − λI| = 0 =⇒ 2
1=⇒ λ2 − λ = 0 =⇒ λ = 0, λ = 1.
1
2

 − λ 2
    2
−1/2 1/2 x1 0
For λ = 1, (A − λI)X = 0 =⇒ = =⇒ −x1 + x2 = 0 =⇒ x2 = x1
1/2
  −1/2 x 2 0
1
Letting x1 = 1 =⇒ x2 = 1 ∴ X =
 1    
1/2 1/2 x1 0
For λ = 0, (A − λI)X = 0 =⇒ = =⇒ x1 + x2 = 0 =⇒ x2 = −x1
1/2 1/2 x 2 0
1
Letting x1 = 1 =⇒ x2 = −1 ∴ X = .
−1
 
0 −1
Example. Find the Eigenvalues and Eigenvectors of the matrix A =
1 0

−λ −1
Solution: |A − λI| = 0 =⇒ =⇒ λ2 + 1 = 0 =⇒ λ = +i, λ = −i.
1
 −λ    
−i −1 x1 0
For λ = i, (A − λI)X = 0 =⇒ = =⇒ −ix1 − x2 = 0 =⇒ x2 = −ix1
1  −i x2 0
1
Letting x1 = 1 =⇒ x2 = −i ∴ X =
 −i     
i −1 x1 0
For λ = −i, (A − λI)X = 0 =⇒ = =⇒ ix1 − x2 = 0 =⇒ x2 = ix1
1
  i x 2 0
1
Letting x1 = 1 =⇒ x2 = i ∴ X =
i
 
3 4 2
Example. Find the Eigenvalues and Eigenvectors of the matrix A = 3 5 4.
0 1 2

3−λ 4 2
Solution: |A − λI| = 0 =⇒ 3 5−λ 4 = 0 =⇒ λ3 − 10λ2 + 15λ = 0
0 1 2−λ
√ √
=⇒ λ = 5 + 10, 5 − 10, 0 √      
√ −2 − 10 4
√ 2 x1 0
For λ = 5 + 10, |A − λI| = 0 =⇒  3 − 10 4√  x2  = 0
0 1 −3 − 10 x3 0
x1 −x2 x3 x1 x2 x
=⇒ √ = √ = =⇒ √ = √ = 3
3 10 +√6 −9 − 3 10 3 2 + 10 3 + 10 1
2 + √10
∴ X = 3 + 10
1

Third Semester Linear Algebra and Probability Theory (MA231TC) 9

 Department of Mathematics

 √    
√ −2 + 10 √4 2 x1 0
For λ = 5 − 10, |A − λI| = 0 =⇒  3 10 4√  x2 = 0
 
0 1 −3 + 10 x3 0
x1 −x2 x3 x1 x2 x3
=⇒ √ = √ = =⇒ √ = √ =
−3 10√+ 6 −9 + 3 10 3 2 − 10 3 − 10 1
2 − √10
∴ x = 3 − 10
1     
3 4 2 x1 0
with λ = 0, |A − λI| = 0 =⇒ 3 5 4
  x2 = 0
 
0 1 2 x3 0
x1 −x2 x3 x1 x2 x3
=⇒ = = =⇒ = =
6  6 3 2 −2 1
2
∴ X = −2
1
 
1 3 3
Example. Find the Eigenvalues and Eigenvectors of the matrix A = −3 −5 −3.
3 3 1
1−λ 3 3
Solution: |A − λI| = 0 =⇒ −3 −5 − λ −3 = 0 =⇒ λ3 + 3λ2 − 4 = 0
3 3 1−λ
=⇒ λ = 1, −2, −2      
0 3 3 x1 0
For λ = 1, |A − λI| = 0 =⇒ −3 −6 −3 x2  = 0
3 3 0 x3   0
1
x1 −x2 x3 x1 x2 x3
=⇒ = = =⇒ = = ∴ X = −1

9 9 9 1 −1 1
   1  
3 3 3 x1 0
For λ = −2, |A − λI| = 0 =⇒ −3 −3 −3
   x2 = 0
 
3 3 3 x3 0
3x1 + 3x2 + 3x3 = 0 =⇒ x1 = −x2 − x3  
−k1 − k2
Letting x2 = k1 , x3 = k2 =⇒ x1 = −k1 − k2 ∴ X =  k1 
    k2
−1 −1
 0  ,  1  are the linearly independent Eigenvectors corresponding to λ = −2.
1 0

Diagonalization of a Matrix
Suppose the n × n matrix A has n linearly independent Eigenvectors. Let P be the matrix
with its columns as n linearly independent vectors of A. Then P −1 AP is a diagonal matrix
D. The diagonal
 entries of D
 the Eigenvalues of A.
λ1
 λ2 
P −1 AP = 
 
.. 
 . 
λn

Third Semester Linear Algebra and Probability Theory (MA231TC) 10

 Department of Mathematics

Note:
1. Any matrix with distinct Eigenvalues can be diagonalized.

2. The diagonalization matrix P is not unique.

3. Not all matrices posses n linearly independent Eigenvectors, so not all matrices are
diagonalizable.

4. Diagonalizability of A depends on enough Eigenvectors.

5. Diagonalizability can fail only if there are repeated Eigenvalues.

6. The Eigenvalues of Ak are λk1 , λk2 . . . , λkn and each Eigenvector of A is still an Eigenvector
of Ak . [Dk = DD . . . D(k times) = (P −1 AP )(P −1 AP ) . . . (P −1 AP ) = P −1 Ak P ].
 
7 2
Example. Diagonalize the matrix A = .
2 4

7−λ 2
Solution: |A−λI| = 0 =⇒ = 0 =⇒ λ2 −11λ+24 = 0 =⇒ (λ−3)(λ−8) =
2 4−λ
0 =⇒ λ = 3, 8.    
4 2 x1 0
For λ = 3, (A − 3I)X = 0 =⇒ = =⇒ 2x1 + x2 = 0. Letting x1 = 1 =⇒
  2 1 x2 0
1
x2 = −2. Hence X1 =
−2    
−1 2 x1 0
For λ = 8, (A − 8I)X = 0 =⇒ = =⇒ −x1 + 2x2 = 0. Letting
  2 −4 x2 0
2
x2 = 1 =⇒ x1 = 2. Hence X2 =
    1  
1 2 3 0 −1 1/5 −2/5
=⇒ P = ,D= ,P =
−2 1 0 8 2/5 1/5
 
1 0
Example. Diagonalize the matrix A = .
0 −3

1−λ 0
Solution: |A − λI| = 0 =⇒ = 0 =⇒ (λ − 1)(λ + 3) = 0 =⇒ λ = −3, 1.
0 −3 −λ  
4 0 x1 0
For λ = −3, (A + 3I)X = 0 =⇒ = =⇒ 4x1 = 0 =⇒ x1 = 0. Let x2 = 1.
  0 0 x2 0
0
Hence X1 =
1    
0 0 x1 0
For λ = 1, (A − I)X = 0 =⇒ = =⇒ 4x2 = 0 =⇒ x2 = 0. Let x1 = 1.
  0 −4 x2 0
1
Hence X2 =
 0    
0 1 −3 0 −1 0 1
=⇒ P = ,D= ,P = .
1 0 0 1 1 0
 
6 −2 −1
Example. Diagonalize the matrix A = −2 6 −1.
−1 −1 5

Third Semester Linear Algebra and Probability Theory (MA231TC) 11

 Department of Mathematics

6 − λ −2 −1
Solution: |A − λI| = 0 =⇒ −2 6 − λ −1 = 0 =⇒ λ3 − 17λ2 + 90λ − 144 = 0
−1 −1 5 − λ
=⇒ λ = 3, 6, 8     
3 −2 −1 x1 0
For λ = 3, (A − λI)X = 0 =⇒ −2 3 −1 x2 = 0
    
−1 −1 2 x3 0
1
x1 −x2 x3 x1 x2 x3
=⇒ = = =⇒ = = ∴ X = 1
5 −5 5 1 1 1
   1  
0 −2 −1 x1 0
For λ = 6, (A − λI)X =⇒ −2 0 −1
   x2 = 0
 
−1 −1 −1 x3  0 
−1
x1 −x2 x3 x1 x2 x3
=⇒ = = =⇒ = = ∴ X = −1
−1 1 2 −1 −1 2
    2 
−2 −2 −1 x1 0
For λ = 8, (A − λI)X =⇒ −2 −2 −1 x2  = 0
−1 −1 −3 x3  0
1
x1 −x2 x3 x1 x2 x3
=⇒ = = =⇒ = = ∴ X = −1
5 5 0 1 −1 2
    0 
1 −1 1 3 0 0 1/3 1/3 1/3
Hence P = 1 −1 −1, D = 0 6 0, P −1 = −1/6 −1/6 1/3.
1 2 0 0 0 8 1/2 −1/2 0
 
3 −2 4
Example. Diagonalize the matrix A = −2 6 2.
4 2 3

3 − λ −2 4
Solution: |A − λI| = 0 =⇒ −2 6 − λ 2 = 0 =⇒ λ3 − 12λ2 − 21λ + 98 = 0
4 2 3−λ
=⇒ λ = −2, 7, 7     
5 −2 4 x1 0
For λ = −2, (A − λI)X =⇒ −2 8 2 x2  = 0
4 2 5 x3 0 
2
x1 −x2 x3 x1 x2 x3
=⇒ = = =⇒ = = ∴ X1 = 1 

36 −18 −36 2 1 −2
    −2 
−4 −2 4 x1 0
For λ = 7, (A − λI)X = 0 =⇒ −2 −1 2   x2 = 0
 
4 2 −4 x3 0
As the second and third row are dependent on the first row, we get only one equation in three
unknowns. i.e., −4x1 −2x2 +4x3 = 0. Letting x1 and x3 as arbitrary implies x2 = −2x1 +2x3 .
With x1 = 1, x3 = 2 we get x2 = 2. With x1 = 2, x3 = 1 we get x2 = −2.
1 2
∴ X2 = 2 and X3 = −2.
  
2 1

Third Semester Linear Algebra and Probability Theory (MA231TC) 12

 Department of Mathematics

Exercise:
Diagonalize the matrices
       
−2 −36 0 7 −4 4 3 1 1 5
3. 4.
1. −36 −23 0 2. −4 5 0 1 3 5 1
0 0 3 4 0 9

Singular Value Decomposition

Definition. Let A be any matrix. The square root of non-zero eigenvalues of AAT (or AT A)
are known as the singular values of A.
Any m × n matrix A can be factored into A = U ΣV T , where U and V are orthogo-
nal matrices and Σ is a diagonal matrix. The columns of U (m × m) are Eigenvectors of
AAT , and the columns of V (n × n) are Eigenvectors of AT A. The r singular values on the
diagonal of Σ(m×n) are the square roots of the non-zero Eigenvalues of both AAT and AT A.

Note:
The diagonal(but rectangular) matrix Σ has Eigenvalues from AT A. These positive en-
tries(also called sigma) will be σ1 , σ2 , . . . , σr , such that σ1 ≥ σ2 ≥ · · · ≥ σr > 0. They are
the singular values of A. When A multiplies a column vj of V , it produces σj times a column
of U (A = U ΣV T =⇒ AV = U Σ).

 
−1
Example. Decompose A =  2  as U ΣV T , where U and V are orthogonal matrices.
2
   
−1   1 −2 −2
Solution: AAT =  2  −1 2 2 = −2 4 4 , |AAT − λI| = 0,
2 −2 4 4
1 − λ −2 −2
=⇒ −2 4 − λ 4 = 0 =⇒ λ3 − 9λ2 = 0 =⇒ λ1 = 0, λ2 = 0, λ3 = 9
−2 4 4−λ     
−8 −2 −2 x1 0
T
For λ = 9, [AA − λI]X = 0 =⇒ −2 −5 4
   x2 = 0 =⇒ −8x1 − 2x2 − 2x3 = 0,
 
−2 4 −5 x3 0 
−1
−18x2 + 18x3 = 0 =⇒ x1 = −(1/2)x3 , x2 = x3 =⇒ X1 = 2  
    2 
1 −2 −2 x1 0
T
For λ = 0, [AA − λI]X = 0 =⇒ −2 4  4   x2 = 0 =⇒ x1 = 2x2 + 2x3
 
    −2 4  4 x3 0 
2 2 −1/3 2/3 2/3
=⇒ X2 = −1 and X3 = 2 . Hence U = 2/3 −1/3 2/3  .
    
2   −1 2/3 2/3 −1/3
  −1  
AT A = −1 2 2  2  = 9 , |AT A − λI| = 0 =⇒ |9 − λ| = 0 =⇒ λ = 9
2

Third Semester Linear Algebra and Probability Theory (MA231TC) 13

 Department of Mathematics

        
Then (AT A − λI)X = 0 =⇒ 0 x1 = 0 . Let x1 = 1. Then X = 1 . Hence   1
V =
  −1
T T T
or V = 1 . 9 is an Eigenvalues of both AA and A A. The rank of A = 2  is 1. 
  2
√ 3
Therefore Σ has only one non-zero entry σ1 = 9 = 3. That is Σ = 0
     0
−1 −1/3 2/3 2/3 3  
Therefore the SVD of A =  2  is  2/3 −1/3 2/3  0 1 .
2 2/3 2/3 −1/3 0
 
1 1
Example. Obtain the SVD of A = .
1 0
    
T 1 1 1 1 2 1
Solution: AA = =
1 0 1 0 1 1
√
T 2−λ 1 2 3− 5
|AA − λI| = 0 =⇒ = 0 =⇒ λ − 3λ + 1 = 0 =⇒ λ1 = ,
1 1−λ 2
√
3+ 5
λ2 =
2  √ 
√ 1+ 5
3− 5  2 1  x1
   
0
For λ = T
, (AA − λI)X = 0 =⇒  √ =
2 −1 + 5 x2 0

1
√ 2√
1+ 5 1+ 5
=⇒ x1 + x2 = 0. Letting x1 = −1, then x2 =
2 2
√
 
−1√  
1+ 5
−1
Therefore X =  1 + 5  = , where α = .
α 2
2 √
1− 5
 
√
3+ 5 1    
For λ = , (AAT − λI)X = 0 =⇒  2
 √  x1 = 0

2 −1 − 5 x2 0
1
√ 2√
1− 5 1− 5
=⇒ x1 + x2 = 0. Letting x1 = −1, then x2 =
2 2
√
 
−1√  
1− 5
−1
Therefore X =  1 − 5  = , where β = .
β 2
2
−1 −1
 
√ p
 1 + α2 1 + β2 
 
Hence U =   

α β
√
 
p
1 + α2 1 + β 2
−1 α

√
 √1 + α2 √1 + α2 

  λ 1 0
As AT A = AAT V T =   and Σ = 
√
.
 −1 β
 
p p
 0 λ2
1+β 2 1+β 2
 
−1 1 0
Example. Obtain the SVD of A = .
0 −1 1

Third Semester Linear Algebra and Probability Theory (MA231TC) 14

 Department of Mathematics

 
  −1 0  
T −1 1 0  2 −1
Solution: AA = 1 −1 = 
0 −1 1 −1 2
0 1
2 − λ −1
|AAT − λI| = 0 =⇒ = 0 =⇒ λ2 − 4λ + 3 = 0
−1 2 − λ
=⇒ λ1 = 1, λ2 = 3     
T −1 −1 x1 0
For λ = 3 (AA − λI)x = 0 =⇒ = =⇒ x1 + x2 = 0 =⇒ x1 = −x2
−1
 −1 x 2 0
−1
Letting x2 = 1 =⇒ x1 = −1 ∴ x =
 1    
T 1 −1 x1 0
For λ = 1 (AA − λI)x = 0 =⇒ = =⇒ x1 − x2 = 0 =⇒ x1 = x2
−1
  1 x 2 0
1
Letting x2 = 1 =⇒ x1 = 1 ∴ x =
1
   
−1 0   1 −1 0
−1 1 0
AT A =  1 −1 = −1 2 −1
0 −1 1
0 1 0 −1 1
1 − λ −1 0
|A A − λI| = 0 =⇒ −1 2 − λ −1 = 0 =⇒ λ3 − 4λ2 + 3λ = 0
T

0 −1 1 − λ
=⇒ λ1 = 0, λ2 = 1, λ3 = 3     
1 −1 0 x1 0
T
For λ = 0 (A A − λI)x = 0 =⇒ −1 2 −1 x2 = 0
    
0 −1 1 x3 0
=⇒ x1 − x2 = 0, x2 − x3 = 0 =⇒ x1 = x2, x2 = x3
1
Letting x3 = 1 =⇒ x2 = 1, x1 = 1 ∴ x = 1 
 1    
0 −1 0 x1 0
T
For λ = 1 (A A − λI)x = 0 =⇒ −1 1 −1 x2 = 0
    
0 −1 0 x3 0
=⇒ −x1 + x2 − x3 = 0, x2 = 0 =⇒ x1 = −x  3 , x2 = 0
−1
Letting x3 = 1 =⇒ x2 = 0, x1 = −1 ∴ x = 0  
 1    
−2 −1 0 x1 0
T
For λ = 3, (A A − λI)x = 0 =⇒ −1 −1 −1 x2 = 0
    
0 −1 −2 x3 0
=⇒ −2x1 − x2 = 0, x2 + 2x3 = 0 =⇒ 2x1 =  x2 ,x2 = −2x3
1
Letting x3 = 1 =⇒ x2 = −2, x1 = 1 ∴ x = −2 
1
 √ √ √ 
 √ √  √  1/ √6 −1/ 2 1/√3
−1/√ 2 1/√2 3 0 0
Hence U = Σ= V = −2/√ 6 0√ 1/√3
1/ 2 1/ 2 0 1 0
1/ 6 1/ 2 1/ 3
 √ √ √ 
1/ √6 −2/ 6 1/√6
T
V = −1/√ 2
 0√ 1/√2
1/ 3 1/ 3 1/ 3

Third Semester Linear Algebra and Probability Theory (MA231TC) 15

 Department of Mathematics

Exercise:
Obtain the SVD of the following matrices

     
2 −1 1 1 3 2 2
1. , 3.
2 2 2.  0 1, 2 3 −2
−1 1

Third Semester Linear Algebra and Probability Theory (MA231TC) 16