Linear Algebra
BN Wijewickrama
IESLCE
Linear Dependence and Independence of
Vectors
• A set of vectors is said to be linearly dependent if one of the vectors
in the set can be defined as a linear combination of the others; if no
vector in the set can be written in this way, then the vectors are said
to be linearly independent.
Linearly Independent Linearly Dependent
Geometric Meaning
• A person describing the location of a certain place might say, "It is 3
miles north and 4 miles east of here." This is sufficient information to
describe the location, because the geographic coordinate system may
be considered as a 2-dimensional vector space.
• In this example the "3 miles north" vector and the "4 miles east"
vector are linearly independent. That is to say, the north vector
cannot be described in terms of the east vector, and vice versa. The
third "5 miles northeast" vector is a linear combination of the other
two vectors, and it makes the set of vectors linearly dependent, that
is, one of the three vectors is unnecessary.
Eigenvalues and Eigenvectors
• Almost all vectors change direction, when they are multiplied by 𝐴.
Certain exceptional vectors 𝑥 are in the same direction as 𝐴𝑥 . Those are
the “eigenvectors”. The vector 𝐴𝑥 is a number 𝜆 times the original 𝑥.
• The basic equation is 𝑨𝒙 = 𝝀𝒙
• The number 𝝀 is an “eigenvalue” of 𝐴.
• The eigenvalue 𝜆 tells whether the special vector
𝑥 is stretched or shrunk or reversed when it is
multiplied by 𝐴.
Transformation Matrices
1 2 0 1 3
• 2 1 0 × 1 = 3
0 0 −3 1 −3
Eigenvector Eigenvalue
1 2 0 1 3 1
• 2 1 0 × 1 = 3 =3 1
0 0 −3 0 0 0
• Eigenvalues and eigenvectors have a wide range of applications,
for example in stability analysis, vibration analysis, atomic
orbitals, facial recognition, and matrix diagonalization.
• The Mona Lisa example pictured provides a simple illustration.
• Each point on the painting can be represented as a vector pointing
from the center of the painting to that point.
• The linear transformation in this example is called a shear mapping.
• Points in the top half are moved to the right and points in the bottom
half are moved to the left proportional to how far they are from the
horizontal axis that goes through the middle of the painting. The
vectors pointing to each point in the original image are therefore
tilted right or left and made longer or shorter by the transformation.
• Notice that points along the horizontal axis do not move at all when
this transformation is applied. Therefore, any vector that points
directly to the right or left with no vertical component is an
eigenvector of this transformation because the mapping does not
change its direction.
• Some properties of Eigenvalues:
(i) The Transpose 𝑨𝑻 of a square matrix A has the same eigenvalues as A.
(ii) The product of the eigenvalues of a matrix A is equal to the determinant
of A.
(iii) If 𝜆1 , 𝜆2 , … 𝜆𝑛 are the eigenvalues of A, then the eigenvalues of
• 𝒌𝑨 are 𝒌𝝀𝟏 , 𝒌𝝀𝟐 , … 𝒌𝝀𝒏
• 𝑨𝒎 are 𝝀𝒎
𝟏 , 𝝀𝒎
𝟐 , … 𝝀𝒎
𝒏
−𝟏 𝟏 𝟏 𝟏
•𝑨 are , ,…
𝝀𝟏 𝝀𝟐 𝝀𝒏
• The eigenvalues are also called proper or characteristic values.
𝐴𝑥 = 𝜆𝑥 → (1)
• Rewriting equation (1) as
𝐴𝑥 = 𝜆𝐼 𝑥 or 𝐴𝑥 − 𝜆𝐼 𝑥 = 0 or
𝐴 − 𝜆𝐼 𝑥 = 0 → (2)
• The homogeneous equation 2 will have a nontrivial solution if and
only if
𝑨 − 𝝀𝑰 = 𝟎 → (𝟑)
• To solve the eigenvalue problem for an n by n matrix, follow these
steps:
1. Compute the determinant of 𝑨 − 𝝀𝑰.
With 𝜆 subtracted along the diagonal, this determinant starts with 𝜆𝑛 or
−𝜆𝑛 . It is a polynomial in 𝜆 of degree n.
2. Find the roots of this polynomial.
By solving 𝐴 − 𝜆𝐼 = 0. The n roots are the n eigenvalues of A.
3. Find eigenvectors
To find eigenvectors 𝑥, Solve 𝐴 − 𝜆𝐼 𝑥 = 0 for each eigenvalue 𝜆.
Example 01:
Find the eigenvalues and eigenvectors of the matrices:
0 2 1
5 −1
(i) 𝐴 = 4 1 0 (ii) 𝐴 =
1 3
4 0 1
3 2 4 0 0 1
(iii) 𝐴 = 2 0 2 (iv) 𝐴 = 1 0 −3
4 2 3 0 1 3
0 2 1
(i) 𝐴 = 4 1 0
4 0 1
• Characteristic equation is :
𝑑𝑒𝑡 𝐴 − 𝜆𝐼 = 0
−𝜆3 + 2𝜆2 + 11𝜆 − 12 = 0
𝜆+3 𝜆−4 𝜆−1 =0
𝜆 = −3 𝑜𝑟 4 𝑜𝑟1
• Eigenvectors :
1 1
𝜆 = −3 ; 𝑣1 = −1 𝜆 = 4 ; 𝑣1 = 4 3
−1 4 3
0
𝜆 = 1 ; 𝑣1 = 1
−2
5 −1
(ii) 𝐴 =
1 3
• Characteristic equation is :
• Eigenvectors
3 2 4
(iii) 𝐴 = 2 0 2
4 2 3
• Characteristic equation is :
• Eigenvectors :
0 0 1
(iv) 𝐴 = 1 0 −3
0 1 3
• Characteristic equation is :
• Eigenvectors :
Diagonalization
• A matrix 𝐴 is diagonalizable if there exist an invertible matrix 𝑃 such that
𝑃−1 𝐴𝑃 is a diagonal matrix. The matrix 𝑃 is said to diagonalize 𝐴.
𝐴𝑃 = 𝑃𝐷
𝑃−1 𝐴𝑃 = 𝑃−1 𝑃𝐷 = 𝐷
D is the Diagonal Matrix.
The diagonal elements of 𝐷 are the eigenvalues of 𝐴.
The columns of matrix 𝑃 are eigenvectors of 𝐴
• Diagonalization is the process of finding a corresponding diagonal matrix for a
diagonalizable matrix.
NOTE:
• The eigenvector 𝑥1 ,𝑥2 , … , 𝑥𝑛 corresponding to distinct eigenvalues
𝜆1 ,𝜆2 ,… , 𝜆𝑛 are linearly independent.
• Matrix having 𝑛 distinct eigenvalues is diagonalizable.
• If all the root of characteristic polynomial are real but not distinct,
then it may or may not be diagonalizable.
• The transformation of a matrix A to 𝑃−1 𝐴𝑃 is known as a similarity
transformation.
NOTE:
• The Square matrix P, which diagonalizes A, is found by grouping
eigenvectors of A into square matrix and resulting diagonal matrix as
the eigenvalues of A as its diagonal elements.
Procedure for diagonalization
1. Find 𝑛 independent eigenvectors of A.
2. Form the matrix 𝑃 with columns as independent eigenvectors.
3. Get 𝑃−1 .
4. Form the product 𝐷 = 𝑃−1 𝐴𝑃.
Example 02:
If possible, diagonalize the following matrices.
0 2 1
5 −1
(i) 𝐴 = 4 1 0 (ii) 𝐴 =
1 3
4 0 1
3 2 4 0 0 1
(iii) 𝐴 = 2 0 2 (iv) 𝐴 = 1 0 −3
4 2 3 0 1 3
0 2 1
(i) 𝐴 = 4 1 0
4 0 1
• Eigenvectors :
• Eigen vectors are linearly independent
5 −1
(ii) 𝐴 =
1 3
• Eigenvalues:
• Eigenvectors ∶
3 2 4
(iii) 𝐴 = 2 0 2
4 2 3
• Eigenvectors :
• Eigen vectors are linearly independent.
0 0 1
(iv)𝐴 = 1 0 −3
0 1 3
• Eigenvalues:
• Eigenvectors :
Powers of a matrix
• We can obtain powers of a matrix by using diagonalization.
• We know that 𝐷 = 𝑃−1 𝐴𝑃
• Where A is the square matrix and P is a non-singular matrix.
𝐷2 = (𝑃−1 𝐴𝑃) (𝑃−1 𝐴𝑃) = (𝑃−1 𝐴(𝑃𝑃−1 )𝐴𝑃) = (𝑃−1 𝐴2 𝑃)
• Similarly
𝐷3 = 𝑃−1 𝐴3 𝑃 𝐴3 = 𝑃𝐷3 𝑃−1
• In General
𝐷𝑛 = 𝑃−1 𝐴𝑛 𝑃 𝐴𝑛 = 𝑃𝐷𝑛 𝑃−1
Example:
0 2 1
(i) Find a matrix P which transform the matrix 𝐴 = 4 1 0 to diagonal
3
4 0 1
form. Hence calculate 𝐴 .
Diagonalization of symmetric matrices
A symmetric matrix is a square matrix that is equal to its transpose.
Matrix A is symmetric if 𝐴𝑇 = 𝐴.
Procedure for diagonalization of symmetric matrices
• Verify all the roots of the characteristic equation of a symmetric
matrix are real.
• Verify eigenvectors for distinct eigenvalues are orthogonal.
• Find an orthogonal matrix P such that 𝑃−1 𝐴𝑃 = 𝐷.
• The order 𝑀𝜆 of an eigenvalue 𝜆 as a root of the characteristic
polynomial is called the algebraic multiplicity of 𝜆.
• The number 𝑚𝜆 of linearly independent eigenvectors corresponding
to 𝜆 is called the geometric multiplicity of 𝜆.
• A is diagonalizable if and only if for every eigenvalue
geometric multiplicity ≤ algebraic multiplicity
Orthogonal Diagonalization
• A square matrix is called orthogonal matrix if transposition gives the inverse
of A.
𝑨𝑻 = 𝑨−𝟏
• A matrix is orthogonally diagonalizable if there exists an orthogonal matrix 𝑃
such that
or 𝑫 = 𝑷𝑻 𝑨𝑷.
• Orthonormal vectors are the unit vectors who are orthogonal to each other.
• To find orthonormal vectors use Gram Schmidt Process.
Example:
Diagonalize the symmetric matrix.
2 1
𝐴=
1 2
Solution:
The characteristic equation
Quadratic Forms
• Quadratic form of some function can be expressed as a product of matrices.
• Quadratic Form = 𝑿𝑻 𝑨𝑿 , Where A is a symmetric matrix.
Example:
• Consider 𝑞 𝑥, 𝑦 = 2𝑥 2 + 3𝑥𝑦 + 𝑦 2
𝑇
𝑥
𝑞 𝑥, 𝑦 = 𝑋 𝐴𝑋 ; 𝑋 = 𝑦
2 3 𝑥 2 3 2 2 𝑥 2 2
(i) 𝑥 𝑦 𝐴= (ii) 𝑥 𝑦 𝐴=
0 1 𝑦 0 1 1 1 𝑦 1 1
𝑥 𝑥
2𝑥 3𝑥 + 𝑦 𝑦 = 2𝑥 2 + 3𝑥𝑦 + 𝑦 2 2𝑥 + 𝑦 2𝑥 + 𝑦 𝑦 = 2𝑥 2 + 𝑥𝑦 + 2𝑥𝑦 + 𝑦 2
• 𝐴 may have different values.
• However the matrix 𝐴 must be a symmetric matrix.
• The product term is divided into two equal parts.
𝑞 𝑥, 𝑦 = 2𝑥 2 + 3𝑥𝑦 2 + 3𝑥𝑦 2 + 𝑦 2
3
2
𝑞 𝑥, 𝑦 = 𝑥 𝑦 2 𝑥
3 𝑦
1
2
For a three variable quadratic form,
• 𝑞 𝑥, 𝑦, 𝑧 = 𝑎𝑥 2 + 𝑑𝑥𝑦 + 𝑏𝑦 2 + 𝑒𝑥𝑧 + 𝑓𝑦𝑧 + 𝑐𝑧 2
𝑥 𝑎11 𝑎12 𝑎13 𝑎11 = 𝑎 𝑎12 = 𝑑/2 𝑎13 = 𝑒/2
𝑋 = 𝑦, 𝐴 = 𝑎21 𝑎22 𝑎23 = 𝑎21 = 𝑑/2 𝑎22 = 𝑏 𝑎23 = 𝑓/2
𝑧 𝑎31 𝑎32 𝑎33 𝑎31 = 𝑒/2 𝑎32 = 𝑓/2 𝑎33 = 𝑐
• 𝑞 𝑥, 𝑦, 𝑧 = 2𝑥 2 + 3𝑥𝑦 + 𝑦 2 + 2𝑥𝑧 + 𝑦𝑧 + 𝑧 2
3
2 1
2
𝑥 3 1
𝑋= 𝑦 𝐴= 1
𝑧 2 2
1
1 1
2
• Example:
(i) Express the quadratic form 𝑥12 + 2𝑥22 + 2𝑥32 − 2𝑥1 𝑥2 − 2𝑥2 𝑥3 as
product of matrices.
(ii) Express the quadratic form
6𝑥 2 + 3𝑦 2 + 3𝑧 2 − 4𝑥𝑦 − 2𝑦𝑧 + 4𝑧𝑥 as product of matrices.
Diagonalization of Quadratic Form
• Let 𝑋 𝑇 𝐴𝑋 be a quadratic form in the variables 𝑥1 , 𝑥2 , 𝑥3 , … where A is
symmetric.
• If P orthogonally diagonalizes A then the new variable 𝑦1 , 𝑦2 , 𝑦3 , …
are defined by the substitution of 𝑃𝑦 to 𝑥.
• 𝑋 𝑇 𝐴𝑋 = 𝑌 𝑇 𝐷𝑌 where 𝐷 = 𝑃𝑇 𝐴𝑃.
• The matrix P is said to orthogonalize the quadratic form.
• Example:
Find a change of variables that reduces the quadratic form
𝑥12 − 𝑥32 − 4𝑥1 𝑥2 a sum of squares and express the quadratic form in
terms of the new variable.
Solution
The quadratic form can be written as
The characteristic equation is
The eigenvalues are
• Orthogonal three vectors:
• Then the substitution 𝑥 = 𝑃𝑦 gives,
• The new quadratic form is
• Or equivalently
