PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

MODULE -5
INNER PRODUCT SPACES

An inner product on a vector space V is a function that associates with each pair of vectors x and y a
scalar denoted by x, y such that, if x, y, and z are vectors and c is a scalar, then
(a) x, y = y, x ;
(b) x, y + z = x, y +  x, z ;
(c) cx, y = cx, y ;
(d) x, y  0; x, x = 0 if and only if x = 0.

The dot product of two vectors x = ( x1 , x2 ,..., xn ) and y = ( y1 , y2 ,..., y n ) in  n is defined to be

x  y = x1 y1 + x2 y2 + ... + xn yn
Note: The dot product in  is an inner product (sometimes called the Euclidean inner product).
n

Thus
x  y =  x, y = x T y

The length of the vector x = ( x1 , x2 ,..., xn ) [denoted by | x | ] is defined as follows:

|x|= ( x  x) = ( x12 + x22 + ... + xn2 )1 / 2 .

Cauchy-Schwarz Inequality: | x  y |  | x | | y |.

The angle  between two nonzero vectors x and y, where 0     , is given by

xy
cos =
x y

The vectors x and y are called orthogonal provided that

xy = 0
The distance between two vectors x = ( x1 , x2 ,..., xn ) and y = ( y1 , y2 ,..., y n ) in  n [denoted by
d (x, y) ], is defined as
d (x, y) = | x - y | = [( x1 − y1 ) 2 + ( x2 − y 2 ) 2 + ... + ( xn − y n )]1 / 2 .

The Triangle Inequality: | x + y |  | x | + | y |.

If the vectors x and y are orthogonal, then the triangle inequality yields the Pythagorean formula:
x+y = x + y
2 2 2

Relation between orthogonality and linear independence: If the nonzero vectors

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 1

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

v 1 , v 2 ,..., v k are mutually orthogonal, then they are linearly independent.

Exercise: Is the converse of the above statement true? Justify your answer.

Note: Any set of n mutually orthogonal nonzero vectors in  n constitutes a basis for  n - called an
orthogonal basis.

Exercise: Give an example of an orthogonal basis for  n .

Note: The vector x in  n is a solution vector of Ax = 0 if and only if x is orthogonal to each vector in
the row space Row (A).

Definition: The vector u is orthogonal to the subspace W of  n provided that u is orthogonal to

every vector in W. The orthogonal complement W ⊥ of W is the set of all those vectors in  n that
are orthogonal to the subspace W.

Note: W ⊥ is itself a subspace.

Theorem: Let A be an m x n matrix. Then the row space Row(A) and the null space Null(A) are
orthogonal complements in  n . That is,

If W is Row(A), then W ⊥ is Null(A).

Finding a basis for the orthogonal complement of a subspace W of  n :

1. Let W be a subspace of  n spanned by the set of vectors v 1 , v 2 ,..., v k ;

2. Let A be the k x n matrix with row vectors v 1 , v 2 ,..., v k ;
3. Reduce the matrix A to echelon form;
4. Find a basis for the solution space Null(A) of Ax = 0.
Since W ⊥ is Null(A), the basis obtained in 4. will be a basis for the orthogonal complement of W.

Note: 1. dim W ⊥ + dim W = n

2. The union of a basis for W and a basis for W ⊥ is a basis for  n .

The linear system Ax = b is called overdetermined if A is m x n with m > n, i.e., the system has
more equations than variables.

Assumption in the following discussion: Matrix A has rank n, which means that its n column
vectors a 1 , a 2 ,..., a n are linearly independent and hence form a basis for Col(A).
Our discussions here are on the situation when the system Ax = b is inconsistent. Observe that the
system may be written as
b = x1a1 + x2a 2 + ... + xn a n ,

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 2

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

and inconsistency of the system simply means that b is not in the n-dimensional subspace Col(A) of
 m . Motivated by the situation, we want to find the orthogonal projection p of b into Col(A) i.e.,
given a vector b in  m that does not lie in the subspace V of  m with basis vectors a 1 , a 2 ,..., a n , we
want to find vectors p in V and q in V ⊥ such that
b=p+q
The unique vector p is called the orthogonal projection of the vector b into the subspace V =
Col(A).

Algorithm for the explicit computation of 'p':

1. We first note that if A is m x n with linearly independent column vectors a 1 , a 2 ,..., a n , then
Ay =p
where y = ( y1 ,y 2 ,...,y n ) ;
2. The normal system associated with the system Ax = b is the system:

AT Ay = AT b
(Note that AT A is nonsingular);
3. Consequently, the normal system has a unique solution
y = ( AT A) −1 AT b ;
4. The orthogonal projection p of b into Col(A) is given by
p = Ay = A( AT A) −1 AT b .

Definition: Let the m x n matrix A have rank n. Then by the least squares solution of the system Ax =
b is meant the solution y of the corresponding normal system AT Ay = AT b .

Note:
1. If the system Ax = b is inconsistent, then its least squares solution is as close as possible to being
a solution to the inconsistent system;
2. If the system is consistent, then its least squares solution is the actual (unique) solution.
3. The 'least sum of squares of errors' is given by | p - b | 2 .

Example: The orthogonal projection of the vector b into the line through the origin in  m , and
determined by the single nonzero vector a, is given by
a Tb ab
p = ay = T a = a
a a aa

Example: To find the straight line y = a + bx that best fits the data points
( x1 , y1 ), ( x2 , y2 ),..., ( xn , yn ) , the least squares solution x = (a, b) is obtained by solving the normal
equations:
n n
na + ( xi )b =  yi
1 1
n n n
( xi )a + ( xi2 )b =  xi yi
1 1 1

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 3

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

Inner Product:

A vector space V along with a specified inner product on V is called an inner product space.

Orthogonality: Let V be an inner product space. The vectors u,v belongs to V are said to be
orthogonal and u is said to be orthogonal to v. If
u, v = 0

1. Consider 𝑓(𝑡) = 3𝑡 − 5 and 𝑔(𝑡) = 𝑡2 , the inner product

< 𝑓, 𝑔 > = 0 ∫ 1 𝑓(𝑡)𝑔(𝑡)𝑑𝑡 . Find < 𝑓, 𝑔 >.

f (t ) = 3t − 5, g (t ) = t 2
1
f,g =  (3t − 5) t
2
dt
0
1
=  (3t − 5 t 2 ) dt
3

0
1
  t4   t3 
= 3
 
−5
 3 

  4    t =0
3 5
= −
4 3
11
=− .
12

Norm, Distance & Angle

Let u and v be vectors in an inner product space V.

1. The norm (or length) of u is ‖𝒖‖ = √⟨𝒖, 𝒖⟩
2. The distance between u and v is 𝑑(𝒖, 𝒗) = ‖𝒖 − 𝒗‖
3. The angle between two nonzero vectors u and v is given by

⟨𝒖, 𝒗⟩
cos 𝜃 = , 0≤𝜃≤𝜋
‖𝒖‖‖𝒗‖

4. u and v are orthogonal if ⟨𝒖, 𝒗⟩ = 0

• If ‖𝒗‖ = 1, then v is called a unit vector.
• If v is any nonzero vector, then the vector 𝒖 = 𝒗⁄‖𝒗‖
is called the unit vector in the direction of v.

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 4

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

Example-1: Let 𝑝(𝑥) = 1 − 2𝑥 2 , 𝑞(𝑥) = 4 − 2𝑥 + 𝑥 2 , 𝑟(𝑥) = 𝑥 + 2𝑥 2

be polynomial in P2, and determine the following.
1. ⟨𝑝, 𝑞⟩ = 𝑎0 𝑏0 + 𝑎1 𝑏1 + 𝑎2 𝑏2 = (1)(4) + (0)(−2) + (−2)(1) = 2
2. ⟨𝑞, 𝑟⟩ = (4)(0) + (−2)(1) + (1)(2) = 0
3. ‖𝑞‖ = √⟨𝑞, 𝑞⟩ = √42 + (−2)2 + 12
= √21
4. 𝑑(𝑝, 𝑞) = ‖𝑝 − 𝑞‖ = ‖(1 − 2𝑥 2 ) − (4 − 2𝑥 + 𝑥 2 )‖ = ‖−3 + 2𝑥 − 3𝑥 2 ‖ =
√(−3)2 + 22 + (−3)2 = √22
Orthogonal:
Two vectors in an inner product space are orthogonal if their inner product is zero. The same definition
applies to Euclidean spaces, where angles are defined, and there orthogonality means that either the
angle between the vectors is π/2 or one of the vectors is zero. So orthogonality is slightly more general
than perpendicularity.
A vector v in an inner product space is a unit vector if Its length is 1.
Orthonormal basis
A set of vectors to be orthonormal if they are all unit vectors and each one is orthogonal to each of
the others. An orthonormal basis is simply a basis that is orthonormal. Note that there is no such
thing as an ‘orthonormal vector’. The property applies to a whole set of vectors, not to an individual
vector.
Theorem: An orthonormal set of vectors {v1, …, vn} is linearly independent.
Proof: Suppose λ 1v1 + … + λ n vn = 0. Then <λ 1v1 + … + λ nvn | vr > = 0 for each r.

Since vr is orthogonal to the other vectors in the set and vr is a unit vector. Hence each λr = 0.
Because of the above theorem, if we want to show that a set of vectors is an orthonormal basis we
need only show that it is orthonormal and that it spans the space. Linear independences come free.
Another important consequence of the above theorem is that it is very easy to find the coordinates of
a vector relative to an orthonormal basis.
Orthogonal Projections- R2:
• Let u and v be vectors in the plane. If v is nonzero, then u can be orthogonally projected onto
v. This projection is denoted by projvu.

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 5

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

• projvu is a scalar multiple of v, i.e., projvu = av.

If a > 0, then cos 𝜃 > 0 and
‖𝒖‖‖𝒗‖ cos 𝜃 𝒖 ⋅ 𝒗
‖𝑎𝒗‖ = 𝑎‖𝒗‖ = ‖𝒖‖ cos 𝜃 = = ⇒ 𝑎 = 𝒖 ⋅ 𝒗⁄‖𝒗‖2 = (𝒖 ⋅ 𝒗)⁄(𝒗 ⋅ 𝒗)
‖𝒗‖ ‖𝒗‖

𝒖⋅𝒗
Then, 𝑝𝑟𝑜𝑗𝒗 𝒖 = 𝒗
𝒗⋅𝒗

Projvu in R2:

• If a < 0, then cos 𝜃 < 0. The orthogonal projection of u onto v is given by the same formula.
• The orthogonal projection of u = (4, 2) onto v = (3, 4) is given by

𝒖⋅𝒗 20 12 16
𝑝𝑟𝑜𝑗𝒗 𝒖 = 𝒗= (3, 4) = ( , )
𝒗⋅𝒗 25 5 5

Orthogonal Projection:

• Let u and v be vectors in an inner product space V. Then the orthogonal projection of u onto
v is given by

⟨𝒖, 𝒗⟩
𝑝𝑟𝑜𝑗𝒗 𝒖 = 𝒗, 𝒗 ≠ 𝟎.
⟨𝒗, 𝒗⟩

• The orthogonal projection of u = (6, 2, 4) onto v = (1, 2, 0) is given by

𝒖⋅𝒗 10
𝑝𝑟𝑜𝑗𝒗 𝒖 = 𝒗= (1, 2, 0) = (2, 4, 0)
𝒗⋅𝒗 5

• Given u = (6, 2, 4), v = (1, 2, 0), and

projvu = (2, 4, 0).

u – projvu = (6, 2, 4) – (2, 4, 0) = (4, –2, 4) is orthogonal to v = (1, 2, 0).

• If u and v are nonzero vectors in an inner product space, then u – projvu is orthogonal to v.

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 6

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

Orthogonal Sets Are Linearly Independent

If S = {v1, v2, …, vn} is an orthogonal set of nonzero vectors in an inner product space V, then S is
linearly independent.
pf: Because S is orthogonal, <vi, vj> = 0, i  j.
<( c1v1+ c2v2 + … + cnvn), vi>
= c1<v1, vi>+ c2<v2, vi>+ …+ ci<vi, vi>+…+ cn<vn, vi>
= ci <vi, vi> = 0

Hence every ci must be zero and the set must be linearly independent.
Corollary
If V is an inner product space of dimension n, then any orthogonal set of n nonzero vectors is a
basis for V. Show that the following set is a basis for R4.
Sol: Because 𝑆 = {(2, 3, 2, −2), (1, 0, 0, 1), (−1, 0, 2, 1), (−1, 2, −1, 1)}
𝒗1 ⋅ 𝒗2 = 0, 𝒗1 ⋅ 𝒗3 = 0, 𝒗1 ⋅ 𝒗4 = 0, 𝒗2 ⋅ 𝒗3 = 0, 𝒗2 ⋅ 𝒗4 = 0, 𝒗3 ⋅ 𝒗4 = 0

Thus, S is orthogonal.

1)Apply the Gram-Schmidt orthonormal process to the following basis for R2: B = {(1, 1), (0, 1)}.
Sol:

𝒘1 √2 √2 𝒗2 ⋅ 𝒘1 1
𝒘1 = 𝒗1 = (1, 1) ⇒ 𝒖1 = =( , ) 𝒘2 = 𝒗2 − 𝒘 = (0, 1) − (1, 1)
‖𝒘1 ‖ 2 2 𝒘1 ⋅ 𝒘1 1 2
−1 1 𝒘2 −√2 √2
= ( , ) ⇒ 𝒖2 = =( , )
2 2 ‖𝒘2 ‖ 2 2

2)Apply the Gram-Schmidt orthonormal process to the following basis for R3: B = {(1, 1, 0), (1, 2, 0),
(0, 1, 2)}.

√2 √2 𝒗2 ⋅ 𝒘1
𝒘1 = 𝒗1 = (1, 1, 0) ⇒ 𝒖1 = 𝒘1 ⁄‖𝒘1 ‖ = ( , , 0) 𝒘2 = 𝒗2 − 𝒘
2 2 𝒘1 ⋅ 𝒘1 1
3 −1 1 −√2 √2
= (1, 1, 0) − (1, 1, 0) = ( , , 0) ⇒ 𝒖2 = 𝒘2 ⁄‖𝒘2 ‖ = ( , , 0) 𝒘3
2 2 2 2 2
𝒗3 ⋅ 𝒘1 𝒗3 ⋅ 𝒘 2 1 1⁄2
= 𝒗3 − 𝒘1 − 𝒘 = 𝒗3 − 𝒘1 − 𝒘 = (0, 0, 2) ⇒ 𝒖3
𝒘1 ⋅ 𝒘1 𝒘2 ⋅ 𝒘 2 2 2 1⁄2 2
= 𝒘3 ⁄‖𝒘3 ‖ = (0, 0, 1)

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 7

 PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

3) The vectors v1 = (0, 1, 0) and v2 = (1, 1, 1) span a plane in R3. Find an orthonormal
basis for this subspace.
Sol: 𝒘1 = 𝒗1 = (0, 1, 0) ⇒ 𝒖1 = 𝒘1 ⁄‖𝒘1 ‖ = (0, 1, 0)

𝒗2 ⋅ 𝒘1 1
𝒘2 = 𝒗2 − 𝒘1 = (1, 1, 1) − (0, 1, 0)
𝒘1 ⋅ 𝒘1 1
√2 √2
= (1, 0, 1) ⇒ 𝒖2 = 𝒘2 ⁄‖𝒘2 ‖ = ( , 0, )
2 2

4) Find an orthonormal basis for the solution space of the following homogeneous
system of linear equations
𝑥1 + 𝑥2 ⥂ +7𝑥4 = 0, 2𝑥1 + 𝑥2 + 2𝑥3 + 6𝑥4 = 0

Let x3 = s and x4 = t,
1 1 0 7 0 1 0 2 −1 0
[ ]⇒[ ]
2 1 2 6 0 0 1 −2 8 0

𝑥1 −2 1
𝑥2 2 −8
[𝑥 ] = 𝑠 [ ] + 𝑡 [ ]
3 1 0
𝑥4 0 1

5. If V is an inner product space of dimension n, then any orthogonal set of n nonzero

vectors is a basis for V. Show that the following set is a basis for R4.
Sol: Because 𝑆 = {(2, 3, 2, −2), (1, 0, 0, 1), (−1, 0, 2, 1), (−1, 2, −1, 1)}

𝒗1 ⋅ 𝒗2 = 0, 𝒗1 ⋅ 𝒗3 = 0, 𝒗1 ⋅ 𝒗4 = 0, 𝒗2 ⋅ 𝒗3 = 0, 𝒗2 ⋅ 𝒗4 = 0, 𝒗3 ⋅
𝒗4 = 0
Thus, S is orthogonal.

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 8

 PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

6. Consider vectors u = (1,2,4), v = (2,−3,5), w = (4,2,−3) in R 3 . Find a) u, v , b) u , w , c ) v, w ,

d ) u + v, w , e) u , f ) v .
Sol. : Given u = (1,2,4), v = (2,−3,5), w = (4,2,−3) in R 3 .
a ) u , v , = 2 − 6 + 20 = 16
b) u , w = 4 + 4 − 12 = −4
c) v, w = 8 − 6 − 15 = −13
d ) u + v, w = 12 − 2 − 27 = −17
e) u = 12 + 2 2 + 4 2 = 21
f ) v = 2 2 + (−3) 2 + 52 = 38

GRAM-SCHMIDT PROCESS:
Every finite-dimensional inner product space V has an orthogonal basis.
Proof: We prove this by induction on the dimension of V.
If dim(V) = 0 then the empty set is an orthonormal basis.
Suppose that every vector space of dimension n has an orthonormal basis and suppose that V is a vector
space of dimension n + 1.
Let (v1, …, vn+1) be a basis for V and let U = v1, …, vn.
By the induction hypothesis U has an orthogonal basis {u1, …, un}.
< 𝒗2 , 𝒘1 > < 𝒗3 , 𝒘1 > < 𝒗3 , 𝒘2 >
𝒘1 = 𝒗1, 𝒘2 = 𝒗2 − 𝒘1, 𝒘3 = 𝒗3 − 𝒘1 − 𝒘 … . 𝒘𝑛
< 𝒘1 , 𝒘1 > < 𝒘1 , 𝒘1 > < 𝒘2 , 𝒘2 > 2
< 𝒗𝑛 , 𝒘1 > < 𝒗𝑛 , 𝒘2 > < 𝒗𝑛−1 , 𝒘𝑛−1 >
= 𝒗𝑛 − 𝒘 − 𝒘 −⋯− 𝒘
< 𝒘1 , 𝒘1 > 1 < 𝒘2 , 𝒘2 > 2 < 𝒘𝑛−1 , 𝒘𝑛−1 > 𝑛−1

Then 𝐵 ′ is an orthogonal basis for V.

1)Apply the Gram-Schmidt orthonormal process to the following basis for

R2: B = {(1, 1), (0, 1)}.

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 9

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

Solution:

𝒘1 √2 √2
𝒘1 = 𝒗1 = (1, 1) ⇒ 𝒖1 = =( , ) 𝒘2
‖𝒘1 ‖ 2 2
𝒗2 ⋅ 𝒘1
= 𝒗2 − 𝒘
𝒘1 ⋅ 𝒘1 1
1 −1 1
= (0, 1) − (1, 1) = ( , )
2 2 2
𝒘2 −√2 √2
⇒ 𝒖2 = =( , )
‖𝒘2 ‖ 2 2

2)Apply the Gram-Schmidt orthonormal process to the following basis for R3: B = {(1, 1, 0), (1, 2, 0),
(0, 1, 2)}.

Sol:
√2 √2 𝒗2 ⋅ 𝒘1
𝒘1 = 𝒗1 = (1, 1, 0) ⇒ 𝒖1 = 𝒘1 ⁄‖𝒘1 ‖ = ( , , 0) 𝒘2 = 𝒗2 − 𝒘
2 2 𝒘1 ⋅ 𝒘1 1
3 −1 1 −√2 √2
= (1, 1, 0) − (1, 1, 0) = ( , , 0) ⇒ 𝒖2 = 𝒘2 ⁄‖𝒘2 ‖ = ( , , 0) 𝒘3
2 2 2 2 2
𝒗3 ⋅ 𝒘1 𝒗3 ⋅ 𝒘2 1 1⁄2
= 𝒗3 − 𝒘 − 𝒘 = 𝒗3 − 𝒘1 − 𝒘 = (0, 0, 2) ⇒ 𝒖3
𝒘1 ⋅ 𝒘1 1 𝒘2 ⋅ 𝒘2 2 2 1⁄2 2
= 𝒘3 ⁄‖𝒘3 ‖ = (0, 0, 1)

QR Factorization:
QR Factorization (or QR Decomposition) is a technique in linear algebra used to decompose a given
matrix A into a product of two matrices:

Q is an orthogonal (or unitary) matrix, meaning QTQ=I (or QHQ=I for complex matrices).
R is an upper triangular matrix.

Methods for QR Factorization:

1. Gram-Schmidt Process: Uses orthogonalization to construct Q.
2. Householder Reflections: More stable and widely used in numerical
computing.

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 10

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

3. Givens Rotations: Used for sparse matrices to introduce zeros efficiently.

Problems
1) Find the QR factorization of the matrix

Solution: Compute the first orthonormal vector q1

The first column of A is:

Compute the second orthonormal vector q2

The second column of A is:

Compute the projection of a2 onto q1:

Compute q1Ta2 :

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 11

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

Normalize u2 to get q2:

The matrix Q is formed by placing q1 and q2 as its columns:

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 12

PROBABILTY DISTRIBUTIONS AND LINEAR ALGEBRA 23ECT401

The upper triangular matrix R is given by: R=QTA

Thus, R is:

The QR factorization of A is:

Dr. Bhavya Shivaraj, Dept Of Mathematics, SJBIT Page 13