Reshma Rastogi (nee Khemchandani)
Linear Algebra and Optimization
1
Vector Spaces
1.1 Vector Spaces over R and C
In this section we present an axiomatic definition of a vector space and illustrate the same
through numerous examples.
Definition 1.1.1 (Vector space) A vector space V over a field of scalars K (taken here
as R or C) is a non-empty set V together with two algebraic operations of ‘+’ (called
vector adition) and ‘·’ (called scalar multiplication) satisfying the following axioms for all
u, v, w ∈ V and α, β ∈ K :
(i) u + v = v + u,
(ii) (u + v) + w = u + (v + w),
(iii) there exists 0 ∈ V (called the zero element or zero vector) such that u + 0 = u,
(iv) for u ∈ V, there exists −u ∈ V such that u + (−u) = 0,
(v) α · (βu) = (α · β)(u),
(vi) 1 · u = u,
(vii) α(u + v) = αu + αv,
(viii) (α + β)u = αu + βu.
Thus ‘+′ : V × V → V and ‘·′ : K × V → V are mappings representing vector addition
and scalar multiplication respectively which satisfy the axioms (i) - (viii) above.
Remark 1.1.1 The scalar field K could be a general field but in our discussion here it is
taken either as the field of reals R or the field of complex numbers C. If K = R, then V is
called a real vector space, and if K = C, then it is called a complex vector space.
Remark 1.1.2 The elements v ∈ V are called vectors. The term vector is generic. It may
represent the usual vector in R2 or R3 , or a real sequence, a polynomial or a function etc.
depending on the elements of V.
Example 1.1.1 Let Rn = {(x1 , x2 , . . . , xn ), xi ∈ R, i = 1, 2, . . . , n}. For u = (u1 , u2 , . . . , un ) ∈
Rn and v = (v1 , v2 , . . . , vn ) ∈ Rn we define u + v = (u1 + v1 , u2 + v2 , . . . , un + vn ) ∈ Rn and
for α ∈ R, αu = (αu1 , αu2 , . . . , αun ) ∈ Rn . The element 0 = (0, 0, . . . , 0) ∈ Rn is taken as
the zero vector and for u ∈ Rn , we define −u = (−u1 , −u2 , . . . , −un ).
VI Linear Algebra and Optimization
We may verify that the ‘addition’ and ‘scalar multiplication’ as defined above satisfy all
the axioms of Definition 1.1.1. Therefore Rn is a real vector space.
For n = 2, i.e. R2 = {(x1 , x2 ), x1 , x2 ∈ R} is a real vector space. Identifying the vector
ˆ we observe that the
(x1 , x2 ) in the real vector space R2 with the usual notation x1 î + x2 j,
usual vector algebra is a special case of the general structure of vector space. Similarly we
may observe that for n = 3, i.e. R3 = {(x1 , x2 , x3 ), x1 , x2 , x3 ∈ R} the vector (x1 , x2 , x3 ) may
be identified with the usual vector x1 î + x2 j,ˆ x3 k̂ in the three dimensional Euclidean space.
In a similar manner if we define Cn = {(z1 , z2 , . . . , zn ), zi ∈ C, i = 1, 2, . . . , n} then Cn is
a complex vector space.
Example 1.1.2 Consider the interval [a, b] on the real line. Consider the set
C[a, b] = { f : [a, b] → R, f is a continuous f unction}.
For f, g ∈ C[a, b], let us define ( f + g)(x) = f (x) + g(x), x ∈ [a, b]. Also for α ∈ R and f ∈
C[a, b], we define (α · f )(x) = α f (x), x ∈ [a, b].
We may verify that the set C[a, b] is a real vector space. We may take zero function
0 · (x) = 0 for all x ∈ [a, b] as zero vector, and for f ∈ C[a, b], define − f in C[a, b] as
(− f )(x) = − f (x), x on [a, b].
The vector space C[a, b] is referred to as the vector space of real valued continuous func-
tions on [a, b].
As per our earlier agreement, we are calling elements of a vector space V as vectors.
Therefore, in that sense, we may call f as a vector in the vector space C[a, b]. This example
illustrates that the word ‘vector’ is generic and its meaning depends on the context.
Example 1.1.3 Let V be the collection of all polynomials of degree less than or equal to
n and with real coefficients. Let p1 (x) = a0 + a1 x + a2 x2 + · · · + an xn and p2 (x) = b0 + b1 x +
b2 x2 + · · · + bn xn be two polynomials in V (if the degree is less than n then we may take the
corresponding coefficients as zero). Let us define p1 (x) + p2 (x) = (a0 + b0 ) + · · · + (an + bn )xn
and αp1 (x) = αa0 + αa1 x + · · · + αan xn , α ∈ R. Then we may check that V is a real vector
space. Here the zero polynomial is taken as zero vector and −p1 (x) is defined as −p1 (x) =
−a0 − a1 x − a2 x2 − · · · − an xn . In case coefficients ai are taken as complex numbers, then V
is a complex vector space. Here again, in a generic sense, a ‘polynomial’ may be called a
vector in V.
Subspace VII
Example 1.1.4 A real sequence ξ : (ξ1 , ξ2 , . . . , ξk , . . . ), ξk ∈ R is said to be bounded if
|ξk | ≤ M for all k. Let V be the collection of all such bounded sequence of reals. For
ξ : (ξ1 , ξ2 , . . . , ξk , . . . ) and η : (η1 , η2 , . . . , ηk , . . . ) belonging to V, we define
ξ + η = (ξ1 + η1 , . . . , ξk + ηk , . . . ),
and for
α ∈ R, αξ = (αξ1 , . . . , αξk , . . . ).
Then V is a real vector space referred as, the space of bounded real sequences.
Example 1.1.5 Let V = {A = (ai j )m×n , ai j ∈ R} be the collection of all real (m×n) matrices.
For A = (ai j )m×n and B = (bi j )m×n , ai j , bi j ∈ R, we define
A + B = (ai j + bi j )m×n ,
and
αA = (αai j )m×n , α ∈ R.
Here the zero matrix 0 = (0)m×n and −A = (−ai j )m×n are respectively defined as zero element
and negative element.
We may again verify that V is a real vector space.
1.2 Subspace
Definition 1.2.1 (Subspace) A nonempty subset M of a given vector space V is called
a subspace of V if
(i) u ∈ M, v ∈ M =⇒ u + v ∈ M,
and
(ii) α scalar, u ∈ M =⇒ αu ∈ M.
Equivalently, M ⊆ V is called a subspace if
u, v ∈ M, α, β scalars =⇒ αu + βv ∈ M.
Obviously M = V and M = {0} are subspaces. These are called improper subspace. Other
subspaces M of V are called proper subspaces. By definition of a subspace, it immediately
follows that the zero vector or the null vector belongs to every subspace M of V. Also
every subspace is a vector space in its own right.
VIII Linear Algebra and Optimization
Example 1.2.1 Consider the real vector space R2 . Let M = {(x1 , 0), x1 ∈ R} ⊆ R2 . Then
M is a subspace of R2 .
Example 1.2.2 Consider the real vector space R2 . Let M = {(x1 , x2 ) : 4x1 +3x2 = 0; x1 , x2 ∈
R} ⊆ R2 . Then M is a subspace of R2 . In fact any straight line in R2 which passes through
the origin is a subspace of R2 . But this is not true if the line does not pass through the
origin. This is because the null vector (in this case the origin) has to be an element in
every subspace.
Example 1.2.3 Let P3 be the real vector space of all polynomials with real coefficients
and with degree less than or equal to three. Let P2 ⊆ P3 consists of all polynomials with
real coefficients and of degree less than or equal to two. Then P2 is a subspace of P3 .
n h i o
Example 1.2.4 Let V = A = ai j , ai j ∈ R . be the real vector space of all (n × n)
n×n
real matrices. Let M be collection of all (n × n) real symmetric matrices i.e. M = {A =
[ai j ]n×n , ai j ∈ R, A = AT }. Then M is a proper subspace of V.
Example 1.2.5 Consider the real vector space C[a, b] of all continuous functions f :
[a, b] → R. Let D[a, b] be the set of all differentiable functions f : [a, b] → R. Then D[a, b]
is a subspace of C[a, b].
Example 1.2.6 Consider the real vector space V of all bounded real sequences. Let M ⊆ V
consists of all sequences from V whose first component ξ1 is zero. Then M is a subspace
of V.
We may prove the following proposition form Definition 1.2.1.
Proposition 1.2.1 Let M1 and M2 be two subspaces of a given vector space V. Then
M1 ∩ M2 is also a subspace of V but M1 ∪ M2 may not be a subspace of V.
Definition 1.2.2 (Linear Combination) Let {v1 , v2 , . . . , vm } be a set of m vectors in
a vector space V. Then an expression of the form
α1 v1 + α2 v2 + · · · + αm vm , (1.1)
where α1 , α2 , . . . , αm are scalars, is called a linear combination of vectors v1 , v2 , . . . , vm .
Linear Independence, Basis and Dimension IX
Definition 1.2.3 (Span of a Subset) Let M ⊂ V be nonempty. Then the collection of
all (finite) linear combinations of elements of M is called the span of M and is denoted
as Span(M). In other words Span(M) is given by
Pp
Span(M) = { i=1 αi vi , vi ∈ M, αi scalars, p f inite}
Proposition 1.2.2 For M ⊂ V, Span(M) is a subspace of V.
Proposition 1.2.2 follows because a linear combination of linear combination is also a
linear combination.
Example 1.2.7 Let M1 = {v1 , v2 } ⊂ R2 where v1 = (1, 1) and v2 = (0, 2). Then
Span(M1 ) = {αv1 + βv2 , α, β ∈ R}
= {(α, α + 2β), α, β ∈ R}
= R2 .
Let M2 = {(1, 2)} ⊆ R2 . Then Span(M2 ) = {(α, 2α), α ∈ R2 }, which consists of all points
on the straight line x2 = 2x1 .
Let M3 = {v1 , v2 , v3 }, where v1 = (1, 1), v2 = (0, 2), and v3 = (−1, 1). Then Span(M3 ) =
R2 .
Example 1.2.1. Let M = {1, t, t2 }. Then Span(M) is the vector space of all polynomials
with real coefficients and of degree less than or equal to 2.
1.3 Linear Independence, Basis and Dimension
In this section we first introduce the concept of linear independence/linear dependence of
a set of vectors from V. This concept is important because it leads to two very important
notions, that of a basis and dimension .
Definition 1.3.1 (Linear Independence) Let {v1 , v2 , . . . , vp } be a set of p vectors from
the given vector space V. We say that vectors v1 , v2 , . . . , vp are linearly independent if
p
X
αi vi = 0,
i=1
=⇒ αi = 0 f or all i = 1, 2, . . . , p.
αi scalars
(1 ≤ i ≤ p)
X Linear Algebra and Optimization
Xp
Here αi vi is the notation used for the sum α1 v1 + α2 v2 + · · · + αp vp , and ‘0′ on the
i=1
right hand side refers to the null vector in V.
The vectors {v1 , v2 , . . . , vp } are said to be linearly dependent if they are not linearly
independent, i.e.
p
X
αi vi = 0,
i=1
=⇒ at least one αi , 0.
αi scalars
(1 ≤ i ≤ p)
So far we have defined linear independence/linear dependence for finitely many vectors
from V. Now let M ⊂ V be an arbitrary set. Then M is said to be linearly independent if
every nonempty finite subset of M is linearly independent. The set M is said to be linearly
dependent if M is not linearly dependent.
Remark 1.3.1 From the definition of linear dependence, it immediately follows that if
{v1 , v2 , . . . , vp } are linearly dependent, then at least one vector (say vk ) can be written as a
linear combination of remaining (p − 1) vectors.
Example 1.3.1 Let V = R2 and M1 = {(1, 0), (0, 1)}, M2 = {(1, 1), (1, −1)}, M3 =
{(1, 1), (2, 2)} and M4 = {(1, 0), (−1, 1), (2, 1)}. Then M1 and M2 are linearly independent,
but M3 and M4 are linearly dependent. (Hint: Use Definition 1.3.1)
Example 1.3.2 Consider the vector space C[a, b]. Let fi (x) = xi , i = 0, 1, 2, . . . , n. Let
M = { f0 (x), f1 (x), . . . , fn (x)}. Then M is linearly independent.
Example 1.3.3 V be the vector
Let space ofall (2 × 2) real matrices. Let M = {E1 , E2 , E3 }
1 0
, E = 0 1, and E = 0 0. Then M is linearly independent.
where E1 = 2 3
0 0 1 0 1 0
Definition 1.3.2 (Basis of a Vector Space) Let V be the given vector space. Let
there exist a subset B ⊆ V such that (i) B is linearly independent and (ii) Span(B) = V.
Then B is called a basis (or Hamel basis) of V.
Definition 1.3.3 (Dimension of a Vector Space) Let V be the given vector space.
Then V is said to be a finite dimensional vector space if there is a positive integer n such
Linear Independence, Basis and Dimension XI
that V contains a linearly independent set of n vectors but any set of (n + 1) or more
vectors of V are linearly dependent. In this case n is called the dimension of the given
vector space V, and we write it as dimV = n.
If V is not finite dimensional, it is said to be an infinite dimensional vector space.
For V = {0}, we define its dimension as zero.
We now have the following propositions:
Proposition 1.3.1 Every vector space V , {0} has a basis.
Proposition 1.3.2 All bases for a given vector space (finite or infinite dimensional) V
have the same cardinality.
Thus a vector space V may have more than one bases but all these bases will have the
same cardinality. In particular, if V is finite dimensional, then all bases of V contain the
same number of elements.
Remark 1.3.2 Let V be a finite dimensional vector space and W be a subspace of V.
Then dim(W) ≤ dim(V).
Example 1.3.4 Let V = R2 . Further let B1 = {(1, 0), (0, 1)}, B2 = {(1, 0), (1, 1)}, B3 =
{(1, −1), (−1, 1)} and B4 = {(1, 2), (2, 5), (−4, 3)} be subsets in R2 .
It can be verified that B1 and B2 form bases for R2 but B3 and B4 do not form bases for
R2 because these are linearly dependent.
Example 1.3.5 Let V = Rn . Let ei = (0, 0, . . . 0, 1, 0, . . . 0) ∈ Rn with ′ 1′ occurring at the
ith place. Then {e1 , e2 , . . . , en } is a basis for Rn . This basis is called the standard basis for
Rn . In particular
e1 = (1, 0, 0),
e2 = (0, 1, 0),
e3 = (0, 0, 1),
is the standard basis for R3 . If we take B = {v1 , v2 , v3 }, where
v1 = (1, 0, 0),
v2 = (1, 1, 0),
v3 = (1, 1, 1),
XII Linear Algebra and Optimization
then B also forms a basis for R3 . Thus, the Euclidean space Rn is of dimension n.
1 0
Example 1.3.6 Let V = {A = (ai j )n×n , ai j ∈ R, A = AT } then {E1 , E2 , E3 } with E1 = ,
0 0
0 1 0 0
E2 = , and E =
3
forms a basis of the vector space of all (2×2) real symmetric
1 0 0 1
matrices. Further it’s dimension is three.
Example 1.3.7 Let V be the real vector space of all polynomials of degree less than or
equal to n, Then a basis for V is {1, t, t2 , . . . , tn }. Therefore the dimension of V is (n + 1).
