Lecture 06

System of Linear
Equations
1. Introduction

System of linear equations is an extremely important problem in linear algebra. The reason is
that these appear in solving many problems, whose initial objective may not be the solution of a
system of linear equations. For example, obtaining solutions of ordinary differential equations,
solutions of partial differential equations, interpolation, and integration require solving a system
of linear equation. Linear systems arise in applications to such areas as business, economics,
sociology, ecology, demography, genetics, electronics, engineering, and physics. With the
availability of faster and more power computers, a lot of effort is going into developing more
efficient and faster algorithms to obtain solution of a system of linear equations.

Algebraically, a linear system of m equations and n unknows x1, x2, ⋅ ⋅ ⋅ xn is written as

𝑎 𝑥 +𝑎 𝑥 +∙ ∙ ∙ 𝑎 𝑥 =𝑏
𝑎 𝑥 +𝑎 𝑥 +∙ ∙ ∙ 𝑎 𝑥 =𝑏 ⎫ ⎪
⬚ (1)
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ⎬
⬚ ⎪
𝑎 𝑥 +𝑎 𝑥 +∙ ∙ ∙ 𝑎 𝑥 =𝑏 ⎭
The system is called linear because each variable xj appears in the first power only. The entities
aij are called the coefficients, and bj are the known quantities, also referred to as the right-hand-
side. If all the bjs are zero, system (1) is called a homogeneous system. If all the bjs are not zero,
system (1) is a nonhomogeneous system.

A solution to (1) is a set of numbers x1, x2, ⋅ ⋅ ⋅ , xn that satisfies all m equations. A solution
vector of (1) is a vector x whose components form a solution of 1). If system (1) is
homogeneous, it always has at least the trivial solution x = (0).

System (1) can be conveniently written in the form of matrices as Ax = b, where A is an m  n

coefficient matrix, x is an n-element column-vector, and b is an m-element column-vector.
𝑥
𝑎 𝑎 ∙ ∙ ∙ 𝑎 𝑏
𝑥
𝑎 𝑎 ∙ ∙ ∙ 𝑎 ⎛ ⎞ 𝑏
⋮ (2)
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ⎜ ⎟= ⋮
⋮
𝑎 𝑎 ⋅ ⋅ ⋅ 𝑎 𝑏
⎝𝑥 ⎠

From an engineering perspective, the system (2) is a input-system-output system. The right-hand-
side vector b is the input to the system whose properties are given by the coefficient matrix A.
We are trying to ascertain the behavior of the system given by the output vector x.

Before we embark upon a method to find solutions of system (2), we will try to understand the
geometric interpretation of a solution.

2. Geometric interpretation of solution

Let us assume that we have a 2  2 system given by
𝑎 𝑥 +𝑎 𝑥 =𝑏
(3)
𝑎 𝑥 +𝑎 𝑥 =𝑏
This system of equation can be rewritten as
𝑎 𝑏
𝑥 =− 𝑥 + ⎫
𝑎 𝑎
(4)
𝑎 𝑏 ⎬
𝑥 =− 𝑥 +
𝑎 𝑎 ⎭
These equations can be thought of as equations of straight lines. The solution of this system of
linear equation is the point of intersection of the two lines. The two lines can intersect in one of
the three ways shown in Figure 1. In Figure 1(a) the two lines are distinctly different, therefore

x2 x2 x2
l2
l2 l2
l1 l1
l1

x1 x1 x1
(a) (b) (c)
Figure 1. Geometrical interpretations of solution of a 2 × 2 system of linear equation.

there is a unique solution. In Figure 1(b), the two lines are concurrent, therefore there are infinite
numbers of solutions. In Figure 1(c) the two lines are parallel to each other. Therefore, there is no
point of intersection. Therefore, the problem is an inconsistent problem.

This behavior of a linear system will be prevailing for larger systems also. These considerations
are therefore important, so that the computational software can branch out in the right direction.

Obtaining a solution of a system of linear equation has been studied for a long time. Very
efficient and sophisticated algorithms have been developed for different scenarios. The
algorithms depend mainly on the characteristics of the coefficient matrix A. There are different
algorithms available for packed-dense matrix, sparse matrix, tridiagonal matrix, blocked matrix,
etc. For the time being, we concentrate on an initial preliminary algorithm for finding a solution.

3. Naïve Gaussian elimination

In this section, the simplest form of Gaussian elimination is explained. The adjective naïve
applies because this form is not suitable for automatic computation unless essential modifications
are made. We will not discuss those modifications here.

The simplest Gaussian eliminations has two steps.

 (i) Forward elimination
(ii) Back substitution
In forward elimination, the coefficient matrix A is converted into an upper triangular matrix. To
achieve this, it employs what are known as elementary row operations. There are three
permissible elementary row operations. These are
(i) multiplying an equation by a constant;
(ii) adding/subtracting two equations, term by term;
(iii) interchanging two equations.
In the back substitution step, the solution vector x is obtained from the upper triangular matrix.
We learn more details through an example

Example 01
Obtain solution of the 4  4 system using Gaussian elimination
6x1 – 2x2 + 2x3 + 4x4 = 16
12x1 – 8x2 + 6x3 + 10x4 = 26
3x1 – 13x2 + 9x3 + 3x4 = –19
–6x1 + 4x2 + x3 – 18x4 = –34

We write the problem as a matrix problem

6 −2 2 4 𝑥 16
12 −8 6 10 𝑥 26
3 −13 9 3 𝑥 = −19
−6 4 1 −18 𝑥 −34
We combine the coefficient matrix and the right-hand-side to construct the augmented matrix
6 −2 2 4 16
12 −8 6 10 26
3 −13 9 3 −19
−6 4 1 −18 −34
At the step the element a11 (= 6) is called the pivot element, and row 1 is called the pivot row. We
will use the pivot element to convert all the elements below that to 0. We achieve this with
following operations
𝑎
𝑅 =𝑅 − 𝑅
𝑎
Similarly, we replace
𝑎
𝑅 =𝑅 − 𝑅
𝑎
𝑎
𝑅 =𝑅 − 𝑅
𝑎
This will convert a21, a31 and a41 to zero. The resulting matrix is
6 −2 2 4 16
0 −4 2 2 −6
0 −12 8 1 −27
0 2 3 −14 −14
In the second step, we mentally ignore the first equation. Now we consider element a22 as the
pivot element, R2 as the pivot row, and repeat the same process as we did earlier. This results in
6 −2 2 4 16
0 −4 2 2 −6
0 0 2 −5 −9
0 0 4 −13 −21
The process is repeated in the third step with a33 as the pivot element and R3 as the pivot row.
This results in
6 −2 2 4 16
0 −4 2 2 −6
0 0 2 −5 −9
0 0 4 −3 −3
The coefficient matrix is in the upper triangular form. This form is referred to as the row-echelon
form. This completed the forward elimination.

We obtain the solution using back substitution. We start with the last equation, solve for x4.
−3
𝑥 = =1
−3
Having found x4, we move backward one equation, we find x3, similarly x2 and x1. Finally, the
vector xT = (3, 1, –2, 1)T.

This system of linear equations has led to a unique solution.

This, perhaps, is also a good place to note what we learnt about matrices at the onset. Matrices
do not have any value. We convert a physical problem into a matrix problem, do some algebra,
and finally go back to the physical problem.

To see the behavior of the procedure for infinite number of solutions, let us consider the
following system.
6x1 – 2x2 + 2x3 + 4x4 = 16
12x1 – 8x2 + 6x3 + 10x4 = 26
3x1 – 13x2 + 9x3 + 3x4 = –19
9x1 + 5x2 – 3x3 + 7x4 = 45
After the forward elimination phase, the system will end up as follows.
6 −2 2 4 16
0 −4 2 2 −6
0 0 2 −5 −9
0 0 0 0 0
The last equation is identically zero on both sides. Therefore, in fact though we though we had
four equations, we actually have three equations. The third equation is
2 x3 – 5 x4 = –9
There are two unknowns, but one equation. Therefore, we select one of the unknowns as an
arbitrary constant, Therefore, let x4 = . Completing the process, we obtain the final solution
 𝐓
11 43 7 3 5 9
𝐱𝐓 = 𝛼+ , 𝛼− , 𝛼− , 𝛼
12 12 4 4 2 2
As we can choose x4 arbitrarily, this gives us infinite solutions. This kind of situation is, actually,
quite common in engineering problems. This situation eventually leads to an optimization
problem.

To see the pattern of an inconsistent problem, once again, let us consider the system
6x1 – 2x2 + 2x3 + 4x4 = 16
12x1 – 8x2 + 6x3 + 10x4 = 26
3x1 – 13x2 + 9x3 + 3x4 = –19
9x1 + 5x2 – 3x3 + 7x4 = 40
After the forward elimination phase, the system will end up as follows.
6 −2 2 4 16
0 −4 2 2 −6
0 0 2 −5 −9
0 0 0 0 −5
The last line leads to a situation where 0 = –5; which is not possible. Therefore, the system is
inconsistent, and it does not have a solution.

We will learn more about the nature of the solution of a system of linear equations more formally
after we learn about finding solution of a homogeneous system of linear equations.

4. Homogeneous system

A system of linear equation whose right-hand side is zero is called a homogeneous system of
linear equation. Taking the example from above,
6x1 – 2x2 + 2x3 + 4x4 = 0
12x1 – 8x2 + 6x3 + 10x4 = 0
3x1 – 13x2 + 9x3 + 3x4 = 0
For the time being, just know this that for a homogeneous system, we can get a solution if and
only if we have one less number of independent equation than the number of unknowns. We will
discuss this more formally shortly. A solution for this system is obviously x1 = x2 = x3 = x4 = 0.
This is called the trivial solution. Obviously, we are not interested in this solution. We are
looking for a non-trivial solution.

To find the non-trivial solution, we proceed with row elimination, as usual. The row-reduced
form will be.
6x1 – 2x2 + 2x3 + 4x4 = 0
– 4x2 + 2x3 + 2x4 = 0
2x3 – 5x4 = 0
To find a solution, we assume x4 = . That will give us x3 = (5/2), x2 = (7/3) and
x1 = (11/12). Since a is chosen arbitrarily, it can be considered any convenient number, so that
the solution can be written as xT = (12, 30, 21, 11)T.
Exercise

Solve the system given explicitly, or by its augmented matrix. Show details
1. 4𝑥 − 6𝑦 = −11 2. 3.0 −0.5 0.6
−3𝑥 + 8𝑦 = 10 1.5 4.5 6.0
3. 𝑥+𝑦−𝑧 = 9 4. 4 1 0 4
8𝑦 + 6𝑧 = −6 5 −3 1 2
−2𝑥 + 4𝑦 − 6𝑧 = 40 −9 2 −1 5
5. 13 12 −6 6. 4 −8 3 16
−4 7 −73 −1 2 −5 −21
11 −13 157 3 −6 1 7
7. 2 4 1 0 8. 4𝑦 + 3𝑧 = 8
−1 1 −2 0 2𝑥 − 𝑧 = 2
4 0 6 0 3𝑥 + 2𝑦 = 5
9. −2𝑦 − 2𝑧 = −8 10. 5 −7 3 17
3𝑥 + 4𝑦 − 5𝑧 = 13 −15 21 −9 50
11. 0 5 5 −10 0 12. 2 −2 4 0 0
2 −3 −3 6 2 −3 3 −6 5 15
4 1 1 −2 4 1 −1 2 0 0
13. 10𝑥 + 4𝑦 − 2𝑧 = −4 14. 2 3 1 −11 1
−3𝑤 − 17𝑥 + 𝑦 + 2𝑧 = 2 5 −2 5 −4 5
𝑤+𝑥+𝑦 = 6 1 −1 3 −3 3
8𝑤 − 34𝑥 + 16𝑦 − 10𝑧 = 4 3 4 −7 2 −7
15. 𝑥 + 𝑦 + 2𝑧 = 4 16. 𝑥 − 2𝑦 + 3𝑧 = 2
2𝑥 + 3𝑦 + 6𝑧 = 10 2𝑥 − 3𝑦 + 8𝑧 = 7
3𝑥 + 6𝑦 + 10𝑧 = 17 3𝑥 − 4𝑦 + 13𝑧 = 8
17. 𝑥 + 2𝑦 + 3𝑧 = 3 18. 𝑥 − 2𝑦 = 5
2𝑥 + 3𝑦 + 8𝑧 = 4 2𝑥 + 3𝑦 = 3
5𝑥 + 8𝑦 + 19𝑧 = 11 3𝑥 + 2𝑦 = 7
19. 2𝑤 + 𝑥 + 2𝑦 − 3𝑧 = 2 20. −5𝑤 + 𝑥 + 2𝑦 + 4𝑧 = 3
3𝑤 + 2𝑥 + 5𝑦 − 8𝑧 = 5 2𝑤 + 3𝑥 − 𝑦 + 5𝑧 = 4
2𝑤 + 3𝑥 + 6𝑦 − 𝑧 = 4 9𝑤 + 5𝑥 − 4𝑦 + 6𝑧 = 2
21. 𝑤 + 𝑥 + 2𝑦 + 3𝑧 = 0 22. 2𝑥 − 4𝑥 + 3𝑥 − 𝑥 + 2𝑥 = 0
4𝑤 + 2𝑥 + 4𝑦 + 7𝑧 = 0 3𝑥 − 6𝑥 + 5𝑥 − 2𝑥 + 4𝑥 = 0
5𝑤 + 3𝑥 + 6𝑦 + 10𝑧 = 0 5𝑥 − 10𝑥 + 7𝑥 − 3𝑥 + 18𝑥 = 0

5. Linear independence, Rank of a matrix, Vector space

We will attempt to learn some concepts those will help us to understand the behavior of a system
of linear equations. What we are trying to understand is that if there are m equations with n
unknowns, what will be the nature of the solution of the system. We will try to ascertain how
many variables we will need to choose independently, how many variables will be dependent.

Linear dependence
Given a set of vectors v1, v2, v3, . . ., vn, a combination
c1v1 + c2v2 + c3v3 + . . . + cnvn (5)
where c1, c2, c3 . . . cn are scalar constant is called a linear combination of v1, v2, v3, . . ., vn.

If a linear combination of vectors

c1v1 + c2v2 + c3v3 + . . . + cnvn = 0 (6)
Provided that not all constants c1, c2, c3 . . . cn are zero, the set v1, v2, v3, . . ., vn is said to be
linearly dependent. If only way (6) can be true for the given set of vectors is if all the constants
are zero, the set v1, v2, v3, . . ., vn is said to be linearly independent. Linear dependence implies
that any vector in the set can be expressed in terms of other vectors.

Example 02
If v1 = 2i + 3j – 4k, v2 = i – 2j + 2k, v3 = 3i + j – 2k, then v1 + v2 – v3 = 0. Therefore, the set v1,
v2, and v3 is linearly dependent.

Example 03
v1 = i, v2 = j, and v3 = k, the only way c1v1 + c2v2 + c3v3 = 0 is if c1 = c2 = c3 = 0. Therefore, the
set v1, v2, and v3 is linearly independent.

Example 04
The set of vectors
𝐯𝟏 3 0 2 2
𝐯𝟐 = −6 42 24 54
𝐯𝟑 21 −21 0 −15
are dependent because
1
6𝐯 − 𝐯 − 𝐯 = 0
2
Although it is easy to check, it is not so obvious. The question is for any given set of vectors how
we ascertain this.

Rank of a matrix
The rank of a matrix, , is defined as the number of non-zero rows of the matrix after row-
reduction.

The rank of a matrix is the basic tool that is used to determine whether a given set of vector is
dependent or independent. For a given set of n vectors v1, v2, v3, . . ., vn, a matrix A is formed by
writing the vectors as rows of the matrix. If A = n, the set is independent. If A < n, the set is
dependent. We see this through an example.

Example 05
Given v1 = [1,2,–3], v2 = [3,–2,–1], v3 = [4,0,–4], v4 = [2,–4,2], determine the dependency.

Writing the vector as rows of a matrix, we obtain

1 2 −3
3 −2 −1
4 0 −4
2 −4 2
After row-reduction, this becomes
1 2 −3
0 −8 8
0 0 0
0 0 0
Therefore, the rank of the matrix is 2. This means that the dimension of the set of vector is 2.
This also means that the set is a dependent set.

Therefore, there exists non-zero values of c1, c2, c3, and c4, such that
c1v1 + c2v2 + c3v3 + c4v4 = 0
Writing the vectors, we obtain
c1[1,2,–3] + c2[3,–2,–1] + c3 [4,0,–4] + c4[2,–4,2] = [0,0,0]
Taking the individual components of the four vectors separately, we obtain
c1 + 3 c2 + 4 c3 + 2 c4 = 0
2 c1 – 2 c2 – 4 c4 = 0
–3 c1 – c2 – 4 c3 + 2 c4 = 0
This leads to a homogeneous system of linear equation with c1, c2, c3, and c4 as the unknowns. To
find the solution, we write the coefficient matrix
1 3 4 2
2 −2 0 −4
−3 −1 −4 2
After row-reduction, this becomes
1 3 4 2
0 8 8 8
0 0 0 0
To find a solution, we write c4 = , c3 = . Therefore, c2 = –( + ), and c1 =  – .

To check, choosing  = 1,  = 2, c1 = –1, c2 = –3, c3 = 2, c4 = 1.

We can check – v1 – 3v2 + 2v3 + v4 = 0.

Formal description of solution of a System of Linear Equations

We have discussed that a system of linear equation can have three possible kinds of solutions,
unique, infinite, and inconsistent. We also discussed the structural form of the coefficient matrix
and the augmented matrix in each of these three cases. After our discussion on rank of a matrix,
we can formalize our discussion.

Let  denote the rank of the coefficient matrix, aug the rank of the augmented matrix, and n be
the number of unknown. We can formalize.

Unique solution  = aug = n

Infinite number of solutions  = aug < n

Inconsistent problem  < aug

In case of infinite number of solutions, n –  is referred to as the nullity of the matrix A.

Vector Space
The following related concepts are of general interest in linear algebra. In the present context
they provide a clarification of essential properties of matrices and their role in connection with
linear systems.

Consider a nonempty set V of vectors where each vector has the same number of components.
The vectors in the set satisfy the following properties.
a) The elements are commutative, A + B = B + A
b) The elements are associative, (A + B) + C = A + (B + C)
c) There is a 0 element, such that A + 0 = A
d) For every element A, there is an element –A in the set, such that A + (–A) = 0

The maximum number of linearly independent vectors in V is called the dimension of V and is
denoted by dim V. Here we assume the dimension to be finite.

A linearly independent set in V consisting of a maximum possible number of vectors in V is

called a basis for V. In other words, any largest possible set of independent vectors in V forms
basis for V. That means, if we add one or more vector to that set, the set will be linearly
dependent. Thus, the number of vectors of a basis for V equals dim V.

The set of all linear combinations of given vectors with the same number of components is called
the span of these vectors. Obviously, a span is a vector space. If in addition, the given vectors
are linearly independent, then they form a basis for that vector space.

This then leads to another equivalent definition of basis. A set of vectors is a basis for a vector
space V if (1) the vectors in the set are linearly independent, and if (2) any vector in V can be
expressed as a linear combination of the vectors in the set. If (2) holds, we also say that the set of
vectors spans the vector space V.
By a subspace of a vector space V we mean a nonempty subset of V (including V itself) that
forms a vector space with respect to the two algebraic operations (addition and scalar
multiplication) defined for the vectors of V.

Exercise

Find rank.

1. 4 −2 6 2. 𝑎 𝑏
−2 1 −3 𝑏 𝑎
3. 0 3 5 4. 6 −4 0
3 5 0 −4 0 2
5 0 10 0 2 6
5. 0.2 −0.1 0.4 6. 0 1 0
0 1.1 −0.3 −1 0 4
0.1 0 −2.1 0 4 0
7. 8 0 4 0 8. 2 4 8 16
0 2 0 4 16 8 4 2
4 0 2 0 4 8 16 2
2 16 8 4
9. 9 0 1 0 10. 5 −2 1 0
0 0 1 0 −2 0 −4 1
1 1 1 1 1 −4 −11 2
0 0 1 0 0 1 2 0

Are the following sets of vectors linearly dependent or independent? If they are dependent, find
the dependency relationship.

11. (3 4 0 2), (2 –1 3 7), (1 16 –12 -22)

12. (1 ½ 1/3 ¼), (1/2 1/3 ¼ 1/5), (1/3 ¼ 1/5 1/6), (1/4 1/5 1/6 1/7)
13. (0 1 1), (1 1 1), (0 0 1)
14. (1 2 3 4), (2 3 4 5), (3 4 5 6), (4 5 6 7)
15. (2 0 0 7), (2 0 0 8), (2 0 0 9), (2 0 1 0)
16. (0.4 –0.2 0.2), (0 0 0), (3.0 –0.6 1.5)
17. (9 8 7 6 5), (9 7 5 3 1)
18. (4 –1 3), (0 8 1), (1 3 –5), (2 6 1)
19. (6 0 –1 3), (2 2 5 0), (–4 –4 –4 –4)

6. Inverse of a matrix
Inverse of a matrix A, indicated by A–1, exists only for a square matrix with  = n. The inverse of
a matrix A is defined such that
 A–1A = A–1A = I
where I is the identity matrix. To find the inverse of a matrix A, we write the matrix A and an
identity matrix I of same size side by side. Now using elementary row operations, we convert the
A matrix into an identity matrix. Simultaneously same operations are performed on the I matrix
also. After the A matrix has been converted to an identity matrix, the transformation of the
original identity matrix is the inverse of A. To convert the matrix into an identity matrix, first we
perform a forward elimination; next we perform a backward elimination. The resulting
elimination process is called Gauss–Jordan elimination. We see this through an example.

Example.
Find A–1 for
−1 1 2
3 −1 1
−1 3 4

−1 1 21 0 0
3 −1 1 0 1 0
−1 3 40 0 1
After performing forward elimination on the left half, we obtain
−1 1 2 1 0 0
0 2 7 3 1 0
0 0 −5 −4 −1 1
To prepare for backward elimination, we convert all the diagonal elements of the left half of the
matrix to 1 by dividing individual rows with –1, 2 and –5 respectively.
1 −1 −2 −1 0 0
0 1 3.5 1.5 0.5 0
0 0 1 0.8 0.2 −0.2
In the backward elimination phase, we use the 1 in the last row to convert 3.5 and –2 above that
to zero. The process will continue to convert the element in (1,2) position to zero. The final result
is
1 0 0 −0.7 0.2 0.3
0 1 0 −1.3 −0.2 0.7
0 0 1 0.8 0.2 −0.2
Final result
−0.7 0.2 0.3
𝐀 = −1.3 −0.2 0.7
0.8 0.2 −0.2

Exercise

Find the inverse.

1. 1.8 −2.32 2. cos 2𝜃 sin 2𝜃
−0.25 0.6 −sin 2𝜃 cos 2𝜃
3. 0.3 −0.1 0.5 4. 0 0 0.1
2 6 4 0 −0.4 0
5 0 9 2.5 0 0
5. 1 0 0 6. −4 0 0
2 1 0 0 8 13
5 4 0 0 3 5
7. 0 1 0 8. 1 2 3
1 0 0 4 5 6
0 0 1 7 8 9
9. 0 8 0 10. 2 1 2
0 0 4
2 0 0 ⎛ 3 3 3⎞
2 2 1
⎜− ⎟
⎜ 3 3 3⎟
1 2 2
⎝ 3 3 −
3⎠