CHAPTER 1: the fundamental ideas of linear algebra.

We start with a vector (i.e. a) surrounded by an infinite space:

We say that vector a belongs to the vector space R or simply a  R if:

1. for any possible scalar value c (e.g. c = 1.3)  (c x a)  R

2. if b is also in R, then for any a and b in R  (a + b)  R

Combining the previous two properties we obtain the third property:

3. any linear combination (c x a + d x b)  R

Any vector space always contains the 0 vector (the origin).

Until now we have simply represented each vector as an object (e.g. an arrow). If we want to
attribute a value to vectors we need to refer them to a common basis. For example, if we
want to represent all the vectors in the space R (imagined as an infinite plane in all directions)
around the origin shown above, we need 2 basis vectors: they don't need to be orthogonal,
but it is important that they be independent, which means that neither one of the two basis
vectors can be obtained by multiplying the other basis vector by a scalar: for example, b
cannot be obtained by multiplying a by any number, because a and b point in different
directions. Therefore, a and b could be two possible basis vectors for space R. We could
chose arbitrarily any other two independent vectors in R as the basis for that vector space.
Since any vector in the vector space R can be represented as a linear combination of a and
b, we say that the dimensions of R are 2, or simply that a and b belong to R2. It is clear from
this description that R2 represents a vector space that intuitively we recognize as a plane.

If we want to represents all possible vectors in what intuitively we recognize as the volume
around the origin 0, we need 3 independent basis vectors (e.g. v1,v2,v3), which means that
none of the 3 vectors can be obtained as a linear combination of the other 2 (e.g., we can't
get v1 as (b x v2 + c x v3), no matter how we choose the two scalars b and c). All the vectors in
the volume around the origin 0 thus belong to vector space R3, a space of 3 dimensions.

1
Since the space has 3 dimensions, any vector a in R3 can be obtained as a linear
combination of the three basis vectors v1,v2,v3:

a = a1v1 + a2v2 + a3v3

v1,v2,v3 are the basis of space R3 and a1, a2, a3 are the 'coordinates' of a in space R3 defined
by the basis vectors v1, v2, v3. We can conveniently represent a as a column vector:

[]
a1
a= a2
a3

In MATLAB we write:

a = [1 ; 3 ; 6];

or as a 'row' vector:

a=[ a 1 a2 a3 ]

a = [1 3 6];

We can also obtain a row vector by transposing a column (we use the superscript T for this),
and viceversa:

[]
a1
a= a2 a T =[ a1 a 2 a3 ]
a3
a = [1 ; 3 ; 6];
a = a';

2
It is a good habit (consistent with common use in many types of software) to represent
vectors as column vectors. We will almost always assume that vectors are given as column
vectors.

Of course we can represent the basis vectors themselves in the same basis:

v 1 = 1 x v 1 + 0 x v2 + 0 x v 3

[] [] []
1 0 0
v 2 = 0 x v 1 + 1 x v2 + 0 x v 3 v 1= 0 v 2= 1 v 3= 0
0 0 1
v 3 = 0 x v 1 + 0 x v2 + 1 x v 3

Notice again how the three basis vectors of R3 are independent: for example you can't obtain
v1 from any combination of v2 and v3. However, we still don't know in what directions these
basis vectors point: for example they could be pointing in three different random directions.

It is important to realize that the 'intrinsic' properties of R3 are 'postulates' that we define
intuitively: in agreement with the way we perceive the world around us, the natural choice of
basis vectors would be to have them orthogonal (perpendicular) to each other; thus we can
choose to have a Cartesian basis for R3 in which v1  v2  v3  v1:

[] [] [] []
1 0 0 1
0  1  0  0
0 0 1 0

However, there is nothing in the representation of these basis vectors that tells us they are
orthogonal to each other: as we said before, the same three vectors v1,v2,v3 could be pointing
in three different random directions, and still be a basis for R3. In other words, we need a
postulate that sets a rule for the orthogonality of the basis vectors. This orthogonality
postulate for the Cartesian basis of R3 is expressed by the following rule:

vectors are orthogonal if their dot (or 'inner') product is equal to 0

If v1, v2, v3 are column vectors we introduce the matrix notation for the dot product:

v1∙v2 = v1Tv2 = 0 ; v1∙v3 = v1Tv3 = 0 ; v2∙v3 = v2Tv3 = 0

[1 0 0] 0 [1 0 0] 0 [0 1 0] 0

1 = (1x0)+(0x1)+(0x0) = 0 0 = 0 0 = 0
0 1 1

v1 = [1 0 0]';
v2 = [0 1 0]';
v3 = [0 0 1]';
v1'*v2
v1'*v3
v2'*v3

3
The orthogonality condition (dot product = 0) is valid for any vector. For example, it’s easy to
show that vectors a=[ 3 5 1 ] and b=[ 3 −2 1 ] are orthogonal:

a = [3; 5 ;1]
b = [3 -2 1] '
a'*b

The notation ||a|| represents the Euclidean norm (= length) or '2-norm' of the vector, which is:
√ a21 +a22 + a23; therefore, the squared length is easily found by multiplying the vector by its
transpose:

[]
2
3
‖a‖ =a a=[ 3 5 1 ] 5 =(9+25+ 1)=35
T

1
format long
a'*a
sqrt(a'*a)
norm(a)

Notice however that the MATLAB function ‘length’ reports on the number of elements in a
vector, not on its Euclidean length.

length(a)

The Cartesian basis vectors v1, v2, v3. are unit vectors, which means their length (2-norm) is
1. For example:
‖v 1‖= √12 +0+ 0=¿ 1.
norm(v1), norm(v2), norm(v3)

a
If we divide a vector by its length, a^ = , we normalize it, which means the new vector is a
‖a‖
unit vector of length 1.

a
a/norm(a)
norm(a/norm(a))

Basis vectors are usually assembled together in a single matrix, and referred to as the basis
of the space:

[ ]
1 0 0
V = [v1 v2 v3] V = 0 1 0 ; for the standard space the basis is the identity matrix I
0 0 1
V = eye(3)

The fact that in order to determine whether two vectors are orthogonal we calculate their dot
product, immediately suggests that, more generally, the dot product between two normalized

4
vectors provides information about the angle between them. In fact, given two vectors a and
b separated by an angle  (in radians) we have:
T T
a b a b
=cos(θ)⟹ a b=‖a‖‖b‖∗cos (θ)
T
=
‖a‖‖b‖ ‖a‖ ‖b‖

We can easily show that this is true by calculating the cosine of the angle between vector a
and itself:

[] []
T T T 3 3
a a a a a a
= 2 = T =[ 3 5 1 ] 5 / [ 3 5 1 ] 5 =1=cos (0)
‖a‖‖a‖ ‖a‖ a a
1 1

In the particular case in which one of the two vectors is a basis unit vector v (a basis vector of
length 1) the cosine is called a direction cosine and we have:
T T
a v a v
= =cos ( θ )
‖a‖‖v‖ ‖a‖

a v=cos(θ)∗‖a‖
T

which means the dot product a T v corresponds to

the length of the projection of a in the direction of
v.

More generally, if bu is a unit vector pointing in the

direction of the b vector, we have:

T T
a b a bu
= =cos ( θ )
‖a‖‖b‖ ‖a‖

a bu=cos(θ)∗‖a‖
T

which means that the dot product a T u of any

vector a times a unit vector ucorresponds to the
length of the projection of a in the direction of u.

An extension of the dot product between two vectors is the dot product between two
matrices, which is what we usually refer to as a matrix product. The result of this product is a
matrix whose elements are the dot products between the row vectors of the 1 st matrix and the
column vectors of the 2nd matrix:

A = [1 2;3 4]
C= A ∙ B= [ ] [ ] [ ][ ]
1 2 5 6
∙
3 4 7 8
=
1 2 5 6
3 4 7 8
=
[7 []
[ 12 ] 5 [ 1 2 ] 6

[]
8
[ 3 4 ] 5 [3 4 ] 6
7 8
=
]
[]
19 22

[]
43 50[ ]
B = [5 6;7 8]

5
C = A*B

In general, matrix product is not commutative:

C= A ∙ B ≠ B ∙ A=D

D = B*A

6
SPECIAL TOPICS
Other types of products between vectors or matrices.

There are other types of products between vectors; the most important ones are:

1. The dyadic or outer product, represented with the symbol ⊗ (old notation) or ∘ (new
notation). Its matrix notation is:

[] [ ]
3 9 −6 3
C=a ⊗ b=a ∘ b=a b = 5 [ 3 −2 1 ]= 15 −10 5
T

1 3 −2 1
C = a*b'

Notice that while the result of a dot product is a scalar (a number), the result of an outer
product is a matrix. Each element of the column vector multiplies the entire row vector. While
the inner product between vectors (a scalar) is obviously commutative, the outer product
between vectors is not commutative, therefore:

[] [ ]
3 9 15 3
D=b ∘a=b a = −2 [ 3 5 1 ] = −6 −10 −2 =(a b ) =C
T T T T

1 3 5 1
D = b*a'
D = (a*b')'

An extension of the outer product to matrices, called the Kronecker tensor product is also
represented with the symbol ⊗. If A is an m×n matrix and B is a p×q matrix, then the
Kronecker product A ⊗ B is the mp×nq block matrix. Its matrix notation is:

[ ]
9 15 3 5
C= A ⊗ B= [
3 1
−2 4
⊗
3 5
=
0
][ ]
6 0 2
0 2 −6 −10 12 20
0 −4 0 8
A = [3 1;-2 4]
B = [3 5;0 2]
C = kron(A,B)

Notice that A ⊗ B is distributive under inversion and transposition such that

−1 −1 −1 T T T
( A ⊗ B) = A ⊗ B ,(A ⊗ B) =A ⊗ B

For example:

[ ]
9 0 −6 0
T T T
D=( A ⊗ B) = A ⊗ B =
3 −2
1 4 [
⊗
3 0
5 2
=
15
3] [ ] 6 −10 −4
0 12 0
=C
T

5 2 20 8
C^(-1)
kron(A^(-1),B^(-1))

7
C'
kron(A',B')

The kron product is very useful to create a block diagonal matrix. For example we can
multiply a 4x4 identity matrix I by a 2x2 matrix that we want to be repeated along the
diagonal.

A = eye(4);
B = [1 -1;-1 1];
K = kron(A,B)

[ ]
1 −1 0 0 0 0 0 0
−1 1 0 0 0 0 0 0

[ ][
1 0 0 0 0 0 1 −1 0 0 0 0
A ⊗ B=
0
0
1
0
0 0
1 0
⊗
1 −1
−1 1 ]
=K =
0
0
0 −1 1
0 0 0
0 0 0
1 −1 0
0
0
0 0 0 1 0 0 0 0 −1 1 0 0
0 0 0 0 0 0 1 −1
0 0 0 0 0 0 −1 1

2. The double dot product, represented with the colon symbol ':'. This is an element wise
multiplication, followed by a summation of all the products and therefore it is commutative.
For vectors it is identical to the simple dot product:

[ ][ ] [ ][ ] [] []
3 3 3 3 3 3
c=a :b=b :a= 5 : −2 = −2 : 5 = [ 3 −2 1 ] 5 =[ 3 5 1 ] −2 =0
1 1 1 1 1 1

c = sum(a.*b)
c = sum(b.*a)

but it is distinctly different for matrices, as the result of a double dot product between two
matrices is a scalar instead of a matrix (in fact it is the dot product between the vectorized
matrices):

[ ][ ][ ][ ][ ]
9 −6 3 9 15 3 9 −6 3 9 15 3 126 210 42
C= A ∙ B=AB= 15 −10 5 ∙ −6 −10 −2 = 15 −10 5 −6 −10 −2 = 210 350 70
3 −2 1 3 5 1 3 −2 1 3 5 1 42 70 14

[ ][ ][ ]
9 15 3 9 −6 3 315 −210 105
D=B ∙ A=BA= −6 −10 −2 15 −10 5 = −210 140 −70
3 5 1 3 −2 1 105 70 35

A = [9 -6 3;15 -10 5;3 -2 1]

B = A'
C = A*B
8
D = B*A

[ ][ ][ ][ ]
9 −6 3 9 15 3 9 15 3 9 −6 3
c= A : B=B : A= 15 −10 5 : −6 −10 −2 = −6 −10 −2 : 15 −10 5 =0
3 −2 1 3 5 1 3 5 1 3 −2 1

C = A.*B
C = B.*A
c = sum(sum(C))
c = sum(C(:))
c = A(:)’*B(:)

3. The cross product, represented with the symbol ×. While the dot product between a and b
is a scalar:

a ∙ b=a b=cos (θ)‖a‖‖b‖

T

the cross product between a and b is a vector:

a × b=sin(θ)‖a‖‖b‖∗n

where sin(θ)‖a‖‖b‖ is the length of the vector, and n is a unit

vector perpendicular to both a and b, directed as the thumb in
the right hand rule (with a being the index finger and b the
anular).

[][ ][ ]
3 3 7
c=a ×b= 5 × −2 = 0
1 1 −21

[ ][][ ]
3 3 −7
d=b ×a= −2 × 5 = 0
1 1 21

Changing the order of vectors in the cross product inverts the direction of the resulting vector,
but does not change its length.

c = cross(a,b)
d = cross(b,a)

Furthermore, while the dot product is defined for vectors of arbitrary dimensions, the cross
product is only defined for vectors of 3 dimensions.

An important property of the cross product u × v is that its length ‖u × v‖equals the area of a
parallelogram with sides u, v regardless of the angle between these sides and their
orientation in space. Likewise the cross product is conveniently used in the calculation of the
volume of a parallelepiped knowing the 3 vectors u, v, w that represent its 3 sides, regardless

9
of the angles between the vectors. In fact, since u × v is a vector, we can take its dot product
with w . The triple product (u × v)∙ w is a scalar quantity that gives the volume of the u, v, w
box. We can represent the vectors u, v, and w as columns of a matrix:

[ ]
u1 v 1 w1
A= u2 v 2 w2
u3 v 3 w3

For a 3x3 matrix A, the triple product (u × v)∙ w equals the determinant of A, det ( A )=| A|:

| |
u 1 v 1 w1
| A|= u 2 v 2 w 2 =u1 ( v 2∗w3−v 3∗w2 ) −v 1 ( u2∗w3−u3∗w 2 )+ w1 ( u2∗v 3−u 3∗v 2 ) =(u × v )∙ w
u 3 v 3 w3

A = [9 -6 3;15 5 5;3 -2 2]
volA = (cross(A(:,1),A(:,2)))'*A(:,3)
volA = dot(cross(A(:,1),A(:,2)), A(:,3))
volA = det(A)
u1 = A(1,1)
u2 = A(2,1)
u3 = A(3,1)
v1 = A(1,2)
v2 = A(2,2)
v3 = A(3,2)
w1 = A(1,3)
w2 = A(2,3)
w3 = A(3,3)
volA = u1*(v2*w3-v3*w2)-v1*(u2*w3-u3*w2)+w1*(u2*v3-u3*v2)

Other types of vector and matrix norms.

An extension of the 2-norm of a vector is the calculation of the p-norm of a vector. For
example, the 3-norm of a=[ 3 −5 1 ] is:

‖a‖3 =√|a1| +|a2| +|a 3|

3 3 3 3

a = [3 -5 1];
norm_3 = (abs(a(1))^3 + abs(a(2))^3 + abs(a(3))^3)^(1/3)
norm_3 = norm(a,3)

p can be any positive real value, but also + ∞∨−∞ . In the latter case we have:

‖a‖∞=max (|a|)
‖a‖−∞=min (|a|)

10
norm_pinf = max(abs(a))
norm_pinf = norm(a,Inf)
norm_minf = min(abs(a))
norm_minf = norm(a,-Inf)

Two types of extension of the norm concept to matrices can also be easily calculated:

1. The p-norm of a matrix A is defined for p=1 , 2 ,+∞ :

A = [1 2 3;0 4 1;5 3 2;7 1 1]

norm_1 = max(sum(abs(A)))
norm_2 = max(svd(A))
norm_1 = norm(A,1)
norm_2 = norm(A,2)

where we use singular value decomposition (SVD, CHAPTER 11) to calculate the
largest singular value of the matrix.

norm_pinf = max(sum(abs(A')))
norm_pinf = norm(A',1)
norm_pinf = norm(A,Inf)

2. The Frobenius-norm is defined for a m x n matrix A as :

√∑
n
‖ A‖fro = ( A T A ) ii
i=1

norm_fro = sqrt(sum(diag(A'*A)))
norm_fro = norm(A,'fro')

and corresponds to the 2-norm of the vectorized matrix:

norm_fro = norm(A(:))

PRACTICE

1. Find 5 vectors perpendicular to the vector a = [ 3 5 1 ] . Calculate the angle between all
pairwise combinations of the 5 vectors. Can you determine if all 5 vectors are linearly
independent from each other? If not, can you explain why?

2. Find an orthonormal basis in which the u1 basis vectors is colinear with the vector a of the
previous problem. Then display both the standard and the new orthonormal basis in the same
plot using the function 'show_frame'.

3. Practice with matrix products. For example, given the three matrices:

A = [1 3;2 3;4 5]

11
B = [3 2 4 2; 4 4 1 2; 1 1 4 5]
C = [ 2 2 1 1; 1 3 1 3]

calculate different combinations of products and try to understand why some of them give you
an error unless you transpose one or more matrices. Then, based on your observations,
derive a transposition law for matrix products. For example, derive the expression equivalent
to:

(ATBCT)T

that does not include a final transposition of the product inside the parenthesis.

12