Module 3: Supplementary Slides
(These additional materials are optional and intended for students who are interested)
Vectors
More Vector Arithmetic
m%%n Modulo operator (gives the remainder of m/n)
%/% Integer division (gives the integer part of m/n)
%*% Matrix multiplication (to be studied later)
%in% Returns TRUE if the left operand occurs in its right operand; FALSE otherwise
> 14%%5
[1] 4
> 14%%5
[1] 2
> 5%in%14
[1] FALSE
> 5%in%c(5,4)
[1] TRUE
General Norm of a Vector
Norm of a Vector
Definition: a norm for a vector is a function that satisfies the following properties:
1. and . (Positive definiteness)
2. (Homogeneity)
3. (Triangular inequality)
A norm is a function that assigns a length to a vector. To compute the distance between two
vectors, we calculate the norm of the difference between those two vectors. For example, the
distance between two column vectors and using the Euclidean norm is
Common Norms
norm is a family of commonly used norms for vectors that are determined by a scalar as:
Examples:
• norm: (Manhattan/ City-block norm)
• norm: (Euclidean norm: we use only this)
• norm: (Maximum norm)
Angle between Two Vectors
(i) Iand are two unit vectors, then
(ii) Cosine Formula: If and are two nonzero vectors then
(iii) Schwartz Inequality: If and are two nonzero vectors then
Angle between Two Vectors
Part (i): First, consider and . Then, clearly . After a rotation through any angle these are still unit
vectors. Call the vectors and . Their dot product is
. Since equals , we have reached the formula .
Parts (ii) and (iii) are immediate, following Part (i)
Other Properties of Matrices
Matrix Multiplication
Block Matrices and Block
Multiplication
The elements of can be cut into blocks, which are smaller matrices. If the cuts between columns of
match the cuts between rows of , then block multiplication is allowed.
Example:
Linear Equations
• Coefficient matrix
• Variable vector
• Vector of right hand side
The product is the combination of columns of . Hence, the system has solution if is inside the
spanned space of the columns of :
Homogeneous systems
Homogeneous System: The system
𝑨𝑚×𝑛 𝒙𝑛 ×1=𝒃𝑚×1 ⟺¿
is called a homogeneous if . The system is non-homogeneous if at least one of the is not 0.
1. if , then the columns of are linearly dependent, i.e., columns are independents and columns
can be written as a linear combination of the other columns.
2. if , then the columns of are linearly independent, i.e., no column can be written as a linear
combination of other columns.
Moore-Penrose Pseudo Inverse
Moore-Penrose Pseudo Inverse: When the system of linear equations can have no solution. In that case, we
can resort to an approximation by using a least square in which we determine the best vector that
minimizes the sum of square of errors . The best fit is obtained as
Note that is invertible because it is square matrix. is sometimes called Moore-Penrose Pseudo Inverse.
The minimum Euclidean norm (i.e., the minimum of
squares of errors) occurs at a point that satisfies:
Example
Moore-Penrose Pseudo Inverse
Solve the system by finding the inverse of the coefficient matrix.
{
𝑥 + 𝑦=2
𝑥 − 𝑦= 0
𝑥 +2 𝑦 =1
no solution
([ ][ ]) [ ][ ] [ ]
−𝟏
1 1 2
[ ]
−𝟏
𝑥 1 1 1 1 1 1 0.71
=( 𝑨 𝑨 )
𝑻 𝑻
𝒙= 𝑨 𝒃= 1 −1 0 =
𝑦 1 −1 2 1 −1 2 0.43
1 2 1
[ ][ [] [ ] [ ]
1 1 1 1 1 .14
𝑨𝒙 = 1
1
−1
2
𝑥
𝑦]=𝑥 1 + 𝑦
1
− 1 = 0.29
2 1.57
Example
Moore-Penrose Pseudo Inverse
library(pracma)
A=matrix(c(1,1,1,-1,1,2),3,2,1)
b=matrix(c(2,0,1),3,1)
x=solve(t(A)%*%A)%*%t(A)%*%b
Norm(A%*%x-b)
f= function(x) {
y=Norm(A%*%x-b);
return(y);
}
optim(x,f)
$par
[,1]
[1,] 0.7142857
[2,] 0.4285714
$value
[1] 1.069045
Basis of a Vector Space
Basis: The linearly set of independent vectors in the vector space that every other vector is a
linear combination vectors from the basis and every linear combination is unique.
Determinants
Determinants
Determinant: The determinant of the symmetric matrix is a recursive function that maps into a real number
by using Laplace Expansion:
In R use: det(A)
Laplace Expansion: For all
• (expansion along column j)
• (expansion along row j)
is a submatrix of that we obtain by deleting row and column .
Remark: Using Laplace expansion along either the first row or the first column, it is not too difficult to verify:
• If then
• If then
• The determinant of a diagonal matrix is the product of the elements on its main diagonal entries.
Determinants
Example
Compute the determinant of
Solution. Using Laplace expansion along the first row, we have
Remark: gives n-dimensional volume of a n-dimensional parallelepiped made by the column vectors
of . If , then this parallelotope has a zero volume in n dimensions. or it is not n-dimensional, which
indicates that the dimension of the image of is less than n (we say the rank of is less than n).
Determinants
Properties of Determinant
1.
2.
3.
4. Adding a multiple of a column/row to another does not change
5. Multiplication of a column/row with scales by . In particular
6. Swapping two rows/columns changes the sign of
7. Determinant of any diagonal matrix is the product of the elements on its main diagonal entries.
8. Similar matrices have the same determinant
o Two matrices are similar if there exists an invertible matrix with
o Using the definition:
Theorem: is invertible and full-rank, i.e., , if and only if
Determinants
Example
Compute the determinant of (this time by using determinant properties).
Solution. Our strategy is to use determinant properties to change the first column to .
To do so, adding -3 times row 1 to row 3 gives:
Now expanding across column 1 is very easy:
This approach is especially helpful for obtaining the determinants for higher dimensional matrices.
Eigenvalues and
Eigenvectors
Eigenvalues and Eigenvectors
Definition: is an eigenvalue of and is the corresponding eigenvector of if:
In R use: eigen(A)
The above equation is known as the eigenvalue equation.
Remark: The following statements are equivalent:
• is an eigenvalue of
• There exists with or equivalently , can be solved non-trivially, i.e., .
• is called singular, i.e., meaning that it is not invertible.
• Remark: is also known as the Characteristic Polynomial
Eigenvalues and Eigenvectors
Properties of Eigenvalues and
Eigenvectors
Theorem (non-uniqueness of eigenvector): If is an eigenvector of associated with the eigenvalue
then for any , is also an eigenvector of with the same eigenvalue.
Theorem: is an eigenvalue of if and only if is a root of the characteristic polynomial of
Other properties:
• and have the same eigenvalues but not necessarily the same eigenvectors.
• Similar matrices have the same eigenvalues.
• Symmetric positive definite matrices always have positive eigenvalues.
• Determinant of a matrix is equal to the product of its eigenvalues.
Eigenvalues and Eigenvectors
Example
Find the eigenvalues and the eigenvectors of
Solution.
Step 1: eigenvalues
Step 2: eigenvectors corresponding to each eigenvalue:
If
If
Matrix Decomposition
(Important tool for obtaining complex computations, e.g., , and many other results)
Matrix Decomposition
Eigendecomposition and
Diagonalization
Similar matrices: Two matrices are similar if there exists an invertible matrix with
Diagonal Matrix: A matrix is diagonal if
Diagonalizable Matrix: A matrix is diagonalizable if it is similar to a diagonal matrix, i.e., if there
exists a diagonal matrix and an invertible matrix such that .
Theorem (Eigendecomposition): A square matrix can be factored into
where is a matrix whose columns are the eigenvectors of and is a diagonal matrix whose diagonal
entries are eigenvalues of .
Matrix Decomposition
Eigendecomposition
Proof.
is diagonalizable if it is similar to a diagonal matrix , i.e., if there exists such that which is the same as .
Let be a diagonal matrix with the eigenvalues on its main diagonal entries and . Then:
This implies that or . Therefore, must be an eigenvector corresponding to .
Matrix Decomposition
Real Powers of a Matrix
Remark: For , we can see:
Continuing this way, we can verify that
It can be shown that the above result holds generally for any not just integer values. This result,
which is based on matrix decomposition, is extremely important in finding when is a very large
number or when it is a real number (e.g., ) in which case the direct approach is not applicable.
Matrix Decomposition
Exponential and Logarithm of a
Matrix
Definition: For a matrix , the exponential of is defined by the Taylor expansion of on as:
Theorem: For a diagonalizable matrix , we have
Proof.
Matrix Decomposition
Exponential and Logarithm of a
Matrix
Theorem: For a diagonalizable matrix , we have
Proof. It is enough to show that the above formula satisfies . We show only as showing the other is very similar.
Matrix Decomposition
Example
If , find the following values:
a.
b. ( the Euler's constant).
c.
Solution. From the previous example’s solution, we have
d.
e.
f.
y$vectors%*%diag(exp(y$values))%*%matrix.inverse(y$vectors)
Matrix Decomposition
Relationship between Eigenvalues and
Determinant
Theorem: Let be the eigenvalues of the matrix . Then
In addition, if is singular (i.e., ) then it has at least an eigenvalue, which is zero.
Proof.
From the eigendecomposition of , we know that where is the matrix of eigenvectors and is the
diagonal matrix whose main diagonal entries are the eigenvalues. Taking the determinant gives:
If is singular, then . Hence, one of the eigenvalues is at least zero.
Matrix Norms
Norm of a matrix: The definition of the corresponding norm for an matrix is
Where is the Euclidean norm.
Remark: From the above, it follows that:
Remark: it can be shown that
where is the maximum eigenvalue of
(The proof needs additional discussion about orthonormal eigenvector bases…)
Positive Definite Matrices
Definition: a symmetric is called positive semidefinite if and only if
And positive definite if and only if
Theorem: is positive semidefinite if and only if all its eigenvalues are greater than or equal to zero.
Proof.
By definition, we have . Choose to be any of the eigenvectors of with the corresponding eigenvalue to .
Hence, we have
Since for any (it is the Euclidean or norm), we must have .
Remark: From above, it follows that if is positive semidefinite then because .
Positive Definite Matrices
Theorem: For a matrix we can always obtain a symmetric positive semidefinite matrix by defining
Proof.
Symmetry requires that . We have
By definition, a PSD matrix, we have .
