Scribd
Upload
654 views

Linear Algebra Cheat-Sheet: Laurent Lessard

The document provides a linear algebra cheat sheet covering topics such as: 1) Matrix basics including definitions, operations, inverses and vector products. 2) Norms including common norms such as 1, 2 and infinity norms. 3) Linear independence, rank, and properties of the matrix rank. 4) Solutions to linear equations, least squares problems, nullspaces and ranges. 5) Orthogonal matrices, projections, and the singular value decomposition.

Uploaded by

AnantDashpute
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
654 views

Linear Algebra Cheat-Sheet: Laurent Lessard

The document provides a linear algebra cheat sheet covering topics such as: 1) Matrix basics including definitions, operations, inverses and vector products. 2) Norms including common norms such as 1, 2 and infinity norms. 3) Linear independence, rank, and properties of the matrix rank. 4) Solutions to linear equations, least squares problems, nullspaces and ranges. 5) Orthogonal matrices, projections, and the singular value decomposition.

Uploaded by

AnantDashpute
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

Linear algebra cheat-sheet

Laurent Lessard
University of Wisconsin–Madison

Last updated: October 12, 2016


Matrix basics
A matrix is an array of numbers. A ∈ Rm×n means that:
 
a11 . . . a1n
A =  ... .. ..  (m rows and n columns)

. . 
am1 . . . amn
Two matrices can be multiplied if inner dimensions agree:
Xn
C = A B where cij = aik bkj
(m×p) (m×n)(n×p)
k=1
T
Transpose: The transpose operator A swaps rows and
columns. If A ∈ Rm×n then AT ∈ Rn×m and (AT )ij = Aji .
• (AT )T = A.
• (AB)T = B T AT .

2
Matrix basics (cont’d)
Vector products. If x, y ∈ Rn are column vectors,
• The inner product is x Ty ∈ R (a.k.a. dot product)
• The outer product is xy T ∈ Rn×n .
These are just ordinary matrix multiplications!

Inverse. Let A ∈ Rn×n (square). If there exists B ∈ Rn×n with


AB = I or BA = I (if one holds, then the other holds with the
same B) then B is called the inverse of A, denoted B = A−1 .

Some properties of the matrix inverse:


• A−1 is unique if it exists.
• (A−1 )−1 = A.
• (A−1 )T = (AT )−1 .
• (AB)−1 = B −1 A−1 .

3
Vector norms
A norm k·k : Rn → R is a function satisfying the properties:
• kxk = 0 if and only if x = 0 (definiteness)
• kcxk = |c|kxk for all c ∈ R (homogeneity)
• kx + y k ≤ kxk + ky k (triangle inequality)

Common examples of norms: 1 x2


• kxk1 = |x1 | + · · · + |xn | (the 1-norm)
p x1
• kxk2 = x12 + · · · + xn2 (the 2-norm)
-1 1
• kxk∞ = max1≤i≤n |xi | (max-norm)
-1
norm ball:
Properties of the 2-norm (Euclidean norm) {x | kxk = 1}
• If you see kxk, think kxk2 (it’s the default)
• x T x = kxk2
• x T y ≤ kxkky k (Cauchy-Schwarz inequality)
4
Linear independence
A set of vectors {x1 , . . . , xn } ∈ Rm is linearly independent if

c1 x1 + · · · + cn xn = 0 if and only if c1 = · · · = cn = 0
 
If we define the matrix A = x1 · · · xn ∈ Rm×n then the
columns of A are linearly independent if

Aw = 0 if and only if w = 0

If the vectors are not linearly independent, then they are


linearly dependent. In this case, at least one of the vectors
is redundant (can be expressed as a linear combination of the
others). i.e. there exists a k and real numbers ci such that
X
xk = ci x i
i6=k

5
The rank of a matrix
rank(A) = maximum number of linearly independent columns
rank(A) = maximum number of linearly independent rows

If A ∈ Rm×n and B ∈ Rn×p then


• rank(A) ≤ min(m, n)
• rank(AB) ≤ min(rank(A), rank(B)) ≤ min(m, n, p)
• if rank(A) = n then rank(AB) = rank(B)
• if rank(B) = n then rank(AB) = rank(A)
So multiplying by an invertible matrix does not alter the rank.

General properties of the matrix rank:


• rank(A + B) ≤ rank(A) + rank(B)
• rank(A) = rank(AT ) = rank(ATA) = rank(AAT )
• A ∈ Rn×n is invertible if and only if rank(A) = n.
6
Linear equations
Given A ∈ Rm×n and b ∈ Rm , linear equations take the form

Ax = b

Where we must solve for x ∈ Rn . Three possibilities:


• No solutions. Example: x1 + x2 = 1 and x1 + x2 = 0.
• Exactly one solution. Example: x1 = 1 and x2 = 0.
• Infinitely many solutions. Example: x1 + x2 = 0.

Two common cases:


• Overdetermined: m > n. Typically no solutions. One
approach is least-squares; find x to minimize kAx − bk2 .
• Underdetermined: m < n. Typically infinitely many
solutions. One approach is regularization; find the
solution to Ax = b such that kxk2 is as small as possible.
7
Least squares
When the linear equations Ax = b are overdetermined and
there is no solution, one approach is to find an x that almost
works by minimizing the 2-norm of the residual:

minimize kAx − bk2 (1)


x

This problem always has a solution (not necessarily unique).


x̂ minimizes (1) iff x̂ satisfies the normal equations:

AT A x̂ = AT b

The normal equations (and therefore (1)) have a unique


solution iff the columns of A are linearly independent. Then,

x̂ = (AT A)−1 AT b

8
Range and nullspace
Given A ∈ Rm×n , we have the definitions:
Range: R(A) = {Ax | x ∈ Rn }. Note: R(A) ⊆ Rm
Nullspace: N(A) = {x ∈ Rn | Ax = 0}. Note: N(A) ⊆ Rn
The following statements are equivalent:
• There exists a solution to the equation Ax = b
• b ∈ R(A)
 
• rank(A) = rank( A b )

The following statements are equivalent:


• Solutions to the equation Ax = b are unique
• N(A) = {0}
• rank(A) = n Remember: rank(A) = dim(R(A))

Theorem: rank(A) + dim(N(A)) = n

9
Orthogonal matrices
A matrix U ∈ Rm×n is orthogonal if U T U = I . Note that we
must have m ≥ n. Some properties of orthogonal U and V :
• Columns are orthonormal: uiT uj = δij .
• Orthogonal transformations preserve angles & distances:
(Ux)T (Uz) = x T z and kUxk2 = kxk2 .
• Certain matrix norms are also invariant:
kUAV T k2 = kAk2 and kUAV T kF = kAkF
• If U is square, U T U = UU T = I and U −1 = U T .
• UV is orthogonal.

Every subspace has an orthonormal basis: For any A ∈ Rm×n ,


there exists an orthogonal U ∈ Rm×r such that R(A) = R(U)
and r = rank(A). One way to find U is using Gram-Schmidt.

10
Projections
If P ∈ Rn×n satisfies P 2 = P, it’s called a projection matrix.
In general, P : Rn → S, where S ⊆ Rn is a subspace. If P is a
projection matrix, so is (I − P). We can uniquely decompose:
x = u + v where u ∈ S, v ∈ S ⊥ (u = Px, v = (I − P)x)
Pythagorean theorem: kxk22 = kuk22 + kv k22

If A ∈ Rm×n has linearly independent columns, then the


projection onto R(A) is given by P = A(AT A)−1 AT .

Least-squares: decompose b using the projection above:


b = A(AT A)−1 AT b + (I − A(AT A)−1 AT )b
= Ax̂ + (b − Ax̂)
Where x̂ = (AT A)−1 AT b is the LS estimate. Therefore the
optimal residual is orthogonal to Ax̂.
11
The singular value decomposition
Every A ∈ Rm×n can be factored as
A = U1 Σ1 V1T (economy SVD)
(m×n) (m×r ) (r ×r ) (n×r )T

U1 is orthogonal, its columns are the left singular vectors


V1 is orthogonal, its columns are the right singular vectors
Σ1 is diagonal. σ1 ≥ · · · ≥ σr > 0 are the singular values

Complete the orthogonal matrices so they become square:


  
  Σ1 0 V1T T
A = U1 U1 T = UΣV (full SVD)
(m×n) (m×m)
0 0 V 2
(m×n) (n×n)

The singular values σi are an intrinsic property of A.


(the SVD is not unique, but every SVD of A has the same Σ1 )

12
Properties of the SVD
Singular vectors ui , vi and singular values σi satisfy
Avi = σi ui and AT ui = σi vi

Suppose A = UΣV T (full SVD) as in previous slide.


• rank: rank(A) = r
• transpose: AT = V ΣU T
• pseudoinverse: A† = V1 Σ−1 T
1 U1

Fundamental subspaces:
• R(U1 ) = R(A) and R(U2 ) = R(A)⊥ (range of A)
• R(V1 ) = N(A)⊥ and R(V2 ) = N(A) (nullspace of A)

Matrix norms:
p
• kAk2 = σ1 and kAkF = σ12 + · · · + σr2
13

You might also like