Module -1 Basics of Linear Algebra for Machine Learning
1. Classes of Spaces Linear algebra involves various classes of spaces that are fundamental to machine
learning, each with unique structures and proper es.
Vector Spaces:
A vector space is a collec on of vectors, where vector addi on and scalar mul plica on are defined.
Formally, a vector space 𝑉 over a field 𝐹 (typically 𝑅 or 𝐶) sa sfies the following proper es:
Closure under addi on and scalar mul plica on.
Existence of zero vector and inverse vectors.
Examples in ML: Feature spaces, parameter spaces, etc.
Metric Spaces:
A metric space is a set where the distance between any two points is defined through a metric func on.
A metric 𝑑 must sa sfy:
d(x,y)≥0 (Non-nega ve)
(x,y)=d(y,x) (Symmetry)
d(x,y)=0 if and only if x=y (Iden ty)
d(x,y)≤d(x,z)+d(z,y) (Triangle inequality)
Examples: Euclidean distance, Manha an distance in machine learning algorithms.
Normed Spaces:
A normed space is a vector space equipped with a norm, a func on that assigns a non-nega ve length
(magnitude) to each vector.
Norm proper es:
∥v∥≥0
∥v∥=0 if and only if
v=0
∥αv∥=∣α∣∥v∥ for any scalar 𝛼
∥v+w∥≤∥v∥+∥w∥ (Triangle inequality)
Example norms:
𝑙1 norm (sum of absolute values), 𝑙2 norm (Euclidean norm).
Inner Product Spaces:
Inner product spaces are vector spaces with an inner product (dot product in 𝑅^𝑛).
Inner product proper es:
Linearity: ⟨u+v,w⟩=⟨u,w⟩+⟨v,w⟩
Symmetry: ⟨u,v⟩=⟨v,u⟩
Posi vity: ⟨v,v⟩≥0 and ⟨v,v⟩=0 if and only if v=0
Inner products are useful for defining orthogonality and projec ng vectors.
2. Pythagorean Theorem in Linear Algebra
The Pythagorean Theorem states that in an inner product space, if two vectors 𝑢 and 𝑣 are orthogonal,
the square of the length of their sum is the sum of the squares of their lengths:
∥u+v∥^2 = ∥u∥^2 + ∥v∥^2
Applica on: This property is crucial in algorithms such as Principal Component Analysis (PCA) and SVD,
where orthogonality plays a key role in dimensionality reduc on.
3. Types of Matrices
Diagonal Matrix: Only non-zero elements appear on the diagonal.
Iden ty Matrix: A special diagonal matrix where all diagonal elements are 1, denoted 𝐼
Symmetric Matrix: A matrix that is equal to its transpose, 𝐴=𝐴^𝑇
Orthogonal Matrix: A square matrix whose rows and columns are orthonormal,
𝐴^𝑇𝐴 = 𝐼
Singular Matrix: A matrix that does not have an inverse (determinant is zero).
Sparse Matrix: A matrix with a majority of zero entries, common in big data applica ons like
recommender systems.
4. Matrix Opera ons
Matrix Addi on:
𝐴+𝐵 adds corresponding elements.
Matrix Mul plica on:
𝐴𝐵 involves the dot product of rows of
𝐴 and columns of 𝐵.
Transpose: The rows and columns are swapped, 𝐴^𝑇
Inverse: A matrix 𝐴^−1 sa sfies AA^−1=I, used in solving linear systems.
Determinant: Scalar value represen ng certain proper es of a square matrix, used to determine
inver bility.
5. Eigenvectors and Eigenvalues
For a matrix 𝐴, a non-zero vector 𝑣 is an eigenvector if: 𝐴𝑣=𝜆𝑣
where 𝜆 is the eigenvalue corresponding to 𝑣.
Eigenvectors represent direc ons that remain unchanged by the transforma on represented by 𝐴, and
eigenvalues describe how vectors are scaled along these direc ons.
Applica ons:
PCA: Eigenvectors are the principal components, direc ons of maximum variance.
Graph algorithms and Google's PageRank.
6. Fundamental Theorem of Linear Algebra
The fundamental theorem of linear algebra concerns four fundamental subspaces of a matrix 𝐴:Column
space (Range of 𝐴): The set of all linear combina ons of the columns of 𝐴.
Null space (Kernel of 𝐴): The set of all vectors 𝑥 such that Ax=0.
Row space: The column space of 𝐴^𝑇.
Le null space: The null space of 𝐴^𝑇.
The theorem states rela onships among these subspaces, such as their dimensions and orthogonality:
The dimension of the column space is the rank of 𝐴.
The row space and null space are orthogonal complements, as are the column space and le null space.
