Dimensionality Reduction

Motivation
• Clustering
• One way to summarize a complex real-valued data point with a single
categorical variable
• Dimensionality reduction
• Another way to simplify complex high-dimensional data
• Summarize data with a lower dimensional real valued vector
 Motivation
• Clustering
• One way to summarize a complex real-valued data point with a single
categorical variable
• Dimensionality reduction
• Another way to simplify complex high-dimensional data
• Summarize data with a lower dimensional real valued vector

• Given data points in d dimensions

• Convert them to data points in r<d dimensions
• With minimal loss of information
Data Compression
Reduce data from
(inches)

2D to 1D

(cm)
Data Compression
Reduce data from
(inches)

2D to 1D

(cm)

Andrew Ng
Data Compression
Reduce data from 3D to 2D
Principal Component Analysis (PCA) problem formulation

Reduce from 2-dimension to 1-dimension: Find a direction (a vector )

onto which to project the data so as to minimize the projection error.
Reduce from n-dimension to k-dimension: Find vectors
onto which to project the data, so as to minimize the projection error.

Andrew Ng
Covariance
• Variance and Covariance:
• Measure of the “spread” of a set of points around their center of mass(mean)
• Variance:
• Measure of the deviation from the mean for points in one dimension
• Covariance:
• Measure of how much each of the dimensions vary from the mean with
respect to each other

• Covariance is measured between two dimensions

• Covariance sees if there is a relation between two dimensions
• Covariance between one dimension is the variance
Positive: Both dimensions increase or decrease together Negative: While one increase the other decrease
 Covariance

• Used to find relationships between dimensions in high dimensional

data sets

The Sample mean

Eigenvector and Eigenvalue
Ax = λx
A: Square Matirx
λ: Eigenvector or characteristic vector
X: Eigenvalue or characteristic value
• The zero vector can not be an eigenvector
• The value zero can be eigenvalue
 Eigenvector and Eigenvalue
Ax = λx
A: Square Matirx
λ: Eigenvector or characteristic vector
X: Eigenvalue or characteristic value

Example
 Eigenvector and Eigenvalue
Ax - λx = 0
Ax = λx
(A – λI)x = 0

If we define a new matrix B:

B = A – λI
Bx = 0
BUT! an eigenvector
If B has an inverse: x=B 0=0-1
cannot be zero!!

x will be an eigenvector of A if and only if B does

not have an inverse, or equivalently det(B)=0 :
det(A – λI) = 0
 Eigenvector and Eigenvalue
Example 1: Find the eigenvalues of
2  12
A 
 1  5 
  2 12
I  A   (  2)(  5)  12
1   5
 2  3  2  (  1)(  2)
two eigenvalues: 1,  2
Note: The roots of the characteristic equation can be repeated. That is, λ1 = λ2 =…= λk. If that happens, the
eigenvalue is said to be of multiplicity k.
2 1 0
Example 2: Find the eigenvalues of
A  0 2 0
 2 1 0 0 0 2
I  A  0  2 0  (  2)3  0
0 0  2
 Principal Component Analysis
Input:

Set of basis vectors:

Summarize a D dimensional vector X with K dimensional

feature vector h(x)
 Principal Component Analysis

Basis vectors are orthonormal

New data representation h(x)

 Principal Component Analysis

New data representation h(x)

Empirical mean of the data

 SIFT feature visualization

• The top three principal components of SIFT descriptors from a set of images are computed
• Map these principal components to the principal components of the RGB space
• pixels with similar colors share similar structures
Application: Image compression

Original Image

• Divide the original 372x492 image into patches:

• Each patch is an instance that contains 12x12 pixels on a grid
• View each as a 144-D vector
PCA compression: 144D  60D
PCA compression: 144D  16D
16 most important eigenvectors
2 2 2 2
4 4 4 4
6 6 6 6
8 8 8 8
10 10 10 10
12 12 12 12
2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

2 2 2 2
4 4 4 4
6 6 6 6
8 8 8 8
10 10 10 10
12 12 12 12
2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

2 2 2 2
4 4 4 4
6 6 6 6
8 8 8 8
10 10 10 10
12 12 12 12
2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

2 2 2 2
4 4 4 4
6 6 6 6
8 8 8 8
10 10 10 10
12 12 12 12
2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12
PCA compression: 144D ) 6D
6 most important eigenvectors

2 2 2
4 4 4
6 6 6
8 8 8
10 10 10
12 12 12
2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

2 2 2
4 4 4
6 6 6
8 8 8
10 10 10
12 12 12
2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12
PCA compression: 144D 
3D
3 most important eigenvectors
2 2

4 4

6 6

8 8

10 10

12 12
2 4 6 8 10 12 2 4 6 8 10 12

10

12
2 4 6 8 10 12
PCA compression: 144D 
1D
 60 most important eigenvectors

Looks like the discrete cosine bases of JPG!...