Week 16 Lecture 01 02 SVD and CUR (Example)
Week 16 Lecture 01 02 SVD and CUR (Example)
material useful in giving your own lectures. Feel free to use these slides verbatim, or to modify
them to fit your own needs. If you make use of a significant portion of these slides in your own
lecture, please include this message, or a link to our web site: http://www.mmds.org
▪ Why? The first two rows are linearly independent, so the rank is at least
2, but all three rows are linearly dependent (the first is equal to the sum
of the second and third) so the rank must be less than 3.
Why do we care about low rank?
▪ We can write A as two “basis” vectors: [1 2 1] [-2 -3 1]
▪ And new coordinates of : [1 0] [0 1] [1 1]
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 5
Cloud of points 3D space:
▪ Think of point positions
as a matrix: A
B A
1 row per point: C
n n
VT
m A m
n
1u1v1 2u2v2
m A +
σi … scalar
ui … vector
vi … vector
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 11
It is always possible to decompose a real
matrix A into A = U VT , where
U, , V: unique
U, V: column orthonormal
▪ UT U = I; VT V = I (I: identity matrix)
▪ (Columns are orthogonal unit vectors)
: diagonal
▪ Entries (singular values) are positive,
and sorted in decreasing order (σ1 σ2 ... 0)
Amelie
Matrix
Alien
n
1 1 1 0 0
SciFi
3 3 3 0 0
4 4 4 0 0 VT
5 5 5 0 0 = m
0 2 0 4 4
Romnce 0 0 0 5 5
0 1 0 2 2 U “Concepts”
AKA Latent dimensions
AKA Latent factors
Amelie
Matrix
Alien
Amelie
Matrix
Romance-concept
Alien
Amelie
Matrix
SciFi-concept Romance-concept
Alien
Amelie
Matrix
SciFi-concept
Alien
Amelie
Matrix
v1
Movie 1 rating
Instead of using two coordinates (𝒙, 𝒚) to describe
point locations, let’s use only one coordinate 𝒛
Point’s position is its location along vector 𝒗𝟏
How to choose 𝒗𝟏 ? Minimize reconstruction error
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 21
Movie 2 rating
Goal: Minimize the sum
of reconstruction errors: first right
singular
𝑁 𝐷 vector
2
𝑥𝑖𝑗 − 𝑧𝑖𝑗 v1
𝑖=1 𝑗=1
Movie 1 rating
▪ where 𝒙𝒊𝒋 are the “old” and 𝒛𝒊𝒋 are the
“new” coordinates
SVD gives ‘best’ axis to project on:
▪ ‘best’ = minimizing the reconstruction errors
In other words, minimum reconstruction
error
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 22
A = U VT - example:
Movie 2 rating
▪ V: “movie-to-concept” matrix first right
singular
▪ U: “user-to-concept” matrix vector
v1
Movie 2 rating
first right
singular
variance (‘spread’) vector
on the v1 axis
v1
Movie 2 rating
U Gives the coordinates first right
singular
of the points in the vector
projection axis v1
1 1 1 0 0 Movie 1 rating
Projection of users
3 3 3 0 0 on the “Sci-Fi” axis
4 4 4 0 0 (U ) : 1.61 0.19 -0.01
5 5 5 0 0 5.08 0.66 -0.03
0 2 0 4 4 6.82 0.85 -0.05
0 0 0 5 5 8.43 1.04 -0.06
0 1 0 2 2 1.86 -5.60 0.84
0.86 -6.93 -0.87
0.86 -2.75 0.41
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 25
More details
Q: How exactly is dim. reduction done?
1 1 1 0 0 0.13 0.02
3 3 3 0 0 0.41 0.07
4 4 4 0 0 0.55 0.09 12.4 0
5 5 5 0 0 0.68 0.11 x 0 9.5 x
0 2 0 4 4 0.15 -0.59
0 0 0 5 5 0.07 -0.73
0 1 0 2 2 0.07 -0.29 0.56 0.59 0.56 0.09 0.09
0.12 -0.02 0.12 -0.69 -0.69
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 30
More details
Q: How exactly is dim. reduction done?
A: Set smallest singular values to zero
1 1 1 0 0 0.92 0.95 0.92 0.01 0.01
3 3 3 0 0 2.91 3.01 2.91 -0.01 -0.01
4 4 4 0 0 3.90 4.04 3.90 0.01 0.01
5 5 5 0 0 4.82 5.00 4.82 0.03 0.03
0 2 0 4 4 0.70 0.53 0.70 4.11 4.11
0 0 0 5 5 -0.69 1.34 -0.69 4.78 4.78
0 1 0 2 2 0.32 0.23 0.32 2.01 2.01
Frobenius norm:
ǁA-BǁF = Σij (Aij-Bij)2
ǁMǁF = Σij Mij2 is “small”
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 31
Sigma
A =
U
VT
B is best approximation of A
Sigma
B = U
VT
𝜎𝑟𝑟
2
𝐴−𝐵 𝐹 = 𝐴𝑖𝑗 − 𝐵𝑖𝑗
𝑖𝑗
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 33
Details!
𝜎𝑟𝑟
We apply:
-- P column orthonormal
-- R row orthonormal
▪ U VT - U S VT = U ( - S) VT -- Q is diagonal
A = U VT , B = U S VT (σ1σ2… 0, rank(A)=r)
▪ S = diagonal nxn matrix where si=σi (i=1…k) else si=0
then B is solution to minB ǁA-BǁF , rank(B)=k
Why?
r
min A − B F = min − S F = min si ( i − si ) 2
B , rank ( B ) = k
i =1
We used: U VT - US VT = U ( - S) VT
2
We want to choose si to minimize σ𝑖 𝜎𝑖 − 𝑠𝑖
Solution is to set si=σi (i=1…k) and other si=0
k r r
= min si ( i − si ) + =
2 2 2
i i
i =1 i = k +1 i = k +1
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 36
Equivalent:
‘spectral decomposition’ of the matrix:
1 1 1 0 0
3 3 3 0 0
4 4 4 0 0 σ1
= x x
5 5 5 0 0 u1 u2
σ2
0 2 0 4 4
0 0 0 5 5 v1
0 1 0 2 2
v2
1 1 1 0 0
3 3 3 0 0
4 4 4 0 0
5 5 5 0 0
= σ1 u1 vT1 + σ2 u2 vT2 +...
n
0 2 0 4 4
0 0 0 5 5 Assume: σ1 σ2 σ3 ...
0 1 0 2 2
Dimensionality reduction:
▪ keep the few largest singular values
(80-90% of ‘energy’)
▪ SVD: picks up linear correlations
Amelie
Matrix
Alien
A C U R
Pseudo-inverse of
the intersection of C and R
A C U R
Rs
Rd
A
Cd Cs Construct a
small U
SVD: A = U VT
CUR: A = C U R
Huge but sparse Big but sparse
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 57
DBLP bibliographic data
▪ Author-to-conference big sparse matrix
▪ Aij: Number of papers published by author i at
conference j
▪ 428K authors (rows), 3659 conferences (columns)
▪ Very sparse
Want to reduce dimensionality
▪ How much time does it take?
▪ What is the reconstruction error?
▪ How much space do we need?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 58