cvpr2009 tutorial: kernel methods in computer vision: part II: Statistics and Clustering with Kernels, Structured Output Learning

Statistics and Clustering with Kernels

Christoph Lampert & Matthew Blaschko

Max-Planck-Institute for Biological Cybernetics
Department Schölkopf: Empirical Inference
Tübingen, Germany

Visual Geometry Group
University of Oxford
United Kingdom

June 20, 2009

Overview

Kernel Ridge Regression
Kernel PCA
Spectral Clustering
Kernel Covariance and Canonical Correlation Analysis
Kernel Measures of Independence


Regularized least squares regression:
n
min (yi − w, xi )2 + λ w 2
w
i=1


Regularized least squares regression:
n
min (yi − w, xi )2 + λ w 2
w
i=1

n
Replace w with i=1 α i xi
 2
n n n n
min yi − xi , xj  + λ αi αj xi , xj
α
i=1 j=1 i=1 j=1

α∗ can be solved in closed form solution

α∗ = (K + λI )−1 y

PCA
Equivalent formulations:
Minimize squared error between original data and a projection of
our data into a lower dimensional subspace
Maximize variance of projected data
Solutions: Eigenvectors of the empirical covariance matrix

PCA continued

Empirical covariance matrix (biased):

ˆ 1
C = (xi − µ)(xi − µ)T
n i

where µ is the sample mean.

ˆ
C is positive (semi-)deﬁnite symmetric

PCA:
ˆ
wT C w
max
w w 2

Data Centering

We use the notation X to denote the design matrix where every
column of X is a data sample
We can deﬁne a centering matrix
1 T
H =I− ee
n
where e is a vector of all ones

Data Centering

1 T
H =I− ee
n
H is idempotent, symmetric, and positive semi-deﬁnite (rank
n − 1)

Data Centering

1 T
H =I− ee
n
H is idempotent, symmetric, and positive semi-deﬁnite (rank
n − 1)
The design matrix of centered data can be written compactly in
matrix form as XH
1
The ith column of XH is equal to xi − µ, where µ = n j xj is
the sample mean

Kernel PCA
PCA:
ˆ
wT C w
max
w w 2
Kernel PCA:
Replace w by i αi (xi − µ) - this can be represented compactly
in matrix form by w = XH α where X is the design matrix, H is
the centering matrix, and α is the coeﬃcient vector.

Kernel PCA
PCA:
ˆ
wT C w
max
w w 2
Kernel PCA:
ˆ ˆ 1
Compute C in matrix form as C = n XHX T

Kernel PCA
PCA:
ˆ
wT C w
max
w w 2
Kernel PCA:
ˆ ˆ 1
Denote the matrix of pairwise inner products K = X T X , i.e.
Kij = xi , xj

Kernel PCA
PCA:
ˆ
wT C w
max
w w 2
Kernel PCA:
ˆ ˆ 1
Denote the matrix of pairwise inner products K = X T X , i.e.
Kij = xi , xj
ˆ
wT C w 1 αT HKHKH α
max = max
w w 2 α n αT HKH α

This is a Rayleigh quotient with known solution

HKH βi = λi βi

Kernel PCA
Set β to be the eigenvectors of HKH , and λ the corresponding
eigenvalues
1
Set α = βλ− 2
Example, image super-resolution:

(ﬁg: Kim et al., PAMI 2005.)

Spectral Clustering

Represent similarity of images by weights on a graph
Normalized cuts optimizes the ratio of the cost of a cut and the
volume of each cluster
k ¯
cut(Ai , Ai )
Ncut(A1 , . . . , Ak ) =
i=1 vol(Ai )

Spectral Clustering

Represent similarity of images by weights on a graph
Normalized cuts optimizes the ratio of the cost of a cut and the
volume of each cluster
k ¯
cut(Ai , Ai )
Ncut(A1 , . . . , Ak ) =
i=1 vol(Ai )

Exact optimization is NP-hard, but relaxed version can be solved
by ﬁnding the eigenvalues of the graph Laplacian
1 1
L = I − D − 2 AD − 2

where D is the diagonal matrix with entries equal to the row
sums of similarity matrix, A.

Spectral Clustering (continued)
1 1
Compute L = I − D − 2 AD − 2 .
Map data points based on the eigenvalues of L
Example, handwritten digits (0-9):

(ﬁg: Xiaofei He)
Cluster in mapped space using k-means

Multimodal Data

A latent aspect relates data that are present in multiple
modalities
e.g. images and text

XYZ[
_^]
z M
qqq MMM
qqq MMM
XYZ[
_^] _^]
XYZ[
q
x &
ϕx (x) ϕy (y) x:
y: “A view from Idyllwild, California,
with pine trees and snow capped Marion
Mountain under a blue sky.”

Multimodal Data

A latent aspect relates data that are present in multiple
modalities
e.g. images and text

XYZ[
_^]
z M
qqq MMM
qqq MMM
XYZ[
_^] _^]
XYZ[
q
x &
ϕx (x) ϕy (y) x:
y: “A view from Idyllwild, California,
with pine trees and snow capped Marion
Mountain under a blue sky.”

Learn kernelized projections that relate both spaces

Kernel Covariance
KPCA is maximization of auto-covariance
Instead maximize cross-covariance
wx Cxy wy
max
w ,w
x y wx wy

Kernel Covariance
wx Cxy wy
max
w ,w
x y wx wy

Can also be kernelized (replace wx by i αi (xi − µx ), etc.)

αT HKx HKy H β
maxα,β
αT HKx H αβ T HKy H β

Kernel Covariance
wx Cxy wy
max
w ,w
x y wx wy

Can also be kernelized (replace wx by i αi (xi − µx ), etc.)

αT HKx HKy H β
maxα,β
αT HKx H αβ T HKy H β

Solution is given by (generalized) eigenproblem

0 HKx HKy H α HKx H 0 α
=λ
HKy HKx H 0 β 0 HKy H β

Kernel Canonical Correlation Analysis
(KCCA)

Alternately, maximize correlation instead of covariance
T
wx Cxy wy
max
wx ,wy T T
wx Cxx wx wy Cyy wy

Kernel Canonical Correlation Analysis
(KCCA)

Alternately, maximize correlation instead of covariance
T
wx Cxy wy
max
wx ,wy T T
wx Cxx wx wy Cyy wy

Kernelization is straightforward as before

αT HKx HKy H β
max
α,β
αT (HKx H )2 αβ T (HKy H )2 β

KCCA (continued)

Problem:
If the data in either modality are linearly independent (as many
dimensions as data points), there exists a projection of the data
that respects any arbitrary ordering
Perfect correlation can always be achieved

KCCA (continued)

Problem:
This is even more likely when a kernel is used (e.g. Gaussian)

KCCA (continued)

Problem:
This is even more likely when a kernel is used (e.g. Gaussian)

Solution: Regularize
T
wx Cxy wy
max
wx ,wy
(wx Cxx wx + εx wx 2 ) wy Cyy wy + εy wy
T T 2

As εx → ∞, εx → ∞, solution approaches maximum covariance

KCCA Algorithm

Compute Kx , Ky
Solve for α and β as the eigenvectors of

0 HKx HKy H α
=
HKy HKx H 0 β
(HKx H )2 + εx HKx H 0 α
λ 2
0 (HKy H ) + εy HKy H β

Content Based Image Retrieval with KCCA

Hardoon et al., 2004
Training data consists of images with text captions
Learn embeddings of both spaces using KCCA and appropriately
chosen image and text kernels
Retrieval consists of ﬁnding images whose embeddings are
related to the embedding of the text query

Content Based Image Retrieval with KCCA

Hardoon et al., 2004
Training data consists of images with text captions
Learn embeddings of both spaces using KCCA and appropriately
chosen image and text kernels
Retrieval consists of ﬁnding images whose embeddings are
related to the embedding of the text query

A kind of multi-variate regression


We know how to measure correlation in the kernelized space



Independence implies zero correlation




Diﬀerent kernels encode diﬀerent statistical properties of the
data




Diﬀerent kernels encode diﬀerent statistical properties of the
data

Use an appropriate kernel such that zero correlation in the
Hilbert space implies independence

Example: Polynomial Kernel
First degree polynomial kernel (i.e. linear) captures correlation
only
Second degree polynomial kernel captures all second order
statistics
...

only
statistics
...
A Gaussian kernel can be written
2
k(xi , xj ) = e −γ xi −xj
= e −γ xi ,xi
e 2γ xi ,xj
e −γ xj ,xj

and we can use the identity
∞
1 i
ez = z
i=1 i!

only
statistics
...
2
= e −γ xi ,xi
e 2γ xi ,xj
e −γ xj ,xj

∞
1 i
ez = z
i=1 i!

We can view the Gaussian kernel as being related to an
appropriately scaled inﬁnite dimensional polynomial kernel

only
statistics
...
2
= e −γ xi ,xi
e 2γ xi ,xj
e −γ xj ,xj

∞
1 i
ez = z
i=1 i!

We can view the Gaussian kernel as being related to an
appropriately scaled inﬁnite dimensional polynomial kernel
captures all order statistics

Hilbert-Schmidt Independence Criterion
F RKHS on X with kernel kx (x, x ), G RKHS on Y with kernel
ky (y, y )

ky (y, y )
Covariance operator: Cxy : G → F such that

f , Cxy g F = Ex,y [f (x)g(y)] − Ex [f (x)]Ey [g(y)]

ky (y, y )


HSIC is the Hilbert-Schmidt norm of Cxy (Fukumizu et al. 2008):
2
HSIC := Cxy HS

ky (y, y )


HSIC is the Hilbert-Schmidt norm of Cxy (Fukumizu et al. 2008):
2
HSIC := Cxy HS

(Biased) empirical HSIC:
1
HSIC := Tr(Kx HKy H )
n2

(continued)

Ring-shaped density, correlation approx. zero
Maximum singular vectors (functions) of Cxy
Dependence witness, X
0.5
Correlation: −0.00
1.5 Correlation: −0.90 COCO: 0.14
f(x) 0 1
1 −0.5
0.5
0.5
−1
−2 0 2

g(Y)
x 0
Y

0
Dependence witness, Y
0.5
−0.5
−0.5
0
−1
g(y)

−0.5 −1
−1.5 −1 −0.5 0 0.5
−2 0 2 f(X)
X −1
−2 0 2
y

Hilbert-Schmidt Normalized Independence
Criterion

Hilbert-Schmidt Independence Criterion analogous to
cross-covariance
Can we construct a version analogous to correlation?

Criterion

cross-covariance
Simple modiﬁcation: decompose Covariance operator (Baker
1973)
1 1
Cxy = Cxx Vxy Cyy
2 2

where Vxy is the normalized cross-covariance operator
(maximum singular value is bounded by 1)

Criterion

cross-covariance
Simple modiﬁcation: decompose Covariance operator (Baker
1973)
1 1
Cxy = Cxx Vxy Cyy
2 2

where Vxy is the normalized cross-covariance operator
(maximum singular value is bounded by 1)
Use norm of Vxy instead of the norm of Cxy

Criterion (continued)

Deﬁne the normalized independence criterion to be the
Hilbert-Schmidt norm of Vxy
1
HSNIC := 2
Tr HKx H (HKx H + εx I )−1
n
HKy H (HKy H + εy I )−1

where εx and εy are regularization parameters as in KCCA

Criterion (continued)

Deﬁne the normalized independence criterion to be the
Hilbert-Schmidt norm of Vxy
1
HSNIC := 2
Tr HKx H (HKx H + εx I )−1
n
HKy H (HKy H + εy I )−1

where εx and εy are regularization parameters as in KCCA

If the kernels on x and y are characteristic (e.g. Gaussian
kernels, see Fukumizu et al., 2008)
Cxy 2 = Vxy 2 = 0 iﬀ x and y are independent!
HS HS

Applications of HS(N)IC
Independence tests - is there anything to gain from the use of
multi-modal data?

multi-modal data?
Kernel ICA

multi-modal data?
Kernel ICA
Maximize dependence with respect to some model parameters
Kernel target alignment (Cristianini et al., 2001)

multi-modal data?
Kernel ICA
Learning spectral clustering (Bach & Jordan, 2003) - relates
kernel learning and clustering

multi-modal data?
Kernel ICA
Learning spectral clustering (Bach & Jordan, 2003) - relates
kernel learning and clustering
Taxonomy discovery (Blaschko & Gretton, 2008)

Summary
In this section we learned how to
Do basic operations in kernel space like:
Regularized least squares regression
Data centering
PCA

Summary
Data centering
PCA
Learn with multi-modal data
Kernel Covariance
KCCA

Summary
Data centering
PCA
Kernel Covariance
KCCA
Use kernels to construct statistical independence tests
Use appropriate kernels to capture relevant statistics
Measure dependence by norm of (normalized) covariance
operator
Closed form solutions requiring only kernel matrices for each
modality

Summary
Data centering
PCA
Kernel Covariance
KCCA
Use kernels to construct statistical independence tests
Use appropriate kernels to capture relevant statistics
Measure dependence by norm of (normalized) covariance
operator
Closed form solutions requiring only kernel matrices for each
modality

Questions?

Structured Output Learning

Christoph Lampert & Matthew Blaschko

Max-Planck-Institute for Biological Cybernetics
Department Schölkopf: Empirical Inference
Tübingen, Germany

Visual Geometry Group
University of Oxford
United Kingdom

June 20, 2009

What is Structured Output Learning?

Regression maps from an input space to an output space

g:X →Y



g:X →Y

In typical scenarios, Y ≡ R (regression) or Y ≡ {−1, 1}
(classiﬁcation)



g:X →Y

In typical scenarios, Y ≡ R (regression) or Y ≡ {−1, 1}
(classiﬁcation)

Structured output learning extends this concept to more
complex and interdependent output spaces

Examples of Structured Output Problems
in Computer Vision

Multi-class classiﬁcation (Crammer & Singer, 2001)
Hierarchical classiﬁcation (Cai & Hofmann, 2004)
Segmentation of 3d scan data (Anguelov et al., 2005)
Learning a CRF model for stereo vision (Li & Huttenlocher,
2008)
Object localization (Blaschko & Lampert, 2008)
Segmentation with a learned CRF model (Szummer et al., 2008)
...
More examples at CVPR 2009

Generalization of Regression

Direct discriminative learning of g : X → Y
Penalize errors for this mapping


Two basic assumptions employed
Use of a compatibility function

f :X ×Y →R

g takes the form of a decoding function

g(x) = argmax f (x, y)
y


Two basic assumptions employed
Use of a compatibility function

f :X ×Y →R

g takes the form of a decoding function

g(x) = argmax f (x, y)
y

linear w.r.t. joint kernel

f (x, y) = w, ϕ(x, y)

Multi-Class Joint Feature Map
Simple joint kernel map:
deﬁne ϕy (yi ) to be the vector with 1 in place of the current
class, and 0 elsewhere

ϕy (yi ) = [0, . . . , 1 , . . . , 0]T
kth position

if yi represents a sample that is a member of class k


ϕy (yi ) = [0, . . . , 1 , . . . , 0]T
kth position

ϕx (xi ) can result from any kernel over X :

kx (xi , xj ) = ϕx (xi ), ϕx (xj )


ϕy (yi ) = [0, . . . , 1 , . . . , 0]T
kth position

ϕx (xi ) can result from any kernel over X :

kx (xi , xj ) = ϕx (xi ), ϕx (xj )

Set ϕ(xi , yi ) = ϕy (yi ) ⊗ ϕx (xi ), where ⊗ represents the
Kronecker product

Multiclass Perceptron

Reminder: we want

w, ϕ(xi , yi ) > w, ϕ(xi , y) ∀y = yi

Multiclass Perceptron

Reminder: we want

w, ϕ(xi , yi ) > w, ϕ(xi , y) ∀y = yi

Example: perceptron training with a multiclass joint feature map
Gradient of loss for example i is

0 if w, ϕ(xi , yi ) ≥ w, ϕ(xi , y) ∀y = yi
∂w (xi , yi , w) =
maxy=y ϕ(xi , yi ) − ϕ(xi , y) otherwise
i

Perceptron Training with Multiclass Joint
Feature Map

(Credit: Lyndsey Pickup)

Perceptron Training with Multiclass Joint
Feature Map

Final result
(Credit: Lyndsey Pickup)

Crammer & Singer Multi-Class SVM
Instead of training using a perceptron, we can enforce a large
margin and do a batch convex optimization:
n
1
min w 2+C ξi
w 2 i=1
s.t. w, ϕ(xi , yi ) − w, ϕ(xi , y) ≥ 1 − ξi ∀y = yi

Crammer & Singer Multi-Class SVM
Instead of training using a perceptron, we can enforce a large
margin and do a batch convex optimization:
n
1
min w 2+C ξi
w 2 i=1
s.t. w, ϕ(xi , yi ) − w, ϕ(xi , y) ≥ 1 − ξi ∀y = yi

Can also be written only in terms of kernels
w= αxy ϕ(x, y)
x y

Can use a joint kernel
k :X ×Y ×X ×Y →R
k(xi , yi , xj , yj ) = ϕ(xi , yi ), ϕ(xj , yj )

Structured Output Support Vector
Machines (SO-SVM)

Frame structured prediction as a multiclass problem
predict a single element of Y and pay a penalty for mistakes

Machines (SO-SVM)

Not all errors are created equally
e.g. in an HMM making only one mistake in a sequence should
be penalized less than making 50 mistakes

Machines (SO-SVM)

Not all errors are created equally
e.g. in an HMM making only one mistake in a sequence should
be penalized less than making 50 mistakes
Pay a loss proportional to the diﬀerence between true and
predicted error (task dependent)

∆(yi , y)

Margin Rescaling
Variant: Margin-Rescaled Joint-Kernel SVM for output space Y
(Tsochantaridis et al., 2005)
Idea: some wrong labels are worse than others: loss ∆(yi , y)
Solve
n
2
min
w
w +C ξi
i=1
s.t. w, ϕ(xi , yi ) − w, ϕ(xi , y) ≥ ∆(yi , y) − ξi ∀y ∈ Y {yi }

Classify new samples using g : X → Y:

g(x) = argmax w, ϕ(x, y)
y∈Y

Margin Rescaling
Variant: Margin-Rescaled Joint-Kernel SVM for output space Y
(Tsochantaridis et al., 2005)
Idea: some wrong labels are worse than others: loss ∆(yi , y)
Solve
n
2
min
w
w +C ξi
i=1

Classify new samples using g : X → Y:

g(x) = argmax w, ϕ(x, y)
y∈Y

Another variant is slack rescaling (see Tsochantaridis et al.,
2005)

Label Sequence Learning

For, e.g., handwritten character recognition, it may be useful to
include a temporal model in addition to learning each character
individually
As a simple example take an HMM

Label Sequence Learning

For, e.g., handwritten character recognition, it may be useful to
include a temporal model in addition to learning each character
individually
As a simple example take an HMM

We need to model emission probabilities and transition
probabilities
Learn these discriminatively

A Joint Kernel Map for Label Sequence
Learning

Emissions (blue)

Learning

Emissions (blue)
fe (xi , yi ) = we , ϕe (xi , yi )

Learning

Emissions (blue)
Can simply use the multi-class joint feature map for ϕe

Learning

Emissions (blue)
Transitions (green)

Learning

Emissions (blue)
Transitions (green)
ft (xi , yi ) = wt , ϕt (yi , yi+1 )
Can use ϕt (yi , yi+1 ) = ϕy (yi ) ⊗ ϕy (yi+1 )

Learning (continued)

p(x, y) ∝ e fe (xi ,yi ) e ft (yi ,yi+1 ) for an HMM
i i

Learning (continued)

p(x, y) ∝ e fe (xi ,yi ) e ft (yi ,yi+1 ) for an HMM
i i
f (x, y) = fe (xi , yi ) + ft (yi , yi+1 )
i i
= we , ϕe (xi , yi ) + wt , ϕt (yi , yi+1 )
i i

Constraint Generation

n
2
min w +C ξi
w
i=1

Constraint Generation

n
2
min w +C ξi
w
i=1

Initialize constraint set to be empty
Iterate until convergence:
Solve optimization using current constraint set
Add maximially violated constraint for current solution

Constraint Generation with the Viterbi
Algorithm

To ﬁnd the maximially violated constraint, we need to maximize
w.r.t. y
w, ϕ(xi , y) + ∆(yi , y)

Algorithm

w.r.t. y
w, ϕ(xi , y) + ∆(yi , y)

For arbitrary output spaces, we would need to iterate over all
elements in Y

Algorithm

w.r.t. y
w, ϕ(xi , y) + ∆(yi , y)

elements in Y
For HMMs, maxy w, ϕ(xi , y) can be found using the Viterbi
algorithm

Algorithm

w.r.t. y
w, ϕ(xi , y) + ∆(yi , y)

elements in Y
For HMMs, maxy w, ϕ(xi , y) can be found using the Viterbi
algorithm
It is a simple modiﬁcation of this procedure to incorporate
∆(yi , y) (Tsochantaridis et al., 2004)

Discriminative Training of Object
Localization

Structured output learning is not restricted to outputs speciﬁed
by graphical models

Discriminative Training of Object
Localization

Structured output learning is not restricted to outputs speciﬁed
by graphical models
We can formulate object localization as a regression from an
image to a bounding box

g:X →Y

X is the space of all images
Y is the space of all bounding boxes

Joint Kernel between Images and Boxes:
Restriction Kernel
Note: x |y (the image restricted to the box region) is again an
image.
Compare two images with boxes by comparing the images within
the boxes:

kjoint ((x, y), (x , y ) ) = kimage (x |y , x |y , )

Any common image kernel is applicable:
linear on cluster histograms: k(h, h ) = i hi hi ,
1 (h −h )2
χ2 -kernel: kχ2 (h, h ) = exp − γ i hii +hi
i
pyramid matching kernel, ...
The resulting joint kernel is positive deﬁnite.

Restriction Kernel: Examples

kjoint , =k ,

is large.

kjoint , =k ,

is small.

kjoint , =k ,

could also be large.
Note: This behaves diﬀerently from the common tensor products
kjoint ( (x, y), (x , y ) ) = k(x, x )k(y, y )) !

Constraint Generation with Branch and
Bound
As before, we must solve

max w, ϕ(xi , y) + ∆(yi , y)
y∈Y

where

1 − Area(yi y)
if yiω = yω = 1
Area(yi y)
∆(yi , y) = 1
1 − (y y + 1) otherwise
2 iω ω

and yiω speciﬁes whether there is an instance of the object at all
present in the image

Constraint Generation with Branch and
Bound
As before, we must solve

max w, ϕ(xi , y) + ∆(yi , y)
y∈Y

where

1 − Area(yi y)
if yiω = yω = 1
Area(yi y)
∆(yi , y) = 1
1 − (y y + 1) otherwise
2 iω ω

and yiω speciﬁes whether there is an instance of the object at all
present in the image
Solution: use branch-and-bound over the space of all rectangles
in the image (Blaschko & Lampert, 2008)

Discriminative Training of Image
Segmentation

Frame discriminative image segmentation as learning parameters
of a random ﬁeld model

Segmentation


Like sequence learning, the problem decomposes over cliques in
the graph

Segmentation


Like sequence learning, the problem decomposes over cliques in
the graph

Set the loss to the number of incorrect pixels

Constraint Generation with Graph Cuts

As the graph is loopy, we cannot use Viterbi



Loopy belief propagation is approximate and can lead to poor
learning performance for structured output learning of graphical
models (Finley & Joachims, 2008)



Loopy belief propagation is approximate and can lead to poor
learning performance for structured output learning of graphical
models (Finley & Joachims, 2008)

Solution: use graph cuts (Szummer et al., 2008)
∆(yi , y) can be easily incorporated into the energy function

Summary of Structured Output Learning

Structured output learning is the prediction of items in complex
and interdependent output spaces



We can train regressors into these spaces using a generalization
of the support vector machine




We have shown examples for
Label sequence learning with Viterbi
Object localization with branch and bound
Image segmentation with graph cuts




We have shown examples for
Label sequence learning with Viterbi
Object localization with branch and bound
Image segmentation with graph cuts

Questions?

cvpr2009 tutorial: kernel methods in computer vision: part II: Statistics and Clustering with Kernels, Structured Output Learning

Recommended

More Related Content

What's hot (20)

Similar to cvpr2009 tutorial: kernel methods in computer vision: part II: Statistics and Clustering with Kernels, Structured Output Learning (20)

More from zukun (20)

Recently uploaded (20)

cvpr2009 tutorial: kernel methods in computer vision: part II: Statistics and Clustering with Kernels, Structured Output Learning