Segmentation and Grouping

• Motivation: not information is • Grouping (or clustering)

evidence – collect together tokens that
• Obtain a compact “belong together”
representation from an • Fitting
image/motion sequence/set of – associate a model with tokens
tokens – issues
• Should support application • which model?
• Broad theory is absent at • which token goes to which
present element?
• how many elements in the
model?

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
 General ideas
• tokens • bottom up segmentation
– whatever we need to group – tokens belong together because
(pixels, points, surface they are locally coherent
elements, etc., etc.) • These two are not mutually
• top down segmentation exclusive
– tokens belong together because
they lie on the same object

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Why do these tokens belong together?

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
 Basic ideas of grouping in humans
• Figure-ground discrimination • Gestalt properties
– grouping can be seen in terms – elements in a collection of
of allocating some elements to elements can have properties
a figure, some to ground that result from relationships
– impoverished theory (Muller-Lyer effect)
• gestaltqualitat
– A series of factors affect
whether elements should be
grouped together
• Gestalt factors

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
 Technique: Shot Boundary Detection
• Find the shots in a sequence of • Possible distances
video – frame differences
– shot boundaries usually result – histogram differences
in big differences between – block comparisons
succeeding frames
– edge differences
• Strategy:
• Applications:
– compute interframe distances
– representation for movies, or
– declare a boundary where video sequences
these are big
• find shot boundaries
• obtain “most
representative” frame
– supports search

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
 Technique: Background Subtraction
• If we know what the • Approach:
background looks like, it is – use a moving average to
easy to identify “interesting estimate background image
bits” – subtract from current frame
• Applications – large absolute values are
– Person in an office interesting pixels
– Tracking cars on a road • trick: use morphological
operations to clean up
– surveillance
pixels

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
 Segmentation as clustering
• Cluster together (pixels, tokens, • Point-Cluster distance
etc.) that belong together – single-link clustering
• Agglomerative clustering – complete-link clustering
– attach closest to cluster it is – group-average clustering
closest to • Dendrograms
– repeat – yield a picture of output as
• Divisive clustering clustering process continues
– split cluster along best
boundary
– repeat

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
 K-Means
• Choose a fixed number of • Algorithm
clusters – fix cluster centers; allocate
points to closest cluster
• Choose cluster centers and – fix allocation; compute best
cluster centers
point-cluster allocations to
minimize error • x could be any set of features
• can’t do this by search, because for which we can compute a
distance (careful about scaling)
there are too many possible
allocations.
 2 
 
iclusters 
 x j   i 
jelements of i'th cluster 
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Image Clusters on intensity Clusters on color

K-means clustering using intensity alone and color alone

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Image Clusters on color

K-means using color alone, 11 segments

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
K-means using
color alone,
11 segments.

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
K-means using colour and
position, 20 segments

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
 Graph theoretic clustering
• Represent tokens using a
weighted graph.
– affinity matrix
• Cut up this graph to get
subgraphs with strong interior
links

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
 Measuring Affinity
Intensity
   2 
aff x, y exp  1 2 I x  I y 
  2 i  
 
Distance
  1  2 
aff x, yexp 
  2 2
d 
x  y 

 
Texture
  1
aff x, yexp 
  2 2
t

  2 
cx  cy 


Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
Scale affects affinity

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
 Eigenvectors and cuts
• Simplest idea: we want a • This is an eigenvalue problem -
vector a giving the association choose the eigenvector of A
between each element and a with largest eigenvalue
cluster
• We want elements within this
cluster to, on the whole, have
strong affinity with one another
• We could maximize
a T Aa
• But need the constraint

a T a 1

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Example eigenvector

points

eigenvector

matrix

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
 More than two segments

• Two options
– Recursively split each side to get a tree, continuing till the
eigenvalues are too small
– Use the other eigenvectors

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
 Normalized cuts
• Current criterion evaluates • Maximize
within cluster similarity, but not
across cluster difference
• Instead, we’d like to maximize assoc(A, A)  assoc(B, B)
   
the within cluster similarity assoc(A,V )  assoc(B,V )
compared to the across cluster
difference
• Write graph as V, one cluster as • i.e. construct A, B such that
A and the other as B their within cluster similarity is
high compared to their
association with the rest of the
graph

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
 Normalized cuts
• Write a vector y whose • This is hard to do, because y’s
elements are 1 if item is in A, -b values are quantized
if it’s in B
• Write the matrix of the graph as
W, and the matrix which has the
row sums of W on its diagonal
as D, 1 is the vector with all
ones. yT D  W y 
• Criterion becomes min y  T 
 y Dy 

• and we have a constraint yT D1 0

Computer Vision - A Modern Approach
Set: Segmentation
Slides by D.A. Forsyth
 Normalized cuts
• Instead, solve the generalized eigenvalue problem

max y yT D  W y subject to yT Dy 1

• which gives

D  W y Dy

• Now look for a quantization threshold that maximises the criterion ---
i.e all components of y above that threshold go to one, all below go to
-b

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
Figure from “Image and video segmentation: the normalised cut framework”,
by Shi and Malik, copyright IEEE, 1998

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth
F igure from “Normalized cuts and image segmentation,” Shi and Malik, copyright IEEE,
2000

Computer Vision - A Modern Approach

Set: Segmentation
Slides by D.A. Forsyth