Motion and Optical Flow
Key Problem
• Main problem in most multiple-image
methods: correspondence
Correspondence
• Small displacements
– Differential algorithms
– Based on gradients in space and time
– Dense correspondence estimates
– Most common with video
• Large displacements
– Matching algorithms
– Based on correlation or features
– Sparse correspondence estimates
– Most common with multiple cameras / stereo
Result of Correspondence
• For points in image i displacements to
corresponding locations in image j
• In stereo, usually called disparity
• In video, usually called motion field
Computing Motion Field
• Basic idea: a small portion of the image
(“local neighborhood”) shifts position
• Assumptions
– No / small changes in reflected light
– No / small changes in scale
– No occlusion or disocclusion
– Neighborhood is correct size: aperture
problem
Actual and Apparent Motion
• If these assumptions violated, can still
use the same methods – apparent
motion
• Result of algorithm is optical flow
(vs. ideal motion field)
• Most obvious effects:
– Aperture problem: can only get motion
perpendicular to edges
– Errors near discontinuities (occlusions)
Aperture Problem
• Too big:
confused by
multiple motions
• Too small:
only get motion
perpendicular
to edge
Computing Optical Flow:
Preliminaries
• Image sequence I(x,y,t)
• Uniform discretization along x,y,t –
“cube” of data
• Differential framework: compute partial
derivatives along x,y,t by convolving
with derivative of Gaussian
Computing Optical Flow:
Image Brightness Constancy
• Basic idea: a small portion of the image
(“local neighborhood”) shifts position,
but
still looks the same
• Brightness constancy
dI assumption
0
dt
Computing Optical Flow:
Image Brightness Constancy
• This does not say that the image
remains
the I
dI same brightness!
dt t
• vs. : total vs. partial derivative
• Use chain rule
dI x(t ), y (t ), t I dx I dy I
dt x dt y dt t
Computing Optical Flow:
Image Brightness Constancy
• Given optical flow v(x,y)
dI x(t ), y (t ), t
0
dt
I dx I dy I
0
x dt y dt t
(I ) T v I t 0
Image brightness constancy equation
Computing Optical Flow:
Discretization
• Look at some neighborhood N:
want
I (i, j ) v I (i, j ) 0
T
t
( i , j )N
want
Av b 0
I (i1 , j1 ) I t (i1 , j1 )
I (i , j ) I (i , j )
A 2 2
b t 2 2
I (in , jn ) I t (in , jn )
Computing Optical Flow:
Least Squares
• In general, overconstrained linear
system
• Solve by least squares
want
Av b 0
( A T A) v A Tb
T 1 T
v ( A A) A b
Computing Optical Flow:
Stability
• Has a solution unless C = ATA is
singular
T
C A A
I (i1 , j1 )
I (i , j )
C I (i1 , j1 ) I (i2 , j2 ) I (in , jn ) 2 2
I (in , jn )
I x2 I x I y
C N N
2
N
IxIy Iy
N
Computing Optical Flow:
Stability
• Where have we encountered C before?
• Corner detector!
• C is singular if constant intensity or edge
• Use eigenvalues of C:
– to evaluate stability of optical flow
computation
– to find good places to compute optical flow
(finding good features to track)
– [Shi-Tomasi]
Computing Optical Flow:
Improvements
• Assumption that optical flow is constant
over neighborhood not always good
• Decreasing size of neighborhood
C more likely to be singular
• Alternative: weighted least-squares
– Points near center = higher weight
– Still use larger neighborhood
Computing Optical Flow:
Weighted Least Squares
• Let W be a matrix of weights
A WA
b Wb
v ( A T A) 1 A T b
T 2 1 T 2
v w ( A W A) A W b
Computing Optical Flow:
Improvements
• What if windows are still bigger?
• Adjust motion model: no longer
constant within a window
• Popular choice: affine model
Computing Optical Flow:
Affine Motion Model
• Translational model
x2 x1 t x
y y t
2 1 y
• Affine model
x2 a b x1 t x
y c d y t
2 1 y
• Solved as before, but 6 unknowns instead
of 2
Computing Optical Flow:
Improvements
• Larger motion: how to maintain
“differential” approximation?
• Solution: iterate
• Even better: adjust window / smoothing
– Early iterations: use larger Gaussians to
allow more motion
– Late iterations: use less blur to find exact
solution, lock on to high-frequency detail
Iteration
• Local refinement of optical flow
estimate
• Sort of equivalent to multiple iterations
of Newton’s method
Computing Optical Flow:
Lucas-Kanade
• Iterative algorithm:
1. Set = large (e.g. 3 pixels)
2. Set I’ I1
3. Set v 0
4. Repeat while SSD(I’, I2) >
1. v += Optical flow(I’ I2)
2. I’ Warp(I1, v)
5. After n iterations,
set = small (e.g. 1.5 pixels)
Computing Optical Flow:
Lucas-Kanade
• I’ always holds warped version of I1
– Best estimate of I2
• Gradually reduce thresholds
• Stop when difference between I’ and I2
small
– Simplest difference metric = sum of
squared differences (SSD) between pixels
Optical Flow Applications
Video Frames
[Feng & Perona]
Optical Flow Applications
Optical Flow Depth Reconstruction
[Feng & Perona]
Optical Flow Applications
Obstacle Detection: Unbalanced Optical Flow
[Temizer]
Optical Flow Applications
• Collision avoidance:
keep optical flow
balanced between
sides of image
[Temizer]
