Lecture 7.

1
Epipolar geometry

Thomas Opsahl
Weekly overview – Two-view geometry

• Epipolar geometry
– Algebraic representation & estimation
– The essential matrix
– The fundamental matrix

• Triangulation
– Sparse 3D scene reconstructing from 2D
correspondences

• Relative pose from epipolar geometry

– Estimating the relative pose from the essential
matrix
– Visual odometry

2
Introduction
• Observing the same points in two views puts a 𝑿
strong geometrical constraint on the cameras

𝒙 𝒙𝒙
• Algebraically this epipolar constraint is usually
represented by two related 3 × 3 matrices
𝐶 𝐶′

𝒖 𝒖𝒖

3
Introduction
• Observing the same points in two views puts a 𝑿
strong geometrical constraint on the cameras

𝒙 𝒙𝒙
• Algebraically this epipolar constraint is usually
𝐸
represented by two related 3 × 3 matrices
𝐶 𝐶′
• The fundamental matrix 𝐹
� ′𝑇 𝐹𝒖
𝒖 � 𝐾 𝐾𝐾

• The essential matrix 𝐸

𝒖 𝐹 𝒖𝒖

�′𝑇 𝐸𝒙
𝒙 �

• These are coupled through the two calibration

matrices 𝐾 and 𝐾′

4
The essential matrix E
• Let 𝐶𝒙 ↔ 𝐶𝐶𝒙′ be corresponding points in the
normalized image planes and let the pose of 𝐶
relative to 𝐶𝐶 be
𝐶𝐶 𝑅 𝒕
𝜉𝐶 =
𝟎 1

𝐶𝒙

𝐶′𝒙′

𝐶
𝒕

𝐶′

5
The essential matrix E
• Let 𝐶𝒙 ↔ 𝐶𝐶𝒙′ be corresponding points in the
normalized image planes and let the pose of 𝐶
relative to 𝐶𝐶 be
𝐶′ 𝑅 𝒕
𝜉𝐶 =
𝟎 1

• In terms of vectors, the equation for the epipolar

plane can be written like 𝐶𝒙
𝐶𝐶𝒙
�′ × 𝒕 ∙ 𝑅 𝐶 𝒙
� =0
𝐶′𝒙′
𝐶𝒙
�
• Rewritten in terms of matrices this takes the form
𝐶𝐶�′𝑇 𝒕 𝑅 𝐶 𝒙
�=0
𝒙 × 𝐶′𝒙
�′
𝐶
𝒕
• This relationship defines the essential matrix
𝐸 = 𝒕 ×𝑅 𝐶′

�′𝑇 𝐸𝒙
𝒙 �=0
6
The essential matrix E

• The essential matrix 𝐸 represents the epipolar

constraint on corresponding normalized points

• �′𝑇 𝐸𝒙
Note that although 𝒙 � = 0 is a necessary
requirement for the correspondence 𝒙 ↔ 𝒙𝒙 to
be geometrically possible, it does not guarantee
its correctness 𝐶𝒙

𝐶′𝒙′
𝐶𝒙
�
𝐶′𝒙
�′
𝐶
𝒕

𝐶′

8
The essential matrix E
𝑿
• Properties of 𝐸
– 𝐸 = 𝒕 ×𝑅
𝒙 𝒙𝒙
– Homogeneous
𝐸
– 𝑟𝑟𝑟𝑟 = 2
𝐶 𝐶′
– det = 0
– 5 degrees of freedom
– 𝐸 can be estimated from a minimum of 5
point correspondences
– If 𝒙 and 𝒙𝒙 are corresponding normalized • It is possible to decompose 𝐸 = 𝒕 × 𝑅 to
image points, then 𝒙 �′𝑇 𝐸𝒙
�=0 determine the relative pose between
– 𝐸 has 2 singular values that are equal and cameras
a third that is zero – Translation only up to scale
– Topic of another lecture

9
The fundamental matrix F
𝑿
• The epipolar constraint on image points is naturally
connected to the essential matrix by the calibration
matrices 𝐾 and 𝐾′ 𝒙 𝒙𝒙

K C x =u ⇒ C x =K −1u 𝐸
C′ C′ C′
𝐶 𝐶′
K ′ x ′ =u ′ ⇒ x ′ =K ′ u ′ ⇒ x ′ =
−1
u ′ K ′
T T −T

• Combined with the epipolar constraint for normalized 𝐾 𝐾𝐾

image points we get
C′
x ′T E C x = 0 𝒖 𝐹 𝒖𝒖

u ′T K ′−T EK −1u = 0

• This defines the fundamental matrix 𝐹 = 𝐾𝐾−𝑇 𝐸𝐾 −1

� ′𝑇 𝐹𝒖
𝒖 �=0

10
The fundamental matrix F
• Properties of 𝐹 𝑿
−𝑇 −1
– 𝐹 = 𝐾𝐾 𝐸𝐾
– Homogeneous
– 𝑟𝑟𝑟𝑟 = 2 𝒙 𝒙𝒙
– det = 0
– 7 degrees of freedom 𝐶 𝐶′
– 𝐹 can be estimated from a minimum of 7 point
correspondences
𝐾 𝐾𝐾
– If 𝒖 and 𝒖𝒖 are corresponding image points, then
� ′𝑇 𝐹𝒖
𝒖 �=0
– For any point 𝒖 in image 1, the corresponding 𝒖 𝐹 𝒖𝒖
epipolar line 𝒍′ in image 2 is given by
𝒍̃′ = 𝐹𝒖�
– The epipole 𝒆𝒆 in image 2 is 𝐹’s left singular
– For any point 𝒖𝒖 in image 2, the corresponding vector corresponding to the zero singular value
epipolar line 𝒍 in image 1 is given by – The epipole 𝒆 in image 1 is 𝐹’s right singular
𝒍̃ = 𝐹 𝑇 𝒖
�′ vector corresponding to the zero singular value

11
Example

𝒖𝒖

• These fundamental lines were determined using the fundamental matrix between images
• Recall that points and lines are dual in ℙ2
u 
l T u = 0 ⇔ [l0 l1 l2 ]  v  = 0 ⇔ l0u + l1v + l2 = 0
 
 1 
12
Estimating F

• Several algorithms • So given several correspondences we get a

– Non-iterative: 7-pt, 8-pt homogeneous system of linear equations that
– Iterative: Minimize epipolar error we can solve by SVD
– Robust: RANSAC with 7-pt
 u1u1′ u1′v1 u1′ u1v1′ v1v1′ v1′ u1 v1 1
          f = 0
• From the definition it follows that each point  
unun′ un′ vn un′ un vn′ vn vn′ vn′ un vn 1
correspondence 𝒖𝑖 ↔ 𝒖𝑖 ′ contributes with 1
equation Af = 0
ui′T Fui = 0
• As before, we see that the matrix A contains
 f1 f 2 f3  ui 
terms that can be very different in scale, so
[ui′ vi′ 1]  f 4 f5 f 6   vi  = 0
  point sets 𝒖𝑖 and 𝒖𝑖 ′ should be normalized
 f 7 f8 f9   1  in advance
[uiu′i ui′vi ui′ ui vi′ vi vi′ vi′ ui vi 1] f = 0 – Centroids  origin
– Mean distance from origin should be 2

13
Estimating F
𝑿
The normalized 8-point algorithm
Given 𝑛 ≥ 8 correspondences 𝒖𝑖 ↔ 𝒖𝑖 ′, do the
following 𝒙 𝒙𝒙

𝐶 𝐶′
1. Normalize points 𝒖𝑖 and 𝒖𝑖 ′ usingsimilarity
transforms 𝑇 and 𝑇𝑇
2. Build matrix 𝐴 from point-correspondences and 𝐾 𝐾𝐾
compute its SVD
3. Extract the “solution” 𝐹� from the right singular 𝒖 𝐹 𝒖𝒖
vector corresponding to the smallest singular
value
4. Compute the SVD of 𝐹� : 𝐹� = 𝑈𝑈𝑉 𝑇 𝑇 𝑇𝑇
5. Enforce zero determinant by setting 𝑠33 = 0 and
compute a proper fundamental matrix �
𝒖
𝐹�
�′
𝒖
𝐹� = 𝑈𝑈𝑉 𝑇
uˆ ′T Fˆuˆ = 0
6. Denormalize 𝐹 = 𝑇𝑇𝑇 𝐹� 𝑇
(T ′u′ ) Fˆ (Tu ) =0 ⇒ F =T ′T FT
T
ˆ

u′T T ′T FT
ˆ u=0 14
Estimating F
The normalized 8-point algorithm The 7-point algorithm
Given 𝑛 ≥ 8 correspondences 𝒖𝑖 ↔ 𝒖𝑖 ′, do the • Given 7 correspondences, 𝐴 will be a 7 × 9 matrix
following which in general will be of rank 7

1. Normalize points 𝒖𝑖 and 𝒖𝑖 ′ usingsimilarity • So the null space of 𝐴 is 2-dimensional and the
fundamental matrix must be a linear combination of
transforms 𝑇 and 𝑇𝑇 the two right null vectors of 𝐴
2. Build matrix 𝐴 from point-correspondences and 𝒇 𝛼 = 𝛼𝒇1 + 1 − 𝛼 𝒇2
compute its SVD 𝐹 𝛼 = 𝛼𝐹1 + 1 − 𝛼 𝐹2
3. Extract the “solution” 𝐹� from the right singular
vector corresponding to the smallest singular • The additional constraint 𝑑𝑑𝑑 𝐹 = 0 leads to a
value cubic polynomial equation in 𝛼 which has 1 or 3
4. Compute the SVD of 𝐹� : 𝐹� = 𝑈𝑈𝑉 𝑇 solutions 𝛼𝑖 which in turn yields 1 or 3 𝐹’s
5. Enforce zero determinant by setting 𝑠33 = 0 and
compute a proper fundamental matrix • This algorithm is to prefer in a RANSAC scheme,
since it is minimal and since for a single sampling
𝐹� = 𝑈𝑈𝑉 𝑇 of 7 correspondences one might get 3 fundamental
6. Denormalize 𝐹 = 𝑇𝑇𝑇 𝐹� 𝑇 matrices to test for inliers

15
Estimating F
• Improved estimates of 𝐹 can be obtained using • OpenCV
iterative methods like Levenberg-Marquardt to – cv::Mat cv::findFundamentalMat
minimize the epipolar error – Arguments are
InputArray points1
�, 𝐹𝑇 𝒖
𝜖 = �𝑑 𝒖 �′ + 𝑑 𝒖
� ′, 𝐹𝒖
�
InputArray points2
int method – {CV_FM_7POINT, CV_FM_8POINT,
𝒖 𝒖𝒖 CV_FM_RANSAC, CV_FM_LMEDS}
𝐹 double param1
double param2
OutputArray mask
𝒍̃ = 𝐹 𝑇 𝒖
�′ 𝒍̃′ = 𝐹𝒖
�
• Matlab
– estimateFundamentalMatrix

16
 Estimating F
• Improved estimates of 𝐹 can be obtained using • OpenCV
iterative methods like Levenberg-Marquardt to – cv::Mat cv::findFundamentalMat
minimize the epipolar error – Arguments are
InputArray points1
�, 𝐹𝑇 𝒖
𝜖 = �𝑑 𝒖 �′ + 𝑑 𝒖
� ′, 𝐹𝒖
�
InputArray points2
int method – {CV_FM_7POINT, CV_FM_8POINT,
𝒖 𝒖𝒖 CV_FM_RANSAC, CV_FM_LMEDS}
�, 𝐹𝑇 𝒖
𝑑 𝒖 �′ 𝐹 double param1
� ′, 𝐹𝒖
𝑑 𝒖 � double param2
OutputArray mask
𝒍̃ = 𝐹 𝑇 𝒖
�′ 𝒍̃′ = 𝐹𝒖
�
• Matlab
– estimateFundamentalMatrix
• � and line
Distance between homogeneous point 𝒖
𝑻
𝒍̃ = 𝑙̃1 , 𝑙̃2 , 𝑙̃3
T

( )
d u , l =
u l
l 2 + l 2
1 2

17
Estimating E

• For calibrated cameras (𝐾 and 𝐾𝐾 are known), • OpenCV

we can first estimate 𝐹 and then compute – cv::Mat cv::findEssentialMat
𝑇
𝐸 = 𝐾𝐾 𝐹𝐹 – 5-pt algorithm

• It is also possible to estimate 𝐸 directly from 5

• Matlab
normalized point correspondences 𝒙𝑖 ↔ 𝒙𝑖 ′
– Currently not available as a built in function in
– Algorithm proposed by David Nistér in 2004
Matlab
– Involves finding the roots of a 10th degree
– MexOpenCV
polynomial

• OpenGV: [Link]
• In RANSAC schemes, the 5-point algorithm is
contains several interesting functions
the fastest alternative
– 5-pt algorithm
– To acieve 99% confidence with 50% outliers, the
5-point algorithm only requires 145 tests while – 2-pt algorithm based on known relative rotation
the 8-point algorithm requires 1177 tests

18
Summary

• Algebraic representation of epipolar geometry 𝑿

– The essential matrix
– The fundamental matrix 𝒙 𝒙𝒙
𝐸
𝐶 𝐶′
• Estimating the epipolar geometry
– Estimate 𝐹: 7pt, 8pt, RANSAC
– Estimate 𝐸: 5pt 𝐾 𝐾𝐾

𝒖 𝐹 𝒖𝒖
• Additional reading:
– Szeliski: 7.2

19