Download as PDF, PPTX
![1995 1999 1998 1998
2004 2005 2005 2007
Computer Vision: Algorithms and Applications [Richard Szeliski]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-6-2048.jpg)
![A decade ago...
A Theory of Space by Space Carving
[ Kutulakos and Seitz, 1999 ]
Reconstructions w/ and w/o color](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-7-2048.jpg)
![Results
[ CVPR 2007 ]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-9-2048.jpg)
![Face Movie
[Ira Kemelmacher-Shlizerman, Eli Shechtman,
Rahul Garg, Steve Seitz]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-16-2048.jpg)
![Lens Blur
[Carlos Hernandez]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-17-2048.jpg)
![Lens Blur
[Carlos Hernandez]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-18-2048.jpg)
![Noise suppression
[reatedigitalmotion.com]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-21-2048.jpg)
![Compression
[bitewallpapers.com]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-22-2048.jpg)
![Uncanny valley
[www.cubo.cc/creepygirl]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-25-2048.jpg)
![Motivations
[Furukawa
and
Ponce,
2007]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-43-2048.jpg)
![Standard Depthmap
Markov Random Field (MRF)
A Comparative Study of Energy Minimization Methods for Markov Random Fields with
Smoothness-Based Priors [Szeliski et al., PAMI 2008]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-47-2048.jpg)
![A Comparative Study of Energy Minimization Methods for Markov Random Fields with
Smoothness-Based Priors [Szeliski et al., PAMI 2008]
Markov Random Field (MRF)](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-48-2048.jpg)
![Piecewise planarity from MRF
1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR
2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial
Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR
2010]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-62-2048.jpg)
![Advanced MRF for
Depthmap Estimation
Global Stereo Reconstruction under Second Order Smoothness Priors
[Woodford et al., CVPR 2008] Best Paper Award
Reference image
Ground truth](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-63-2048.jpg)
![Experimental results
Reference image Neighboring image
Output depthmap
[Woodford et al.]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-77-2048.jpg)
![Comparative experiment
Reference image Ground truth
Standard MRF MRF with a triple clique
[Woodford et al.]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-78-2048.jpg)
![How to enforce piecewise planar
1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR
2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial
Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR
2010]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-79-2048.jpg)
![Planemap
[Furukawa
and
Ponce,
2007]
1. Extract Manhattan
directions
2. Extract planes](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-81-2048.jpg)
![How to enforce piecewise planar
1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR
2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial
Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR
2010]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-96-2048.jpg)
![How to enforce piecewise planar
1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR
2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial
Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR
2010]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-98-2048.jpg)
![Relaxing Mahnattan
Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Use sparse lines + sparse points to detect planes
• MRF + Graph-cuts](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-99-2048.jpg)
![Relaxing Manhattan
[Sinha et al.]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-100-2048.jpg)
![Relaxing Manhattan
[Sinha et al.]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-101-2048.jpg)
![Enable Curved Surfaces
• Building reconstruction from a top down view
Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial
Imagery [Zebedin et al., ECCV 2008]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-102-2048.jpg)
![Enable Curved Surfaces[Zebedin et al.]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-103-2048.jpg)
![How to enforce piecewise planar
1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR
2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial
Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR
2010]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-104-2048.jpg)
![Priors through primitives for
large-scale indoor modeling
“Reconstructing the World’s Museums” [Xiao and Furukawa, 2012]
(Best Student Paper Award)](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-117-2048.jpg)
![[ Cabral and Furukawa, 2014 ]](https://image.slidesharecdn.com/lecture02-yasutakafurukawa-3dreconstructionwithpriors-150306074719-conversion-gate01/75/Lecture-02-yasutaka-furukawa-3-d-reconstruction-with-priors-160-2048.jpg)
Uploaded bymustafa sarac
Lecture 02 yasutaka furukawa - 3 d reconstruction with priors
This document summarizes a talk on 3D reconstruction with priors. It discusses using priors to improve 3D reconstruction in cases where standard multi-view stereo fails, such as in the presence of noise, for compression purposes, or to avoid the uncanny valley effect. Methods for enforcing priors discussed include using Markov random fields with advanced smoothness terms or piecewise planar maps. Integrating top-down primitive detection approaches with bottom-up multi-view stereo can also help enforce piecewise planar or curved surface priors for scenes.
More Related Content
![2008 brokerage 03 scalable 3 d models [compatibility mode]](https://cdn.slidesharecdn.com/ss_thumbnails/2008brokerage03-scalable3dmodelscompatibilitymode-100413035231-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Lecture 02 yasutaka furukawa - 3 d reconstruction with priors
Lecture 02 yasutaka furukawa - 3 d reconstruction with priors
- 1. ICVSS 2014 3D Reconstructionwith Priors Yasutaka Furukawa Washington University in St. Louis
- 2.
- 3. Outline • Why priors? •Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 4. Brief Introduction of Multi-ViewStereo (MVS) Camera pose 3d reconstruction Input: Calibrated photographs Output: 3D geometry
- 5.
- 6. 1995 1999 19981998 2004 2005 2005 2007 Computer Vision: Algorithms and Applications [Richard Szeliski]
- 7. A decade ago... ATheory of Space by Space Carving [ Kutulakos and Seitz, 1999 ] Reconstructions w/ and w/o color
- 8. Volumetric graph cuts byVogiatzis, Torr, and Cipolla CVPR 2005 1. This is called Haniwa. 2. This is Roberto Cipolla’s Haniwa. This will be a final quiz!
- 9.
- 10.
- 11.
- 12. ComputerVision in Industry •Amazon • Apple • Facebook • Google • Microsoft • Nokia
- 13. ComputerVision in Industry •Amazon • Apple • Facebook • Google • Microsoft • Nokia Mountain View Seattle Zurich NYC LA …
- 14. ComputerVision in Industry •Amazon • Apple • Facebook • Google • Microsoft • Nokia Mountain View Seattle Zurich NYC LA …
- 15. Google Seattle -3DVision Team • Bundling streetview data (by Sameer Agarwal) • Photo Tours (by the team) • Picasa face movie (by Ira Kemelmacher- Shlizerman and Rahul Garg) • Lens Blur (by Carlos Hernandez) Steve Seitz, Sameer Agarwal, Carlos Hernandez, David Gallup, Changchang Wu, Li Zhang
- 16. Face Movie [Ira Kemelmacher-Shlizerman,Eli Shechtman, Rahul Garg, Steve Seitz]
- 17.
- 18.
- 19. Why am Ihere?
- 20. What if notexture?
- 21.
- 22.
- 23. Why priors? • Failsin some important cases • Noise suppression • Compression
- 24. Why priors? • Failsin some important cases • Noise suppression • Compression • Uncanny valley for 3D reconstruction
- 25.
- 26. Uncanny valley for3D reconstruction
- 27. More accurate, but looksworse Less accurate, but looks better
- 28. People like everything aboutthis picture
- 29. 3D photography hateseverything about this picture
- 38. Uncanny Valley for3D Reconstruction Ten Things You Should Know About Large Scale 3D Reconstruction (3DV 2013) Martin Byröd Apple Maps
- 39. Why priors? • Failsin some important cases • Noise suppression • Compression • Uncanny valley for 3D reconstruction
- 40. Outline • Why priors? •Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 41.
- 42.
- 43.
- 44. Enforce Prior inMVS • Manhattan-world assumption • Planarity • Orthogonality
- 45.
- 46. 8.0 8.2 8.5 8.9 9.3 camera center image screen Depthmap Problem Discretizeddepth values { 0.1, 0.2, …, 8.0, 8.1, … 9.8, 9.9}
- 47. Standard Depthmap Markov RandomField (MRF) A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors [Szeliski et al., PAMI 2008]
- 48. A Comparative Studyof Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors [Szeliski et al., PAMI 2008] Markov Random Field (MRF)
- 49. Markov Random Field(MRF)
- 50. Markov Random Field(MRF) 7 7 6 0 6 7
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58. Markov Random Field(MRF) Graph-cuts (alpha-expansion) gives you very good solutions
- 59. Prefers front-parallel surfaces MarkovRandom Field (MRF)
- 60. We want piecewiseplanar
- 62. Piecewise planarity fromMRF 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 63. Advanced MRF for DepthmapEstimation Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award Reference image Ground truth
- 64.
- 65.
- 66.
- 67.
- 68. Standard MRF p q MRFwith a triple clique p q r
- 69. Standard MRF p q MRFwith a triple clique p q r
- 70. Standard MRF p q MRFwith a triple clique p q r 0 = |3 + 5 - 2 x 4|
- 71. Standard MRF p q MRFwith a triple clique p q r 0 = |3 + 5 - 2 x 4|
- 72. Standard MRF p q MRFwith a triple clique p q r
- 73. Standard MRF p q MRFwith a triple clique p q r
- 74. MRF with atriple clique
- 75. Optimization becomes a challenge StandardMRF (submodular) MRF with a triple clique (non-submodular) • Graph-cuts (alpha-expansion) • Belief propagation (TRW) • QPBO • Belief propagation (TRW?)
- 76. Optimization becomes a challenge StandardMRF (submodular) MRF with a triple clique (non-submodular) • Graph-cuts (alpha-expansion) • Belief propagation (TRW) • QPBO • Belief propagation (TRW?) Fusion moves…
- 77. Experimental results Reference imageNeighboring image Output depthmap [Woodford et al.]
- 78. Comparative experiment Reference imageGround truth Standard MRF MRF with a triple clique [Woodford et al.]
- 79. How to enforcepiecewise planar 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 80.
- 81. Planemap [Furukawa and Ponce, 2007] 1. Extract Manhattan directions 2. Extract planes
- 82. Planemap Possible planes a b c d ef g h 1. Extract Manhattan directions 2. Extract planes
- 83. Planemap Possible plane idsa b c d e f g h e e e e e 1. Extract Manhattan directions 2. Extract planes
- 84. Depthmap to Planemap Depthmap( ) Planemap ( )
- 85.
- 86.
- 87. Planemap MRF p 7 60 6 7
- 88.
- 89.
- 90.
- 91.
- 92.
- 93. Comparison Standard method Manha9an Planemap
- 94. Kitchen -‐ 22 images house -‐ 148 images gallery -‐ 492 images Reconstruction Results
- 96. How to enforcepiecewise planar 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 97.
- 98. How to enforcepiecewise planar 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 99. Relaxing Mahnattan Piecewise PlanarStereo for Image-based rendering [Sinha et al., ICCV 2009] • Use sparse lines + sparse points to detect planes • MRF + Graph-cuts
- 100.
- 101.
- 102. Enable Curved Surfaces •Building reconstruction from a top down view Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]
- 103.
- 104. How to enforcepiecewise planar 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 105. Piecewise Planar and Non-‐Planar Stereo for Urban Scene Reconstruction David Gallup Jan-‐Michael Frahm Marc Pollefeys University of North Carolina ETH Zurich
- 106. 3D Reconstruction from Video Street-‐Side Video Real-‐Time Stereo 106
- 107. Idea 3D ReconstructionVideo Frames 107 Enforceplanarity where it looks like a building
- 108. First step: Planar and Non-‐Planar Stereo 108 Video Real-‐Time Stereo Plane Detection Planemap (w/ non-‐planar)) Labels = planes non-‐plane discard
- 109. Second step: Appearance Prior 109 Video Frames
- 110. Classification • divide image into regular grid • RGB, HSV, hue histogram, edge orientation histogram • 5000 human-‐labeled examples (5 images) • k-‐nearest-‐neighbor classifier non-‐planar planar 110 Hoiem et al. 2005, Xiao et al. 2009
- 111. Algorithm Video Real-‐Time Stereo Planar/Non-‐Planar Classification Plane Detection Planar and Non-‐Planar MRF 111 non-‐planar planar
- 112. Effect of Classifier 112 Without image prior With image prior non-‐planar planar
- 113. Results 113 Video Frame Plane Detection Planar Classification Reconstruction non-‐planar planar
- 114.
- 115. Summary: Shape priors inDepthmap MVS MRF with a triple term Manhattan planemap Planemap Planemap w/ curved surfaces Mix of planemap and depthmap
- 116. Outline • Why priors? •Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 117. Priors through primitivesfor large-scale indoor modeling “Reconstructing the World’s Museums” [Xiao and Furukawa, 2012] (Best Student Paper Award)
- 119. Technical Contributions • Architecturalshape priors through primitives • Single consistent 3D model
- 120. Technical Contributions • Architecturalshape priors through primitives • Single consistent 3D model Inverse Constructive Solid Geometry (CSG)
- 121. Technical Contributions Inverse ConstructiveSolid Geometry (CSG) • Architectural shape priors through primitives! • Single consistent 3D model (real merging)
- 122. + Technical Contributions Inverse ConstructiveSolid Geometry (CSG) • Architectural shape priors through primitives • Single consistent 3D model (real merging) + + - + + -
- 123. CGAL • From CSGto a mesh
- 124.
- 125.
- 126.
- 127.
- 128.
- 129. Bottom-up approach point 2D line 2Drectangle 2D CSG
- 130. 2D CSG Reconstruction pointà line
- 131. 2D CSG Reconstruction pointà line line à rectangle From 4 line segments
- 132. 2D CSG Reconstruction pointà line line à rectangle rectangle à 2D CSG
- 133. 2D CSG Reconstruction Explainthe data • Laser points • Free space !
- 134. 2D CSG Reconstruction Repeatin each slice
- 135.
- 136. Bottom-up approach point 2D line 2Drectangle 2D CSG
- 137. Bottom-up approach point 2D line 2Drectangle 2D CSG
- 138. Bottom-up approach point 2D line 2Drectangle
- 139. point 2D line 2D rectangle 3Drectangle 3D CSG
- 140. 2D CSG to3D CSG
- 141. 2D CSG Rectangle primitive 2D CSGto 3D CSG
- 142. 2D CSG Rectangle primitive 2D CSGto 3D CSG
- 143. 1. Generate primitives(cuboids) 2D CSG Rectangle primitive 2D CSG to 3D CSG
- 144. 3D CSG Reconstruction 1.Generate primitives (cuboids) 2. Build a 3D CSG
- 145.
- 146. Last Step 1. RemoveCeiling 2. Texture Mapping
- 148. Challenging Navigation Frick CollectionGallery (NewYork City) Goal Start
- 151. Statistics # of subproblems
- 152. Ground Ground+Aerial Outdoor Indoor Ground vs.Aerial à Ground+Aerial Google Streetview Google MapsGL Furukawa et al. Aerial Google/Bing/NASA … This paper This paper
- 153. So, museum recons. from veryhigh-end devices
- 154.
- 155.
- 156.
- 157. {# of museums}<< {# of restaurants} + {# of grocery stores} + {# of clinic} + ...
- 158. Millions of small/medium- scalebusinesses • Image-based for scalable deployment (especially for emerging markets) • Compact model (for browser on low-end PCs)
- 159. Millions of small/medium- scalebusinesses • Image-based for scalable deployment (especially for emerging markets) • Compact mesh model (for browser on low-end PCs)
- 160. [ Cabral andFurukawa, 2014 ]
- 161. Outline • Why priors? •Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 162.
- 163.
- 164. Why priors? • Onlyreconstruct planar outline • Texture mapping looks better • Challenge is how to IGNORE clutter
- 165.
- 166. 1. Panorama images2. MVS points 3. Point evidence 4. Free-space evidence 5. 2D room outline 6. 3D model
- 167. 2D room outlinereconstruction 3. Point evidence 4. Free-space evidence Outline should pass through this point Threshold
- 168. 2D room outlinereconstruction 4. Free-space evidence3. Point evidence 5. Shortest path formulation
- 169.
- 170.
- 171.
- 172. 2D room outlinereconstruction 5. Shortest path formulation
- 173. 2D room outlinereconstruction 5. Shortest path formulation
- 174. 2D room outlinereconstruction 5. Shortest path formulation Model complexity penalty added to each edge
- 175. 2D room outlinereconstruction 5. Shortest path formulation
- 176. 2D room outlinereconstruction Standard Ours
- 177. Features • Regularize on{# of line segments} • Piecewise planarity enforcement • With Dynamic Programming, exactly control the number of vertices
- 178.
- 179.
- 180.
- 181.
- 182.
- 183.
- 184. Exact complexity control 13verts. 15 verts. 17 verts. 19 verts. 23 verts. . . . . . .
- 186.
- 189. Summary • Why priors? •Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 190.
- 191. A. Who ownsthe Haniwa in the volumetric graph-cuts paper? 1. Yasutaka Furukawa. 2. Roberto Cipolla. 3. British Museum. B. What is Uncanny valley for 3D reconstruction? 1. As the number of authors in a paper goes up, the model quality goes down. 2. As the face realism goes up, the model looks creepier. 3. As the model accuracy goes up, the rendering quality goes down. C. How can one enforce piecewise planarity with a standard MRF formulation easily? 1. Use super-pixel segmentation. 2. Use primitive detection. 3. Use object recognition.
- 192.