Comp4010 lecture3-AR Technology
4 likes•1,208 views
The document discusses the perception of reality in relation to augmented and virtual reality technologies, focusing on the senses involved in human perception like sight, sound, and touch. It outlines key concepts of presence, tracking, and interaction techniques essential for creating immersive experiences, as well as the technological advancements needed for lifelike virtual reality headsets. Various types of display technologies, input methods, and challenges in registration and tracking for augmented reality are also presented.
Comp4010 lecture3-AR Technology
- 1. AR TECHNOLOGY COMP 4010 Lecture Three Mark Billinghurst August 10th 2021 [email protected]
- 2. REVIEW
- 3. How do We Perceive Reality? • We understand the world through our senses: • Sight, Hearing, Touch, Taste, Smell (and others..) • Two basic processes: • Sensation – Gathering information • Perception – Interpreting information
- 5. Reality vs. Virtual Reality • In a VR system there are input and output devices between human perception and action
- 6. Presence .. “The subjective experience of being in one place or environment even when physically situated in another” Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and virtual environments, 7(3), 225-240.
- 7. Types of Presence • Spatial Presence • Feeling that you are in another space • Object Presence • Feeling that an object is really in your space • Social Presence • Feeling that someone is really with you
- 8. Object Presence • What makes an object appear real? • Touch/Haptic feedback • Appearance • Lighting • Audio cues • Occlusion • Etc..
- 9. Social Presence • What makes a Person appear real? • Interactivity • Visual appearance • Audio cues • Touch • Contextual cues • Etc.. Oh, C. S., Bailenson, J. N., & Welch, G. F. (2018). A systematic review of social presence: Definition, antecedents, and implications. Frontiers in Robotics and AI, 5, 114.
- 10. Senses • How an organism obtains information for perception: • Sensation part of Somatic Division of Peripheral Nervous System • Integration and perception requires the Central Nervous System • Five major senses (but there are more..): • Sight (Opthalamoception) • Hearing (Audioception) • Taste (Gustaoception) • Smell (Olfacaoception) • Touch (Tactioception)
- 11. The Human Visual System • Purpose is to convert visual input to signals in the brain
- 12. Human Horizontal and Vertical FOV • Humans can see ~135 o vertical (60 o above, 75 o below) • See up to ~ 210o horizontal FOV, ~ 115o stereo overlap • Colour/stereo in centre, Black & White/mono in periphery
- 13. Vergence-Accommodation Conflict • Looking at real objects, vergence and focal distance match • In VR, vergence and accommodation can miss-match • Focusing on HMD screen, but accommodating for virtual object behind screen
- 14. Stereo Perception/Stereopsis • Eyes separated by IPD • Inter pupillary distance • 5 – 7.5cm (avge. 6.5cm) • Each eye sees diff. image • Separated by image parallax • Images fused to create 3D stereo view
- 15. Depth Perception • The visual system uses a range of different Stereoscopic and Monocular cues for depth perception Stereoscopic Monocular eye convergence angle disparity between left and right images diplopia eye accommodation perspective atmospheric artifacts (fog) relative sizes image blur occlusion motion parallax shadows texture Parallax can be more important for depth perception! Stereoscopy is important for size and distance evaluation
- 16. Creating the Perfect Illusion Cuervo, E., Chintalapudi, K., & Kotaru, M. (2018, February). Creating the perfect illusion: What will it take to create life-like virtual reality headsets?. In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (pp. 7-12). • Technology to create life-like VR HMDs • Compared to current HMDs • 6 − 10× higher pixel density • 20 − 30× higher frame rate
- 17. Comparison between Eyes and HMD
- 18. Sound Localization • Humans have two ears • localize sound in space • Sound can be localized using 3 coordinates • Azimuth, elevation, distance
- 19. Sound Localization (Azimuth Cues) Interaural Time Difference
- 20. Haptic Sensation • Somatosensory System • complex system of nerve cells that responds to changes to the surface or internal state of the body • Skin is the largest organ • 1.3-1.7 square m in adults • Tactile: Surface properties • Receptors not evenly spread • Most densely populated area is the tongue • Kinesthetic: Muscles, Tendons, etc. • Also known as proprioception
- 21. Spatial Resolution • Sensitivity varies greatly • Two-point discrimination Body Site Threshold Distance Finger 2-3mm Cheek 6mm Nose 7mm Palm 10mm Forehead 15mm Foot 20mm Belly 30mm Forearm 35mm Upper Arm 39mm Back 39mm Shoulder 41mm Thigh 42mm Calf 45mm http://faculty.washington.edu/chudler/chsense.html
- 22. Proprioception/Kinaesthesia • Proprioception (joint position sense) • Awareness of movement and positions of body parts • Due to nerve endings and Pacinian and Ruffini corpuscles at joints • Enables us to touch nose with eyes closed • Joints closer to body more accurately sensed • Users know hand position accurate to 8cm without looking at them • Kinaesthesia (joint movement sense) • Sensing muscle contraction or stretching • Cutaneous mechanoreceptors measuring skin stretching • Helps with force sensation
- 23. AR TECHNOLOGY
- 25. Augmented Reality Definition • Combines Real and Virtual Images • Both can be seen at the same time • Interactive in real-time • The virtual content can be interacted with • Registered in 3D • Virtual objects appear fixed in space
- 26. Augmented RealityTechnology • Combines Real and Virtual Images • Needs: Display technology • Interactive in real-time • Needs: Input and interaction technology • Registered in 3D • Needs: Viewpoint tracking technology
- 27. Example: MagicLeap ML-1 AR Display •Display • Multi-layered Waveguide display •Tracking • Inside out SLAM tracking •Input • 6DOF wand, gesture input
- 30. MagicLeap Display • Optical see-through AR display • Overlay graphics directly on real world • 40o x 30o FOV, 1280 x 960 pixels/eye • Waveguide based display • Holographic optical element • Very thin physical display • Two sets of waveguides • Different focal planes • Overcomes vergence/accommodation problem • Eye tracking for selecting focal plane • Separate CPU/GPU unit
- 31. AR Vergence and Accommodation • Fixed focal distance for OST displays • Accommodation conflict between real and virtual object
- 33. Tracking • Inside out tracking • Sensors on the user’s head • Using multiple sensors • Time of Flight Depth Sensor • IR light projector • Wide angle cameras • Internal accelerometer (IMU) • Creates 3D model of real world • Tracks from model
- 34. Time of Flight Sensor • Timed pulses of light • Fast depth sensing • Long range • Inexpensive
- 35. Spatial Mapping https://www.youtube.com/watch?v=_XJsP3uo2Bs
- 37. Input • Multiple input methods • Handheld Controller • Multiple buttons, trackpad input • 6 DOF magnetic tracking • Eye gaze • Integrated eye tracking • Hand tracking • Natural hand input
- 38. Controller Input • Handheld controller • Magnetic tracking – six degree of freedom Transmitter Coils Receiver Coils
- 39. Magnetic Tracking Operation • Moving coil through magnetic field generates voltage • Position accuracy decreases over distance
- 41. Eye Tracking
- 43. 1: AR DISPLAYS
- 44. AR Display Technologies • Classification (Bimber/Raskar 2005) • Head attached • Head mounted display/projector • Body attached • Handheld display/projector • Spatial • Spatially aligned projector/monitor
- 45. Bimber, O., & Raskar, R. (2005). Spatial augmented reality: merging real and virtual worlds. CRC press. DisplayTaxonomy
- 47. Types of Head Mounted Displays Occluded See-thru Multiplexed
- 48. Optical see-through Head-Mounted Display Virtual images from monitors Real World Optical Combiners
- 49. ViewThrough Optical See-Through HMD
- 50. Optical Design - Birdbath ▪ Reflect off beam splitter
- 51. Optical Design – Curved Mirror ▪ Reflect off free-space curved mirror
- 52. Example: Meta2 • https://www.youtube.com/watch?v=e1W29w63W4g
- 53. Optical Design - Prism
- 54. Epson Moverio BT-300 ▪ Stereo see-through display ($700) ▪ 1280 RGB x 720 pixels, 23 degree FOV, 30Hz, 69g ▪ Android Powered, separate controller ▪ VGA camera, GPS, gyro, accelerometer
- 55. Optical Design - Waveguide • Use prisms/grating elements
- 56. Lumus Display
- 58. Example: Sony Smart EyeGlasses https://www.youtube.com/watch?v=kYPWaMsarss
- 60. AR HMDs • Microsoft HoloLens2 - $3,500 USD • Wearable computer, 47 degree FOV • Waveguide displays, optical see-through • Vuzix Blade - $1000 USD • 30 degree FOV, optical see-through • Self contained, Monocular, Android OS • Epson BT 30C - $499 USD • 25 degree FOV, see-through • Tethered display, USB-C connector
- 63. Pros and Cons of Optical see-throughAR • Pros • Simpler design (cheaper) • Direct view of real world • No eye displacement • Socially acceptable (glasses form factor) • Cons • Difficult to occlude real world • Image washout outdoors/bright lights • Wide field of view challenging • Can’t delay the real world
- 65. ViewThrough aVideo See-Through HMD
- 66. Example: Varjo XR-1 • Wide field of view • 87 degrees • High resolution • 1920 x 1080 pixel/eye • 1440 x 1600 pixel insert • Low latency stereo cameras • 2 x 12 megapixel • < 20 ms delay • Integrated Eye Tracking
- 67. Varjo XR-1 Image Quality
- 69. Handheld AR • Camera + display = handheld AR • Mobile phone/Tablet display
- 70. Pros and Cons ofVideo See-ThroughAR • Pros • True occlusion • Digitized image of real world • Registration, calibration, matchable time delay • Wide FOV is easier to support • Cons • Larger, bulkier hardware • Can’t see real world with natural eyes
- 71. Multiplexed Display Virtual Image ‘inset’ into Real World
- 74. See-Through Display Taxonomy Example Products Binocular See-Through Displays Monocular Optical See-Through Video See-Through Stereoscopic Overlays Monoscopic Overlays Single Camera Dual Camera Monoscopic Overlays Stereoscopic Overlays Stereoscopic Overlays Monoscopic Overlays Video See-Through E.g.: smartphone- or tablet-based hand-held AR Also: Google Glass in VST mode E.g.: Lumos DK-40 E.g.: Microsoft HoloLens, Epson Moverio BT-200, Vuzix STAR 1200XLD E.g.: Trivisio ARVision E.g.: Vuzix iWear VR920 with iWear CamAR Possible, but no clear advantage E.g.: Canon COASTAR, Vuzix Wrap 1200DXAR Optical See-Through E.g.: Microvision Nomad, DigiLens DL40, TacEye ST, Vuzix M2000AR
- 75. More on Head Mounted Displays • Karl Guttag Blog - https://kguttag.com/
- 76. HANDHELD AR
- 77. Handheld AR • Camera + display = handheld AR • Mobile phone/Tablet display • Video see-through AR
- 78. User Perspective Rendering for AR
- 79. User-Perspective Hand-Held Display Handheld display with device perspective Handheld display with user perspective Image: Domagoj Baričević
- 83. SpatialAugmented Reality • Project onto irregular surfaces • Geometric Registration • Projector blending, High dynamic range • Book: Bimber, Rasker “Spatial Augmented Reality”
- 84. Lightform • Depth sensor + projector • Create 3D model of space • Deform image mapping • Content creation tools
- 86. Steerable Projector Image: Claudio Pinhanez, IBM Research Everywhere Projector Display A steerable, tracked projector can display images anywhere
- 87. Head Mounted Projector • NVIS P-50 HMPD • 1280x1024/eye • Stereoscopic • 50 degree FOV • www.nvis.com
- 88. HMD vs.HMPD Head Mounted Display Head Mounted Projected Display
- 89. Tilt5 - https://www.tiltfive.com/ • Stereo head worn projectors • Interactive wand • Roll-able retro-reflective sheet • Designed for shared interaction
- 90. • Retroreflective roll-able mat incident light diffusion retro-reflection reflection Lambertian reflector (e.g. unfinished wood) Mirror reflector Retro-reflector
- 93. Video MonitorAR Video cameras Monitor Graphics Combiner Video Stereo glasses
- 94. Examples
- 95. Magic Mirror AR Experience • See AR overlay of an image of yourself
- 97. OtherTypes ofAR Display • Audio • spatial sound • ambient audio • Tactile • physical sensation • Haptic • virtual touch
- 98. Haptic Input • AR Haptic Workbench • CSIRO 2003 – Adcock et. al.
- 99. Phantom • SensableTechnologies (www.sensable.com) • 6 DOF Force Feedback Device
- 100. AR Haptic Interface • Phantom, ARToolKit, Magellan
- 101. Olfactory Display MetaCookie: An olfactory display is combined with visual augmentation of a plain cookie to provide the illusion of a flavored cookie (chocolate, in the inset). Image: Takuji Narumi
- 103. 2: AR TRACKING
- 105. AR RequiresTracking and Registration • Registration • Positioning virtual object wrt real world • Fixing virtual object on real object when view is fixed • Calibration • Offline measurements • Measure camera relative to head mounted display • Tracking • Continually locating the user’s viewpoint when view moving • Position (x,y,z), Orientation (r,p,y)
- 107. Coordinate Systems Local object coordinates Global world coordinates Eye coordinates Model transformation • Track for moving objects, if there are static objects as well View transformation • Track for moving objects, if there are no static objects • Track for moving observer Perspective transformation • Calibrate offline • For both camera and display
- 108. Spatial Registration
- 109. The Registration Problem • Virtual and Real content must stay properly aligned • If not: • Breaks the illusion that the two coexist • Prevents acceptance of many serious applications t = 0 seconds t = 0.5 second
- 110. Sources of Registration Errors •Static errors • Optical distortions (in HMD) • Mechanical misalignments • Tracker errors • Incorrect viewing parameters •Dynamic errors • System delays (largest source of error) • 1 ms delay = 1/3 mm registration error
- 111. Reducing Static Errors •Distortion compensation • For lens or display distortions •Manual adjustments • Have user manually alighn AR andVR content •View-based or direct measurements • Have user measure eye position •Camera calibration (video AR) • Measuring camera properties
- 112. View Based Calibration (Azuma 94)
- 113. Uncalibrated Calibrated The Benefit of Calibration
- 114. Dynamic errors • Total Delay = 50 + 2 + 33 + 17 = 102 ms • 1 ms delay = 1/3 mm = 33mm error Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
- 115. Reducing dynamic errors (1) •Reduce system lag •Faster components/system modules •Reduce apparent lag •Image deflection •Image warping
- 116. Reducing System Lag Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop Faster Tracker Faster CPU Faster GPU Faster Display
- 117. ReducingApparent Lag Tracking Update x,y,z r,p,y Virtual Display Physical Display (640x480) 1280 x 960 Last known position Virtual Display Physical Display (640x480) 1280 x 960 Latest position Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop
- 118. Reducing dynamic errors (2) • Match video + graphics input streams (video AR) • Delay video of real world to match system lag • User doesn’t notice • Predictive Tracking • Inertial sensors helpful Azuma / Bishop 1994
- 119. PredictiveTracking Time Position Past Future Can predict up to 80 ms in future (Holloway) Now
- 121. TRACKING
- 122. Frames of Reference • Word-stabilized • E.g., billboard or signpost • Body-stabilized • E.g., virtual tool-belt • Screen-stabilized • Heads-up display
- 123. Tracking Requirements • Augmented Reality Information Display • World Stabilized • Body Stabilized • Head Stabilized Increasing Tracking Requirements Head Stabilized Body Stabilized World Stabilized
- 124. Tracking Technologies § Active • Mechanical, Magnetic, Ultrasonic • GPS, Wifi, cell location § Passive • Inertial sensors (compass, accelerometer, gyro) • Computer Vision • Marker based, Natural feature tracking § Hybrid Tracking • Combined sensors (eg Vision + Inertial)
- 126. MechanicalTracker •Idea: mechanical arms with joint sensors •++: high accuracy, haptic feedback •-- : cumbersome, expensive Microscribe
- 127. MagneticTracker • Idea: coil generates current when moved in magnetic field. Measuring current gives position and orientation relative to magnetic source. • ++: 6DOF, robust • -- : wired, sensible to metal, noisy, expensive Flock of Birds (Ascension)
- 128. InertialTracker • Idea: measuring linear and angular orientation rates (accelerometer/gyroscope) • ++: no transmitter, cheap, small, high frequency, wireless • -- : drifts over time, hysteresis effect, only 3DOF IS300 (Intersense) Wii Remote
- 129. UltrasonicTracker • Idea: time of Flight or phase-Coherence SoundWaves • ++: Small, Cheap • -- : 3DOF, Line of Sight, Low resolution, Affected by environmental conditons (pressure, temperature) Ultrasonic Logitech IS600
- 130. Global Positioning System (GPS) • Created by US in 1978 • Currently 29 satellites • Satellites send position + time • GPS Receiver positioning • 4 satellites need to be visible • Differential time of arrival • Triangulation • Accuracy • 5-30m+, blocked by weather, buildings etc.
- 131. Mobile Sensors • Inertial compass • Earth’s magnetic field • Measures absolute orientation • Accelerometers • Measures acceleration about axis • Used for tilt, relative rotation • Can drift over time