Augmented reality for consumer devices
- 1. Augmented Reality for Consumer Devices (and shortcuts for Windows Phone) Jared Bienz [email protected] jaredbienz.wordpress.com @jbienz
- 2. What will be covered What is Augmented Reality? How AR is being used today Different approaches to AR (platform agnostic) Tools for building AR on Windows Phone Getting “crafty” Windows Phone Microsoft Corporation.
- 3. What is Augmented Reality? “A live direct or indirect view of a physical, real-world environment whose elements are augmented by computer- generated sensory input such as sound, video, graphics or GPS data.” - Wikipedia Windows Phone Microsoft Corporation.
- 4. What is Augmented Reality? “A view of the world through a digital “lens”, which allows the scene to be changed or appear differently than the real- world environment.” - Jared Bienz Windows Phone Microsoft Corporation.
- 5. AR Approaches Windows Phone Microsoft Corporation.
- 6. Vision-Based (Games) “These Magic Cards Are The „Holy Shit‟ Moment of Nintendo‟s New Toy” - Gizmodo (March2011) Windows Phone Microsoft Corporation.
- 7. Vision-Based (Apps) Kogan concept app Windows Phone Microsoft Corporation.
- 8. Sensor-Based (Games) Live Butterflies Viewer / Game Windows Phone Microsoft Corporation.
- 9. Sensor-Based (Apps) PhotoSynth for iPhone (yes, we know) Windows Phone Microsoft Corporation.
- 10. Vision vs. Sensor Vision + No sensors required (only access to camera feed) + Existing libraries for tracking tags (code is fairly simple) - Tied to specially printed tags (in range, good angle) - CPU intensive Sensor + Requires no printed tags + Not nearly as CPU intensive - Code is more complex (due to dealing with sensors) - Most current examples are tied to a fixed location - Requires sensors Windows Phone Microsoft Corporation.
- 11. Steps for Building Vision-Based AR 1. Capture video from the camera 2. Using a vision library to locate tag in scene (position) and determine its scale and skew 3. Working backward, calculate the real-world camera location and direction 4. Transform virtual scene (possibly using a virtual camera) 5. Draw virtual scene on top of camera feed 6. Repeat Windows Phone Microsoft Corporation.
- 12. Vision-Based AR on Windows Phone 00:37 00:37 SLAR Toolkit - slartoolkit.codeplex.com Windows Phone Microsoft Corporation.
- 13. Sensor-Based AR Virtual World + Camera and Sensors = Augmented World Windows Phone Microsoft Corporation.
- 14. Steps for Building Sensor-Based AR 1. Position items in “virtual world” (WP use XNA) 2. Determine how device is rotated using Motion APIs 3. Rotate virtual world to match device rotation 4. Capture video from camera 5. Draw virtual world on top of camera feed a) (WP) Use XNA to render in XNA objects – or – b) (WP) Convert XNA space to Silverlight transforms 6. Repeat Windows Phone Microsoft Corporation.
- 15. What Sensors are Required? Accelerometer Camera Compass Gyro (best) Windows Phone Microsoft Corporation.
- 16. Connecting the camera to UI (WP) // Create PhotoCamera camera = new PhotoCamera(CameraType.Primary); // Create a VideoBrush and connect it to the camera cameraBrush = new VideoBrush(); cameraBrush.SetSource(camera); // Fill rectangle with video from the camera someRectangle.Fill = cameraBrush; Windows Phone Microsoft Corporation.
- 17. Sensors Accelerometer Gyro + Very fast + Fast + 3 Axis (X, Y, Z) - Not a good reference point + 3 Axis (X, Y, Z) in motion + Always a reference point, even in motion Compass - Drifts + Simple to use - Jitters - Not always available - 1 Axis only (heading) Windows Phone Microsoft Corporation.
- 18. Motion “Sensor” (WP) Windows Phone Microsoft Corporation.
- 19. Using Motion “sensor” (WP) // Create Motion “sensor” if (Motion.IsSupported) { motion = newMotion(); motion.TimeBetweenUpdates = TimeSpan.FromMilliseconds(20); motion.CurrentValueChanged += motion_CurrentValueChanged; motion.Start(); } // Handle movement void motion_CurrentValueChanged(object sender, SensorReadingEventArgs<MotionReading> e) { … } Windows Phone Microsoft Corporation.
- 20. Hardware Availability (WP) Accelerometer & Compass are required for motion. Highest quality if Gyro is available. All 7.0 devices have Accelerometer Some 7.0 devices have Compass (Motion enabled) All 7.5 devices will have Compass (Motion enabled) Many 7.5 devices will have Gyro (Best motion) Windows Phone Microsoft Corporation.
- 21. Debugging and Testing Only Accelerometer supported in the emulator today Test for Motion.IsSupported and fail gracefully! AR should always be tested on a physical device Windows Phone Microsoft Corporation.
- 22. Sensor-Based AR on Windows Phone Step-by-step video - bit.ly/WPARBasic Windows Phone Microsoft Corporation.
- 23. Geo Augmented Reality TexMex TexMex Automatically Augmented Real World Object + Your Location and + Generated = World Location Direction (sensors) Virtual World Windows Phone Microsoft Corporation.
- 24. Steps for Building Geo-Based AR 1. Get location of objects in real world (web service, etc.) 2. Get location of the user (Location APIs) 3. Calculate actual distance between user and objects 4. Generate “virtual world” where user is at center 5. Capture the direction the user is looking (Motion APIs) 6. Rotate virtual world to match device rotation 7. Capture video from camera 8. Draw virtual items on top of camera feed 9. Repeat Windows Phone Microsoft Corporation.
- 25. Introducing: GART 00:18 Geo AR Toolkit - gart.codeplex.com Windows Phone Microsoft Corporation.
- 26. Steps for using GART 1. New Windows Phone project 2. Add a reference to GART.dll 3. Add ARDisplay control to page (use Expression Blend) 4. Add child views (map, heading, camera, 3D) 5. Create an ARItem to represent a real-world thing 6. Add ARItem to ARDisplay.ARItems collection Windows Phone Microsoft Corporation.
- 27. GART Manages Sensors GART automatically manages the camera, motion and location APIs for you! Call ARDisplay.StartServices in Page.OnNavigatedTo Call ARDisplay.StopServices in Page.OnNavigatedFrom If there is a problem starting or stopping any sensor it’s bubbled through the ARDisplay.ServiceError event. Windows Phone Microsoft Corporation.
- 28. GART Demo Windows Phone Microsoft Corporation.
- 29. Getting Crafty Windows Phone Microsoft Corporation.
- 30. Targus TG-42TT $14 Cable Ties: 30 cm $3 Cable Ties: 10 cm $3 Friction Mount $16 Spray Grip $3 Mainstays Fabric Bin $6 Total: $45 Windows Phone Microsoft Corporation.
- 31. Please let me help you build Windows Phone apps gart.codeplex.com [email protected] jaredbienz.wordpress.com @jbienz on Twitter Windows Phone Microsoft Corporation.
- 32. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. Windows Phone Microsoft Corporation.