Architectural Augmented Reality Sensor Fusion on iPad 2

Demo of sensor fusion for augmented reality running on an iPad 2. The front of this building has been preprocessed to serve as a visual marker. The iPad's camera detects the image to get an initial estimate of where the user is standing and how the iPad is oriented in space. After that, the camera and the gyros/accelerometer in the iPad work together to keep the overlay aligned, even when the building goes out of view or isn't detected by the vision algorithm.

Right now it's not rending anything interesting-- the red-green-blue lines represent the x-y-z axis as calculated by the camera and sensors. The background grid is drawn as a large cube surrounding the user-- you can see the corners when the camera pans up and left. The white rectangle with the X in the center only shows up when the camera detects the building facade; as you can see, it isn't detecting the facade every frame, but it doesn't have to as the gyros provide plenty of readings to fill in between the camera estimates. As a result, the animation runs at a nice smooth 60fps.

Pipeline: FAST corner detector - Ferns keypoint classifier - RANSAC homography estimator - Kalman filter (with CoreMotion attitude matrix) - OpenGL Modelview matrix

Category:

Science & Technology

Tags:

• augmented
• reality
• ar
• sensor
• fusion
• architecture
• building
• facade
• city
• urban
• outdoor
• outside
• gyro
• accelerometer
• compass
• gps
• ipad
• apple
• iphone
• ipod
• camera
• computer
• vision
• orientation
• pose
• estimate
• estimation
• extrinsic
• game
• rotation
• invariant
• ferns
• fast
• ransac
• kalman
• filter
• prototype
• software
• app

[megatonyproductions]

What is the name of the app

[megatonyproductions]

What is the nMe of the app

[perkelation]

I'm so excited... are you working on a mobile Minecraft client? This is totally stable enough for an immersive augmented reality environment. I want to create a collaborative augmentation of my city with my friends on our server!