3D Multi-touch Prototype for Augmented and Virtual Reality

In this video I demonstrate the 3D interaction prototype that I have developed during my Master Thesis at Fraunhofer FIT in Bonn, Germany. The system uses a depth sensing camera, which is based on the time-of-flight principle and measures phase shifts of a reflected infrared signal. My contribution is the computer vision algorithm, the gesture recognition technique as well as the concept and implementation of the demo application.

Category:

Science & Technology

Tags:

• multi touch
• 3d interaction
• augmented reality
• virtual reality
• depth sensing
• 3d camera
• computer vision
• gesture recognition
• opengl
• human computer interaction
• hci
• fraunhofer
• master thesis
• barehanded interaction
• minority report
• prototype

[ghackenberg]

There is now a paper out about the technology: IEEE VR 2011, Lightweight Palm and Finger Tracking for Real-Time 3D Gesture Control

[orange7886ro]

cool stuff !! how many cameras did u use?

[ghackenberg]

@orange7886ro Thanks! The system uses only one camera for detecting and tracking human hand-like objects in 3D. The camera is a little bit special though, as it uses the time-of-flight technique for measuring the distance of objects instead of the color. This means that your algorithm's calculations operate on 3D geometry directly.

[rraallvv]

COOL!!! have you thought in implementing a kind of interface like the one seen in the traffic control center in Zion, in the movie The Matrix? I think you could do something like that it using some old red/cyan anaglyph glasses. Please give it a try... I cant wait to have something like that ;-)

The the Zion Traffic Control Center as seen in the movie The Matrix has the youtube ID ZPPBzmn6Fh4

[ghackenberg]

@rraallvv I think there is soon to come first applications of the technology. It will be an interesting time seeing applications emerge from this interaction concept. If it's gonna be something like the traffic control center in Zion I cannot tell at the moment. But there seems to be potential for quite some diversity.

[ghackenberg]

@nutbag36 I agree, this kind of interface also has its drawbacks in certain situations/tasks. Maybe Hollywood provides currently the most interesting visions towards how such technology can be really exploited fully (Minority Report, Iron Man). In sum in Germany we would probably say: "it's not a universal cure", but I also believe we will see people enjoying these types of interfaces for frequent activities.

[nutbag36]

cool tech, but wouldnt it be way quicker to use a keyboard and mouse?

[PCwiz43233445669]

NOW THATS FUCKIN COOL ive never seen one of those

[sisisisi1111]

But now, there is kinect,,,,

[XxXRangerXxX]

Playstation move uses similar technology

[jonathanw65]

Masterfull , just like Minority Report!!

[Rick31112]

Nice, great stuff good work..

[rraallvv]

@ghackenberg Is your application free for testing? Using a free driver like iz3d, you could make a proof of concept adding deep perception. The driver is for a 3d monitor, although it comes with an option to use with those old red/cyan anaglyph glasses that works with any monitor/projector. It only requires that your application works either with Direc3D or OpenGL, then it runs in the background making possible the deep perception of all the 3d objects in any Direct3D or OpenGL application.

[Jhaeson]

@nutbag36 Yeah well, nope if you can't reach the mouse.... Try to look this for a work group or even for speechs. I really want to see this with holograms, looks like technology is closer to virtual reality very soon.

[ghackenberg]

@MudRat02 Actually we were considering implementing OS events (e.g. mouse movement and multi-touch). Unfortunately there was not the time to work on it till now.