Deep Augmented Reality?

testing the idea of a markerless augmented reality system that is aware of depth (ie: you a foreground and a background). uses the kinect, openCV, OpenFrameworks

Category:

Science & Technology

Tags:

• kinect
• openkinect
• ofxkinect
• openframeworks
• opencv
• augmented reality
• depth camera

[BiggBenn56]

Dude this is just PURE awesomeness!

I have a kinect and at first i thought this vid was again something like eye-toy (remember that? ;D) But its with kinect!

what software do you need to do that?

[naus3ayt]

@BiggBenn56 thanks :) this demo was coded in c++/OpenFrameworks; it uses libFreenect to speak to the kinect and openCV to handle the depth segmentation.

[xSacha]

@naus3ayt

hi, is the source of this project available anywhere?

[naus3ayt]

@xSacha not the one from this particular sketch; I'm working on a newer, more articulated version and when my current project will be over I'll probably publish a few snippets.

Anyway, if in the meantime you want to try something similar, here's what I did:

- segmented the depth data at a certain depth and extracted a contour (highlighted in the top right box)

- used the contour to mask the rgb image (using OpenGL blending)

- created the 3d scene with the sphere, added some basic physics

[naus3ayt]

@xSacha - if the the sphere has the same Z of my user pool, I checked for collisions

- finally I blended background, masked images and 3d stuff together

[CryingSinger]

This was actually something that I raised to one other Kinect hacker on YouTube a while back concerning this. It's called occlusion, and there are a few other videos demonstrating Kinect AR occlusion. I think markerless occlusion AR like yours could be useful for, say, creating AR interfaces with which we could type, like those old Japanese videos of AR-based keyboards.