Will the iPhone 4 replace the Kinect controller ?

[visScience]

cool! next generation of phones will have stereo vision by default. what technique did you use to generate the surface mesh?

[KeithWasHere1]

looks like an electric afro

[albertoju1997]

That is Albert Einstein

[JohnnyKidder]

HAN SOLO LIVES!!!!!

[DanFrederiksen]

what is the noisy backdrop and why so shallow?

I'm guessing the kinect is more precise, larger volume and more robust with less computation

[helmutkungl]

The background is noisy because it was black space. We could have assigned a "mask" to remove the space behind the head, but we were just too lazy.

Based on tests using static diffuse object the Kinect is far less accurate/precise, but it's without a doubt a great real-time camera for larger FOV's. I have never tested the accuracy of the Kinect for large areas so I can not comment on its accuracy.

[solCruiser]

the IR-projection from kinect is better for low light conditions

[helmutkungl]

Do you mean the Kinect works better under low light conditions ? If so yes, it has a harder time resolving its active IR coded pattern in the presence of sunlight which has an IR component that washes out it's own pattern.

[solCruiser]

@helmutkungl (i 'am also from germany so...) - guter Einwand - stimmt !!! dann ist der Einsatz als Sensor für Roboter im Ausseneinsatz schonmal nicht geeignet. Im Innenraum ist ja der Infrarotanteil auch bei Tageslicht sehr gering (Fensterscheiben, wenig Reflexion)

[helmutkungl]

@solCruiser

My German is at about the level of a 5 year old- which is when we immigrated to Canada... Leider mein Deutsche ist nicht sehr gut.

[EdogaruTech]

@helmutkungl But you need 2 iPhones, Kinect is cheaper. And the CPU of the PC or laptop is higher

[jaysteels]

Is there an app for rhis?

[helmutkungl]

There will be next week for $399 on the Scanner Killer site

[BLinDaTbESt]

@helmutkungl You're kidding...

[goku21youtub]

so when and where can this be purchased if ever ?

[helmutkungl]

It can be purchased from XYZ RGB Inc., or you can use it online at Scanner Killer

[goku21youtub]

so where can i sign up for the beta? :D at the moment i´m looking hard into facial motioncapture at home , already got 2 cameras which record with 75 frames per second at 640x480 sitting right in front of me ... give me a shout if you ever consider a betaprogram! thanx

[helmutkungl]

The software we use is called Scanner Killer, but it's currently setup more for higher resolutions so running a video stream using the online service would be too expensive.... however check the website in about 1-2 months as there will be a big new release that may interest you :)

[goku21youtub]

@helmutkungl hey thats awessome , will check the site rropm time to time ... btw are you from germany, helmut?

[helmutkungl]

Yes I am :) I'm full of chocolate and beer.

[helmutkungl]

As of August 8 the online solution is free :)

[markymark2036]

It could work great with to make augmented reality apps more immersive, displaying partial virtual objects when behind objects detected to be closer in the foreground.

[helmutkungl]

@markymark2036

Good idea :) I'll keep that one in my back pocket.

[markymark2036]

Nice work. Can this software process the depth map live on the fly or do you need to record first then process?

[helmutkungl]

@markymark2036

No yet. We record and then post process the 3D data :)

[helmutkungl]

@markymark2036

No yet. We record and then post process the 3D data.

[goku21youtub]

Scanner Killer are there already programs which can utilise this algorythm ?

an aftereffectsplugin would cool imho

[helmutkungl]

@goku21youtub

Currently not, the SK software is a standalone suite which outputs standard 3D formats like .obj +UV's, .ply, .stl etc. Which you can then import into any 3D package supporting those file types.

[goku21youtub]

@helmutkungl

standalone is awesome too :D  ... will you eventually make a facial-motion-capture software out of this or is it just a proof of concept of your awesome algorythm? (looked at your site and the fullbodyscanner looks badass )

[helmutkungl]

Thanks :) Yes, various companies have purchased the software for 4D facial performance and we continue to add functionality. Currently it's the base alogorithm we're most proud of wether used on a single stereo-pair or a video stream.

[ThereRLG2006]

Maybe you should clarify that this was done with a Cellphone Camera, not anything that's specifically unique to the Apple device. This same trick could be done with any Smart-phone, Web Camera, or just about any other video capture device on the planet.

[helmutkungl]

@ThereRLG2006

Yes, I should have been clearer. But you are correct any input will work provided the focus can be locked and the camera calibrated.

[johngomm]

Ok, develop a small clip-on mirror/lens system for the ipod touch to get the second view and have the processing done on the ipod and you have a winner for cheap, portable 3d capture. Until then, nobody is going to use two iphones for realistic scanning applications.

[helmutkungl]

@johngomm

The purpose of the video wasn't for people to use 2 iPhone's for 3D scanning but to show the robustness of the Scanner Killer algorithm i.e. that it works even with a 1MP cell phone image at 30 fps.

If Apple was to add a second camera for "stereo vision", then you would essentially have a 3D iPhone scanner.

[helmutkungl]

@johngomm

I think the mirrors are a great idea... if you build it they will come :)

[roidroid]

it looks very high quality, like the results from a "fringe pattern" structured light technique.

I don't really understand why you are using the camera from an iPhone (or did you say 2 iPhones? i wasn't sure coz iirc the iPhone4 has 2 cameras on it, and both could be probably be used at once if you had a mirror).

Is your software RUNNING on the iphone4 or something?

[helmutkungl]

@roidroid

Simply to show the possibilities :) If you can do it with a cell phone camera, then you can do it anything.

We placed two iPhone's side by side and recorded a video stream from both cameras, then processed the data offline.

No, this is not an iPhone App... just yet.

[FlyingD]

This is a joke right? The Kinect costs $149. Period. No contracts and connects to any computer via USB with no tom foolery.

[helmutkungl]

@FlyingD

The Kinect system is a great XBOX accessory. However it is a closed "active" stereo system which uses a coded IR pattern. The Scanner Killer technology is an open architecture Pro tool which works with any camera input, varying FOV's, in both passive and active mode, supports high-end DSLR cameras or MV cameras etc., and produces high accuracy data in the millions of polygons.

...Image an iPhone with two cameras running the SK algorithm... that was the point.

[GrahamWiebe]

how would this replace the kinect camera? can you do this for $150?

[helmutkungl]

@GrahamWiebe

Only if Apple adds a second camera for stereo-vision and buys our company :)

[tezzy4lyf]

u spend a little more and u can have a proper mocap device why spend on 2 iphones for this....its a good phone don't get me wrong if anyone's buying it for mocap purpose its a little stupid buy the kinect instead its only 200 USD max.

[InfiniteRealities]

@tezzy4lyf Kinect has allot of draw backs. It's in it's infancy, it's low quality and you can't connect more than 2 because of interference issues from the IR patterns. This video demonstrates the versatility of the Scanner-Killer software as well as the higher quality mesh resolution it can produce, than compared to a Kinect.

[FlyingD]

@InfiniteRealities Really? For $149 I think the results shown 3D Facial Perfomance Capture using Kinect look fantastic. The method used here can work with any 2 decent camera devices I assume and I'm not sure why the poster is pimping the iPhone 4 in particular. Also, I can't post links, but I've read that people are using up to 4 Kinects at a time for capture.

[InfiniteRealities]

@FlyingD Have you any examples of the fantastic 3D Facial Performance Capture from Kinect? I haven't seen anything like that. Very interested to hear more. In theory yes this technique can be used on other camera products. The issue is syncing the cameras together. Not easy to do. For example Canon has some limitation in it's SDK for this kind of control.

[FlyingD]

@InfiniteRealities Yes, search the term 3D Facial Performance Capture from Kinect. Using 2 iPhones to do this is retarded overkill and I can't help but think it's an Apple fanboy trying to make a desperate poke at a great and cheap Microsoft technology.

[InfiniteRealities]

@FlyingD Yeah I did that. Same one that's listed on the side here. The results are very low resolution and far too noisy to be directly usable. Maybe when Kinect 3000 comes out in a few years. Don't think anyone is a fan boy here, all technology if it works well, is always useful : )

[pang1oss]

"At 30 frames per second" shot at 30, but not solved at 30fps, correct?

This is taking two videos from two iphone 4's and running a stereo matching algorithm to generate a depth map, like the freely available openCV, correct?

I mean it does look tweaked for dropouts and data continuity, which you have to do with the opencv stuff too.

[helmutkungl]

Yes it was indeed solved at 30 ftps.

No it's not tweaked at all- in fact this was our first attempt without any adjustments what so ever.

It's not extracted from a depth map, but from a a very similar type of map.

No it's not freely available and no it's not openCV. We spent 4 years developing this particular technology. If you're interested check out our online solution at: scannerkiller dot you know what.

Cheers,

HK

[helmutkungl]

Sorry, I misunderstood your question. This was shot at 30 fps and post processed at 30 fps, but not in real-time. Each frame (stereo-pair) took 25 seconds in post to extract the 3D data shown here.

HK

[sethgotham]

More information please...

[helmutkungl]

What would you like to know specifically ?

[mertdejedi]

What did i just watch! You look like clay!

[helmutkungl]

Read the description. Your comment does not make sense.