@madVal8 I'm using lossless compression on the depth frames, at native resolution (640x480 pixels, 11 bits per pixel, 30 frames per second). The resulting bandwidth in typical settings is about 500 KB/s. The color frames are compressed using a standard lossy video codec, in this case Theora (mostly due to its nice API and licensing terms), also at 640x480 pixels, 24 bits per pixel, 30 frames per second). H.264 or something similar would work as well.
@okreylos Thanks for the info!! had another question - Does your version of Vrui toolkit support connecting multiple Kinects to one single PC or did you use multiple of them to connect the Kinects and used some kind of a central server to work on them?
@madVal8 Vrui doesn't know anything about Kinect; it's just a VR / 3D graphics / UI development toolkit (think Qt or GTK+). Kinect is handled by the Kinect package, which uses Vrui for its graphical applications.
Anyway, the Kinect package handles multiple Kinects per PC, as many as you have USB buses (there can only be one Kinect per USB bus due to bandwidth constraints). There is no direct support to pool multiple PCs, but that's a simple addition to the software.
Imagine if this will be how they record people for games in the future, then the environments can be digital and the actors be photorealistic. Or in movies... REAL 3d movies.... awesome man, just awesome.
Are you guys working on the quality of the video itself, one thing i noticed in your video with the Kinect Box and 2 Cameras is that it is really blurry and you couldn't make out an image. Another question regarding quality is if it is possible to fix those black and white blotches that appear on the screen when the subject is moving. If these have been asked before i'm sorry. Thanks
Very nice start. Your videos have answered the first question I had about using multiple Kinects to correct for the eye-line issue of webcam video calls. Thanks.
awesome videos, don't listen to the haters... you beat them to it... hehe thanks for putting your work up for everyone to see.... this is paving the way to technology that is cheap and easily accessible to everyone! As a software engineer myself whenever i get free time from my own job i can't wait to try out the musical and artist potential of this.
Thank you for showing what amazing things you can do with the Kinect - I would love to see what you are able to do with better motion cameras! Maybe microsoft should hire you to update the kinect!
Btw, full echolocation could be used in order to complete shapes that aren't visible... Kinect just leaves them incomplete... And to texture invisible mesh parts, some sort of deductive algorithm could be used like those found in recent Photoshop versions
with enought kinect-like devices you can actually dynamically move through the room and for example observe the girls' but without her knowing it. i.e. you can move like solely from your site. with green screens, in order to move througth the space, the real camera at the girl's site must be moved. kinect owns green screens.
can you elaborate on what she's actually looking at in front of her when she's "pointing at the camera"? is it a tv or projector and what is it actually showing?
@boabie Normally there would be two-way video, and she would be looking at a 3D representation just like hers. The way the sites are connected is that if she looks at that virtual person's eyes, her virtual representation is looking directly into the real person's eyes on the other side.
In this test setup, she only saw a glyph (a sphere) representing my eyes, and was looking at that.
And yes, she's sitting in front of a 3D display similar to the 3D TV shown in the video.
@askingxforxit007 Correct, it's not a hologram. However, it is displayed on a head-tracked 3D TV, and therefore looks just like a hologram (or, rather, a somewhat blurry real person) to the viewer. Hence, "holographic" in quotes.
From what it looks like, the kinects should be at 120 degrees from each other to get the best coverage. a third kinect at the back would get full coverage as well.
Would it be better visually if the kinects covered the front and back, so that the seams are at the side rather than in the center of the face? The 90' separation between the kinects seems to add artifacts rather than adding detail (might be hard to tell from vids). So, If your aim is good coverage, then the two kinects should cover as different angles as possible, no?
Does your set-up clean up background subtraction? Looks cleaner than most vids so far.
@johngomm This is being looked into. The thing is, you want good coverage of the face, and a single Kinect will still get shadows even if it's head-on (ignoring the problem that you can't have it head-on because that's where the display screen is). Take the nose, which will cast a shadow on the rest of the face if the user turns her head. And for video conferencing, the back of the head is not that important (how often do you talk to the back of someone's head).
@michaeljtandy Yup...we have some Optrima TOF's. I suspect we are about to see some significant price drops on TOF. Has anyone done side by side comparisons of TOF vs. structured light in high IR environments?
@FoundUnder Took two days to develop the 3D reprojection method once Hector Martin figured out the low-level USB protocol, and the rest wasn't really that Kinect-specific. Custom depth image compression, color compression using a standard codec, TCP network protocol, calibration method, etc. Much of that I had lying around already.
@toshironikko It only requires one Kinect for each side of the conversation who wants to be sent in 3D, and some means of head tracking, e.g., a Kinect, for each side who wants to see the other side in 3D. I'm using two Kinects on one side here simply to get better coverage.
In a nutshell: if you want to talk to someone else via 3D video chat, both of you need one Kinect.
@okreylos well ive seen the one of the other videos regarding this - it would show only half of the person/thing in 3D - the half facing the camera while everything else would be "shadowed" or "hollowed" . To experience a Full 3D chat experience it would require 4 kinects and the cheapest face tracking recognition wiimote sensors. Im not saying it couldnt be done cheaper im just saying the $150 is still a understatement.
can you mount a wiimote and a camera on top of your head, place the 2 kinects in front of you pointing at you and project yourself into a 3d space on that tv. combine the position of the wiimote and the camera like you did and adjust so we see what you see?
I really value the work you are doing. I've been dreaming of a time when we can walk around 3d in environments and interact with other people. Literally I've imagined this for at least 15 years. I wonder if you could use the Kinect camera's in a way that you could capture objects in 3d and then save them as models for a game or even to make an entire game look completely real by scanning all 3d surfaces of a room and having it compile in a way that it's useful? Thoughts?
I saw this video here, then i saw it on g4s iphone app.
Im tellin you, if you could get like good quality 3d video chat streaming online, make sure you, idk, patent it or somethin so people can't steal your work without giving you credit.
@rekneslon Each "facade" (a 3D view from a single point) is quite accurate, especially now that I have some temporal filtering. It's more difficult to join multiple facades, like the two shown in this video, into a single consistent 3D model (look at the line running down her face). There's existing software to do it, like Geomagic, but it's not in real time as far as I know.
@okreylos no not really. Why does it take people like him to solve our world issues when other people who get paid more than us can't figure this shit out.
Great job again! Thanks for keeping us updated. I wonder what the improved video quality will look like, that you wrote about in the comments. I guess I'll have to wait for a future video. I hope it will fix most of the holes in the face, because they are still a bit irritating, but that is surely just a temporary thing until some improvements are made. The capabilities already are amazing for a software so early in it's developement. Gruß aus Hamburg! (naja fast Hamburg).
By expensive cameras, I am assuming you TOF cameras like those from MESA, Optrima and Canesta. From a quality viewpoint, I'd be curious where you see the advantages of Structured light/Kinect vs. TOF solutions, particularly in less controlled (more IR intense lighting conditions)?
BTW. Are you actually from the future? We've been taking bets.
@pointanddo No, I was referring to depth-from-stereo cameras such as Point Grey's bumblebee, or the dragonfly clusters that formed our previous 3D video system. I don't have experience with TOF cameras, but I know an expert who says that those feature odd nonlinear distortion, and don't deal well with surfaces of different IR absorption coefficients (dark appears farther). That might be why Microsoft didn't use PrimeSense's initial TOF design for the Kinect.
@pointanddo There are some pretty impressive things that can be done with time-of-flight - companies like ASC have cameras that can look through Venetian blinds by detecting multiple returns.
I think the big downside to SwissRanger is the $9095 price tag - some day maybe they can be made for $150 but I think we've got a few years to wait yet!
By expensive cameras, I am assuming you TOF cameras like those from MESA, Optrima and Canesta. From a quality viewpoint, I'd be curious where you see the advantages of Structured light/Kinect vs. TOF solutions, particularly in less controlled (more IR intense lighting conditions)?
BTW. Are you actually from the future? We've been taking bets.
How much does the orientation of the Kenitic effect the overlap distortion? For instance in this case you have the hardware place 120° from the target with it aimed directly at the target. What if you were to have the Kenitic aimed 1m behind the intended target, but still placed at the same 120° from the target? Would this lessen the confusion between the two units that causes the line down the middle of the targets face?
here is a question, can you use the Kinects to get a 3d "scape" and then use high res cameras to put the textures (facial features) over top of the scape by adding key follow points?
@lich109 Yes. It wouldn't be based on follow points, but on careful camera registration during setup. In a way I'm already doing that, since the Kinect's color and depth camera are not in the same place; they're not even carefully locked into place -- different Kinects have quite different camera placements. So ignoring the color camera and just putting a higher-resolution camera somewhere close by wouldn't change the software at all.
how about using 3 kinects angled 120° each around her? :) or only 2 but with a larger angle (again, like 120°) so that there is less laser dots overlap for each sensor.
@Ceasar425 Well, we have expensive 3D cameras already, and they're not really all that much better (I'd say worse).
The main motivation here is to create a system where the whole enchilada -- head-tracked stereoscopic 3D display on a big screen, 3D tracked input devices, remote collaboration capability, 3D video -- is all bundled and costs less than $8K.
@Ceasar425 I should elaborate. The kind of 3D cameras you probably mean, the ones they used to film Avatar, are fundamentally different beasts from this thing. Once you've filmed a scene using one of those, you can't go into the editing room and say "That's nice, but now let's change the viewpoint and look from over the shoulder of that blue guy over there on the right." With a real 3D camera like the Kinect, you can do that.
Oh, btw, since there's overlap in the camera FOV, it seems you could try trimming 0.5-1% of the edge off before compositing it together to get rid of the jaggedness.
@WyattEpp One of our grad students invented a much better blending method with transparency based on line-of-sight angle of each triangle. Haven't had a chance to apply that yet; it will improve video quality significantly.
Wow, you have made some amazing progress, very interesting..this has many applications...I had no idea the future was so near. Keep up the wonderful work!
Just to be sure I've got this right... so for each kinect, you RLE the depth map delta and theora the colour image (Try AVC? might get you a lower bitrate. Or maybe something wavelet-based.); then bundle them together; on receipt, you unpack the depth map and just apply each video frame as a texture? How are you syncing frames? I should probably just go look at the code...do you have some sort of source control?
@WyattEpp Yes, that's pretty accurate. It's more than just RLE, though. I use RLE to skip background pixels entirely, but I encode spans of valid pixels using Huffman-encoded deltas, since, due to the Kinect's sensing technology, neighboring pixels are highly depth-correlated. It cranks a single depth frame from 422KB to around 30KB, depending on how much non-background is in there.
@WyattEpp I'm using Theora because it's rock-solid and has a good API. x264 is somewhat flaky; as they say, "60 percent of the time, it works every time." The higher bitrate of Theora is not an issue, since overall bandwidth is dominated by the depth frames anyways. Lossy depth compression looks like crap.
Frame synching is a sore point right now; the current method is essentially broken. Need to improve that.
Source will be out soon, once I'm not totally embarrassed by it. Bit hacky.
@okreylos awesome. I'm extremely anxious and excited to see you continue your work. Be careful not to expose too many secrets. Your work is so extremely valuable. I'd hate to see a big company come in and swoop it up.
I am simply amazed by the work you are doing. Incredible to watch the growth from your first 3d video capture with the Kinect through tonight's video - very inspiring. Can't wait for the next vid!
Are those empty spaces there because the infrared is not making it back to the Kinect? What would happen if you add an infrared spotlight to the face?
@frankcoffee The line running down the middle of her face is where the 3D "facades" from the two Kinects overlap; the cameras are almost facing each other. The fact that the 3D glasses don't show up at all is because they're too shiny, so the depth-probing IR dots don't make it back to the IR camera.
That is just mind blowingly amazing. You are doing top shelf work on this stuff. Please continue and please continue to make these videos and allow us to be a part of your incredible advances.
@r2d2art2005 The Wiimote is a core component of our line of low-cost virtual reality environments. The Kinect (or Kinect-like cameras) will be the backbone of our tele-collaboration infrastructure, which is exactly what I'm showing here.
@Fry790 Video quality is one issue, proper calibration another. I finally figured out how to build a good target for automatic calibration, which should really improve how the individual 3D streams fit together. And the current implementation of the 3D video network protocol is embarrassingly unreliable. Need to fix that.
@okreylos Thank you for your reply. I can't wait until you polish the video, calibration, and network protocol. You are consistently delivering the most groundbreaking Kinect projects, and I will continue to enjoy (and hopefully one day directly benefit from) this trend.
This is so cool!
1337Yazman 3 weeks ago in playlist Uploaded videos
Y don't make new videos anymore?
MrTonski1 2 months ago
aaaaaawesome! This is the FUTURE of video chatting!!!
MrSamstaman 5 months ago
You're not great at explaining things, and this looks like crap at the moment -- but this is still amazing. Good job LOL
CooldudeXXL 7 months ago
Comment removed
FreeMeFromG 9 months ago
You're so cool man. I wish i had the knowledge you had :( i would like to dive into this virtual world :)
Sciencegek 9 months ago
You mention you are using lossless compression. What is the resolution of each frame (rgb/depth)? the native 640x480 or are you downsampling it?
madVal8 9 months ago
@madVal8 I'm using lossless compression on the depth frames, at native resolution (640x480 pixels, 11 bits per pixel, 30 frames per second). The resulting bandwidth in typical settings is about 500 KB/s. The color frames are compressed using a standard lossy video codec, in this case Theora (mostly due to its nice API and licensing terms), also at 640x480 pixels, 24 bits per pixel, 30 frames per second). H.264 or something similar would work as well.
okreylos 9 months ago
@okreylos Thanks for the info!! had another question - Does your version of Vrui toolkit support connecting multiple Kinects to one single PC or did you use multiple of them to connect the Kinects and used some kind of a central server to work on them?
madVal8 9 months ago
@madVal8 Vrui doesn't know anything about Kinect; it's just a VR / 3D graphics / UI development toolkit (think Qt or GTK+). Kinect is handled by the Kinect package, which uses Vrui for its graphical applications.
Anyway, the Kinect package handles multiple Kinects per PC, as many as you have USB buses (there can only be one Kinect per USB bus due to bandwidth constraints). There is no direct support to pool multiple PCs, but that's a simple addition to the software.
okreylos 9 months ago
Comment removed
madVal8 9 months ago
Imagine if this will be how they record people for games in the future, then the environments can be digital and the actors be photorealistic. Or in movies... REAL 3d movies.... awesome man, just awesome.
MASproductionss 10 months ago
Dude, this is amazing! People gets distracted from the artifacts on the picture, but the possibilities are insane.
medallish 10 months ago
Is it something similar as ars electronicas cave?
ThaFacka 10 months ago
- "you must see this droid safely delivered to Alderon. This is our most desperate hour. Help me O(biwan)K(enobi)reylos, you are my only hope"
Konquistadoren 11 months ago
Are you guys working on the quality of the video itself, one thing i noticed in your video with the Kinect Box and 2 Cameras is that it is really blurry and you couldn't make out an image. Another question regarding quality is if it is possible to fix those black and white blotches that appear on the screen when the subject is moving. If these have been asked before i'm sorry. Thanks
rslayer9 1 year ago
Very nice start. Your videos have answered the first question I had about using multiple Kinects to correct for the eye-line issue of webcam video calls. Thanks.
krhumbles 1 year ago
awesome videos, don't listen to the haters... you beat them to it... hehe thanks for putting your work up for everyone to see.... this is paving the way to technology that is cheap and easily accessible to everyone! As a software engineer myself whenever i get free time from my own job i can't wait to try out the musical and artist potential of this.
digid 1 year ago 3
@okreylos please ignore the haters, internet anonimity is a bitch. You're doing incredible work, keep it up!
FreedomReign66 1 year ago 2
Why use a wiimote?
stupidjunk978 1 year ago
@stupidjunk978 because of the infared tracking I guess.
Designandrew 1 year ago
You and your team are amazing!
Thank you for showing what amazing things you can do with the Kinect - I would love to see what you are able to do with better motion cameras! Maybe microsoft should hire you to update the kinect!
TheDarkemotion21 1 year ago
Incredible!
IIDMII 1 year ago
Very impressive.
I cant wait to see a version without GFX glitches even though thats probably not your top priority ;)
A lot of kudo's on managing to create eye contact properly.
alexicon89 1 year ago
Pretty cool but I can't even tell if she's hot.
Dawgtha 1 year ago 2
@Dawgtha Trust me, she is.
okreylos 1 year ago 11
Btw, full echolocation could be used in order to complete shapes that aren't visible... Kinect just leaves them incomplete... And to texture invisible mesh parts, some sort of deductive algorithm could be used like those found in recent Photoshop versions
alexgrinkov 1 year ago
hehe, they can't invent holography that would bring 3d into real world... so why not bring real world to 3d :)
alexgrinkov 1 year ago
green screen owns kinect
1Nekit1 1 year ago
@1Nekit1
with enought kinect-like devices you can actually dynamically move through the room and for example observe the girls' but without her knowing it. i.e. you can move like solely from your site. with green screens, in order to move througth the space, the real camera at the girl's site must be moved. kinect owns green screens.
alexgrinkov 1 year ago
@1Nekit1
and of course green screens should be set up like everywhere. with 3d scanners devices you can remotely chat from a cafe, for example
alexgrinkov 1 year ago
can you elaborate on what she's actually looking at in front of her when she's "pointing at the camera"? is it a tv or projector and what is it actually showing?
boabie 1 year ago
@boabie Normally there would be two-way video, and she would be looking at a 3D representation just like hers. The way the sites are connected is that if she looks at that virtual person's eyes, her virtual representation is looking directly into the real person's eyes on the other side.
In this test setup, she only saw a glyph (a sphere) representing my eyes, and was looking at that.
And yes, she's sitting in front of a 3D display similar to the 3D TV shown in the video.
okreylos 1 year ago
Live 3d telepresence. Awesome!
knightowljoe 1 year ago
its still on a tv. get rid of the tv and keep the hologram then we can call it something. right now its still just video chat lol
pcrat54 1 year ago
Comment removed
thygate 1 year ago
what's the point?
Totz666666 1 year ago
This comment has received too many negative votes show
Its not holographic if its on tv. what a waste of money
askingxforxit007 1 year ago
@askingxforxit007 Correct, it's not a hologram. However, it is displayed on a head-tracked 3D TV, and therefore looks just like a hologram (or, rather, a somewhat blurry real person) to the viewer. Hence, "holographic" in quotes.
okreylos 1 year ago 9
From what it looks like, the kinects should be at 120 degrees from each other to get the best coverage. a third kinect at the back would get full coverage as well.
Wraithofvolsunga 1 year ago
the answer to your problem is 3 kinects. you're welcome.
Jasen1337 1 year ago
Another Fail. Go for more.
appleseedas 1 year ago
@appleseedas Thank you again!
okreylos 1 year ago
If you don't have an amazing job I hope some sort of technology giant sees this and hires you.
TeenageMuse 1 year ago
@TeenageMuse I have a pretty sweet job, that allows me to do these things. But thanks for the sentiment.
okreylos 1 year ago
ITS ALIVEEEEEE
featheredmusic 1 year ago 2
That is amazing..!
Thimmet 1 year ago
Looks cool, but I think it would look much better with a third Kinect camera.
Winstonsicle 1 year ago
I can't wait till some company sees this and gets ideas, thus throwing you a couple million dollars to produce it.
niftycanoffree 1 year ago 4
could you somehow impliment radar? to ge behIND what the camrea cant see?
theomick 1 year ago
Unbelieveable !
samsoulee 1 year ago
Happy Holidays Guys, Merry Christmas.
HomeWindmills7 1 year ago
Realy interesting series of videos. Im looking forward to the next one :)
N3CR1S 1 year ago
Are you using a Wiimote?
wyattsawesome 1 year ago
@wyattsawesome Yes.
okreylos 1 year ago
@okreylos are you aware of youtubes new homepage? lol
don't flatter yourself ;)
p.s keep pretending, I mean pioneering lol
shawnio 1 year ago
Virtual reality games... here we come!!! :D
iatedeadpeople 1 year ago
Great work, I'm following it closely.
Would it be better visually if the kinects covered the front and back, so that the seams are at the side rather than in the center of the face? The 90' separation between the kinects seems to add artifacts rather than adding detail (might be hard to tell from vids). So, If your aim is good coverage, then the two kinects should cover as different angles as possible, no?
Does your set-up clean up background subtraction? Looks cleaner than most vids so far.
johngomm 1 year ago
@johngomm This is being looked into. The thing is, you want good coverage of the face, and a single Kinect will still get shadows even if it's head-on (ignoring the problem that you can't have it head-on because that's where the display screen is). Take the nose, which will cast a shadow on the rest of the face if the user turns her head. And for video conferencing, the back of the head is not that important (how often do you talk to the back of someone's head).
okreylos 1 year ago
consistently amazing
cruxphotography 1 year ago
This comment has received too many negative votes show
please stop making pointless videos.
shawnio 1 year ago
@shawnio That's funny, considering you can't seem to stop watching them. Can I have my own creepy Internet stalker?
okreylos 1 year ago 14
@shawnio You obviously didn't watch it all the way through, plenty of "pointing" in the second half ;)
ajeoae 1 year ago
That why i say what kinect will be a bomb.
blackbirdru 1 year ago
kinect is gonna be pretty damn kool to use at uni + due to higher tuition fees, education cuts, :P
MrZAlmao 1 year ago
FUCKING GENIUS!!!
MrShebaw 1 year ago
Interesting, but the most important thing -- the middle of her face -- has the worst quality.
nroose 1 year ago
@michaeljtandy Yup...we have some Optrima TOF's. I suspect we are about to see some significant price drops on TOF. Has anyone done side by side comparisons of TOF vs. structured light in high IR environments?
pointanddo 1 year ago
think of the porn possibilities!
canucks16 1 year ago
how long did it take u to hack the kinect to do this
FoundUnder 1 year ago
@FoundUnder Took two days to develop the 3D reprojection method once Hector Martin figured out the low-level USB protocol, and the rest wasn't really that Kinect-specific. Custom depth image compression, color compression using a standard codec, TCP network protocol, calibration method, etc. Much of that I had lying around already.
okreylos 1 year ago
thats cool, i fuck supermodels.
0110111101101011 1 year ago
Fuck thats annoying to look at. use an HD cam or something.
fr3d420 1 year ago
why are you using kinect to do all this ? there is alot better technology available.
oBLACKIECHANoo 1 year ago
@oBLACKIECHANoo Show me something better that costs $150, and I'll buy it.
okreylos 1 year ago 49
@okreylos that would work too haha
ILZProductions 1 year ago
@okreylos wouldnt this cost closer to $350? since it takes two kinects and a wiimote....
toshironikko 1 year ago
@toshironikko It only requires one Kinect for each side of the conversation who wants to be sent in 3D, and some means of head tracking, e.g., a Kinect, for each side who wants to see the other side in 3D. I'm using two Kinects on one side here simply to get better coverage.
In a nutshell: if you want to talk to someone else via 3D video chat, both of you need one Kinect.
okreylos 1 year ago
@okreylos well ive seen the one of the other videos regarding this - it would show only half of the person/thing in 3D - the half facing the camera while everything else would be "shadowed" or "hollowed" . To experience a Full 3D chat experience it would require 4 kinects and the cheapest face tracking recognition wiimote sensors. Im not saying it couldnt be done cheaper im just saying the $150 is still a understatement.
toshironikko 1 year ago
@toshironikko If you insist -- OK.
okreylos 1 year ago
This is going to be like the movie, "GAMER" soon.
TeheRJ 1 year ago
Reminds me of Mass Effect 2 in the Illusive Man's office.
Shadow9392 1 year ago 5
I've been dreaming for this since the invention of the internet. Thank you for posting this, sir - and I wish you the best of luck in the future.
MartinTrankell 1 year ago
Amazing...great job sir!
Hello from Reddit.
shizyo 1 year ago
To bad the hardware is cheapass, the tech none the less isn't.
sypros 1 year ago
genius
Prof16440 1 year ago
holy shit!
peeweejd 1 year ago
Very nice - now I'm going to have to convince my advisor that this could be fun and related to my work... somehow...
CircleTheSkies 1 year ago
Now we only have to wait for a new version of this tech so it doesn't look so bad
teamaquadan 1 year ago
I like how he's using a wiimote for the kinect technology...
xguitarguyx77 1 year ago
can you mount a wiimote and a camera on top of your head, place the 2 kinects in front of you pointing at you and project yourself into a 3d space on that tv. combine the position of the wiimote and the camera like you did and adjust so we see what you see?
dustybikeeeeeeeer 1 year ago 2
Absolutely amazing!
youtubasoarus 1 year ago
You have an awesome job.
shwhjw 1 year ago
Howdy
vinniyo 1 year ago
this could revolutionize porn, imagine the possabilities
jimbob15101510 1 year ago 3
@jimbob15101510 search for ThriXXX for first steps into this direction
595o 1 year ago
wow, amazing
Caeseur 1 year ago
This has got to be some of the most amazing work I've seen in a while. You sir, are a genius!
DoctaProfessah 1 year ago
Am i watching porn?
KarlNuevo16 1 year ago
@KarlNuevo16 If you are, I don't want to know.
okreylos 1 year ago 3
Wow, you were right, this technology is coming along faster than I could have predicted! Keep up the good work!
jake2beck 1 year ago
awesome work sir, you will go far with this. keep up the great work, this will change a lot of things with this device.
madmax2069 1 year ago
this is awesome
xg4bx 1 year ago
I really value the work you are doing. I've been dreaming of a time when we can walk around 3d in environments and interact with other people. Literally I've imagined this for at least 15 years. I wonder if you could use the Kinect camera's in a way that you could capture objects in 3d and then save them as models for a game or even to make an entire game look completely real by scanning all 3d surfaces of a room and having it compile in a way that it's useful? Thoughts?
ieatthingshigh 1 year ago
....
Mrtheben 1 year ago
Have you tried one camera facing down on the subject, and one in front of the subject?
MrSpc4 1 year ago
I saw this video here, then i saw it on g4s iphone app.
Im tellin you, if you could get like good quality 3d video chat streaming online, make sure you, idk, patent it or somethin so people can't steal your work without giving you credit.
pb0o6 1 year ago
You are so awesome!
BadLactose 1 year ago
@flawless2c Most impressive, indeed.
vadersdemise93 1 year ago
I don't know how accurate it would be but would be but could you create a 3d model of the person or whatever you have in front of the Kinect(s)?
rekneslon 1 year ago
@rekneslon Each "facade" (a 3D view from a single point) is quite accurate, especially now that I have some temporal filtering. It's more difficult to join multiple facades, like the two shown in this video, into a single consistent 3D model (look at the line running down her face). There's existing software to do it, like Geomagic, but it's not in real time as far as I know.
okreylos 1 year ago
Do I care?
luffy11234 1 year ago
@luffy11234 I don't know. Maybe?
okreylos 1 year ago
@okreylos no not really. Why does it take people like him to solve our world issues when other people who get paid more than us can't figure this shit out.
luffy11234 1 year ago
@luffy11234 better question would be "Do you matter?" The answer to this would be "no."
eternalautumn 1 year ago
@luffy11234 if you dont care then why in the hell did you watch the video, and why waste time commenting ?
should you care ? yes you should. or your just being the typical Youtube troll
madmax2069 1 year ago
Remarkable!
rocshaboc 1 year ago
Great job again! Thanks for keeping us updated. I wonder what the improved video quality will look like, that you wrote about in the comments. I guess I'll have to wait for a future video. I hope it will fix most of the holes in the face, because they are still a bit irritating, but that is surely just a temporary thing until some improvements are made. The capabilities already are amazing for a software so early in it's developement. Gruß aus Hamburg! (naja fast Hamburg).
CptSoda 1 year ago
This comment has received too many negative votes show
Ask her to take her top off.
peperonyandchease 1 year ago
@peperonyandchease Watch it! That's Mrs. Okreylos you're talking about.
okreylos 1 year ago 45
Very Impressive :) thanks for sharing this with us :)
Elistios 1 year ago
By expensive cameras, I am assuming you TOF cameras like those from MESA, Optrima and Canesta. From a quality viewpoint, I'd be curious where you see the advantages of Structured light/Kinect vs. TOF solutions, particularly in less controlled (more IR intense lighting conditions)?
BTW. Are you actually from the future? We've been taking bets.
pointanddo 1 year ago 5
@pointanddo No, I was referring to depth-from-stereo cameras such as Point Grey's bumblebee, or the dragonfly clusters that formed our previous 3D video system. I don't have experience with TOF cameras, but I know an expert who says that those feature odd nonlinear distortion, and don't deal well with surfaces of different IR absorption coefficients (dark appears farther). That might be why Microsoft didn't use PrimeSense's initial TOF design for the Kinect.
okreylos 1 year ago
@pointanddo There are some pretty impressive things that can be done with time-of-flight - companies like ASC have cameras that can look through Venetian blinds by detecting multiple returns.
I think the big downside to SwissRanger is the $9095 price tag - some day maybe they can be made for $150 but I think we've got a few years to wait yet!
michaeljtandy 1 year ago
By expensive cameras, I am assuming you TOF cameras like those from MESA, Optrima and Canesta. From a quality viewpoint, I'd be curious where you see the advantages of Structured light/Kinect vs. TOF solutions, particularly in less controlled (more IR intense lighting conditions)?
BTW. Are you actually from the future? We've been taking bets.
pointanddo 1 year ago
How much does the orientation of the Kenitic effect the overlap distortion? For instance in this case you have the hardware place 120° from the target with it aimed directly at the target. What if you were to have the Kenitic aimed 1m behind the intended target, but still placed at the same 120° from the target? Would this lessen the confusion between the two units that causes the line down the middle of the targets face?
retsaoter 1 year ago
here is a question, can you use the Kinects to get a 3d "scape" and then use high res cameras to put the textures (facial features) over top of the scape by adding key follow points?
lich109 1 year ago
@lich109 Yes. It wouldn't be based on follow points, but on careful camera registration during setup. In a way I'm already doing that, since the Kinect's color and depth camera are not in the same place; they're not even carefully locked into place -- different Kinects have quite different camera placements. So ignoring the color camera and just putting a higher-resolution camera somewhere close by wouldn't change the software at all.
okreylos 1 year ago
This would make porn so much awesome
UerMom 1 year ago
@okreylos Now do 4.
BurnZeZ 1 year ago
One step closer to virtual reality
gordon1201 1 year ago
Ich hab dich abboniert :De
momentomorinacko 1 year ago
Amazing!!! Keep up the awesome vids!!!
qdog1342 1 year ago
Watching history in the making... Fantastic!
davidbaumseattle 1 year ago
how about using 3 kinects angled 120° each around her? :) or only 2 but with a larger angle (again, like 120°) so that there is less laser dots overlap for each sensor.
awaken69 1 year ago
mann, i'm so glad that you are sharing all this thing...
gersinhw15 1 year ago
Amazing! Would you consider buying a higher quality (and very expensive) 3D camera?
Ceasar425 1 year ago
@Ceasar425 Well, we have expensive 3D cameras already, and they're not really all that much better (I'd say worse).
The main motivation here is to create a system where the whole enchilada -- head-tracked stereoscopic 3D display on a big screen, 3D tracked input devices, remote collaboration capability, 3D video -- is all bundled and costs less than $8K.
We finally have that.
okreylos 1 year ago 2
@Ceasar425 I should elaborate. The kind of 3D cameras you probably mean, the ones they used to film Avatar, are fundamentally different beasts from this thing. Once you've filmed a scene using one of those, you can't go into the editing room and say "That's nice, but now let's change the viewpoint and look from over the shoulder of that blue guy over there on the right." With a real 3D camera like the Kinect, you can do that.
okreylos 1 year ago 6
Oh, btw, since there's overlap in the camera FOV, it seems you could try trimming 0.5-1% of the edge off before compositing it together to get rid of the jaggedness.
WyattEpp 1 year ago
@WyattEpp One of our grad students invented a much better blending method with transparency based on line-of-sight angle of each triangle. Haven't had a chance to apply that yet; it will improve video quality significantly.
okreylos 1 year ago 23
Wow, you have made some amazing progress, very interesting..this has many applications...I had no idea the future was so near. Keep up the wonderful work!
Figilswif 1 year ago
Just to be sure I've got this right... so for each kinect, you RLE the depth map delta and theora the colour image (Try AVC? might get you a lower bitrate. Or maybe something wavelet-based.); then bundle them together; on receipt, you unpack the depth map and just apply each video frame as a texture? How are you syncing frames? I should probably just go look at the code...do you have some sort of source control?
WyattEpp 1 year ago
@WyattEpp Yes, that's pretty accurate. It's more than just RLE, though. I use RLE to skip background pixels entirely, but I encode spans of valid pixels using Huffman-encoded deltas, since, due to the Kinect's sensing technology, neighboring pixels are highly depth-correlated. It cranks a single depth frame from 422KB to around 30KB, depending on how much non-background is in there.
okreylos 1 year ago
@WyattEpp I'm using Theora because it's rock-solid and has a good API. x264 is somewhat flaky; as they say, "60 percent of the time, it works every time." The higher bitrate of Theora is not an issue, since overall bandwidth is dominated by the depth frames anyways. Lossy depth compression looks like crap.
Frame synching is a sore point right now; the current method is essentially broken. Need to improve that.
Source will be out soon, once I'm not totally embarrassed by it. Bit hacky.
okreylos 1 year ago
do you think its possible for you to make a flawless image at some point with the technology given? you're getting pretty dang close.
MichaelMusic1 1 year ago
@MichaelMusic1 There are some really low-hanging fruit left to improve this. Now that the big step is done, next come the incremental improvements.
okreylos 1 year ago
@okreylos awesome. I'm extremely anxious and excited to see you continue your work. Be careful not to expose too many secrets. Your work is so extremely valuable. I'd hate to see a big company come in and swoop it up.
MichaelMusic1 1 year ago
gr8 work, amazing,,,,,,,
jibiabraham 1 year ago
This is... the greatest thing I have ever gazed upon. You will be revolutionizing how we communicate! BE PROUD!
JBGUY2K 1 year ago
gr8 work, amazing
jibiabraham 1 year ago
home-based strippers anyone?
hgpwns 1 year ago
I am simply amazed by the work you are doing. Incredible to watch the growth from your first 3d video capture with the Kinect through tonight's video - very inspiring. Can't wait for the next vid!
jbix 1 year ago
she's a cutie
skinnyluigi 1 year ago
That is absolutely amazing to see how well done that is so far. Keep up the amazing work!
Foxtropics 1 year ago
Are those empty spaces there because the infrared is not making it back to the Kinect? What would happen if you add an infrared spotlight to the face?
frankcoffee 1 year ago
@frankcoffee The line running down the middle of her face is where the 3D "facades" from the two Kinects overlap; the cameras are almost facing each other. The fact that the 3D glasses don't show up at all is because they're too shiny, so the depth-probing IR dots don't make it back to the IR camera.
okreylos 1 year ago
That is just mind blowingly amazing. You are doing top shelf work on this stuff. Please continue and please continue to make these videos and allow us to be a part of your incredible advances.
GHRiz 1 year ago
omg wtf? man you have to do an more advanced version of this!!! keep workin on that man it just looks so awesome!
KevboardGames 1 year ago
I've been waiting for a new vid! Great work.
On the down low, it would be cool to see you demonstrate a Star Wars training remote program ;)
MudRat02 1 year ago
May I ask what you do for a living?
LenntBear 1 year ago
Awesome, you combine the technologies of both the Kinect and the Wiimote!
r2d2art2005 1 year ago
@r2d2art2005 The Wiimote is a core component of our line of low-cost virtual reality environments. The Kinect (or Kinect-like cameras) will be the backbone of our tele-collaboration infrastructure, which is exactly what I'm showing here.
okreylos 1 year ago
@okreylos Does the IR from the WiiMote IR bar interfere with the Kinect camera?
adfhau 1 year ago
amazing work my man. don't let microsoft hire you for less than 7 figures!!
ile3000gt 1 year ago
Great job. What is the next step/goal for you?
Fry790 1 year ago
@Fry790 Video quality is one issue, proper calibration another. I finally figured out how to build a good target for automatic calibration, which should really improve how the individual 3D streams fit together. And the current implementation of the 3D video network protocol is embarrassingly unreliable. Need to fix that.
okreylos 1 year ago 2
@okreylos Thank you for your reply. I can't wait until you polish the video, calibration, and network protocol. You are consistently delivering the most groundbreaking Kinect projects, and I will continue to enjoy (and hopefully one day directly benefit from) this trend.
Fry790 1 year ago
@Fry790 Well, get a Kinect, a PC, install Linux and the software, and call grandma in 3D right away. That's an immediate benefit right there!
At least as soon as I release the new software version, that is... :)
okreylos 1 year ago 2
wow, that's cool ! what does it do under different colored lighting?
MrOne2watch 1 year ago
@MrOne2watch You mean the 3D capture? It's not affected by lighting, since it uses active sensing. The colors would look different, of course.
If you're asking about the background removal, that's lighting-independent as well since it's based on depth.
okreylos 1 year ago