 Great, thank you Now for All of us there are certain moments in our lives that change the way we think about things a certain defining events or Bits of wisdom that are imparted by other people They either make us look at things differently to the way we did before or they reinforce our existing beliefs and And make us more sure of ourselves and for me one of those events was in a home economics class in year seven and We were given an exercise of designing an optimal kitchen layout And so we had to work out the position of the stove position of the fridge where the workspace would be all of that sort of thing and then we had to do a bit of time and motion study and this was the the first time I'd ever been exposed to this concept and the whole idea of usability and thinking about how things are arranged for optimal efficiency and It was a silly little class exercise like over in half an hour But it was one of those things that stuck with me and it's led to what some people would call probably rather obsessive behaviors For example when I'm letting the dishwasher I always pre-sort the cutlery because it's faster to do that than to sort it when you're taking it back out and And when I'm putting times into the microwave I always do multiples of 11 seconds because there's less seek time on the keypad There's actually another reason for that. There's a double reason my previous microwave Yeah, I see there. Yes My previous microwave had a feature where wanted to finish cooking it would continue turning Until it had returned everything back to the front. It had about an 11.4 second cycle time So what happens is if you put in say 20 seconds, you got to wait 2.8 seconds from the time it cooks to the time It says it's ready. Whereas 22 seconds. You've only got to wait point eight. So it's much much better so my very first LCA was not far from here at UQ and one of the one of the keynotes was Mischie Henning who talked about computing fallacies and Usability and all of the stupid things that are done when people really should know better and One of the things you brought up was this issue of user interfaces in the real world physical environment so natural interface and The obvious one is door handles. I mean a door is something that we go through dozens of times a day Why should we need instructions to know how to use a door? That's just stupid And so the question that he put to the audience was what is the optimal design for a door that you've got to push to go Through and so someone give me an answer Yeah, a flat plate exactly So I was already thinking along these lines when When I saw this keynote back in 2002 so it sort of reinforced my obsession with this sort of thing and Over time I ended up with sort of mantra, which is the best UI is no UI Everything should disappear and it should simply do exactly what it is meant to do at the right time It should not necessarily take conscious thought to make something behave in a logical way Objects should be context-aware and they should be intent aware, but of course that's really really difficult to achieve So one of the things That I did This comes back to the door example once again. Okay, think about a door So Mischie's example was the best interface for a door is a flat plate So you simply push it it's unambiguous, but if you think back more fundamentally to what a door does think about all the The states it needs to be in and the reasons it should be in that state if I want to walk through it I want the door to be open. I shouldn't have to do anything. It should just not be there If I'm not walking through it and it's windy. I want it to be closed so that you know dust and the neighbors cat don't come in and if Joe burglar comes and tries to open the door it shouldn't open for him So it should be open in some cases closed and others locked in some cases locked unlocked in other cases Or maybe it's in the evening and the house near in the house is hot and outside It's started to cool down So you want the door to be open to allow cool breeze to come in all these sorts of scenarios But the door itself should be able to take care of this. You shouldn't have to think about it So one of the things I set out to do was to remove keys. I wanted Doors to simply be unlocked when I went through them and to be locked when it whenever anybody else tried to go through them Now I'm not going to dwell on this because I've already done lots of talks about this sort of stuff So a few years ago. I implanted nerf ID tag in my arm partly to experiment with this sort of thing Partly also to figure out some of the exploits and things associated with it So the end result is that For me my front door is unlocked all the time I can walk up to it empty-handed and it just lets me in for other people. It happens to be locked So I won't go into that any more detail because a lot of you have already heard me talk about that But one of the one of the interesting little side stories is that there's some really cool stuff on eBay nowadays A friend of mine a few months ago bought an ultrasound machine on eBay. He set it up in his garage We have a nice little surgery in there now and And so of course we're going to his garage and just scan random things see what we can see and So this is a scan of my arm and the crosses up in the top show the position of the RFID tag Which is just under the skin so that was verifying. Yes, it really is still there hasn't gone away But on the theme of things behaving automatically some of you may remember how many people were at the myth TV minicons a couple of years ago Now a few of you okay, I did a talk there about presence awareness for myth TV and I've done a little experiment at the time where I had an infrared badge like an infrared transmitter that sent out a beacon So you click the badge on there's a receiver at the TV if the TV sees a receiver then it knows you're there and if you stand up walk out of the room TV pauses and You can extend that a little bit further and have follow me as well So you walk out of one room and then into another room and you show pops up on the TV that of the room that you walked into She thinks it's pretty cool But it's got a few problems firstly sticking a badge on is just naff and stupid and at the time I I said I wanted to experiment with a bit of face detection and things like that so I've been doing that and It actually works pretty well those little webcams sitting up on top of the TV And this is once again an example of the environment around you simply doing what it should do without you having to Decide I am going to pause the TV or I need to do something to make it make it behave the way it should and So let's try this out. This is probably going to be one of those big demo fail moments But hey, that's what makes this fun. So Let's stick up a movie. I'm not normally I do this with myth TV, but I don't have myth TV on my laptop so I Just run up in player and when I was trying this earlier it was having a hell of a time with the lighting conditions in here So we will see what happens Okay, so what we should have now Bring that to the foreground is if it sees me in front of it or if it detects any faces The movie should play and if I walk away, it's probably gonna detect faces and random patterns on the wall now Then the movie pauses and it requires a number of consecutive frames I've set it up with enough several consecutive frames So that false positives don't trigger at the moment it's triggering like crazy in this room, but it actually is not that bad in the lounge room Inconceivable so That actually works. Okay. So this is just one of the little things that I've done experimenting with trying to to make the world behave the way it should and So there are all sorts of sensors and control interfaces and things like that that you can you can bring into use and Part of the point of this and this is something I've said in previous talks as well is to think of everything as modular So think of the world as Lego you can have a whole lot of different actuators You can have a whole lot of different sensors You can have different ways of controlling things and once you start putting these various elements into place You can then tie them together in new ways So if you already have an actuator in place to control something in your physical environment If you dream up a new way of controlling it It's then a relatively simple thing to tie it together and then you can get different behaviors and one of the most interesting things to come out recently I'll tell you a little story about it. So a long time ago in a universe far far away a Powerful Empire had plans for a new game console peripheral and This thing is actually really really cool I mean, I know there are at this sort of conference there are people that like to bash Microsoft But in this case they have built some amazing hardware. This stuff is really really cool What they've done is taken technology that was previously only available to Researchers at very high price point and they produced it literally in the millions I think they sold a million in the first 14 days or something and that's pulled the price point down to the point where Research equipment that used to have to spend many thousands of dollars for you can now get for a hundred and ninety each dollars And you can walk into a store and buy it And it's got some really interesting stuff in it There's an accelerometer which may sound kind of stupid for something that you stick on front of your TV and never touch But there's a reason for that There's a pitch control servo so it can adjust its angle and the idea is that you can put it down And it will self level so you can and you can also have it track faces and things like that It's got a 640 by 480 RGB camera in it so you can do regular Video pick up and it also has a four-way microphone array which in itself is really interesting I haven't done anything with this yet But it has a lot of potential The one of the major problems you have when you're picking up audio from a large room is that you get everything you get the Echoes and you get ambient noise and you get multiple people talking so if you're trying to do say voice control or if you want to have a skype conversation with someone on your TV and You're standing there talking to them and someone else is talking in the background You get all sorts of problems with sound sound is one of those things that is really really hard to do right I mean on Julian I'm sure would agree with me on this one and people underestimate how difficult it is to get audio Working correctly What they've done with this is put in a for array for my way microphone array so they can actually do spatial analysis of the sound and Distinguish the position that sound comes from in the room and cancel out everything else so what they could do is use the the position derived from things like video sensors in the depth sensor and Then ignore sound that comes from anywhere else So you could stand in a noisy room and talk and your voice will be picked up clearly it also has a IR camera, which is 11 bit so 2048 levels and a Structured light IR projector, and this is where stuff gets really really interesting The way this works is a little bit like radar It's a technique called LiDAR so it uses light instead of radio waves But instead of using Time to reflect like radar what it does is use dispersal of light pattern across a surface to detect Distance and deformation So the way to think about this is imagine that there is a grid of little colored dots infrared dots being projected out from this in a cone and As the dots strike objects that are further away They'll be further apart and so with the infrared camera by doing analysis of the position of the dots in the image It makes it really easy to get the distance to any object that is in front of the camera and that is really cool so let's try that and See how it works. Did I get yeah, okay, so Actually, no, I'll tell you before I get to that. I'll tell you a little story about how the the drivers for this came about because it is actually kind of intriguing November 4th was when this was launched and on the day it was launched the more freed Who many of you will know from Adafruit who's just a little hardware company in the US? I thought it would be a cool peripheral for robotics projects So she bought one and ran a USB packets niffer on it and thought yeah This looks like it's got some good stuff in it, but I don't have time to write drivers So I want someone else to do it. So later that day She published the US the packet dumps and this was at 340 p.m. And offered a $1,000 bounty So the first person to release an open-source driver for it CNET got on the case really really fast called their local Microsoft rep and They made a statement 20 minutes later saying Microsoft will continue to make advances in these types of safeguards and work closely with law enforcement and product safety groups to keep connect Tampa resistant Okay, so they don't want anybody messing with their stuff so an hour later they're more increased about it and Over the next couple of days there was a bit of discussion about it and she thought about the implications of okay She's putting up money for someone to reverse engineer something and Microsoft really does not like this So she increased it again This was after the US launch on November the 4th what happened next was that the the European launch for schedule for November the 10th and on November the 10th they went on sale in stores in a number of places including Spain and This pleasant-looking gentleman named Hector Martin walked into a store About nine o'clock on that morning and he's a pleasant-looking chap That he doesn't actually own an Xbox. He just wanted to buy the Kinect But what he is is one of the members of the the group that reverse engineered the Sony DRM on the ps3 at noon he published video proof and Source code of a working driver So later that day Lee Moore says you've done it and not only did he get $3,000 She decided that because of the threat to people's rights to To reverse engineer and to develop interoperability with existing products. She donated an extra $2,000 to the EFF So then Microsoft made a statement saying the interface was opened by design and they were inspired Okay, of course it's a big company. They don't talk internally and stuff like that But I thought it was when you look at it in as a microcosm like that. It's quite an interesting little sequence of events So this is what it actually does What we're looking at here are the two video streams one on the left is the depth camera one on the right is the RGB camera and There are some really interesting things you can see out of this What this is doing is applying false color mapping to the the depth values that are coming out of the camera So the closer something is to the camera the hotter it is Now there are think a few things you'll notice here if you look at my hand for example in that picture It's reasonably close in color to the wall behind me And you also notice that there are things like a hard line intersecting fingers So if you are applying You know edge detection algorithms and things like that they have a bit of trouble with the RGB image So one of the interesting things about this is not just that it allows a whole new class of approaches to solving problems but it means you can take Existing problem-solving approaches and things you know things in open CV for example Instead of throwing the algorithm at the RGB values throw them at the depth values and all of a sudden it works really really well So as you can see depth edge detection is a piece of cake when you're dealing with the the depth values So I'll fire it up and we'll see it working for real. So as I get closer you can see it gets hotter Hey, let's have a look at you guys Hey At this sort of technology Six months ago. You would have been paying many many thousand of dollars for something they could do Anything even remotely like that. So that is really really cool. And this what I'm running right now is the example application That Hector released as part of his that runs with his driver. So this is with the the open-source driver So what I did was sort of mucking around with with things like hand tracking and mapping 3d zones in space to various actions and It can start bending your brain a little bit. I'm not used to dealing with these sorts of things so It was quite a stretch for me. But basically what's happening is that? The camera itself is outputting like if for example, you run a hand tracking system You end up with XYZ coordinates of whatever whatever your point of interest is. So say your hand But they are projective coordinates. They're not absolute coordinates So if you think about a room being a cube with XYZ axes on it It is not an absolute position That's the same everywhere through a scale the same everywhere throughout it because what we're looking at is a conical field of view What's actually a slightly flattened cone an X value, which is the horizontal value? Will be a different offset depending on how far away you are from it So an X offset on the edge of the field of view will actually move in towards the edge So when you start defining volumes of space, it can become a bit brain bending But I drew lots of sketches and worked out some Some positions and played around with it. So Just as a demo of this we've got a thing hooked up This is another major opportunity for demo fail. So This could be fun So what we're doing here is I'm running the the hand tracking software and There are a number of points in space. I've been careful not to walk in front of it right now There are a number of points of space that have been defined as having different meanings When it detects one of those events those little it's a loop basically which has little state machine in it so when it detects a transition of say your hand moving into a hot zone it then Sends a network connection which is going through at the moment to Andy's laptop sitting on the bottom there Which is associated wirelessly with the parrot. So if I I'll angle that up a little bit What do I have I can sit it on and we'll do I think the self-level of Ren and locked it up. Okay, so See if I can get it to detect my hand So I can now send an up command Push Come to me my pretty Practice we got it to work about one time in ten. I had no expectation. It was going to work Actually, I just go back on that one. No back Okay, so one of the problems that you have what you would have noticed is that when I was doing this I was looking back at the computer screen the whole time and I was watching the what you were seeing which was the position that it was interpreting my hand as being and Trying to figure out where my hand is relative to those hot zones in space Because the thing is if you wave your hands around you go into a hot zone like you have a transition into it It sends an event that is really really annoying and Something that I discovered as soon as I started playing with this is that it's hard to have any sort of interface That doesn't give you any feedback like we're used to To either physical feedback if we're dealing with physical devices or with some kind of visual feedback if we're using software or maybe audible or something like that and so over the Christmas holidays, I was relaxing and playing with the kids Christmas presents and And getting any inspiration and one of the things I realized is that in a computer game They often highlight for example if you're running through a virtual environment and there are multiple objects you can have An action on then when you select it often it will have like a blue glow or something So you know, okay, that's the action. That's the object. I've selected so I can Open that door using the force and then I can run through and zap all the guards and So I decided that it was worth giving it a go I would try to replicate that sort of visual feedback but in the physical world So because I've already got a bunch of things at home that are connected to actuators in the home automation system I set up some high-intensity blue LEDs So that if I select an object the object blows blue so I can tell what I selected Curtains, yes, I know this was one of the other things that Playing with the curtains So what you're actually looking at in that last little demo of the the curtains is the connect connected to a machine running Ubuntu, which is talking via the local the land in my house and I have an Arduino compatible board called a 2010 and an ethernet adapter on it Which is using power over ethernet and that's embedded basically outside the wall where the the curtains are and It's exposing a web services interface So when the Ubuntu machine sees an event It can then just sends a connection to the web services interface Which then fires off the appropriate physical actuators and I have it connected to the high-intensity blue LEDs above their curtains And also to big 240 volt relays that fire the the curtain motors So they control it to run up and down and it's got a feedback mechanism and timers and things like that interlocks built into it So you can't do You can't make a mistake like I did originally and try to activate both up and down simultaneously on the motors They don't like that But there are big downsides to this There is a problem here. Well, there are several problems here Firstly pretty isn't it Doesn't exactly blend in with the decor The other big problem, of course is look at what's invisible in the mirror there So you're sitting on the toilet and you're looking at a camera And it doesn't matter if you explain to people that there is no human watching this video stream There is a computer analyzing the scene and doing stuff it it still just creeps people out Yeah, that's not so good There's some other stuff in that that picture That I've also linked up we have a just-in-time hot water system So it's a hot water service that you punch in the temperature you want and it comes out of that temperature So what I've done was got an extra controller and I pack into it and once again I've got a little Arduino and some relays controlling it so that instead of having to go over and set the water temperature I can do things like have the connect Determined if I've walked into the shower So if I go to the shower it goes hmm. He's gonna have a shower I'll set it to 42 degrees and the temperature changes So all sorts of different things like that that you can do There are other sensors as well that I've been playing around with I mean the connect is really cool But there are lots of sensors that are much simpler and that don't draw about two amps and And don't freak people out looking at a camera things like IR range sensors are fantastic I've used a heap of these there are only a few dollars each and It's a reflective sensor. So what happens is that they're quite small You can attach it to something or hide it behind something and it'll give you a reading Which is proportional to the distance of whatever object is in front of it So I can do things like attach that to a surface and then you wave in front of it And it detects your hand is past in front of it and you can then trigger off actuators to do things like Close a drawer in the kitchen by waving at it Which is what you do with things like this This is a pneumatic actuator I The system in at the company I work at now Used to work in pneumatics and he has lots and lots of knowledge of this sort of stuff. So He arranged a one of these for me to experiment with so what you can see there is the it's a Bidirectional so a push-pull pneumatic actuator in the bottom and a bunch of hose There are some pressure bleed valves there and the gray block in the middle with the fittings coming off It is a two-way solenoid so I can control the piston and You may you probably didn't notice but in the previous video where we were looking at the curtains being closed in the bathroom You may have noticed that those windows are actually held closed with sticky tape right now That is because I'm intending to put electric either electric or pneumatic actuators in there because I want to be able to You know wave the curtains up and then push the windows open So thank you very much Yes, you have been warned If anybody has ideas for other things that could be done with this I would love to hear them and there is some really really cool stuff happening with hacking in the connect at the moment So there are some sites there that you can check out and see lots of cool projects and videos They're also there's a video actual stick it up now There are some videos that we took this morning just in case Stuff didn't work and of course we had stuff that didn't work So there are a few crashes and things like that and there's also a video up That actually did work So I'll put those URLs back up in a moment. Yeah, there was a gentleman here A two-way mirror Yeah Yes, I'm Julian you had a comment have you added ambient light sensors so your obnoxious blue glow isn't Extremely obnoxious when it's 3 a.m. And you hung over I'm never hung over at 3 a.m No, I haven't actually I've at the moment. I've just got them running hard on I could pulse with modulate them or something like that The problem I've got is not actually turning them down. It's getting them bright enough to be seen in daylight I've got some here, so I'll pull them out later if anybody's interested But I've actually used multiple strips of high-intensity blue LEDs, you know stuck end-to-end and Angled them so they're reflecting back against the wall so they're visible and even then in broad daylight It's not really obvious when they are activated But one thing actually it's quite handy having some form of illumination to walk around the house when it's when it's dark Oh, and this was the other video of stuff that did work. I'll show you that one as well Any more questions, or are you ready to start hurling those time? I thought that some of possible long-term consequences of things like those automatic doors is like people will forget How a door actually works? You get a point where there is a pirate down in the house and people don't know how to get out Yeah, I suppose there's always that danger Thinking forward, and this is starting to get into the realm of real conjecture here If you take what I've been talking about to you know It's logical conclusion where the entire world around you simply does what it should do I mean we're starting to talk about things like the technological singularity here imagine for example I wanted to sit down. I should be able to just sit down and there should be a chair there I shouldn't have to it should just be there and it wasn't there three seconds ago There are all sorts of things that they can be achieved with technology to to make your world react to you I'll stick those URLs back up Yeah, how well does it work in bright Sun outside? They connect. Yeah, like it's our lighter. I haven't used it outside actually. That's a good. I that's a good question I've used it in brightly lit rooms like with sunlight coming through and it seems to be okay in that Sorry better in low light generally. Yeah Well, one of the advantages of it one is that it actually does work in total darkness and So there are applications for it for security monitoring and various things like that that are quite cool Yes question up there. Have you Recently was looking at the video of the connect controlling a VM where ESX and the gentleman's just like turning the X on turning it off Changing clusters, etc. Have you played with it with controlling software? So controlling, you know the hardware through software Yeah, I have a little bit in stuff I have a little bit and I was intending to do something for this presentation And I chickened out because my presentation remote died During the Arduino mini-con and last night. I was sitting there thinking I could link this so I could change slides and And I thought no, I've got more pressing things to do. So I did other preparation instead But yeah, you can basically You can do anything you like with it Like what we've been well and you know, we're just doing then was when it detected event It sent a network connection, but it could do anything you like Yes Have you tried? Regarding your earlier demo where you stopped or pause the video by walking out of frame from the camera Have you tried combining the two because obviously with the connect you can have a better chance of face recognition because you can tell From context that the thing that's in the foreground is probably your head. Yeah In fact, I had an interesting conversation with a friend of mine when I first got the connect one of the very first things he said was If you really break it down you don't even need to do face detection anymore because what you do is Use the connect to do a 3d volumetric analysis of the room and you have a baseline and if you if someone walks into the room It doesn't matter if they're lying on the couch with their back to the TV You still know the volume has changed and so you can figure out. Okay, there's someone in the room So, yeah, I haven't tried it yet, but there are some interesting things that could be done with that Yes So I gather that Microsoft's connect stuff does things like skeleton tracking. Is there any yeah? Yeah, there are open tools for doing that now as well. I could demo that There are a whole bunch of things. I could just play around with this while people asking questions. How close are we on time? Ten minutes, okay, we have about ten minutes. Okay, John I tracking like I've always wanted to focus for those eyeball Can you see that being doable in the near future? I tracking the resolution is not that great one of the One of the One of the things I heard was that they actually degraded the resolution because I simply discovered it wasn't necessary for the games They wanted to run on Xbox and so one of the slight niggles about this is that this is a great piece of hardware But it could have been just that much bit better Maybe we'll see some more high-resilence come out. Actually another interesting thing is that Originally, it's not Microsoft technology. It's from a company called prime sense who did a whole lot of early work with with skeleton tracking and depth Analysis and various other things and so what Microsoft have done is taken the prime sense reference design and implemented it Prime sense have now been contracted to I believe it might be a sooth or someone like that and there is another Version of this coming out Developed by prime sense probably in the second quarter of this year So right now, this is the only game in town But I expect that now that everybody's realizing how cool this is that'll change pretty fast And then we might see specs increase will see a greater resolution and other things Okay Yeah, I think that might be just one of their reference designs But the one that they're working on at the moment is going to be mark It's going to be like a mass market product. You'll be able to bite and came out or wherever. I Didn't know about that. Maybe they have Last I heard was I had a contract to produce this design from the company There is actually a skeleton tracking example here somewhere as well John yeah, also I know I guess a similar line. Can you see Microsoft building it into I don't know windows phone eight nine or ten Basically, I know you to do similar gestures from your mobile device Is it like do you know what what's the processing power in the in the box the What was discovered is that I was actually just talking to bought about this before the session Their original concept was that it was going to take an awful lot of processing power within the connect itself in order to do the analysis and then Send the final results through to whatever device was connected to it so the Xbox or whatever and it was discovered that it took far less CPU power than was expected and In the so they actually had provisioned early on for an extra processor and some other stuff directly within the connect To do on-board scene analysis and they found they didn't need to so they ripped that hardware out And it only uses a few percent of the Xbox's CPU in order to to do all the stuff that you know You're seeing here and a whole lot more So it would be quite conceivable from the point of view of a CPU point of view To do this in a small device the big problem is that these things take a lot of juice They're actually using a lot of power like I wasn't kidding earlier when it pulls a couple of amps You can't run it directly off your computer. It needs a power booster in order to run it So and they get quite hot or you can feel that physically hot. There's a little fan inside it It must be cranking out quite a lot of IR in order to To throw spots of light you know six meters away and then be able to analyze it What what's with the sorry? Where is this? What's with the shadow on the false color depth? It looks like you actually have a shadow when you shouldn't yes that technically is a shadow In that it is an area that is occluded Because what we're seeing is the the projection, you know of the points from a fixed point If I stand here and there will be a shadow basically behind the area. Yeah Yeah, okay, what's to go with the four-way microphone array is that have people made use of that yet with the open source drivers? Or is I haven't heard of anybody doing anything with it? I know there's been some work investigating how it works There there's been a lot of releases of the open drivers just recently So there may be better support for it now the early version I don't think the version of the drivers that I started playing with which was right after It was released had no support for it at all Has anybody done anything with the microphones on this anyone? Clarify I don't know You're an audio geek Okay, so audio is binary blobs. No one's been able to do anything with it was the answer Hey, yes Hey, John How's it supporting multiple people trying to interface with the same device? So you're just a single person interface at the moment or no if in fact if I did players Someone else come down the front, please Okay, so I'm player one. I need player two. Oh Yeah, now it's doing. Yeah, it's now doing Sorry, I Think that's I think that's Andy messing with my head So you can see it's doing it's doing a skeleton tracking of some sort here it probably is yes It's very confused So yeah, you can have support for multiple people and skeleton tracking and various things as well Yeah Have you ever tried taking apart the connect hardware for example may be looking at hacking it or repackaging the casing to suit your bathroom date or better I haven't pulled this one apart I've seen a few teardowns online people have done put a bit of work into that There does seem to be a fair bit of space in it that isn't necessary Partly I think because they're trying to get separation for the microphones So if it was trimmed down to just the cameras, it could probably be you know a third that length Easy to hide. Yeah Okay, I think we're out of questions. So thank you very much