 I'd like to thank you all for coming. I hope you've all been having a great ELC. I know this is the the last talk on the last day So I promise you this is probably not this talk is probably not going to go over time or anything like that I'll try to keep things short and sweet and we can talk about you know, whatever you want towards the end here So a little bit about who I am. My name is Scott Garmin. I am I work at Intel's open source technology center I've been working on the Octo project as a software engineer for the last I guess I started about three and a half years ago there And then more recently this minnow board project has come out the minnow board is one of the first Open hardware platforms that has an Intel atom processor on it And I'm acting basically as a technical evangelist for the minnow board So I'm going to talk a little bit about the minnow board because that's kind of what drives this this demo here And one of the reasons that I've been doing it And then in other areas I live in portland, oregon and the us and portland is known for being a very kind of bike culture Kind of city in the us which Maybe to folks in europe is not quite a big deal But it is nice to to live in a city in in the us that is fairly friendly to Bicycling and getting around transportation by bike. So I'm kind of in the maker thing I have a background that I started off in college as an electrical engineering major and then switched over to computer science Actually when I discovered linux and the open source movement Really kind of motivated me But I do have a little bit of a background in me and I still like playing around with things soldering stuff up breadboarding little prototypes and hacking robot kits, so Another thing people who know me will tell you is that I have a slight obsession that some of us in the geek community have with puns And so especially when it comes to fish puns if you can think of any good ones be sure to let minnow That works well given that most of you know, this is in at least in in the uk You know, I wouldn't necessarily try that if I was giving the talk in asia someplace, you know Because the translation doesn't always come through So this isn't going to be a big sales pitch about the minnow board But I do want to tell people about what it is make sure there is some awareness about it because again This is one of the motivations behind doing this project So the minnow board was basically designed as a platform for the yachto project as a matter of fact We wanted an open hardware and a fairly affordable platform the people that could use for yachto project development Being from intel we wanted it to be based on the intel atom processor And we worked with a third party company circuit co Which many of you may know is the manufacturer of the beagle board and the beagle bone On some of those other kind of successful community oriented boards So we saw a good fit working with them to develop this kind of board Which is kind of like our version of that We want the minnow board to be kind of a successful community board with a healthy open source community And one of my roles as technical evangelist is to try to kind of develop that community to kind of foster it And and get people excited about this So the minnow board was designed we like to say with performance flexibility openness and standards in mind And so like I said, it's it's the first open hardware atom platform And it's a pretty exciting time to be at intel now because The whole concept of open hardware and having the schematics of design files and everything available for Embedded system boards is something that you know, I I finally see as being embraced Some of you may have heard about the announcement of galileo, which is a new Kind of like an arduino on steroid. It's got a the new quark processor in it And that is also going to be an open hardware platform You'll be able to get all the design files available for that So I'm seeing a little bit of a paradigm shift with an intel and embracing these things and I think that's a great thing And it's something I'm pretty excited to be a part of Specwise the the minnow board has a gigahertz cpu. It's based on the tunnel creek processor It has a gigabyte of ram It is a single core system, but it has hyper threading. So for some multi-threaded applications It'll act as if it's a two core system And one of the things that's unique about the minnow board is it's got really strong i o performance So we've got pc i express on this board That gives us good i o Capabilities, so we've got say it on it. We've got full gigabit ethernet as opposed to You know tying a 10 100 nick to a usb line We've tried to make this board kind of straddle some different markets I think it's going to be very popular in the professional kind of like embedded product development market because of that open hardware nature of it And the fact that it's one of the more affordable Adam platforms we can get today But we also did some things with the minnow board like adding some GPio buttons and leds and things so that you could take the thing right out of the box and immediately start Start working on it hacking on it and interacting with it And it has a kind of an expansion capability so that you can add on these We call the minnow board lures, which are these add-on expansion boards to add other capabilities to it too So we like to say that the board is friendly to hobbyists It's not as cheap as some of the other options out there But it it it's friendly to hobbyists, but it also scales up to kind of more serious embedded workloads And I see a a rule for this board in areas where you need additional processing performance You need that i o throughput capability And there are many people who are getting into embedded development nowadays It's becoming more and more common It used to be kind of like a niche thing that people were doing I feel like other platforms like the raspberry pi have really enabled More people to get into this embedded community and you know People are going to want to have plenty of options. So this is one of those options that fits in to that The minnow board has various embedded i o so you know if you're used to things like spy bus i squared c GPio lines And even controller area network support is on the minnow board The board is 10.7 centimeters square The msrp is 189 us dollars If you want to buy it in europe, uh, our two main distributors in europe are farnell and tigl, which i think is based in austria And we ship it with the angstrom linux distribution, which some of you may be familiar with The nice thing about angstrom is that it's a binary distribution You can install binary packages using a package manager online and everything But it's also a distro that's the octo project compatible And that helps us to continue our message with getting people involved Potentially in using the octo project on this as well So that's my little pitch about the minnow board. You can learn more at our website minnowboard.org And uh I heard some rumors that uh, there might be some minnow boards That would be available at the last session of the day So you might want to show up at the the last session I think there are going to be a few of them given away. There's going to be a little little contest so Let's talk about the robot arm itself. So this this project is the minnow board. I called the minnow board fish picker upper and The purpose of of this project was I wanted to create a demo that Uses the minnow board and does something that's kind of fun and kind of like maker oriented type of thing And I wanted to specifically take a technology that a lot of people find intimidating Which in this case is computer vision and object detection And see how simple I could make it see if I could make it into a project that a family could do with their kids over the course of a weekend And I wanted to make it as affordable as possible I wanted the the robot arm I wanted to keep that as as cheap as possible and that ended up Driving some compromises I had to make this robot arm doesn't have any servo motors in it for example And I wanted to you know, specifically make use of open cv because it's a pretty well known computer vision library So the way this works you can see the robot arm right here And what I've done is this is the owi robot arm. You can buy it on amazon at least in the states And it is a very affordable robot arm. Oh look at that It's uh, I think it's around 40 or 50 dollars and it's a kit so you can put it together You can see how the motors and everything are set up in it Um And for another 20 or 25 dollars, you can buy a little usb control module So you can control it via basically a usb serial to kind of interface I think by default it comes with a joystick control But if you want to do computer automated control over it, you want that additional board for it too Uh, the next thing I did was I took just a standard logitech usb web camera And I mounted it to the base of the robot So this thing is kind of looking out as the robot kind of rotates around And we're going to make use of that And I wanted to basically have this robot arm Pick up a small object. I have a nice spongy foam. I call it foam fish. It's kind of in the shape of a fish I didn't bring it with me Um, and it can identify this object pick it up and then move it over to like a dinner plate or something like that So, uh, that's that's the the summary of the the minnow board fish picker upper So, uh, there were some challenges with the approach that I initially wanted to take with it Uh, at first I was being a little bit more ambitious I wanted to mount the web camera to the top of the robot arm So I could try to maybe get some three-dimensional control and have it kind of zero in on the object It's trying to pick up But there were a couple of challenges with that one of them is that the field of view on the camera wasn't that great And the other is just that this is because it's such a cheap robot arm I think the the heaviness the weight of the camera itself is is is pretty strong So you'd have to go to something more specialty like a little mini camera or something like that on there If you wanted to to give that a try and that would have added more expense and complexity to it so rather than Putting the the camera on those the kind of jaws up here. I said, okay, let's try to Constrain things a little bit more What we'll do is we'll just put the camera down here on the base and we'll just Use the computer vision to control the one one degree of freedom The kind of the rotation of the base as it zeroes in And then the reaching down and the picking up of the the fish object would have to be like a canned routine The other thing is the lack of servo motors here Can anybody tell me what the difference is between just a plain dc motor and a servo motor? Yes Right exactly. So you get you get some feedback from servo motors. You can figure out where it is and its range of motion And this robot arm does not have servo motors It just has a plain dc motor. So when i'm controlling it, I'm basically issuing commands to say Rotate this motor in this direction for and you know sleep for a second or something like that and then you know wake it up and stop it So that what that means is that the uh, I can't I can't figure out really if there's um If the the motor if the arm is is moving in one direction. It's hit its limits And it's kind of like, you know, I'm grinding through the motor So you have to be careful to to do that So I try to model that in software. I say, you know, I have so many seconds of range of motion But it's not even that simple because Then you get into the situation where if I make a number of movements in one direction and then back It's not the the steps that it's it's going through It doesn't put you back into the exact same situation if you're moving back and forth So there's some slop in those those motors too So that provided some challenges I had to deal with as well The other thing is I ended up using OpenCV's Har Wavelet based object detection and that is not an exact Science it's there are some difficulties, especially if there are different lighting conditions that the robot is in I will get either more or less accurate Object detection and sometimes I'll have to tweak some of the parameters to get that to work just right So plenty of challenges involved in this and I tried to do the best I can to to work around them One of those challenges too that I didn't mention Was the fact that this base has a bunch of desol batteries in it And I also knew that I didn't want to run this off of batteries because the motion again would be dependent on how You know how healthy the batteries are So one of the first things I did was I I modified the base I drilled some holes in it and I'm making use of some power supplies And unfortunately these power supplies are not very friendly even using a power converter With the the power outlets out here and I found that I couldn't get this working Otherwise this would have been in the yachto project booth over the last couple of days But basically so you know I have a couple of three volt power supplies that are powering this and this allows The the robot arm to work consistently given that that power source This here is a close-up of the the robot space where I've got the web camera mounted I basically just took some double-sided tape and taped it down. It's pretty securely attached to the the base And so that you know that summarizes some of the modifications I made to the actual robot awi robot kit itself So open cv how many people in here have worked with open cv before? Okay, so a number of you it sounds like most of you that were probably new to it So basically what open cv is it's the most popular computer vision library you can get that's open source It's a bsd licensed library. It actually was started by intel 1999 although. I believe it's maintained currently by a different organization And there's a lot to it. So there's a lot more than just object detection In open cv. There are something like 2,500 algorithms They can do all sorts of thing motion tracking complex image processing taking the negative of images filtering things out by color All sorts of stuff you can do with with open cv And it's a cross-platform library. So you can work with it in a number of languages like cc plus plus python java I think even lisp I'm not sure about that, but I think I read about that to you so The object detection System basically how does it work? How do we train a computer to recognize an object? So Open cv has a couple of different algorithms you can use to do this object detection But i'm using what's called har classifiers And the way it works is basically we feed this computer learning program A set of images that include the object that we're trying to detect and those are our positive images And then we have a bunch of basically false negative images or Images that don't have the object that we're trying to detect We feed it into this kind of machine learning algorithm And it basically you know gets trained it learns to identify what are the similarities between a lot of these these Pictures that that have the object we're trying to detect and make sure that it's not giving false positives in the the pictures That it knows doesn't have the object And it does this through this this har classifier system And the nice thing about har classifiers is that they're computationally efficient and what they do I'm I'm probably not going to explain it very well I do have a little youtube video that I can show you that kind of goes into a little bit more detail But in a over generalized way Har classifiers are a series of tests that you make against the a picture that you're trying to figure out if it has an object in it And the first initial test you run you do it in a series Will give you very accurate information if it knows for certain the object is not in the picture And so if you run these computationally efficient classifier levels You can immediately kind of rule out certain images if it doesn't have it But if it thinks okay, it It might have it in it Then you go on to the next kind of more computationally intensive test to determine if the object is in the image And then it keeps going down down the the chain until it gets to the more Seriously computationally intensive ones that give you a much more accurate view of whether that object exists there So to train a hard classifier you have to start with fairly large sets And it's recommended that you know a hundred or more images of the object that you're trying to detect To make sure that there's plenty of data for it to work with And in addition it's it's unfortunately It's not as simple as just taking the photo and feeding it into these this this machine learning algorithm You have to help it out a little bit. You have to mark out with a box this region of interest That shows where that object is within those pictures And there's a utility for doing this to help you out doing this called object marker And I'll give you a quick demo of that And that will I'll give you an idea of the process that you have to go through And then there are a couple of commands that you go through that that basically take The output from object marker, which is like a vector file And then you feed it into this open cv hard training program, which is actually what does the machine learning So, uh, so this is just going to be a quick demo of what I've got here. So let me bring over a terminal screen And so what I'm going to run here is That object marker program And I'm going to generate this this vector type file positive description dot text With a directory that I have in this case just a handful of images that have the Have the object in it. So what you see is This little this is the foam fish object I'm referring to So you go you draw a box around it. You try to make it as accurate as possible And you move on and what you're trying to do is you're taking photos Especially of the object in different contexts different backgrounds So that you can get as much information about this as possible So you might say take the object and put it on your spouse's head Or like in you know low contrast types of backgrounds And so on and so forth on the barbecue grill It was lots of fun walking around and taking this little fish thing and trying to take photos of it everywhere And if you are doing things that are more common such as you're doing trying to do facial recognition Which is a really popular thing to do with opencv. There are pre Created databases of information that of this hard classification that you can make use of It's you have to go through this this effort generally if you're trying to identify an object that You know that doesn't have such a wide audience of people making use of it So if we take a look now On positive Descriptions you'll see that we have basically these are vector files that describe where in that image the coordinates of the rectangle that you know We created and that can be fed into later steps along the way And then as far as the The explanation of har that there's a guy on youtube who did a much better job at it than I did So let me put this right here It's not quite big enough to and uh, let's see if I can grab the microphone and actually you can hear the audio from this Duct face detection we chose har object detection as opposed to other methods like sips due to its reliability and speed Har object detection acts as a funnel where an every region of an image is analyzed Using a set of classifiers called harm features that act as a funnel called a har cascade The classifiers at the top of the cascade are extremely fast and have extremely low false negative rates to immediately remove regions of an image That do not contain a face the hard features become more and more complex further down the cascade is an optimization Images are rejected as soon as possible if their features do not resemble a face The har object detector works by calculating an integral image of a great scale image Similar to an integral of a functioning calculus every pixel of an integral image contains the sum of the intensities of every pixel Above it and to its left any original image Allowing the average intensity of any rectangular portion of an image to be obtained by accessing only four pixel values rather than hundreds at a time A har object detector takes advantage of three different types of rectangular features Edges lines and combinations of four rectangles in order to detect a single object A har object detector defines thousands of these Rectangular features in different regions and combines these features in order to define an object and detect it efficiently Each of these features is recognized extremely quickly in an integral image Since thousands of features are verified thousands of times for every single frame of video The integral image of the frame allows the software to be much more efficient by effectively calculating the images intensities and events For face detection specifically We used an xml file containing a cascade with thousands of rectangular features that are present in the face For example, the cascade contained definitions for the cheeks being brighter than the eye socket The nose being brighter than the eye socket and the forehead being one of the brightest portions of the face overall Thousands of these comparisons are conducted in real time to ensure that a single region of a single frame of video contains the face So I don't know if the audio came through too well on that or not But I I felt like he gave a much better explanation than I could have at this point So I'd like to thank Varun Ramesh for that video And that gives you a little education on the har object detection Okay, well actually why don't I give you a demo? So I like I said, I didn't have the the robot i'm working here and I apologize for that I do have a video I can show you though of The robot actually working So this is in a hotel room earlier this year So I've got this kind of blue placement placemat back a background down Which is kind of like the back of an aquarium, you know aquarium wallpaper So that's kind of water and that's the fish in the water And I've got the minnow board set up there And we're running the it's kind of hard to see the minnow board fish picker upper code Which is on github. I'll share a link with that at the end And you see the robot arm is looking for it in that red box around the fish object is open cv's object detection Saying okay, we've detected this object And it'll zero in on it Using a smaller movements and then reach down and and try to pick it up And then it goes and drops it off in the plate there Or the bowl So again, it's in in one sense. It's a really simple thing But you know like a kid learning how to do this I mean, you know kids love robots to begin with to be able to control them and give them like autonomous You know capabilities such as this is pretty exciting So that was the goal of the project is to try to enable This kind of excitement that you can get around autonomously controlled robots that can see So there you go. All right. Let's see if I can start the slide show at the right slide Yes, okay So open cv obviously isn't the only way that you can do computer vision And I just wanted to mention briefly that there there are other methods So I talked a little bit about the fact that the Using the open cv with the web camera can be a little bit sensitive to lighting conditions One of the alternatives you could use is something like a microsoft connect Which actually uses infrared to do its its vision control and the infrared is The the amount of light in the room is irrelevant to that And one example of a robot that you can make use of is something called the turtle bot Which is made of pretty commodity parts as well So it actually uses a kind of a Roomba as its base They actually sell Roomba's that don't actually vacuum but are just can be used as kind of robot Experimenter kits And then it's got the microsoft connect up there as As a it's computer vision and so it's got a whole platform It uses I think the robot operating system that you can make use of To control this and you could put either like a net book or you know a minnow board or some little embedded board in there to To actually control it So that's an alternative thing here So one of the other purposes of this project was to use it for advocacy of the yachto project And so while you can use the angstrom distribution and you can install open cb and the dependencies that this thing requires I wanted to briefly go over to how you would enable this kind of application using the yachto project So basically you needed to You know grab the yachto project release and it's Metadata there and you'll find that you don't have certain programs that you need So for example open cv and open cv has a number of dependencies such as libe v If you're using the web camera you want video for linux utilities and so on But if you look around when you're in this situation With with the oe core layer, you don't have the software you need You don't necessarily have to jump in and start writing your own recipes yourself So it turns out that the meta open embedded repository had just about all the recipes that we needed We were able to grab the recipes from there and create our own layer based on that. So The layer name I gave this was meta robot open cv demo And I created an image recipe that basically just booted up to x and gave me a terminal window And that included all the required open cv stuff And there's a website you can go to that. I believe it's layers dot open embedded dot org Which is a searchable index of all the different layers you can get for open embedded Yachter project layers basically and so you can enter in the name of a recipe that you're looking for And it will return, you know where you can find that So that's a really handy tool to use when you're trying to grab some of these other recipes So what is a what is a yachter project? What is an oe layer? Basically, it has a directory structure here where you need a configuration file This is your layer dot comp file and that will take tell the bit big tool where you can find the remaining recipes in your layer That's kind of like a boilerplate that you would make use of And then you have a recipes sub directory and then you have the different You know applications that you need into it and you would put the the recipe for swig or recipes for swig within the swig directory and enable that in your Build bblayers dot comp file you would you point to both your oe core Base layer and then this layer and then the the bsp layer for the machine that you're building for etc And to show you what an image recipe looks like in open embedded We have this robot open cv demo image dot bb And it's pretty basic. We've got a description. We have something called image features and so uh, you know, what's going to bring in most of this is x11 base will bring in everything all the depend all the sub dependencies you need to get to a working xorg base system And I just added a couple other things, you know splash screen the drop air ssh server and so on And then we append Here an image install we can add the additional specific packages that we want to have for this so open cv apps There was a calibration utility that I use for manually controlling the robot arm that uses wx python So I wanted to include that as well. Um, I think I threw mesa demos in there just to do some benchmarking And and so that's an example of you know, how quickly you can write an image recipe for the octo project So I had been working on this I think I got this first working for linux fest northwest In april of this year and I've been kind of developing it and adding more to it as time went on And there are some other things I'd still love to do with this The code for this is basically a big state machine But i'm not using any python state machine libraries So it would probably make the code a little bit more maintainable if I migrated over to a python state machine library One of the ways you could improve the reliability of the system would also be to increase the number of samples That are used for the open cv training One thing I've been wanting to do too So, you know, the middle board has a number of gpio headers that you can connect to And I thought it'd be really cool to have it Detect a fish by color so that you could have like, you know, some push button switches attached to it And you could pick whether you want it to pick up the green fish or the blue fish or the yellow fish And then have the you know, the the program be able to do that And then and then like I said the control of it via some of the gpio buttons Some resources I've got here so information about the minnow board the octo project This is a link to the the actual specific model of that robot arm you can get Some stuff about open cv a link to that video that we watched And most importantly if you want to actually take a look at the code that's running this as well as the octo project layers that you make use of You go to github.com slash minnow board. We have our own organization. You'll find some repositories for that stuff too So really, I mean that was what Half an hour worth of worth of speaking here. I'm happy to answer any questions people may have about this kind of thing Yes True yeah, that would be something else you could do you could add some more object detection Capabilities with with recognizing the bowl and then you could have the bowl in arbitrary locations. Yep Yeah, it's it because it's just two-dimensional and again because I'm only using that one Access of rotation what I did was on you couldn't see it on the blue placemat But I just had a couple of dots set out. There were a six inch radius from the base So as long as it's along that six inch radius, it should be able to to pick it up So it depends on how fast you want the frame rate to be It turns out one of the things that's the most computationally intensive with this is actually just getting the frames from the From the web camera itself. I think I was getting I think I settled on About five frames per second. I wanted to get and by contrast So I tried running this on a raspberry pi and it was really choppy. It was actually barely usable at that point So, you know, you'll see some performance differences between minnow and some of the other platforms And that we're making use of definitely the cpu, but the actual open cv detection wasn't especially High it might I might have only been using 40 or 50 percent of the cpu for that aspect of it And if you do want to replicate this on another board In the code, I include some comments that identify the parts where you would you can slow down the frame rate to To make it more usable for, you know, lower performing options Back there first Yes Yeah, that would actually probably be a more reliable way and a simpler way of doing this And the reason I didn't do that it was again because I wanted to get into this whole concept of teaching people to use something complicated like object detection And Because of the object detection too you can even bring in the whole idea of like you're using Artificial intelligence and machine learning algorithm to get this to work So that was part of the story behind this but that would be another way that you could go about doing this certainly Yes, there's a question here So it's it's pretty simple like the script that's really controlling this is I think around only 400 lines of python itself So what you do is you use open cv to open up the The web camera device that you're making use of you're pulling frames from it and then you run code that Analyzes those frames using the har detection method Which you're pointing at the database that you generated through those steps that I talked about where you You define the region of interest and then you you run that open cv har training command and what that outputs is Kind of a directory structure that acts as a database of sorts so you need to to Be pulling in images and then pointing the python code to that that database So Right, and then what it does is when it identifies the object You can get a function that returns and what it returns is the coordinates of the box that it's drawing around it And then I have to figure out what is the center within that box Right. Yep, exactly Yep That that would probably Might be possible in some ways There's some things you could do that would make it easier if you had a camera that was looking at the robot arm There are these Hatch patterns There's a name form and I can free out what it is You can use that are actually really easy to identify with open cv And you can put those on the axes that are rotating and so you could do things like You know figure out what what the degree of rotation is Making use of open cv again, but it would have to be kind of like a third party view of it Not necessarily something that um, you know, it's got a first Person view out of it. I guess Maybe yeah, sure. Yeah Yeah, yes Oh, I think it's uh, yeah, I think it's just a kind of a motion jpeg where I'm pulling the frames out of Yeah, yeah, something that might have h264 Encoding built into it might be more efficient too. Yeah Anything else? All right. Well, like I said last session of the day You know, definitely you show up to the the closing session and thank you all for coming Appreciate it