 So I guess we shall make a start. Thank you everybody for coming to this which is the first hour of the student innovation section of the research and student innovation mini conf here at LCA. This sort of section of the mini conf is something that various people have been trying to get accepted into LCA for quite a few years now and we're very very glad that it has been. This is basically being made available to provide an opportunity for students who are doing interesting things with free and open source software to share their experiences with the rest of the delegates to LCA and I hope this is a beneficial experience for both our presenters and you the audience. Our first presenter today is David Claridge who's a student at UNSW and is the current leader of the RoboCup team and he's going to be presenting RoboCup world-class soccer players running Linux. Please make him welcome. Thanks Chris. So today I want to talk to you a little bit about what RoboCup is and the various leagues of the competition. Introduce the hardware that we've been working with the Aldebaran now robot and what I want to spend most of our time together on is talking about the software we've developed using a number of open-source tools including Linux to be able to go into the RoboCup competition and play soccer and doing fairly reasonably well at that task. Also at the end I'll talk a little bit about the open-source nature of the competition and how we've managed to contribute a few things back to the open-source community through this research. So RoboCup this is the official definition it's an international robotics competition that's been around since 97 and the aim is to develop autonomous soccer robots with the intention of promoting research and education in robotics and AI. Now there are several leagues the most recently introduced of these is RoboCup at home where you see robots like this one who are designed to help people around the house get a beer out of the fridge, do your cleaning that sort of thing. There's also the RoboCup rescue league where there are simulated disaster sites such as a collapsed building or otherwise situations would be too dangerous for human rescuers to go into without an idea of the lay of the land and where the victims are. So these robots go in there and perform triage basically on behalf of the human rescuers at creating maps and that sort of thing and then we have the soccer leagues which is where it all started. For new entrants to the competition the easiest way in is through the simulation league since you don't actually have to buy a physical robot and they play full 11 versus 11 soccer games which are simulated in this RCSS sim software and there's the 2D and a 3D simulator. The 2D simulation focusing more on strategy and player positioning, the 3D focusing more on actually developing gate engines for humanoid robots and the other aspects of robotics that can be simulated without the need for the physical hardware. In terms of the physical robot competitions we have various leagues playing soccer. The small size league has shared sensor information so there's an overhead camera that all the robots have access to and you can see the color coded dots on top of the robots allows them to very quickly figure out where their players are on the field and where the opponents are and also the ball and processing is all done off-board on computers on the side of the field and just basic IO is done on the robots. Then in the middle size soccer league you have these wheeled robots that are slightly larger and have their own sensors. Again they're wheeled robots so the focus is more on strategy kicking these sort of things not having to worry about the difficulties of humanoid robotic gate engines. Then moving on to the humanoid soccer leagues there are three of those starting with the kid size. In this league universities or companies that enter build their own robots that they enter the competition with certain height and weight restrictions. There's a teen size league similar thing but with larger robots and finally the standard platform league. The idea of the standard platform league is to provide a way that people who are more interested in software so AI or that side of robotics rather than actually building the robots themselves to answer the competition. The standard platform league started off with the Sony Abo which has now been deprecated and is no longer available and instead we're using the Aldevarin now humanoid robot the one on the right there and that's been the case since 2008. So the now robot is produced by a French company called Aldevarin based in Paris and here the basic specifications of the machine. It has a 500 megahertz geode processor, 256 megs of onboard RAM, a 2 gigabyte USB stick which is what it boots off and where you store your software, a couple of microphones, webcams, speakers, sonars, two gyroscopes, a three axis accelerometer and eight pressure sensors in the four in each of the feet. Now the actual layout of where this hardware is in the robot itself is a little unusual. This is a diagram of where the cameras are located in the forehead and in the chin, not in the eye sockets as you would expect. That's where the infrared sensors go obviously and as you can also see from this diagram there's a blind spot in the field of view of the two cameras and to make it even more fun you can only use one camera at a time. They're on the same I2C bus and while I don't understand a lot of the hardware details for some reason that precludes them both being used simultaneously. The microphones in the ears which is probably the only sensible part of the design of this robot. You can also get an academic addition that has two more microphones, one in the front of the head and one in the back of the head but the robot cup addition doesn't include that. The speakers also go in the ears for whatever reason. Then in the chest plate we have some ultrasonic sensors to detect obstacles you're about to run into and as I mentioned earlier the foot sensors these are probably the most useful for the purposes of developing a walk engine. They provide feedback about where the center of mass, center of pressure is of the robot while it's walking around. The robot, the robot cup version of the robot I should say has 19 degrees of freedom. The hands aren't actually able to be actuated in the robot cup addition but an academic version which is considerably more expensive has hand actuators if you wanted to use those. And probably the most unusual feature of the robot's kinematic chain is the hip your pitch joint. The left and right hips are connected so if these are the two sort of the robot's legs are coming out this way as you turn one in the other one goes in as well so they're connected in this really bizarre way that makes developing walks on it pretty interesting to say the least. We've had a few issues with the hardware for a start. We're in Sydney. The manufacturers are in Paris so we're looking at at least one month's turnaround for repairs. Mostly the company that manufactures the robots blames French customs for taking most of the time which wouldn't surprise me but compared to for instance the German teams and other European teams in the competition who have sort of less than one week turnaround that can be quite an issue. And the robots really only last about two weeks in our lab. That's because we work pretty hard on them. We're intensely developing walk engines and inevitably when you're doing experimental work it's going to damage the robot. Ideally we would use a simulator for all of this more risky development but the simulators really aren't up to the level where they're accurately modeling what the robot hardware does so if you want to be competitive you really have to try things out on the real hardware. To get around this problem of repairs taking a long time we have nine of the robots and however shipping is still pretty expensive so we have to send them off in batches and often we find ourselves in a situation where we have only one or two working robots. The competition at the moment is of three versus three in 2010 and in this year's competition it's going up to four and four versus four in the games. So it's going to be even more of a problem if the hardware doesn't improve. Some standard software comes pre-installed on the robot so if you buy one of these you get Linux with the RT pre-empt patches built and installed on it. Busy box including so you have a shell some basic bin utilities various libraries and other applications that have been built with open embedded. Open embedded is a suite of tools for building things for other platforms basically for embedded platforms and so we've put together a tool chain across a cross compiler tool chain that allows us to build additional libraries that haven't been made available by the manufacturer. So a 32-bit Linux tool chain is available already but we've built a 64-bit one as well for those users who are using 64-bit Linux and finally there's some proprietary software on the robot called Naoki and it is the only way to actually talk to the sensors and motors. So there's an arm, nine chip in the chest that the Naoki software talks to and that's the only approved way sort of within the scope of the warranty of the robot that you're allowed to talk to to the hardware. Now Naoki is designed to be an extensible platform where you build shared object libraries that can talk to the other Naoki threads that do motion, read sensors and so forth but we found that their inter-process communication and other other aspects of the hardware just difficult to work with or too slow so we've sort of built our own system on top of this. So I guess that's what I want to focus on here, our Runswift robotic software and how it talks to Naoki, how it talks to Linux and uses various tools and libraries to actually work. So there are four key components to our software. LibAgent is the Naoki extension module that basically allows us to interact with the robot physically. The LibAgent uses a shared memory block to share information with our other main application which is called Runswift. So this is the executable that actually contains all the game logic, all the computer vision, the walk engine and so forth. Now Sync is just a simple script, wrapping our sync and a few other tools for uploading code to the robots, managing what software is installed on them and so forth. Despite its simplicity it's actually been incredibly useful. I'll talk about that a bit later and often now it's a QT application that allows us to do remote debugging to make record files of what's going on on the robot and see in real time what the robot's thinking, where it thinks it is, where it thinks the opponents are and so on. So LibAgent, the extension module for Naoki communicates with the Runswift motion thread by using some callback functions that Naoki allows you to register. So the DCM or Device Communications Manager is the thread of Naoki that reads sensor values, writes actuation values back on. And you can register a callback to run before and after the DCM. In the Runswift application we have this motion thread that and the two of them use a semaphore to synchronize when each one should be doing different things. So basically the way it operates, the Runswift motion thread tries to acquire the semaphore. It then reads the information in this agent data structure in the middle, which contains hopefully sensor values, where all the joints are at, where the center of pressure is, and so forth. It then goes through and calculates what it wants to do with the actuators, so the next phase of the walk. Also LEDs, sonars, and so forth are all controlled through this, and it writes that information back to agent data. Then in the pre-DCM cycle callback the values in this agent data shared memory block are simply loaded and sent off to the DCM, to Naoki. And then after the DCM actually updates and moves the robot and reads all the sensors, those values are written back to agent data. It updates the semaphore so that the motion thread can continue on the next cycle. Now this happens 100 times a second, so the sort of sense plan act cycle is occurring at 100 hertz, and if for some reason our Runswift executable fails, unlike modules that are tied into Naoki directly, if it fails it doesn't actually crash Naoki, which has the nice side effect of not making the robot fall over and break into a thousand pieces. The way we accomplish this is when we go to post the semaphore in lib agent we check if the semaphore value is zero, and if it isn't we increment a miss cycle count, and if that happens 50 times, so for half a second, we assume that the Runswift executable is crashed, or someone's hit control C, or something like that, and we send the robot into this safely sit routine where it crashes down and waits for you to come and look after it. So my phone's just gone to sleep, there we go. So the Runswift executable itself is where all the interesting logic happens. So this is just sort of an overall system diagram, so we have Linux providing things like the networking stack for reading information from our teammates and from the referee, video for Linux, which is used to get frames from the camera, and then the top sort of line going across here is where most of, where one set of processing gets done, calculating the forward kinematic chain, that's where you read all the angles of the various joints, and calculate a transformation matrix from the feet of the robot through to the camera, so that given any given pixel you can identify where that is physically around you on the ground in terms of the distance and heading from the robot. Once that matrix has been computed, we go through and run a bunch of vision algorithms on the on the image that we've received from video for Linux, and project those identified objects, whether they be the ball, goals, other robots and so on, into the ground plane, and pass that information into localization or the world model. So this is where all of the information coming from vision and other sensors is filtered, to try and we use Kalman filters and particle filters and these other sort of Markov models to come up with an estimate for where the robot is, where the ball is, and where our teammates are, and so finally with all that information our behavior, our high-level behaviors that actually plan what to do and give the robot instructions, which is implemented in Python, I'll get more on that later, reads the current state of the world model and uses that as its input to decide what to do. The motion thread I've already talked about, so that's that runs in parallel with the perception thread and plans the walk and accepts action commands from behavior. So there are actually six p-threads running at any given point in time in the run-swift executable. The most important of these being the first two, motion and perception. Motion uses, is making use of the RT preempt extensions, the Linux kernel, to try and basically, if it wants to run it should absolutely get the first priority because if vision algorithms which are the most expensive part of our routines are busy hogging the CPU and we don't get that update done to the joint angles within the one hundredth of a second that they need to be done in, the robot will fall over or and it will break into a thousand pieces which is very undesirable when you're trying to play soccer. The perception thread is where most of the heavy processing goes on, the things I just talked about, kinematics vision, localization and behavior and then there are four other completely IO-bound threads that sit there listening to the network from information from teammates, information from the referee, the receiving ones and so they just block and receive information about five times a second each and also there are two transmitters, one which broadcasts five times a second the current state of the robot to all of its teammates and the off now transmitter which communicates with that QT application I mentioned earlier for debugging. So it's sort of a variable rate thing you can plug in the ethernet jack and stream more images at sort of full ethernet speeds or you can be doing it over the wireless and just read some basic information like the XY coordinate of the robot to kind of see what it's thinking as it's running around. So in a little more detail the perception thread where a lot of the more CPU-bound processes are going on. The kinematics module I already mentioned how we compute the pose matrix that allows us to project points in the camera plane into the ground plane. Video for Linux is used to grab a frame and pass it into the detection algorithms. We support various methods of reading from the camera using video for Linux so you've got you can use a simple read call you can use MmapDio to get a pointer to a kernel buffer where the images are coming in or you can use user pointers which is sort of the other way around you give the kernel a pointer to a buffer that you own and ask it to store the images in there and we've noticed a pretty significant performance difference between those. So the moment we're using Mmap which seems to be the fastest way to actually load information from the camera into a buffer however it has the downside that unlike user pointers it can't be cached for some reason. We're working on that. So the world model as I mentioned this is what I'm actually going to be spending most of my time on for the next six months. For my honest thesis is developing a series of unscented Kalman filters that model the position of robots on your own team the ball and anything else that we may wish to model incorporating all of these noisy sensor values we're getting including the fact that vision might falsely detect a ball where there is none or it might you know the kinematic chain might be inaccurate because of inaccuracies in the actual manufacturing of the robot so the angle between two joints may not be really what the specs say they should be the odometry of the robot as read by the accelerometers and gyroscopes may not be precisely what it truly is so you've got all this noisy information and you need to combine it into a coherent model that your behavior can use. So that's what that's all about and then behavior. So I was mentioning earlier before the talk that at the competition we often completely rewrite our strategy from scratch based on our observations of what the other teams are doing. Now most of the system is written in C++ and completely rewriting our behavior engine from scratch in C++ in the few hours before a game is unlikely to be a very successful endeavor. So what we've done is we've used the Python C API to embed a Python interpreter inside the C++ application which in turn allows us to even at runtime load new behaviors and modify things on the fly so we go back or forward whichever way I didn't mean to go just then. So we're using a feature of the Linux kernel called iNotify which hooks onto the file system and notifies you of changes to a particular directory you're interested in so we have a certain directory we've added to the Python path that is modified that is monitored and whenever a change occurs we unload the Python interpreter and load it up again reloading all our behavior classes and because this is happening in the perception thread motion is completely uninterrupted so whatever action command was last written to motion is what it will keep on doing so it'll keep walking or standing still or kicking and you shouldn't even have to get out of your chair so you should be able to sit there coding and decide actually the goalie should dive right in this situation not left and just change the file and watch it start doing the right thing. Compare this to the workflow we used to have when we were just using pure C++ where you would have to get out of your chair go to the robot sit it down carefully so that when you kill the program it doesn't hurt itself go back to your chair compile the code upload the code go back to the robot started again you're looking at at least a two minute turnaround if not more just to change a single parameter or variable somewhere in the code so using using dynamic language like python has really aided our ability to do rapid behavior development including at the competition of course we're not allowed to modify the code during games that would be a little bit unfair however even when we're just working in our lab in the six months leading up to the competition it helps to be able to do this sort of rapid behavior development. Lately I've been working on automatically generating the interface between python and C++ using sython or swig and these other tools but it hasn't quite got there yet so at the moment we're manually crafting python to C API conversion functions which is a bit of a pain but it's still better than doing everything in C++ so and so the behavior is what really takes care of complying with the rules making sure you go off when the referee tells you you've been sent off for pushing the strategy involved and cooperation between teammates so now sync is the third tool that we've developed the simple shell script it's about two or three hundred lines and it allows you to figure out where your robot is on the network using a vahi it in our lab it detects whether you're on the run swift wireless network or if you're on the school's wired network and routes things appropriately it allows you to configure a new robot which involves installing ssh keys setting up the app repository and so on and the most commonly used feature is every time you change something on the robot you run now sync and it uploads any changed files both to your behavior to your c++ executable to live agent is required so we found just having access to our sync and shell scripting on our robots has made development a whole lot faster the last tool that i mentioned is off now so this is a qt application that links against the same library that we run on the robot so you can simulate a lot of the code that you would run on the robot on your own machine in particular vision so if you're developing vision algorithms you don't want to say oh it didn't see the ball in that frame let's try and make the robot walk that way again tilting its head that way in these lighting conditions and reproduce this bug instead we use off now to record a dump file of all of the frames you could see while walking around and then you can run it over and over again and do reproducible sort of testing there's also a tab in off now that lets you see the robot's live state so where the circle over there is where the robot thinks it is in the top right corner is it's what it's currently seeing and then there's also a whole bunch of internal state variables that you can monitor during execution off now also lets us calibrate the colors so this is just a yuv point cloud rather than rgb we're using the yuv color space and this lets you see how we've classified everything in that area of the yuv space as being ball orange that area is being gold blue and so on and you can sort of visualize how they mapped out and find you in the color calibration so that's the software the competition tries to encourage open source collaboration by getting teams especially teams who do well to publish their source code on the web for other teams to use so every year the starting point is the winners of the previous year our most recent software from singapore where we came second is up on github at github slash unisw computing slash runswift the team that won be human from germany also published their code so if you're going to start working on these robots you might prefer to look at their code but ours isn't too bad and we're continuing with our with our code base from last year rather than starting from theirs and finally we maintain a public app repository for the now that and we also take requests if people want something to be packaged for the now that isn't available from al-debaran you can add a deb line to your sources dot list on the now pointing to our app repository and you get access to all of this cool stuff so that is the end of the presentation um any questions jeremy so um rubber cups are one of the research things going on at unisw that really focuses on pushing out a source as well as papers is there anything that you or that is going on generally at unisw to try encourage all of our um cs papers to come with source because i know that's a big problem in research that people write a paper about an algorithm but right well so as an undergraduate student i guess i don't have a lot of sway in terms of the policies the university has to with whether or not you should have to release source um in the case of robo cup it was actually myself and other teammates going to our leader saying hey we want to do a code release and be cool like the german teams and they said okay fine you can release your code so it's not like there's a really big push to do it but uh for people who want to the option seems to be available yeah any other questions um you mentioned the long turnaround on parts yes i i guess yes there is no option to harden harden up one of the robots so so as as like a crash dummy version we have tried a few things like this for instance last year we built a harness that um hangs from the ceiling that the robot is sort of held onto loosely by as we're doing walk development so when it inevitably stumbles and falls it doesn't hit the ground at least not with full force but it was difficult to do that in a way that didn't obstruct the actual motion of the robot um oh it's more thinking of like actually a replica of the robot that's got soft padding on it or something i was more thinking of actually patting padding the robot or actually hardening up the common weak points i guess the only downside to that would be the possible effects that could have on causing slightly different behavior than what you get on the actual robot so if you sort of optimize your walk engine for this robot while it's got all this padding on it and then for the competition you take it all off and the weight distribution is different uh your walk may not work correctly so there are probably there are certainly situations where that would be useful um we often we just leave foam padding lying all around over the field so when it and people just walk around moving it around the robot so when it falls over it falls on that instead the other thing is we can't actually open the robots up it's against our warranty agreement so we can't internally harden it up only externally we have access to the head where the usb stick and wireless card are but uh the body is sort of a closed black box okay that that leaves you in a very hard position when yes so actually uh with the most recent set of repairs and upgrades all the everyone are coming out to australia in a couple of weeks um to upgrade robots or they haven't given us an exact date yet but i think uts gruffith and newcastle are also going to be competing this year and so they want to come out here and because it'll be cheaper to do all our upgrades onsite but it's sort of a one-off thing most of the time uh yeah we have to send it back to paris there's also going to be in australia an australian open uh i think in may that'll be hosted at unsw and al dever and may come out to run a repair workshop at the australian open yeah um can our next presenters come down to set up while is robercup all universities in principle no um like companies are welcome to enter in there are other leagues such as the humanoid leagues where other companies have participated in the past um also there's robercup junior which i didn't mention but it's for school students specifically and uh and they work with sort of lego-based robots um so there's suddenly a wide variety of groups getting involved in robercup but the standard platform league at the moment is 28 universities yeah okay well join me in thanking david for that talk on robercup