 All right, okay, so let's get started. My name is Kevin Hillman. This is Patrick Sixman. We're part of Bay Libra, and we're going to talk about this thing called I'm in a box. So first a little quick free advertising. We're Bay Libra. We're a small embeddagenics shop doing all sorts of stuff for hardware and software on embeddagenics products. A bunch of upstream Linux kernel work as well. And more relevant to this talk, we've done a lot of work for the kernelci.org project, which is one of the people who are hitting all the CI and testing talks. You've probably heard from them already, so I'll do a little bit of talking on that as well. So this is the lab in a box. If you wanted to stay for five minutes and go see somebody else's talk, the short version is, it's a PC with a bunch of embedded board shoved inside of it right now. That's the lab in a box. That's the TLBR. But the talk is we're going to go inside and see a little bit about the hardware and about the software, how we put it all together. So kernelci.org is the project that is focused on kernel testing. It's about three years or something now, focused primarily on upstream Linux kernel testing. And it's heavily leveraging lava. So the main idea behind kernelci.org is that the labs and boards are distributed all over the place and the kernel is tested on all sorts of platforms. So as of yesterday, I think we looked at these stats. We've done over 3 million boots of the kernel on over 250 unique boards, three different architectures, and we're averaging over 2,500 boots every day, or 2,700 boots per day now actually. The kernel is booting on lots of different hardware and we're detecting, we're catching all sorts of little things. And notice this isn't testing. This is just booting to a shell around us and it can pass. And Linux next to things like that, don't do that. We reliably have lots of different hardware, so that's kind of where kernelci was born out of this idea. We have lots of hardware that runs things, but not a lot. It's breaking often and not getting tested often. So that's kind of where the part of the case came from. So there's the kernelci.org is a website where some of this is reported. We also do some mailing lists and reports. The target audience for this is primarily the internal maintainers and internal developers who care about certain subsystems and certain platforms that break. They want to know where people go to break. So just a quick thing where we've come from. When we first started, we weren't even building the kernel. A lot of different depth configs that are in the kernel. Over the last several cycles, we've gotten to the point where we're pretty reliably building the kernel without failures. And boots are getting pretty stable at least on stable and kind of mainline. Linux next is still kind of a wild west. It's not building a lot in Linux next. We just need more eyes looking at those regressions and stuff. Anyway, that's just a little bit about kernelci. And so I wanted to talk about kernelci just because of how this kind of box fit inside the kernelci type of CI. So this is what kernelci looks like today. I don't know if those boxes are legible, but basically these are. So we have Gantt and Git repos here, centralized build servers and some testing scripts here. But this is the side where lava with the lab in a box fits in. So the idea in kernelci is this stuff is kind of centralized in the cloud, build servers and lava jobs and they generate artifacts that are made available for labs to then pull down and test whatever hardware they have. So in kernelci, most of these labs over here are running lava with maybe one lava master, maybe a few slaves and a bunch of boards down here. So kernelci has, I think we're up to 11 or maybe 12 different labs that are kind of distributed over the planet. Now some labs have two or three boards, some labs have eight boards. And so the idea for kernelci is all this stuff is distributed. So but the lab in a box is basically fits in this model. So this is one of the ways that you could use a lab in the box is to add a few boards into a PC and have them start contributing to kernelci. So that's why I wanted to show this kind of kernelci piece to see how it kind of easily would fit in there. And then after the boards are booted, lava kind of collects results and push them to a kernelci backing, which is just a big database essentially where all the results are collected. And then the kernelci.org web UI is actually just kind of querying the database and presenting the results. So the lab, all the labs report to again is centralized back and in the front and it's kind of like that. So that's the way kernelci.org is working today. And another place that's using kernelci is recently is the automotive grid Linux part of the Linux foundation is actually using, is starting to work on their CID that actually is being based on lava and kernelci. So they've been using lava for a while. Right now, Belieber is actually helping them extend kind of kernelci to do some of their testing. So their infrastructure is basically following that same type of loop with lava labs. The idea being the same kind of idea where AGL member companies can actually have labs into their companies and contribute to a centralized testing infrastructure. And then their goal is also to do a bunch more generic testing and some automotive specific testing. So now to the actual lava labs. Thank you Belieber. Okay, so let's now talk about our concept of the lab. First, let's talk about our motivations. How did we get there? Well, that was the first motivation. Moving away from this mess, fuzzy, un-maintainable shares and desk and boards everywhere while it's experimental. So we have to go through that stack initially to understand all the mechanics and get more experience with that. But at some point we have to go to the next level and provide something that is easily maintainable, scalable and that can be shared with everyone. Go back and one of those heads will make a comment. This is my lab and this is actually the organized shelf. The one that you can't see that's actually hiding around the corner is much less organized. So we're not trying to shame anybody else. This is our office in France and this is my office. This is all ours. We're not pointing fingers at anybody else. We have also the motivations. We also want to simplify the administration of a lab. Today we are building labs based in Milava, which is a very nice technology, but it's still difficult to get into. The random time is quite difficult. The installation process has been also dangerous with Lava V1, with V2 and the user containers. It was a bit more easy. We also have to deal with those device times where you have to describe the device, set it up, etc. It's not easy. So we want to simplify that. We also have this issue, so in order to control and to run the test and the devices, we need to have access to the device console and have this serial cable connected to the post. And then come into the picture of having to find which TTIO device the board is, particularly when you have multiple hands. So we want to simplify that. Because we think that ultimately the users of a lab should not be aware of the technologies inside the lab. They should just leave it and run it. All the other motivations, as I explained before already, we want to use the duplication, we want to make it scalable, and in other words, we want to accelerate the deployment of the CI labs so that we get more and more stuff tested. Okay, so based on those motivations, we have listed those requirements. Those main requirements. Equipment number one integrates everything into a single case. That includes also all the software. All the Lava software or all the commercial software. We think in the same place that it includes also all the devices that test the activities. All the connectivity, all the wiring, everything. And because we are doing, we started this work in the context of AGL, we had the requirement to integrate the reference board for AGL. Other requirements, we want to keep the cost reasonable as much as possible because this is the first experiment. But we also want, if the cost is reasonable, more and more people can leave it. We want it to be scalable, reproducible, maintainable, safe. We don't want your card to remove the duplication and everything. We want the installation to be easy, both in terms of software hardware. So we would like to provide, we want to provide pre-installed, pre-configured software components as well as an administration control plan which is I think missing today. We want that to fit in our department so that all workers can still do CI, contribute to CI. And last but not least, we want to document all that. Based on those challenges, all of those requirements, we have identified those two main challenges. Integration, integration we don't really have a lot of things to put in a single case. How to power control the devices because they all have different supply voltages. We need to have all the network connections done. USB also is a real cable. So we know that in terms of integration, this is sort of true. That's why we include all the shelves and desks. And also we want this to be more technical. It's not because we managed to give you all have that you cannot change a board and a new board and new board. Every time you need to install a new board or change one, you don't have to spend a day and break the board. So here it is, this is all a lot. It's a simple PC tower that fits everything. And if we unbox it a little bit, this is how it works. So we have selected components. From the best, there was a lot of problems with selecting the components. And I'm going to describe you one by one the main components we have used and why. So first of all the boards. So Lava is still quite a lot of processing and memory. So our first idea was to use one of the cheap development boards we have based on more processors. To actually consider the brand usage and the necessity to have a pretty large storage. Particularly for AGL artifacts that was limited. The use of this board because it's only two gigs of RAM, eventually three. We wanted to have SSD connected to store all the big artifacts. So the choice was pretty limited and we prefer to stick with a cheap PC solution. So we selected mini ITX interboard. It's still a quad core running two gigs or more CPU speed. We have 8 gigs of RAM and SSD from 20 gigabits. We used the regular ATX power supply. And what is interesting here is that this power supply is also powering all the devices. It's been quite common in the community that the ATX power units are the cheapest and powerful power units we can buy. For 50 bucks or more you have six or seven hundred watts. Super regulated, protected and everything. If you ever wanted to buy the same as a RAM equipment, you multiply by 10 the price or more. So the good thing is that it provides many power rates plus 5 volts plus 12 volts. And that is probably most of the pros we are using today. So that is great. So that's why we have a single power unit for everyone. Also you may know that we need to reset or power cycle the device on the test. So for doing that we have used our internal home main power measurement and power control solution called ACME. So it's a case for the big mud rat and it's using power control measurement products. We see a pros of the next one. So what is good here, why we have picked up this solution is because we are also ready to take power measurement. Doing CI is not only about testing features and functionalities. It's potentially so doing power and performance profile. So using that solution, integrating already our ACME solution means that we are ready for the next level of tests. And there's a USB hub because we need to get all the serial consoles. There's a network switch that is because we also connect a network of all the devices on the test. And in case later we display a little more power choice in terms of configuration. And it's a regular network switch. And in terms of devices on the test, we have selected the Raspberry Pi M3. The TID cable is red. We have the brand new low potato board. It's an AI logic S905 based on a new board which is maintaining. We have integrated the joggerboard from Qualcomm. We have integrated the Granitas Arc M3 starter kit. This is the reference board for HDR. And as well as the NXP server light IMX6. So that makes six devices on the test in a single PC. What is interesting is that the device on the test, we have selected a PC case that has the drive base. So it means that we can just, it's very easy to insert and remove the board. So here is a more enclosure for the power control and measurement unit. So you have the pickleboard pack with the IPK. And that's all what we call an IP overprobes. There is one per device on the test. So it powers switch the power. And as well it's written power control. And everything is controlled from the radar. That's it. What do you think is the scope of the device? Alright, so a little bit more about the hardware. So you saw in there the USB hub. So there's a lot of USB connections here. But the primary one is the USB serial console. So we're using USB serial cables. And in our case we're using FTDI cables. Because over experience with lots of boards. I've found that, or we've found that the lower, there's lots of cheap USB serial cables out there. And they all work at first. And some of them just stopped working. And some of them stopped working altogether. Some of them you actually have to physically unplug them and replay them. And these don't work again. And we've just found that the FTDA cables are much, much more reliable. So that's what we're using. All USB cables. Some boards like that Renaissance board actually has an onboard FTDI. So it's just a straight up USB cable. And it uses the onboard FTDI chip. But most of the others are just USB serial to a USB header on the board. Also USB devices like the Dragon board that's in the Lotto box. It actually is a fast boot device. So you need, there's another USB cable to actually do the, for Lotto to drive that board over fast boot. In the Lotto box today there's not any of the required gadgets on the internet. There's sometimes boards that don't have physical ethernet that you need to, that for you to the boot loader actually supports gadget ethernet. So there's sometimes get another USB cable to do gadget ethernet. Or some boards that don't have that have the USB, the boot loader or USB mass storage more where they can show up as a mass storage device. And that's how you put an internal or something on some of the mass storage devices. The point being there's, there's lots of room for USB, lots of need for USB cables and USB hub jam in there. Right now we have an eight port and it's full. So even with six boards we'd have to, we'd have to get a bigger hub if we wanted to add some more boards. So, and then Patrick mentioned the power a little bit. So all these, some of these devices actually power over USB, which means you have to switch USB power, not just the barrel connector power. And acne probes that are in there can switch kind of barrel jack connectors where they can switch USB, USB plugs as well. So on power like Patrick said we're using ATX supply and that's, and that's great for all, because all these boards are either 5 volt or 12 volt. In fact, all my boards too I think are all 5 volt or 12 volt. Everyone's thought there's something a little bit strange, but 5 volt or 12 volt covers most of the boards that we've seen. So we're using that and we're just switching that, we're switching DC directly with the acne probes. So there's a couple ways of doing this. I mean in the Lada box here we're doing with an acne, but there's also a lot of pretty cheap, just simple USB or GPIO controlled relay boxes. I use several of those in my lab along with acne. So there's a board farming buff yesterday where we talked a little bit about the diversity of solutions available here. But there are some pretty cheap ones that are quite reliable and then there's some pretty fancy ones too. And then there are standard kind of off the shelf wrap mount devices for switching as well. Obviously those aren't in the fit and set of PC. So we picked acne partly because we make it so it's easy to have them and also because we want to do not just power switching for the power measurement. And then networking, you can see the switch, the small switch, shunned in the box there too for all the networking. So all the devices you have to test and the Lava dispatcher are kind of on their own separate internal ads. So they're actually not, they have access to the outside world, but they can talk amongst themselves on the internal ads. So that kind of helps to keep things isolated. But the Lava box itself doesn't need internet access for, at least for working with kernel CI because kernel CI can push down jobs. But if you're not wanting to do this on a kernel CI type of project, you can just have this on your own LAN. The external interface of the Lava box will connect your LAN and all the devices you test here are kind of on an internal isolated LAN. So on the software, Lava itself is broken into two pieces. There's a Lava master and a Lava slave, also called a Lava server and a Lava dispatcher. So I think in Lava documentation, these terms are mixed together interchangeably and now probably the same. So on the slave side, slave is basically a Lava slave is where the devices actually are kind of, you can think of them as being physically connected to the slave. So the Lava slave is what manages all the USB serial connections and provides services to the boards over the network. And so, for example, when a device under tests a new sub, the boot loader might need to do a DHCP or even TFTP to kernel. It's the Lava dispatcher software that actually provides the DHCP and the TFTP and NFS and all the things. So after the board boots, they might want to wrap them up, they might want to allow them to run file system over NFS or NBD or something like that. And so the dispatcher software provides all those services for the board. And the Lava dispatcher is also what's managing all the USB serial connections so it knows directly how to connect to the serial console and send commands to the boot loader and the boot loader and so on. So it's where, if you have a bunch of USB serial cables, that's where the UDEP rules get created to basically manage all the serial ports. So they make sure they come back between the same and the same kind of thing boots. And then also where, so the Lava Labs are using Ser2Net to conducts or something like that to be the actual managing of the URs themselves. So that's also on the Lava dispatcher where those things are all kind of configured. It's also where the USB connections are managed so like the fast-moving stuff and the gadget ether and stuff, that's all managed by the Lava dispatcher, the Lava slave and as well as all the power stuff. So basically all the main stuff to make the board run is all kind of managed by the Lava dispatcher, the slave. And the Lava master's job is basically to, Lava provides a web interface to manage boards where you can kind of take boards offline and provides a web interface for all it. It also provides an XML RPC API to do all those things, but it also provides a web interface. But it's where jobs are submitted to the Lava master, so if you want to run a whole bunch of jobs on one particular board, that Lava master is where it decides how to schedule those jobs and if things are high priority, low priority, and sorts all that out. So it's kind of where the scheduling and priority management is done all on the Lava master's side. And kind of confusingly, it's also where you describe the boards, even though boards are actually physically connected to slaves, you describe them in configuration files that live on the master, and then it's the Lava master that gives those configurations to the Lava slaves. That's one of the things that I find quite confusing about Lava is all the slave configuration of the device. The slave to manage the device is where the configuration files for those actually live on the master. But those configuration files are where you define all the things that I was just talking about. So what its console is and what its power port is this way. So Lava has this concept of device type and this concept of device. So device type is like a board, so it's like a beagle bone block. So all beagle bone blocks, if you have multiple beagle bone blocks in the lab, they all share the same device type. But each instance of the beagle bone block will have a different serial cable, it will have a different power switch and so on. So the device is the specific to that instance, so where the specifics of the board, specific things are set up, and then they share a device type. And in Lava these descriptions are done in the wonderful format of Jinja 2. So if you're not a web developer and you basically have L-Templating or things like that, you may not be familiar with Jinja 2. It's not very easy on the eyes I find, but these files tend to be kind of small, so once you've seen a couple they're not too difficult, but I find it lovely personally. But that's what exists. It does though make it because it comes out of the world of kind of templating web pages and so on. It also makes it easy to extend. So a device file is actually an extension of the device type that you can override. You can set up hierarchies and override pins with common files. So like the beagle bone block device type actually inherits a bunch of stuff that's common to all U-loop devices. And then the beagle bone device actually inherits the beagle bone device type and get override things so it can set up the rest of the way. And all this is over, I forgot to mention on the slave too, but on the bottom here I have the links to the Docker container. So the kernel CI project itself is making Docker containers for the lava master and the lava slave as separate containers. So they can be combined or they can be run as separate containers. So there are certainly cases where you want to run a lava master and a lava slave on different boxes around different machines, or perhaps one master in multiple slaves. In the lava box example everything's just round on the same PC obviously because there's only one PC. But you can re-containerize it in a way that you can actually run it on separate machines. So on these github links, so github.com slash kernel CI actually has all of the various projects that are related to kernel CI including the Docker projects. I can't see very well by the way, but if you guys have questions while we're going, don't hesitate. You might have to yell because you're not seeing hands, but if there are questions, go ahead and we can do it all the way too. So in terms of bringing all the software together, because we're doing all the pieces with Docker containers, we're using Docker compose, which is just a tool for projects that actually are made up of multiple containers. You may not be able to read that either, but it's basically a compose file. This is the lava master where it's Docker config is and then a lava slave where it's Docker config is. And a few variables to go with the container and so on. And then the third container it actually is, is a squid proxy. I didn't mention that because the lava devices are actually often going and fetching the same. Each board might be doing the same kernel. So it's going to be downloading sometimes the same images. There's a squid proxy actually here to cache it. So it's not doing just saves a little bit of the mix and that works faster if you're multiple boards trying to get the same thing. But that's all contained in the container. Right, so this lava box here is kind of designed just to be an example to show how we kind of constrained the lab down into a small enough component that we could make things simple, make things an example. But obviously if you can build your own, this is not the only way to do this, this is just an example. And obviously if you can do it with a PC or everything in the case, you can obviously do it with a PC with a bunch of stuff laid around your case. And with your massive cables if you prefer. We do that, we do that pretty well too. And then you've got a big board, you might need a bigger case if you really want to give it a case, obviously. So yeah, this doesn't depend on any of the underlying hardware, obviously except the device that tests the configuration files. But because of the lava docker containers, we actually have generalized kind of how you define the board and that can actually generate the lava configuration file where you just kind of define a board as where's it's you are and where's it's power switch and everything else kind of gets handled by the lava docker containers. So have we achieved all those? Well, at least the first one, yes. We went from this mess to a nice PC case. The maintenance nightmare is kind of soft. More seriously in terms of achievements, yeah, we managed to have a fully functional serial that everything integrated into a single PC case. We have no more wiring or board laying or desk chips. So it's a case, it's a PC case with the power supply of the PC and with the network cable of the PC and that's it. So it fits well in an apartment. If you are a home worker, you might not be living in a small apartment or all this allowed to get all your office space. So that's also a shift. We believe it's a good demonstrator. It's very nice to go to conferences like this one, technical showcase, just demonstrate a PC. We put all the device on the test on drive trace so it makes it easier to maintain. We need to change the board or put a new one. It's just a trade that we take. In terms of cost, it's always an important question. We believe the cost is reasonable. The board, it's not the cost, it's really the reactable. If you exclude the devices on the test itself, it's about 400 euros, buying everything new from Amazon. We're searching for the lowest price ever. So we believe it's okay when you fully actually use this, all the trades are used. This could be reduced, obviously, if you have a spare PC, you don't need a brand new one. There's no real hardware specifics, so you could use any PC, any USB or network switch or power supply. You just need a power fully enough and with enough run. In terms of software automation, we are partially there. We've done some of the automation, but it's still under work, so we haven't completed it yet, but we are still on track. We know we can do it and everything has been internalized and it's scalable. Still, having achieved all those points, we still have found some limitations in our solution. The first one is that it's still pretty dangerous to build. It took us quite a lot of time to integrate all that into a PC case, so it makes it difficult if you think about mass production. It will try some good tinkering skills. You need to solder, you need to drill holes. It's fun if you like that, but it's not fun at all if you don't. Everything is pretty packed inside, as you can see. Obviously, there's a constraint on the size of the device in the test. It has to fit in a drive bay of 3.5 inches or 5.5 inches. The 8 is a little bit limited between two trays. Well, it's a limitation, but we don't think it's a minor limitation because of the plus 5, plus 12 volt power supplies. I think you'll remember me and Vidya, both having requested more than that because otherwise, it's cool. There's a little trick to do is you have to get... Yeah, so we are eating, so it's not shown on the picture, but if you come to the technical showcase, you'll see that. We have fans, we have selected a PC case that allows us to put front fans and back fans and top fans. We can make a storm with it, so no problem. So yeah, it's a good question, thanks. Eating could have given problems, so we put large fans, so no problem of it. I think that is a little bit tricky also is to balance the power consumption of the device in the test. If you consider the max power that it may consume, it could be 1, 2, or up to 4 or even more amps. So we cannot connect all the devices in the test in a single way. You have to balance those devices in the test across the available way. You really try to not exceed 4 amps per 88 power range. Something we believe is that using a larger PC case may not allow us to put much more devices in it because of the excessive wiring. And something that is really limiting is that today we don't have standard CI connector that will provide all the necessary connections at the device in the test. So you have to custom wire all of your devices. That's why it makes it pretty tedious. So it's not an imitation, it's a non-imitation of any lab, just that we haven't really solved it. So yeah, what could be also important? We have selected a regular power supply, actually we should have picked a more powerful one. That's actually one of our advice for you if you want to build one. It's put a high power ATX unit. It's not because it misses power, but it's because it provides more power range, as simple as that. So the balancing thing I just described before is easier than that. We should try to see how we could integrate a larger board, particularly in the context of 8G and 890 boards. They tend to use larger board. So this is not really a question here. The administration control panel, you may have noticed that they haven't shown it. I'm sure this is not there. But the idea behind that is that we don't want lab builders and users to go into Slash Dev and look for the TTYs and everything to protect everything. We want them just to keep some more bugger-pasting. So we would like to have a web page that serves as an administration control panel and that would highlight all the valuable permissions. Pick up the right device types and do the job free. Right now the administration control panel is a YAML file. Exactly. One thing we missed with that solution is that it's too complex and expensive for a work world lab. Some of you may just want to contribute to any CI solution or do CI on your own development. And so actually you just need one more. Or even you may consider some companies that are okay to do CI with their board but don't want to use a competition. So this is too much for a work world case. So this is something we need to address. In the documentation we also are showing it to you. So what's next? Considering all this, what's next? The lab in the box was our first experimentation of the lab in the box concept. We wanted to validate this concept. I think we did. It's low cost. It's affordable. It's targeting individual or limited groups. It was only a few points. Next we want to address other cases. The first one as I just explained is the one board lab. So we could name it that. We would like to address the professional grade lab in the box. We could use rights and everything would be mounted in nice rights like we have in the hallway. Because if it has to scale, we don't have to go through. And we also have to put all the wires in tiny spaces. One interesting thing to do would be to try to work with all manufacturers and try to define some kind of standard CI connector. That would be so great because plugging in a new board in a lab would take 5 minutes. Because the connection is standard. It's not impossible. But it has to be done. We haven't addressed the connectivity, the wireless connectivity, the Wi-Fi. That's what we need to tackle. And then we would like to start integrating. Now that we have integrated the lab infrastructure in the other software, we want to work with the test systems and integrate startup jobs. And work with the documentation. So I think this is it. Thanks a lot for coming if you have a question. That's the right time to ask. But you also have to see it live. And you will see it tomorrow at the technical showcase. So you can actually see the lab box running test jobs tonight. So please come.