 First, I'd like to introduce you to my colleague Manuel Traut, and me, I'm Anna-Maria Gleix, and we both are working for Lina Trunix, and in the next 40 minutes, we would like to explain the background of continuous integration with Jenkins, Lippert, and real hardware. So, let you know what will happen in the next 40 minutes. First, I will give a short overview what's the background, why we did the development we did. We shortly explained a challenge to control real hardware. We explained our solution, Lippert connection, and Jenkins connection, and we'll point out some future extensions. So, first, the project was, or why we made this development was because we are working for the RGL project, and there we have continuous integration, and we use Jenkins and Lippert, and the RGL project for everybody who do not know this, you should know it. It's the real-time Linux project, and it's a Linux foundation collaborative project, which was first announced in October 2015. The members fund Linux kernel-preempt-at-a-patch set mainlining, maintenance, and documentation, therefore, we create a new Viki, and another important part is that we, with this project, we try to establish a community, which is required for long-term maintenance. When you make a development, it's required that you also test it, and so we have to develop and make a continuous integration system for real-time preempt-at-a-patch set. So, our goal is that we can test Linux kernel with real-time capabilities on different machines. Different machines could be real hardware or virtual machines, and so we need to be able to power-control the machines, and if we boot a kernel that crashes, we need to get the boot lock so that we can examine what happened during boot, and why did we make something wrong with the development. So, we need to schedule tests on different machines, and so first, we compile the kernel. Does it work? Yeah. We compile the kernel, deploy the kernel on the device on the test, boot the kernel on the device on the test, and last but not least, we want to run different tests like cyclic tests on the machines. So, you need infrastructure so that you can make those continuous integration, and we decided to use Jenkins because it's the defector standard for continuous integration, and with Jenkins, you are able to schedule tests on different machines. And for Jenkins, there are a lot of plugins available, and there are also two plugins so that you are available to control machines. The one possibility is that you control the machines via Docker, or the other possibility is that you control the machines via Lipford. Docker is very restricted, so you are only able to use Docker containers, and Lipford there you can use hardware, virtual machines, and based on different hypervises. So, we decided us because Docker is really restricted to use Lipford. Lipford is a standardised interface to enumerate and control virtual machines, and you can use different hypervises or talk or communicate to different hypervises via drivers. And the main advantage of Lipford is that you have a graphical front and light like the command light interface like VIRSH. So, here is how Lipford works. So, you use the VIRSH interface, command line interface, and you communicate via TCP or SSH with the Lipford daemon. The Lipford daemon then use those drivers, for example, QEMU driver, so that you can communicate with the virtual machine, or another possibility would be that you use the Alex C driver that you communicate with containers. But Lipford is not, or there is no driver that you can communicate with real hardware and control them. So, this was basically the challenge that we had. So, we wanted to use Jenkins and Lipford, and we wanted to control real hardware. So, now I will explain you the ideas we had and the basic architecture where we fit in our own software that we developed. So, the reasons why we wanted to control real hardware was we needed to be able to restart a node in our test setup. For example, if the kernel crashes, you don't have software possibilities to reboot the machine and try another kernel or something like that. So, you need some ability to power control them. And another requirement we had was we wanted tests to behave in Jenkins like on other virtual machines. So, we wanted to change as little as possible other software components and get a lot out of them. And at least it's also interesting that we can put off nodes that are currently not used because we all want to save energy on this planet. And this is also possible with this tooling. So, our basic setup is we have existing racks at Linux where our test hardware is mounted in. So, we have 90-inch industrial racks typically with eight slots in it. And David can put in into such dashes as you can see on the left-hand side are slots. We can mount some embedded hardware. And we have some standardized interfaces on the back. So, one for plugging in the power on the very left. And then we have two connectors here, one for the serial port of the hardware and one for the ethernet port of the hardware. And inside this rack we have also mounted a network switch and power control switch that is able to power and unpower those slots via ethernet or via HTTP or SNMP. And we have an eight-port serial device server in there that just grabs the output of the serial console and puts it into some ethernet stream. So, our vision was we have Jenkins, we have the lib word slave plugin and Jenkins, and we have our racks and we wanted to connect them somehow. And one part of our solution is R4D, which means remote for devices under tests. And now I want you to explain the basic architecture of R4D, the features, and the usage. And R4D is just for controlling the racks. So, you can use R4D as a standalone tool. And the next part of the talk will explain you how to combine it with lib word and so on. So, you have this R4D daemon we decided to have a centralized daemon to communicate with that we can implement something like logging and so on on our complete farm. And this centralized daemon communicates with a database. The database has a model that more or less is the same than our farm in hardware. So, we have the racks. We have each rack has some slots where you can put hardware in. Then we have entities for the power control and serial devices so that we can store our persistent information of our lab inside the database. And then we have a soap interface on the other hand of our R4D daemon. We use soap because trust you are familiar with soap, but it's quite easy to exchange this interface or add another interface for RESTful AP, for example, or something you like more. And we have this web interface we are able to communicate with the daemon and redrive the information out of the database, what hardware is available, where is it located, and which hardware uses which power control switch or serial device server. And we are even able to control via the soap interface such as the power control management switches. So, we can power off and power on a single board or we can redrive the real status. So, each time we ask the soap interface, what's the status of this embedded board? We get back it's powered or it's unpowered and we ask the switch. Each time we don't cache this value in the database because you have also the possibility to change the power on the switch directly with some buttons and so we get always the real state. We also added a plug-in interface to support multiple different vendors of power control and serial control switches to be easily extendable here. For the database back end, we used SQL Alchemy. The whole R4D daemon is written in Python. So, this was an easy decision and so we can also support multiple database back ends there. But we just use SQLite at Linux at the moment because there is not much traffic on this database SQLite or scales well for our use case. Then we developed this command line interface tool to interact. We are soap with our daemon. There we have the possibility to configure our lab so we can add new racks, add power controls to this racks. I will show you how to use that in some seconds. And we are also able to move a board out of one rack and put it into another one, for example. And of course, we are able to switch boards on and off with this tooling. So, you can just add a board there, use the tooling to switch it on and off and look if the correct board is powered and unpowered as expected. And we can dump the whole database layout or parts of them to see what's in our test lab. So, the basic usage of this tooling is quite simple. First, you need to add a rack. Therefore, you need to give a name of the rack. This is CIRT1 here in this example. And you need to give a location where the rack is located. Then you can add a new power control switch to this rack. There you also give the name of the rack. Then the name of the plugin of the power control switch you want to use. And the URI or the IP address of this Ethernet control power switch. Then, in the same way, you can add a serial server for this rack. And if you've done those three steps, you can start adding boards to the rack. For example, we added here a board with name C at all in two slots, six of our first CIRT test rack. So, then you can just dump the content of your database here where you can see we have now five machines inside our rack. You can see the ports here in which they are located and the IP address or name, host name, and port of the power control of the embedded serial device server to connect to the serial console of this device. So, the next part will be the LibVirt connection of R4D. So, now you have your R4D demon and you can manage your racks. And now we integrated this that you can use R4D with LibVirt. So, now it looks the following. It's the same at the slide before, only a little bit illustrated, a little bit different. So, you use VIRSCH and you communicate via LibVirt over TCP or SSH with a LibVirt demon. And the LibVirt demon now is linked against LibR4D. So, the LibVirt demon is now able to communicate with the R4D demon via the SOAP interface. You need on both sides a patched version of LibVirt so that both versions of LibVirt are linked against LibR4D. Here you see a div stat, what we had to change to reach this. The main part of the change you can see here is the R4D driver we had to implement. And we decided that we will introduce a configure switch so that you can decide if you want to build LibVirt linked against LibR4D or if you do not want to link it against it. So, these are the adaptations we have to make. And LibVirt has a wide variety of feature sets and we do not use the feature set where you can create a new machine or we only use the feature set of LibVirt that you can open and close connections to the demon, that you can get the number and the list of the boards you have defined, that you can power cycle a board, get the current state of the board, power state of the board and we implemented also that you can open a serial console stream which is required that we can get the boot lock even if the board crashes. So before, Manu showed you how you can list all the boards using R4D and now you can do the same using VIRSCH. So, I make a connection to my R4D, via R4D to my R4D demon and then I can list all the boards which are now in this, which are defined in the demon and you can see it's the same list you saw before. Here you can see you can call the console command of VIRSCH, specify the domain and then you can use it as you are used to it. So, the next part of the game was how to integrate all this stuff into Jenkins. So, even in Jenkins we needed to do a little code change for that and then it was quite straightforward. So, I can show you how to add a new node into Jenkins and how to control this node via Jenkins now. So, the code change was really just one liner that we needed to do in the Lippard slave plugin of Jenkins because there is a list of types that are supported as hypervisors and we just added our, I call it pseudo hypervisor because it's not really a hypervisor. We control real hardware with this one, but we just added such a type here into the list of available hypervisors in the Lippard slave plugin. This little change gave us the possibility to go to the Jenkins configuration and there you can, if you have installed the Lippard slave plugin, you have such a cloud configuration method where you can specify a hypervisor. This is drop-down field where you can see all those fields specified on the last slide and so we have also R4D available now here. Additional, we need to specify the host where R4D is running. This can be the same host as your Jenkins master instance, but it can also be another host located maybe clearly nearly your test lab or something like that. For us it's a different host, so we needed to enter varus here, the host name of this machine, and then we needed to specify a port. This is no SSH port. This is the port where the soap interface after R4D daemon is running, but the UI development of this plugin just didn't think about that there can be things done like we did, so they called it just SSH port. And then we added another thing called concrete specified another thing called concrete slave capacity. They will tell that only one slave can be used at one time inside this cloud. Then we have some repeats from the last slide here and oh no, sorry. Then we switch to the node configuration in Jenkins. It's located in the main configuration below nodes and there we said add a new node to Jenkins and then we were able to add a new node by selecting the hypervisor, configure it on the last page of Jenkins, and then we just redrive the list of boards inside our test lab and this is quite the same way you see VMs listed here if you use the QAMO protocol or VMware protocol up there. So we use our five flags in this example here and what we did is this is the host name of our embedded board. This is also the name we configured it in R4D and so we used the same name here to use it in Jenkins. You can give different names in each step, but that just leads to unneeded confusion. So we recommend to use always the same name for this board. Then we have a startup idle time that can be specified. This is the time Jenkins waits until it makes its first connection to the device after it is powered on. So we worked with real hardware, so we power it, then the boot loader is loading, a kernel is loaded, the user space needs to be loaded and then Jenkins can connect to it so we need about one minute here until Jenkins can schedule a job on our real hardware. Then we set the number of executors to one so that only one job can be executed in parallel on this hardware. This is specific to our case because we wanted, for example, to run cyclic test and we want to generate a load with hackbench and therefore we just wrote a test script that starts the load generator and starts cyclic test, for example, and this is just one executor, this shell script. You could also do a job that generates hackbench load and another one that generates the cyclic test and then schedule them in Jenkins in parallel, but we felt this would be harder to implement in this way. So we decided to have always just one executor, which is also nice because then Jenkins ensures that there's just one test case running on my real hardware and I don't need additional locking on my target or in R4D or something like that. Currently, we have one shutdown method that can be used. There are just a couple of methods that are allowed in LibWord. We just implemented the destroy method, which is the same and just press five seconds to button off your PC and then it's just cut off. So no soft power off or something like that is supported at the moment and this leads to just power off the port on the power control switch. Then we used another setting. We say we want only build jobs be scheduled on this real hardware that is explicitly scheduled by Jenkins on some kind of this node. So this is a follow-up of the last page. It just scrolled down a bit and there I say a setting for saving energy. I say take this agent online in demand and offline when idle. So Jenkins decides when to power on and power off our boards and we can even give a delay here. For example, we give an in-demand delay for 30 seconds and if we have the same board five times in our rack then we can give a label to it and schedule jobs by label and then it waits for the specified amount of time until powering a new board of the same type because you know we take 60 seconds to boot so maybe it's not efficient to always boot and shut down the system and the same is the idle delay. So if a job is finished, Jenkins waits now for one second until it power off the device. So we always want our port boards to start and stop after each job but this is also a setting that might be different for other test purposes. So as I specified we need to explicitly specify the node where test is scheduled and if you use Jenkins pipelines to specify your test cases you can do something like that. You say node, specify the name of the node and then you can just write down your test case. In our case it's just a one line up because we added this to a resource of a cruvy library so that the cyclic test transcript is deployed along with the test to the boards but you can also add along our test specification here and I just right rubbed the whole block with a try and a timeout requirement so you know we worked with real hardware and real hardware might have issues. So for example a root vices they might be destroyed, you might have deployed a kernel that doesn't boot or something like that so you have a timeout of 10 minutes and if the job can't be scheduled for 10 minutes you just get this error here and mark the current build as unstable because our hardware seems to be broken. If you have a look in the Jenkins lock then you can see it looks pretty the same then your schedule something on a virtual machine the only difference is the name of the machine here so it's really transparent it's the same for real hardware and a VM. So our conclusion is we have now three instances of tools where we can do the same thing we have our command line tool to directly communicate with the r4d daemon to control our boards we can do it we are lip word by the way we implemented lip r4d that can be used from c code quite easily so that you don't need to do the soap calls from c over and over again and this is this library linked against lip word so if you want to control this infrastructure vrc it's quite easy if you want to do it with another language you need to implement such a soap adapter but that's typically also quite easy so we have our lip word connection here and for this reason we have the same in Jenkins so we have the same list of boards here here and here and if you want to add a new board to our test infrastructure we just need to add it in r4d create a new node in Jenkins and we can schedule jobs on this new real hardware so this is the current status of our project but as it's software so it's never finished you know so let's talk about some ideas we have what we can do and then Anna Maria will show you another piece that really fits nicely in our infrastructure called r4d test box which is nearly my laptop so one of the nice improvements would be to have the possibility to soft power off boards in some way we think about using maybe it's just requests for that to send via the serial line to do a gracefully shutdown of a system or maybe we have other ideas for that but this is would be a nice thing that you don't run five system checks on standard root five system on each boot or something like that then we think about that it can be interesting to have some kind of authentication on the r4d level so that you can for example only admins can add new boards on the urx and configure stuff there and you have for example user groups for several kinds of boards at the moment we don't need to settle in otronics because we have multiple instances of r4d for this case but it would be also nice to support this in r4d and we had this in mind as we designed the interfaces that it is easy to add this in a later step and of course we had another to do on our list if we had to find wider acceptance to control real hardware this way we want to of course we want to post our changes on libwrit that implements this new r4d driver in libwrit we want to post those patches and we also want to post the one line of patch for the libwrit slave block and for chankin standard so another development we had to do was the r4d test box the reason for this was if you use embedded boards you have the problem for example you get a board you should insert or integrate it into the test infrastructure and then you have another real serial console so you have the problem that you cannot communicate or cannot use the serial device server to get the serial console lock from the device from the embedded board so we need an instance in between so that you can redirect the output and can use it with the serial console another problem you face when you work with those embedded boards and like to integrate it we use the powers switch to start the board and to stop it so we reset the power and you have a board or you have boards where they do not boot if you power them so you have to push a button and I think there are not enough employers that we can stand some people in front of the rack and say please push the button I need to schedule a test so we have to think about a technical solution also it is great if you can make interface tests for example for can so we decided to build a test box it's a r4d test box and there you have all those features integrated it's based on a banana pi router board and we developed a shield on top so that you all the required connections are there so Mano shows you now another advantage of this test box is that it could serve as a all-in-one solution so you put this test box on next to your embedded device and then the box can power on the board or power control the board and you it can act like as a serial device server and the only thing you need is that you have to implement those plugins for r4d so that r4d is able to communicate with the test box so if you now like to have some more information or some detailed information about for example the continuous integration of rg there's the link to it they can get the measurement results uh we make during development then we um published all the source code on the c irt github project so they can found find the source code for r4d and lip r4d and also for the patched versions of libwirt and the libwirt slave plugin and we have to say thanks to ben and because uh without his ideas we could not talk about this stuff in the we talked about the last 33 minutes and um so he had a lot of ideas um and he also did a lot of work and to get it or to make proof of concepts and to get it running so thank you so now if you have some question or something like this do not hesitate to ask is there a mic if you can hand over to him okay um so is your r4d test box is that a commercial product or is that just a prototype right now or what's the status uh the status is uh it's a development version at the moment so we didn't publish the schematic sensor on off that uh but it's based on debian linux and open source software and we we are willing to publish the schematics and so on uh on github but it's not ready yet if you are interested in the further development of the r4d test box just drop Anna Maria me and mail and then we will inform you as soon as we push the stuff on github or if we have the possibility to uh produce some shields or something like that okay thanks some more questions there was one question there was one question and here so there are different uh solutions already available so what's the motivation to push this project and also to design own hardware because hardware um breaks down so you have something to to to maintain you have uh some stuff we which uh why didn't you use standard components uh and so for instance there's also think lab grid from pangatronics i think um so what is what is the main advantage of of uh this solution and the other thing is do we also get like when the board hangs in in u-boat or or or kernel do you get uh all of the console and uh sort um have you also written uh interface um tests or did you just focus on uh integrating the hardware into Jenkins uh we at the first point we we focused on integrating the hardware in the in Jenkins and we're driving as you asked on the second point we're driving the boot lock from the devices therefore we just added a rubber test around the real uh test running cyclic test on that that calls virge with the console tag to retrieve the the console like and artifact it along with the build um and the first question why did we develop our own test box if there are others available um it's quite simple answer uh as we started with this uh we thought we were the first one so we looked around and didn't find a proper solution for us uh we started to do it and now we i just learned about about the things pangatronics did uh as i uh looked at again now of uh the conference and they are they have something similar and of course uh if they are the same interests we should collaborate because the main difference is the the one thing you focus on on of the shelf hardware and there you you um build your own and when you work in in large companies then you have like um external companies coming and inspecting your network for security issues and if you have uh devices like that in your in your local error network then you will get get problems because those devices might get hacked and you you need to support them by your own so that i i see to the maintenance i i see the main the main problem um but did you also this was also the reason why we decided to use the serial controls from zena and the power control switches from goode because then we just have standardized hardware to put in our rack but did you also write some some interface tests or or uh so um or a test bench for that no that will be the next iteration to have that um i think over here do you have a standard way of loading the kernel in root file system to the devices or how do you handle that because i guess the kernel is part of the test flow pardon can you please repeat for me i guess building the kernel and therefore getting its own the device is part of the test flow how do you do that so um we built the kernel and make um build debian packages then we load the debian packages on our uh devices and we install it remove the package but um keep the important parts like um the kernel image and then we use kexec to test it so if you um boot a broken kernel um you're can ensure you're um you can reboot the system and then you have your standard kernel and the system comes up again so you do not have to go there and have to flash reflash it or something like this some more questions any other question yeah yeah you smaller rooms you mentioned that on your test box you have the can interface to test uh such interfaces on the device on the test do you also manage that via lib world also some support in addition to serial and power control no not so far but you got the idea some more questions okay so thank you very much for your attention