 Yeah, welcome. I'm Jan Simon Möller. I'm working on Automotive Grade Linux and I'm the release manager. I'm also lead of the Seat Expert Group and today I want to talk about some, well, experiences I made while dealing with testing AGL. So it will be basically a quick overview what systems we looked at, what we use already, what we might use or plan to use, and, well, some observations. So take that with some grain of salt. So I want to talk about a few systems that are here on this list. And I want to introduce you to Yoctopetest. So who uses Yocto? Raise your hand. Okay, that's a good share. So Yoctopetest, I will introduce it to Fuego, Lava, KernelCI, Labgrid and R4D. So that's basically the ones I looked into more closely. And of course there are more, but that's for another day. Hello. Okay, so P-Tests. Who used the P-Test already? Okay, that's like 10. Okay, so we have something to do. So what is P-Test? P-Test is a part of, well, the Yoctoproject open embedded to be more specifically. And you can find the documentation here in the Yocto wiki. It's also in the manuals. We just need to stumble across it. So what is it basically? So TLDR, it is a way to package up the, well, actually most of the time, the tests that a package library ships already. And we can then execute it on the target and collect some results. So let's say, battery is dead, I'll use that one. So in principle, P-Tests are a convention how things are packaged within Yocto, OE. In principle, we create sub-packages. So if your package is called foo, we create a foo-p-test sub-package. The output format is basically a result, like pass, fail, skip, and then the test name. Basically, it aligns with the auto tools and the test suits that make test hat. It has some limitations because, well, you have a lot of paths, a couple of fails, maybe a few skips. How do you present that? So that's not handled here. P-Test is also a convention how to call them. There's two things that are applied here. A, we call that thing run-p-test, so we have a wrapper script with that name. And at the final image that we deploy on the target, we have a small C application called p-test-runner, which goes through either a single test if I specify it or all installed P-Tests, which can take quite some time. Let's take a little closer look so you can write your own P-Test and submit it to OE and Yocto. So a P-Test, so I took this from the C-LiP recipe and I just, well, distilled the parts out that are relevant for P-Test. So we have at the very top our wrapper run-p-test as by the convention. We have the class in her repeat test, which enables all the magic to actually package it up and so on. We have one step where we can actually compile the test. In this case, it's calling make-test. It might be different for your other applications or libraries. But in principle here, we compile the test suite. In the next step, we install the test suite. Look at this. It's a special path where we actually install the files into. And yes, we might have to fix up some path that are in the, well, probably test scripts or in the make file. That is likely unavoidable in some cases because they still think they are with, we were compiled within the Yocto cross compilation and we don't run on the target. So usually a little fix up can be expected. Okay. Well, so far that is not hard. And in the end, we just tell which package or which dependency the p-test sub-package has, which cannot be auto-detected. That is usually something like make or some, if you use a Python or Perl, that might not get well-detected easily. So otherwise, it's not hard because we just package something that already exists. So until now, we didn't have to develop any test. We just package up what we already have. Open embedded Yocto has already, I think, in the number of 50 p-test packages, if I'm not wrong. Sorry? 64. 64. So we have the most common packages covered, but there are probably a few more libraries that have a p-test which is not yet packaged. So patches, welcome. Here you know how it works. Now, how do we use that thing? So we did all the work. We have to do things. Well, we need to enable the p-test machinery during building. That is the first line. That is distro features append p-test. And that will make sure all the p-test packages are created. The second line makes sure all the p-test packages are installed in our final image. And then I can go ahead and call p-test runner. Alternative. If you just want, like, five p-tests that are of interest to you, well, you can always install only the sub-packages. This will install all p-test packages, but you can also just use image underscore install plus equals and then name minus p-test. So a few, well, personal things that I found good or bad, take with a grain of salt. So p-test is nice. It compiles the tests as we compile the binary. So it's exactly the matching test. It's cross compilation. It's ahead of time. It's well integrated with BitBake. And there is also a feature called the test image class, which allows you to basically automatically spin a QMO VM and basically run the tests on it. So there is some nice simple automation available here. What I found a little hard is we have a lot of output that comes out of this. And it's not very easy to digest that. The run takes quite long. Yes, sure. I mean, if we do a full pass of all tests used and GCC and whatnot, yeah, that's like, depending on the hardware we run that on, five hours or even more or less. What we get out of it is quite a large log here and now. So there are helper scripts in the Yocto QA that allow us to diff between the previous run and the current run, which allow us to do some analysis kind of what changed. That is good. But digging through that log is quite hard. So the actual visualization of the results needs a lot of post-processing or additional tools. Okay. So references. So here is, that's the wiki. And this link is the BitBake manual. Okay. Let's take a look at the next system, Fuego. So what is Fuego? It's a system. Well, if I would condense it to one sentence, TLDR. So it's automated testing of an embedded target from a host system with a large set of prepackage tests. So what does that mean? We do not depend on the target system. We compile the needed tests, the needed resources from the host, which is a nice feature, actually, because we test the production image at ESEs. We don't have additional resources installed, right? So p-test would be an additional package. We can remove it, yeah, but I mean, choose your weapon. So what is Fuego? So if you set up Fuego, you get a Jenkins instance that's preloaded with a lot of tests, ranging from LTP to all kinds of stuff. Simple stress tests, but you can also write up, I don't know. We have a command line tool that lets us instantiate those tests. In principle, what it will do, we have scripts to compile the tests for the target, so we have to define what the target is, what compiler to use. It will then connect to the target, upload and run the tests on the target. It will grab the results. It has the capability to parse them and visualize the results for the tests where that is configured, makes sense, and so on. So here's an example. So this is the configured set of tests for a Beaglebone board. We have different categories, benchmark, functional tests, so this list goes on very long, probably around about 100 or even more. The visualization is possible, so Jenkins can store the results and we can actually draw graphs and configure Jenkins to have those graphs for us, and then we see how things develop over time, which is quite nice. So a few notes again with a grain of salt. Large tests out of the box, that's a big plus. The fact that we do not require anything besides basically SSH and a shell on the target is good. The results parsers are very, very good strength. Now, for the use case that I have, or AGL has, if you are working on this on your desk with your board next to it, it's great. We have the assumption that we can actually connect to the board from the Jenkins. Now, if we go with an open source project, the board might not be where the Jenkins is, or the Jenkins is actually somewhere in the cloud. Then we would need a VPN. We do not... Well, what is not out in the box by default is deploying the board. We have our own deployment Jenkins job and bring it in the chain, but in principle we assume board is deployed and we have SSH over network. Each board, even if it's the same type of board, is its own executor in Jenkins, is its own configuration. Now, it might be interesting to have some sort of pooling. I have three beagle bones. We could basically execute some of those actually in parallel. If we would apply the concept of executor and just crank up the number of CPUs to stay in the Jenkins language, or the number of executors. If we make that not a board, but a device type, device category, we could just crank up the Jenkins executors on that node. Yeah, okay. A few references. If you grab the slides later, we have the FuegoTest.org wiki, which is kept up to date. There is a nice presentation about what Fuego is and a quick start guide how to set up Fuego using a set of containers. So it's quite easy to get jump-started here. We also looked into Lava. So what is Lava now? The Lava project just read it, or just got a new website, so this is just like two weeks old. It was started within Linaro, and their need was to test on, like, well, initially there are pictures like ten panda boards. So we had a stack of ten panda boards. All right. So they started to scratch that itch and developed Lava. In principle, its strength is device automation, and we can execute then tests. Initially we are able to power up the device, pump the file system up, usually over network, net boot, and then we can, well, do something. Execute a test and so on. So what does Lava do? So it manages basically the devices in your board farm, so having ten panda boards. Yeah, we have a device type panda board and ten instances. So it schedules, it has a scheduler. It manages the deployment of the file systems, power off, booting, commands for the bootloader, test execution, again, multiple devices per type. And meanwhile, in the latest Lava, we have templates for more than 150 devices that come as part of Lava. So adding a device is quite easy meanwhile. The setup supports the actual, well, workers where the devices under test are attached to be remote. So you can have one master instance, which is basically accepting the job requests, and multiple labs that actually have the devices attached, which could be in a different rack, different building. Yeah, they can be remote. So here's an example. That's from the public Lava instance. Here you see we have different worker hosts. We have different devices of a specific type. They are either in good condition, bad condition, retired. So those ones are actually idle accepting jobs. Here we see the output of one execution run. Before that, we have the whole downloading the resources, getting the bootloader over serial to load the right files and so on. And finally, we have the user space up, and here we see actually how tests can be run, and we see those marks here. So we run a U-name. That's the output. But we have here those marks for the test cases, which are then picked up by the Lava engine, so we can identify that this portion here belongs to this test case, and the result is pass. So what's good about this here, we have multiple instances for each type, which makes it really easy to allow multiple tests to run in parallel. That would be similar if we go for the Jenkins and crank up the number of executors. That is in here. The master worker split allows you to have multiple labs. Let's say for a larger project, you might not have the boards on your desk or in your department, so it's possible to have boards where they actually are, and it's not hard to set up such a worker. The simplest case that you can actually do is, well, don't expect miracles, but you can take a Raspberry Pi with a relay hat, and that is your worker. What is hard, the initial setup. It got, meanwhile, better also by using containers. So there's a project, Lava Dash Docker, which is hosted on GitHub by KernelCI, and that makes the whole deployment easier. The parsing here is not that sophisticated as in Fuego. So we have just basically markers here and the ability to collect results. We can collect some measurement, like a single number, but that's limited. So the actual result parsing is less advanced than in Fuego. A few references to the documentation of Lava and to the Lava Docker, which will help you set it up in case you want to try it out. So Lava Docker has one configuration file that's needed. It's called boards.yaml, and there's an example. If you copy that over, follow the instructions. You get a Lava setup with a QMU machine, and that will get you jump-started quickly. Then if we are talking about, you see, there's a pattern. We are going more and more remote. If we are talking about multiple people working on a platform on different boards, maybe the same boards, then we need a way to collect results. That applies to Lava, possibly to Fuego, or different Fuego instances. We want a way to collect the results, but because, well, to be honest, do we care that much about the actual testing? The testing is actually a tool. We actually care about the results, so we need a way to collect them. We looked into Kernel CI for that. Now, Kernel CI started out as a project to collect boot reports for the Linux kernel, and it was quite successful in what it tried to achieve. Meanwhile, the project is organizing itself, and we can't expect some more development going on here. The initial tests were related to the Linux kernel, and for the Linux kernel, we are interested in, does it boot? Does it pass? Well, LS is the network up. Some very simple tests. And that was the test case. Basically, Kernel related. So, Kernel CI is good at aggregation and visualization of your tests. What is Kernel CI? It's a database. It's a backend to parse the incoming results and a web front-end. The format for the results is JSON, and it is then capable of visualizing the results. In this case, that's the official Kernel CI instance. We see that for a given kernel version here, we had a couple of boards doing the run, the boot, and we see here the parse and fail rate. So, for this use case, it's quite good, and we can actually collect the results of a lot of labs which are actually distributed around the globe. So, it's a very distributed system. Most of those labs run lava, some run custom tools, right? The upload is independent of the tool, so we could run whatever underneath. Wego, lava, doesn't matter. This is still centric to testing the kernel, so this is basically boot reports, what we care about. Now, for our use case, we have to take that to another level because we are actually not interested in, does it boot? We are actually interested in tests at runtime, different tests at runtime. So, there has been the addition of... So, this is the boot tab on Kernel CI. We have now a test tab, and that's being further developed. So, what's good about this? We can actually aggregate results from multiple sources. It's independent of the tool. We have JSON format, how to submit the results. The setup, yeah, initially the setup, again, a little bit painful. Meanwhile, there's a Kernel CI docker, so that makes the setup fairly easy and repeatable. What is hard is that tailoring Kernel CI now for different use cases like the user space tests. That is hard, and Kernel CI, they are working on a redesign to make that easier. A few references. So, there's the Kernel CI docker if you want to set it up. There's a tool with Kernel CI admin, which then helps with creating the tokens. One example how it can be modified is PowerCI. So, that's a modification of Kernel CI, and it's used to measure the power consumption of the system, which brings in quite nice a few graphs and so on, but it's a lot of work to do that. Now you see the pattern. We are getting more and more into distributed, and we are getting more and more into larger labs, so more than one board. So, if you set up kind of your own lab, and you add it to one of the engines, whatever engine you use, you sooner or later have the problem that you want to jump into one board and debug it, and you want to grab a board for local development, local debugging. So, you need to take over control, block the board from further processing, and there's a project called LabGrid, which scratches that itch. LabGrid was started by Pengotronics, and they maintained that project. So, what's it? In principle, it's an abstraction of the hardware control. What does that mean? Well, power serial, and it lets you basically reserve a board, take over control, get the terminal from the board in a unified manner. It has such a coordinator, and basically a developer or admin could then access, reserve a board, a specific board, and access it. So, you can expose the devices under test, either to the test tool or to the developer with this wrapper or with LabGrid. So, what's good is we can then easily jump into the boards on the farm, debug them, without, yeah, big issues, and without basically going through the, if you talk about Lava, without going through the worker node, right? You don't need access to the worker node. You get access through LabGrid. It abstracts the hardware specifics. So, how do we get the serial? How do we power on the board? That is basically encapsulated in LabGrid, which makes things like configuring it for, let's say, using Fuego or using Lava easier because you have the same commands for all boards. You can hide the nitty-gritty details away. The setup, it's actually, yeah, written in Python, installed with PIP. I still had a little nitty-gritty problems here and there, so, but that should be better, meanwhile. The docs are quite good. They don't read the docs, so that will get you started with it. Similar, slightly different use case, but quite interesting idea is R4D. R4D was started by the RT patch guys, and they said, okay, we want to automate now a rack of, let's say, whatever device, and we want to hook it up to, let's say, Jenkins. So, it is kind of similar to what LabGrid does, but it has an interesting difference. So, R4D, which is, well, maintained under the CI-RT project, so it provides infrastructure for power control and console access, and the interesting feature, it plugs into Libbird. What does that mean? So, the idea was that we have a rack with some boards, could be an x86 system, could be an arm system, could be some whatever board, doesn't matter, but it's sitting in a rack. We have a network switch, power switch, serial server, done. So, kind of standardized components, a specific network switch, a specific power switch, a specific serial server. So, it's not that flexible like the other tools, but it allows us to hook into Libbird, there is a Libbird slave plugin, which can talk to this R4D stack. And basically, to configure that thing, we would add a rack, we would add a power controller, so this specific power switch, a serial server, this specific serial server, and so on, and add a board. So, we have power, we have serial, that's all we need to control that for basic stuff. So, we will manage power serial here, we will allow remote control and we plug into Libbird. That plugging into Libbird is actually the very interesting idea, because that thing shows up in Jenkins like an executor, like a note. So, it shows up as if it was a cloud machine. You can configure it like a cloud machine. And Jenkins has the necessary calls to hook into Libbird and make that happen. Actually, the boards will show up in Jenkins with those descriptions, where's the board? So, that will actually show up in Jenkins, which is a really, really nice idea to do that. So, it's something for Fuego, right? It's well suited, yeah. So, hooking it up with Libbird, great idea, I like it. It's rather small. The downside is currently only a specific set of power switches, serial server, and so on is supported. There is also like a tiny machine with some relays. I don't remember that, it wasn't a talk. There is this possibility to extend that, to do it also with other hardware. Downside delivered patches don't seem to be upstream, but we have Debian packages to our rescue. All right. Now, of course, there are more and other systems. This was basically a quick tour around the systems that I got in touch with, if you... Well, I need to... Oh, yeah, that's how it works, just a second. I need to move to here, and then I can show this over here, yes. So, if you are interested in what other systems exist, way more ideas than I could give you in the time available, then take a look at the test stack survey, which is on elinux.org. This is part of the testing summit, which has been organized, and there are a lot of... Well, way more than I already knew. So that's a very interesting list of test systems. They all have their specialty, so check them out. They have some interesting ideas and solutions, so you can pick your favorite weapon. So, with that, we are wrap up on QA. So, if you have questions, somewhere was the mic over here. So, of course, the frameworks all have their own strength, but what is important is, in my opinion, is that we can collaborate on those tests, that we can share actually the test results, because the more... Well, it gets a big data problem in the end, but the more we can share here, the more data points we have, the more results we can actually evaluate, and then it becomes a problem of aggregating that, evaluating and visualization. So, the big part is we need better tools to collaborate, to share the results, and to visualize the results so we can make something out of it and don't have, like, five megabytes of pass-skip in a log file. And more boards, more boards, that actually we are running the tests on, so that will help to raise the bar here, and that's my call to action here. Any questions? We have microphones over here. It's a little bright over here. Questions? Yep? Okay, the question is about, do you know any test frameworks that deal with real hardware attached to the boards? So, like, bare board is fun, but if you need a microphone, a driver, something like that, do you know any ways that automate this, like, nicely? Thank you. Can you specify, I mean, a microphone attached, yeah. What? I mean, if the board has some hardware that works with a microphone, like a digitizer or something, and you need to play a sound into that microphone, and you need to record a picture from the HDMI or something like that, so if you need to drive external stimuli to the board, and you have to, like, control it some way, something like that. Okay, okay. So, I'm sure you can write that up with Fuego in a job. It will not be easy. Similar in Lava, you can, so in Lava you can attach a label to a board, which means has some extra hardware, so you can identify that we have special hardware on that board, and you can basically steer a drop to that very board. Second, if you need some external input, in Lava you can, in the job, it gets pretty long, then in the job description, you can start, you can either grab another board, which is connected, or you can create a container, which runs at the same time, and which then does some poking, yeah? I don't say the writing that test description is easy, it's pretty spaghetti-ish, but it's possible. Multinode as well. Multinode, yeah, yeah. Okay, could you run the microphone? Who has a question? Yeah, a couple of things, yes, we have done things with audio in Lava in the lab before. There is some limited video testing, but as was noted, it really is down to the test writer, and having the right support hardware there to be able to do it. The second thing I'd note is, you mentioned of LabGrid and having access to the board, there is also within Lava the concept of a hacking session. You can submit a job, which will give you SSH access as long as you are granted that access to the particular sub-debt in the lab. You can SSH onto the board once the image has been deployed, so there is that sort of access, and it will do all the power control, all the deployment for you, so just wanted to make that note. Yes, the hacking session is there, and I actually use that myself Now, with LabGrid, you get kind of a set of boards abstracted. So, yep. Okay, so to the first question, we are also developing the Slavstack. It's also mentioned in the survey, and there is a custom-made board. It's called Moogspy. It has add-ons by its own. It provides internet, USB, some kind of switches, etc. But recently we started developing some add-ons to it, and the microphone and the speakers are one of the add-ons. So we use this, for example, to automate tests of, like, NLP and that kind of stuff. I'm also going to have a talk on add-on, which allows you to grab a video and to provide some kind of input so that you can emulate some user clicking or to get fully remote access, like hardware version of the RDP. So, generally, yes, we are doing stuff like that, and if you are interested, find us. Okay. Are the boards available? Where can I get a board? Yep. Okay. But I have to produce it myself. Hi. Okay, one last question. I think we are running late already. For the audio testing, I think Google provides a loopback module which you can use in order to also deploy sound and also record it at the same time with a delay introduced in the loopback module. I was wondering if you have any solutions regarding the deployment of the image. You said about LabGrid that it's an abstraction layer for hardware, but the main problem I had was deploying the whole image, and the only solution was using relays and stuff like that in order to... That is the usual case. You need some sort of power control, either kind of a remote PDU with the sockets or a relay. That is what you require in most solutions. So, R4D, same deal, LabGrid, you need also, well, some command. Okay, so it provides the layer, but I'll have to take care of the deployment of the image. You need to wire up the board. In any case, yes. Okay, thanks. That's true. So, there will be a wiring. You need to provide serial, some sort of power switch, whatever that is. Yep. Just to add to that, so a lot of the modern dev boards are using SD cards, and so there are some solutions out there where people have made SD card muxers, so you can switch an SD card image from being attached to the device under test to some host system, and then... Yeah, which is like, I'd like to buy that from Seed Studio. That's quite nifty, yeah, and I'm sure there will be some... some boards being manufactured soon. Everyone needs it, but it's hard to get by those, right? Yeah, you can't find them just off the shelf. Would be good if we come up with a board that does that for us. All right. Thanks for joining, and I'm around all week. If you have any more questions or want to deep dive into any of those, find me. And thanks for joining.