 Hello everyone. I hope you have a nice conference. Welcome to my talk about AGL's Continuous Integration and Automated Testing Infrastructure and about some info updates. My name is Jan Simon-Möller. I'm the release manager for AGL and I'm also taking care of the backend infrastructure. So, AGL's Continuous Integration and Automated Testing is there to fulfill a couple of goals. A, we want to ensure the stability of the AGL Unified Codebase. This is a Linux distribution for use in the automotive industry and it's based on top of the OctoProject. We also want to provide a fast feedback loop to the developer. So, this should happen for each code change on the available platforms but also for the applications that we develop. And the tests need to run on each supported hardware and on each image variant out there. So, how do we do this? We host our code in Garrett. We have a CI builder which in our case is Jenkins. And, well, we need something to run tests either in virtual machines or even better on real hardware. The catch is that the hardware might be in different locations. The hardware might not be publicly available. So, it's hard to come by some of these boards which is why such an automated setup makes it easy for developers to get feedback on their changes even if they do not have a zoo of different hardware on their table. So, for the board farm management we use Lava. The feedback is then kept currently in Garrett with messages back to Garrett. The initial plan was to use the kernel CI web UI and extend that with a test tab. That's work in progress. The kernel CI web UI is redone. So, we are waiting for the upstream project here to be redone. So, currently we close the loop by messaging straight to Garrett to the developer. This is how it looks like. We have our Git repos. The changes get built. We host then the build artifacts. And then we trigger a job through Jenkins in the Lava lab. This will then push the job down into a worker, a slave. Basically, there can be multiple. They are remote. They can be distributed. And then we collect the results. And this gives us the possibility to do a fast feedback loop for the developers. Let's take a look. For Garrett we host our Git repository there. It also provides us with a patch review workflow and a patch management. It has a web UI. And you can integrate it also with your LDAP. We can trigger jobs in Jenkins. So, there's a Jenkins plugin to pull Garrett. Our server is at the URL below here, gary.automotivelinux.org. Our Jenkins is the build scheduler CI builder. It will retrieve the updates from Garrett. It will trigger jobs. We have two main categories. One is a platform build, which builds full images. And one is an application type build, which only builds a single application, which is, of course, faster. To give you an idea, a full image build would take four hours from scratch. If it's using the cache, we are down to 45 minutes an hour, depending on the speed of the builder. It can go further down. The best time was like 30 minutes. But of course it depends on the size of the image. If you have an image with Chromium in, then that will take long, for example. Now, the application jobs, they are very smaller. They use the SDK, they use a previously built image, and they only build the application in question. So, the build phase is a matter of minutes, let's say five to ten minutes, and then we have the application build and can run testing on a previously created disk image. All that happens on build.automotivelinux.org and is triggered through Garrett. A platform build, as I said, is a build of whole images. It's a whole stack of Yocto layers that we built, and it takes quite long. The shortest path is about 30 minutes, but we have a few images that take longer. We use a templating tool called Jenkins Job Builder, which is developed by the OpenStack community, and with this we do statically pre-create our Jenkins jobs. You can also use, for example, Jenkins DSL or similar. Comparable is also GitLab CI or whatever other new ones are there. Downside of Jenkins Job Builder is that you tend to create a lot of jobs because you basically expand the matrix for each machine, for each branch, so you create a lot of jobs. Lava is our board farm tool. It's basically a scheduler, which allows us to execute tasks either on real physical board hardware, that's for what it was made for, or on a virtual hardware. It's a board farm, board lab management tool, essentially, and it splits the workflow between the test developer on the one side, the developers on the one side, and the admins that manage the boards, the actual physical hardware in the lab. So you basically deal with the boards as if they were servers in a rack. It has a distributed approach, which is very powerful because you can have remote labs wherever the boards are and still access them. As I said, it takes away the board maintenance from the developer tester and it allows also for easy access of multiple developers to a development board. And the one important feature is that it saves you from a lot of the SD card juggling, so that will be a good benefit because you can actually run tests faster if you do not need to juggle and write out SD cards, which takes a long time. This is the structure. Basically, we host a lava master, which has the VEP UI, which distributes the jobs. We have workers with the devices under test attached. There can be multiple of them and that's how the setup looks like. Now, in reality, things can then look like this. On the left side, you have here a set of devices under test with power control, serial and network. And the worker node is up here provided by this minoboard in this case. Or the larger you get up here in a rack with multiple devices under test in a stack. Now, if you are curious how to set this up, we use lava docker, which is developed by the Kernel CI project. This is a dockerized version of lava. So there are two components in there. One is the master and one is the worker. And on the worker, you connect the devices under test. And that is where you connect your boards. You need power control, you need serial, and you need a network for network boot. You can set up things like SD card moxers and so on. That is possible, a little more complicated to wire up in the test job, but that's possible. So if you want to try that at home, the simplest setup is in the README there with just a master, a worker and QMU. In the drag kit jobs, we have two helpers. One is called lava CLI, which is the CLI tool to connect to the lava server. And we have a little template generator in relink scripts. So these create jobs basically using the AGL URLs and using the test description that we host. So in lava, we execute a set of actions and we host these in our QA test definitions. And relink scripts is just a wrapper, a template generator that allows us to generate our CI jobs and send them off to lava. So that's a quick intro. What happens then in lava once you send a job is basically lava will boot the board to a shell, which means it needs to power up. It needs to instruct the boot loader, which is usually QMU, but also can be UEFI, Pixi, fastboot. There are a lot of options there. And in the end, we need to be able to drive the board over the serial connection or over SSH is also possible. Basically, we can then do anything that we can drive over a shell or terminal. We can test software components either in QMU VMs. We can test our software running on a real target. We can also test peripheral hardware components with some effort. You can also create a device group and wire up a test, which means that multiple boards do actually boot up and interact with each other. That is possible. Lava is not good for UI side tests. If you need that, then go and look at OpenQA from the SUSE project, which is there for dealing with UI work, where you say, I need this rectangle look like this type of tests. Also, we need to contribute more Lava test definitions to enhance the code base to make the UCB more stable. And we also welcome new Lava worker labs for more boards, for community boards, IMx8 to enhance the board coverage here. This now brings me to the updates on our CIAT infrastructure that happened lately and that will show you the progress that we made there in the last couple of months. So we added PyHGL. PyHGL is a PyTest-based test framework. It's a new test framework for the HGL services. It's written in Python. It's using PyTest, so it's very simple to develop on and very simple to use. For example, you can just invoke PyHGL. There are markers, for example, for jobs that require hardware like radio, for example. And those can be excluded if you said not hardware required. There are tests that require internet access. If that's not the case in your test environment, you can say not internet. Or you can combine both not internet and not hardware required. So that's pretty straightforward to deploy. And we have this active in CI already. The tests have been developed by Eddy and the integration has been done by Scott, so kudos to them. Here's an example run. I just started up a QMU virtual machine. I limited the Ethernet, I limited the network interfaces to Ethernet and then said it's no special hardware attached and we don't have access to the internet, which rules out GeoClue and weather to some extent as they require working network. The next change that was done was that for all the service widgets, we do now build the coverage widget, which means we do a GCOF based instrumentation of the service. And at runtime, we have a wrapper which will produce the coverage data installing the widget in question. It will run the tests and then report the coverage of our services. This is deployed in CI and it will report back the lines and branch coverage of the services. The heavy lifting was done by Scott and we have also a few more things in the pipeline. So quarantine will add the up square board back. That means we will boot with the QMU artifacts. We built x86 through the machine QMU x86 and the up square board will have to use these artifacts. Also, we want to see that the compositor runs and we see a home screen or we see a screen coming up. Marius is working to add a snapshot tool and write a comparison test. So we can actually test that the compositor comes up on all boards. We will not basically test all the UI parts, but we will make sure that the compositor is up. And we have more work coming up on GitLab to extend the mirror to allow more features to be used there. Now what's next? We also will build more images. So currently the main image is the AGL demo platform, but there is an instrument cluster expert group and they will define, come up with new images that use containers. So there are more images that will need to be built. Also in the instrument cluster, there are requirements to do static code analysis, to do linting on the recipes, so there is more coming on that front. We will have new hardware being added. So the AGL reference hardware has been shipped and the board that you see here is now available. AGL will put it into CI and will also research ways how to easily allow remote access to the boards in the farm. There are also experiments being done how to enable a video capture so we can actually take a closer look at what the boards actually display and make that easily available. Now, what infra updates are there? So we have now a GitLab mirror available and that is on gitlab.com.automotivegradelinux. Currently we mirror all our sources there and the first step is to provide a way to easily switch the source URIs between Garrett and the mirror to provide additional redundancy and allow further development here. Also, we have done a larger rework of our dock site. Shankar Barangosh was our Google season of docks intern and he reworked the documentation site. Now it's all in one Git repo and the rendering is done by read the docks, so that's easy to update. Check out the new dock site and there's a guide how to add new documentation there. Thanks for joining. I will be available for questions in the chat and online. Enjoy the conference and we welcome your contributions to AGL. Thank you.