 Back everybody our next speakers are here from code think giving us a talk on open QA They've updated their talk in the last few months. So if you saw the one in Berlin, it's gonna be there's something some new content Lawrence and I can't pronounce your last name And James Thomas are your are your presenters, so please welcome them Thank you very much for the introduction. Yeah, so the title of this org is open QA testing on hardware automated full system tests actually open QA is one of the main tools that we've used in this kind of journey that we've been on over the last few years when we're Thinking about functional testing on on hardware and automating that But actually in addition to that there's quite a few other tools that we've created that we're also going to be talking about today so Yeah, my wave introductions. Well, just did a great job of that already really but My name is Lawrence the project manager at code think James a senior engineer at code think and We've been working on automotive projects for many years now And we're also both involved in free and open source software as well And I'll talk a little bit more about what code think is and what code think does in a couple of slides But yeah today's talk is going to be really Talking through Like I say this this journey that we've been on thinking about automated testing Functional testing on hardware so I'll start with a bit of context around why we think testing is so important and Trying to set the scene and then I'll do a little recap on what we've talked about before and Then we're going to hand over to James and we'll talk about some some new advancements that we've made some new developments in testing and automation So why testing matters in automotive? So actually even before we start talking about testing I think it's useful to set the scene and talk about software complexity in the vehicle and Probably I don't really need to labor this point because I think everyone in the audience will be aware of this kind of overarching challenge that we're facing at the moment as an industry I Think I read I read in an article a few weeks ago Lines of code in a vehicle 10 or 15 years ago was about 10,000 lines of code on average And now today if you look at a GL IVI plus the cluster It's just shy of 200 million lines of code and I can't find that article anywhere So if anyone does find if anyone sees that please link it to me I can't I can't find it but I definitely did read that and You know as an industry I think that's just indicative of the challenge that's being faced And this article by McKinsey. I won't read it through all of this, but it summarizes it quite nicely the level of complexity of the features that consumers are demanding the difficulties posed by integration of those features and Especially software updates over the air it points out specifically Essentially the automotive industry is kind of struggling to keep pace the productivity is not quite much in the complexity that we're seeing And I've put this comment on the bottom here, which is about Most automakers are not really set up to support software for the life cycle of the vehicle I just bought a car in December and I'm pretty sure I won't get any software updates Although it's going to be on the road for 10 15 maybe 20 years It's a pretty basic car. Maybe a better car might get more software updates and I know that You know this situation is changing people are working on this and trying to do You know full system updates more regularly and easier, but Generally speaking I think most most products out there not just vehicles, but most products are not really Designed to be maintained for the life cycle of the whole product Okay, so code thinking our view so could think is a software services company and what that means is that We work on lots of different automotive projects So we have quite an interesting vantage point I think we work with lots of different tier ones and lots of different OEMs as well and invariably We we see the same kind of things happening on projects, which has led us into thinking about testing We're an open-source consultancy with a strong expertise in embedded Linux and Excuse me. Also, we Spent a lot. We have spent a lot of time thinking about The integration story and the construction of software. So, you know, the software delivery life cycle is called build test deploy integration That's kind of you know an area where we've been really focused on So the theory of constraints I wanted to add a slide about this because I Think it's a very useful frame of reference to think about problems On on projects and generally, you know in software delivery as on an ongoing basis so I first learned about this from a book called the Phoenix project which is it's kind of about dev ops and an IT services and how this the theory of constraints it originally comes from production line manufacturing process, but how that can be applied to Dev Ops and IT and it's it's a novel actually it's it's it's a really really good book I recommend it to anyone who if you haven't read it the Phoenix project. It's really good but the theory of constraints is is useful because It's kind of founded on this principle of bottlenecks in the system. So in a rough summary the Maximum throughput of your system is constrained by the maximum throughput of your your bottleneck So if you can alleviate the bottleneck the Overall capacity of your system will be alleviated in line with that so an improvement of the bottleneck is an improvement for the whole system if you think about this example of a Highway with six lanes for all the different vehicles, and then there's a crash in the middle Or there's something that's happened. You can see that you know that the maximum throughput is only As wide as the the bottleneck is so Yeah, improvements at the bottleneck are improvements to the whole system and then conversely any effort that you put in either side of the bottleneck is Not really having any impact. It's it's good. It's an illusion. It's a complete waste of energy so when code think comes onto projects and If things are delayed or something like that and We often try to look for the actual bottlenecks and see where they are So yes what that usually means I mentioned earlier. We see the same patterns repeating almost always we start to think about testing and Usually that's trying to make testing more reliable more repeatable More robust almost always we're pushing for automated testing in CI pipelines But also we try to bring testing as early in the development cycle as we possibly can what people call shift left Because the earlier that you find an issue the less expensive it is to fix And I don't just mean in terms of dollar signs, but in you know people people's time and and all the other resources that go into projects So why is it more? Why is it less expensive to fix an issue the earlier you find it? Well testing is generally an expensive part of a project overall We we still believe that testing is generally overlooked on a lot of projects a Lot of projects rely solely on manual testing And on large automotive projects you can be looking at hundreds of people Sometimes in the biggest projects, you know, it can be into a thousand people who are validating the vehicle at any one time and Hardware is quite a prized resource especially in the early days of a project, you know people can Kind of not literally fight over it Certainly, you know, it you know, it's difficult to get your allocated time with the hardware in the early days of the projects in our experience, so if you have For example, if you have a really critical issue on the software that it doesn't boot or something like that You really want to find that in emulation Before you put it onto the hardware because if you know that you have a really critical issue You don't need to waste that valuable resource. You don't need to waste people's time putting it onto the hardware and Then if we extend that one step further The vehicle environment is the most expensive testing environment on a project, you know, it takes a long time to get the Though the vehicles wired up into a prototype state You have to have a special license to operate them and all this kind of thing so Any issue that you find in the vehicle environment that you could have found earlier is again It's a it's wasting people's time basically and in an extreme example on the previous project that James and I worked on If you've bricked a unit in the vehicle, you had to phone Bob from some other department. He had to come down Take out the seats change all the wiring and replace the unit and everyone's just stood around for a few hours You know days your day of testing is down the down the drain and yeah, just a waste of time And it's it's these kind of hidden costs. I think that makes testing so important because It's very hard to kind of quantify that kind of thing, but but it's always there So and conversely if you improve your testing and you start finding more issues earlier That's very hard to sort of to measure because you're measuring something that never happened in a way You know, you know that you're finding issues earlier or you're not letting them in the software But it's it's kind of hard to justify. But so it's it's these hidden costs. We found So, yes, we want to catch them as early as possible Also, we want to automate repetitive and manual tasks wherever possible. I'm certainly not saying that Manual testing can completely be removed. It absolutely can't it's it's essential, but obviously in any we want repeatable tests and manual testing by its definition Can't really achieve proper repeatability. So The current project that I'm working on They have a full Full system validation. It's a three week test and I think they do it every every two months or so at the moment the tier one that I'm working with and I asked the testing manager How many times do you plug and unplug the USB media in order to test it? Does anyone want to guess in a three week full system validation test? How many times? Thousand's not bad. It was 800 Which you know, I thought was fair play. It's quite a robust test. That's good. But yeah a huge amount of tests so If you were able to automate the USB media test not only would you be able to Not have somebody doing it during the day and you could make better use of the the hardware resource because you could do it Overnight but that person would actually be free to do more important things stress test the system Try and find edge cases use it as a real user would something much more interesting So it's not about removing manual testing or removing people who are manual testers It's just about freeing them to do things that are more valuable and more interesting And also maybe saving the sanity of this poor soul who had to do it I think if we worked it out, I think if you think it's 30 seconds per test Which is probably quite optimistic because you have to check everything loads properly It's obviously 400 minutes, which is about a day. So a full day of someone's time doing that Yeah, so a USB media was Was something we were focused on but it's just a good specific example Okay, I'll do a quick recap of some of the previous things that we've spoken about So we spoke at ELC 21 and FOSDEM 2022 and that was talk was lava and open QA was automated continuous system testing, so I'll talk a little bit about How we started how we sort of identified open QA as a tool that we wanted to use and How we combine that with lava to get some testing on on hardware and As Walt said, we I gave this talk at the AGL AML AMM in Berlin in March But we have some Advancements in Sten as well So the talk that we gave at FOSDEM and ELC We sort of use this to mind the gap To sort of set the context here. So it wasn't specifically Focused on automotive at the time, but I think the same principle does apply so Modern testing modern software platforms is extremely time-consuming and there's generally a lot of manual testing And there's a lot of products like I said at the beginning where you might You might use the kernel version that supported for two years But you know that your device is going to be in the field for ten years or plus So if you speak to upstreams, they will you know, upstream projects will always say Use the latest version and that's absolutely right. You know, that's that's why they work on the latest versions But in reality, we've found that there is this gap because the the sheer amount of Regression testing that you need to do before you have the confidence to upgrade In many cases, you know, it wasn't happening and certainly This wasn't a novel observation. There's a lot of people have had this observation before as well I think there was a talk by a Pengu Tronix from FOSDEM. I think a few years back Which is very good on the same topic So obviously, you know upstream first that kind of thing is is all going in the same direction So we thought that basically more testing Making testing more available is Is going to help maybe reduce that gap So we looked at OpenQA OpenQA is a very well-established testing framework. It's For the OpenSUSE project and maintained by OpenSUSE. So it's designed for Linux desktop environments and Linux distributions But we wanted to see if we could use this in an embedded context it works on a principle of screenshot comparisons using these things called needles and I'll talk about that in the next slide Well, basically, it's a click and a search and a time our value and a screenshot so you can compare what you What you expected to get in the UI versus what you did get and what we really liked about it was that It's actually replicating how a user is using the software because it's a touch input. So it's not You know, it's not sending loads of can messages from Canalizer as a simulation or something like that It's really is how the user would use the software And it's designed to work with QMU So what's a needle and test in this context? So a needle is simply it's the screenshot of what you expect to see and then it's also some JSON coordinates so you can do quite clever things like Outline an area on the screenshot where you need to click and then you can have an area that you want to mask so for example if the clock is changing in the corner you don't want that to fail your test so you can mask that area and you can Change the tolerance that you expect in different areas if you expect the colors in the background to be different you know shades of brightness and so forth and Then a test is separate to that and the test operates on the needle So in the test you can say where you want to click and how long you want to wait for So with this system you can you can run multiple different tests on the same needle to navigate through to different areas and things like that so you can relatively quickly you can you can get the workflow of the All the different screens in the UI So brief overview of the architecture The test themselves are in git and that's quite useful because You can tag them with every release of the software so obviously as your project matures You're going to be adding more tests so you can always keep track of where they were up to And this we this is git lab because that's our weapon of choice But it works with any other CI orchestration as well So that acts as a launcher and it spins up an open QA worker The open QA worker can be Anything we use git lab runners, which is a VM of course where you can be a laptop or a Raspberry Pi And then separately you need an open QA server and this does two things It stores the needles because they can get quite large so you you store them separately And then also it has this kind of dashboard of results Which I've expanded on another slide as well so Yeah, and then every time you want to do a test the open QA worker will spin up an instance of QMU And that's done ad hoc so it's per test and then it reports the results back to the open QA server And a view would look like that so you can see here just on the bottom We've got a mainly green test run with one fail at the end you could drill into that and see exactly You know what what you expected to see versus what you got So that's open QA with QMU as it's designed to work We push this to the GNOME project So GNOME OS is now using open QA in its CI which is very nice There's a GNOME open QA server But like I say we wanted to get these exact same tests running on hardware So we look to lava and I think probably most people will have heard of lava here It's was originally a linearal project. It's used a lot by the kernel CI projects as well essentially you can it's orchestration of hardware testing so you can set up what's called a Board farm and have lots of different boards with different architectures on and you flash your flash your image to it And then run your tests on it So using VNC we got that working with open QA so that we could run the same tests there instead of QMU in emulation and Yeah, we have a lava instance at code think now and we have an open QA instance as well and We are testing the Linux kernel upstream project every night with this so we've got a devian image that tests and That's had some good feedback from Linus, which is great. So that's actually being used So yeah, we managed to get the open QA tests working on a dev board The next step after that was getting it on to representative hardware on For example an embedded project an automotive project and also getting something in place that allowed us to put it onto any generic hardware and this led us to developing QAD and A host of other things so I'm going to hand over to James now and he'll take it from there So, okay. What is QAD? QAD is a very very lightweight Linux Daemon That you can put onto pretty much any Linux system that uses DRM for rendering and EV dev for input So for example, this actually works on Android you can run it on natively compile it for Android automotive and Pretty much any type of system that uses DRM for rendering So this runs on real hardware you can slot it in the dependencies are incredibly minimal So it's very easy to compile. You don't have to change anything about the software you're trying to test And that was a design goal. We did want to make this In a way that didn't require a special test version of the software We want to consume the version of the software from the CI that will end up in production and run the tests on that So you build this statically linked binary and you put it on there And it's essentially creating this Automatable test engineer. So what can it do? QD is very simple. So we have a very very simple HTTP API So you run this Daemon on your system And it exposes to HTTPM points. So you can get HTTP get the screenshots of the display Or multiple displays and you can also post input events to it. So this is JSON formatted input data So it's very very simple. You specify the X the Y duration the type of event velocities and things like that So byproducts of this it was originally designed for testing But because we know how access where you can get a screenshot and send an input event you can actually use this as a method to remote the access the hardware that you want to test so say you're working from home or You want to test something in a vehicle This could be installed in the vehicle and then you can remotely connect to it and then control the UI So if you want to test a new piece of software you want to see if your changes have gone into effect you can use this and then do that Depending on your network configuration of course What isn't QAD It's it's very dumb. So it will not do anything by itself. You have to call The screenshot so you have to get that and you have to post an input event It doesn't provide any type of remote shell access. So keep on using SSH It doesn't do any type of Test orchestration so it will not do any automation for the tests at all. You need to use something else for that And it's not a silver bullet. So just because you have this Doesn't mean your software is going to magically get better So how do we actually automate QAD then? As I said, we have a very very simple get and post API so we get the screenshot and we send input events so this ties in very nicely with the The architecture of open QA and how you write open QA tests Because open QA is based on this concept of asserting a screen and then clicking a point on there and making sure that worked So for this we've added a open QA console and back end into the QAD repository So you can actually integrate this with your open QA tests So if you go back to this slide where we're using lava for the hardware orchestration This talks VNC to the Ubuntu instance. That's running on the Target hardware there. So lava has orchestrated this It's provisioned the board. It's given you access to that boots. Ubuntu is running VNC and then you use open QA And VNC is kind of the default protocol that open QA works Works with But that's not what we want. So We want to be able to run these tests on an automotive rig So this is a big piece of hardware. It could be on a developer's desk. It could be a vehicle You know, it's got multiple ECUs speakers and all that type of thing And we need some method of communicating with it That isn't going to introduce software that is going to change the parameters of the actual system So I've never come across an automated project that had a VNC server on for example. So we can't use VNC And whenever people use Western That does have a remote desktop protocol, but that usually is installed afterwards So you're kind of making a test build in order to to do that So this is the architecture that we use for the client that we currently working with So we have a git lab instance And then a developer will make a merge request into the yachto recipes for the system. So they want to update the system That merge request is then built. So we build the image and Then we send this to a git lab runner. Now the git lab runner itself is a small It's a small Intel machine, but it doesn't have to be that powerful. It could be a Raspberry Pi or something It's something even more minimal, but this Git lab runner is essentially doing the orchestration for the test hardware So this is physically attached to the automotive rack. So it will have a can USB interface It will have serial interface to the rig. It will have ethernet to the rig and that Particular device is running open QA worker and that's the thing that spawns the tests So we download the image we just built from the CI pipeline on to that runner We then flash it onto the rig So this is going to be rigged dependence, of course, but we send it over and then run the Flashing procedure on that new image. We reboot and then we start QAD So once we started QAD we can now run open QA tests Using the QAD console that we've developed for open QA One of the real benefits of actually using open QA for this type of thing is if you're a developer as part in the CI Pipeline during the testing stage You can get instant feedback. So you get a link to the test reports that one showed earlier But this will also give you a live view of the testing. So as the tests are running you can see The tests running on hardware in real time So you do get this instantaneous feedback there And then of course once it's finished you can pass or fail the pipeline depending on the policies there You know some tests are not as important as others But perhaps if Cal Play didn't start you would want to fail that pipeline And here's a video so this is a demo of this so the rigging question here. This is actually the AGL demo platform It's running on a Raspberry Pi and there's a multi-touch touchscreen attached. So I Think we'll go to the terminal in a minute So on the right hand side there, this is the open QA worker being Run on a piece of hardware on the right hand side now They're launching QAD on the device on the left hand side is EV test just to show you that the input events are coming through So now they start the tests and this is the type of thing that would be done by the CI So the CI would start the test and you can see there. We've got tests that goes on the home screen to the HVAC And you can see the input events arriving now. We have another test that goes back to the home screen and You can see the input events there Now that test is finished. That's the only test we did for that. So it's a very very simple demo So it's finished you can now go to the open QA Server and see the results of that Now this is actually one of my favorite things about open QA the UI for this is exceptional for reviewing tests So you can there's a slider to compare screenshots if something fails There's a needle editor So if you need to update the needles if you need to update the screenshot because something's changed in the UI You can use that So it makes it very very easy to kind of update these tests Now as I said, there is a needle editor in the web UI So if you need to update the test and you took the Jason in the test then Use that is very very simple and it will push it to get and then you can start using that test straight away But what we found when trying to retrofit this approach onto an existing platform that had no testing Was creating those needles in the first instance was a bit of a chore so we've got Dozens of tests here and we have to create the needle and the Jason file for them in the first instance and It wasn't that easy Which is why we invented This QAD web UI So this actually is useful without writing tests because you can actually use this to remotely access your Hardware here, so you can connect to QAD running on a piece of hardware and use this to then Navigate through the UI and what have you But its main purpose is actually to create those Jason files and take the screenshot for a particular needle so you can specify your needle directory you can give the project directory you can edit existing needles from there you can easily as a whizzy wig editor so you can define the Areas that you want to mask and the areas that you're interested in because at the threshold and then you can export all of that to the Png Jason pairing there Now you don't have to use this. This is I don't think it's actually currently open source, but we're about to press the button on that But I think it's very useful and it's very useful as this remote access technology So whenever there's a failure on a piece of hardware that is running in a different country we use this to see what's happening So the future work for QAD It's it's a new project. So it's not the most mature thing in the world. It does work We need to open source the web UI And we probably need to do more screen capture back ends So the default one that we use now is a DRM back end. So it literally just grabs them Rendered frame buff from the DRM subsystem. We do have a Western back end, but that's a Western back end specifically for the Geneva layer manager So this is the ILM protocol and that that that is useful because you can then speak ILM to anything that's Using layer manager on a system so you can ask for the screen and Configure it that way More input devices so in terms of the input device We either piggyback on an existing piece of hardware that's on there So let's say the multi-touch monitor or we do have a new input back end so you can create these virtual Touchscreen keyboard and mouse devices If for some bizarre reason your Linux system is not using EV Dev for the input you would have to create that yourself But it's relatively trivial. You just have to implement the primitives in the input It's MIT licensed and if you scan that QR code, that's the repository for it So this gets us to a very good place. So now we have a lot of tests where we can exercise the UI We can see what's going wrong. We can verify that The system looks good, but of course an automotive Project is is a lot more than just touching the screen and seeing what the reaction is Now we need to automate things like the can interactions So if I went to the climate application and I increased the temperature That will send a can message that which we then need to respond to so if we did want to have a test that did that to Check that the temperature was increased We also need to provide that can simulation and we want to do this without Relying on a Windows laptop running canalizer You know, we want to put this on the control device So this is an example that just came up with before the talk So we have a very very simple JSON parsing can Damon that runs on the control unit and it's connected out using a USB can device and we can just simply define The types of signals that we want to send the period whether or not we want to send them at the start and we can map them to Signals that we receive so for the example of the HVAC unit there. We will get a fan Event received in a particular format. We then need to send that back with the Temperature or the fan speed and we can do that quite easily with a Simple JSON signal map there It that definitely needs more work And That's not open at the moment, but this is something we are thinking about Now another piece of hardware that we developed for this project is This USB switcher so the example Lawrence gave of inserting a USB stick taking the USB stick out 800 times Can now be automated using this piece of hardware This this actually has been Brilliant for the testing because now we can actually test for every merge request USB media Does it does it come up is the salt order correct is Does it play? If we remove it does it go away if we put it back in does it come back? If we change the language is the salt order in Chinese correct is the salt order in Korean correct and all those types of things are Open using this piece of hardware Again, this is attached to the control unit that's attached to the rig and Then you can put two USB devices in and switch them using a very very simple serial API there And another amazing thing is that we can now test Cal Play as well so We can have a test where we plug an iPhone in Does it come up? Does it come up in two seconds? Does the functionality within the Cal Play application actually work? So here's a bit more about the switch It has so you can connect two devices to one host That's to a USB stick and an iPhone for example to one rig if you did want to connect connect more so if you wanted an Android device an iPhone and a USB stick you can daisy chain these and Use it that way so you can switch them on and off So you can have as many devices plugged in as you you want That's completely open. So it's open source hardware the firmware everything about it is open source and It's it's Generically useful. So it's it's not just for automotive testing if you need to automate any type of USB procedure This type of thing is it is an absolute godsend So the QR code there will link you to the Git lab repository for all of that Now all of that Brings us to something that code think has been working on for the last couple of months and that is testing in a box So everything that was just described there in the process kind of looks like that You can see on the left there's a control unit, which is the git lab runner in the open QA worker There's an iPhone attached a USB switch Ethernet dongles there's a can device there peak can USB device And and it's it's a lot And it's not that easy to put into a rack or To stack up anywhere and it's certainly not the type of thing You would want to take into a vehicle for testing because it's just a bit of a mess and all of that requires setup So you need to set up the control unit. You need to install the software. You need to Find provision all of these can devices. You need to buy the hardware. You need to plug it in And that's not ideal and that takes time so the solution is to literally put all of that into one box and it does exist is is a Is the very first one? I believe and so what this does is it allows you to Connect to a piece of hardware that you want to test and it has a number of interfaces, so it has a serial IO interface has can bus emulation The USB switch that was previously mentioned is built into that. It has a USB hub It has a Bluetooth and Wi-Fi Chip, I think it's a risk 5 chip in there And that actually is quite exciting because now we can do tests Where we mock a phone for example, so in our current test rig, we actually have a Google pixel That's close to the test hardware so we can then test whether or not contacts download or whether or not it's paired correctly Or whether or not the music's available But with this we could then mock that we could send contacts We could change the name we could un-pair we could repair we can do all that in an automatable way Which isn't that easy because you know you'd have to do that on the Android side Has I swear see SPI HID emulation so you can use this as a keyboard and mouse for a device so if you Wanted to automate sending keyboard presses or mouse movements You can use it for that and it also has a host PC and I Can't remember what the actual host PC in there, but it's some lightweight Armboard and that allows you to also then put the software directly into the box so You can install the GitLab runner You can install actually run an instance of GitLab CI in there And it has all of the software QAD open QA ready to go and ready to deploy You know this this is this is really going to be very useful because you can take this on to any project Plug it in and then start writing these tests almost immediately And the provisioning for it for this it's It's all in Ansible so you can deploy to this using Ansible It contains all of the software We're planning to add CI templates so you can more easily integrate this type of thing into your Existing CI so you know we prefer GitLab, but you could be it you could use this on anything And we're working on a version 2 of the board so there's going to be more features Now this is also going to include software again that we have open sourcing to monitor performance of tests So we can gather data about a test run and Then compare it with a previous one. So let's say you have a merge request That comes in you can do the test get that to performance data and compare it against the one That's in the current master branch so you can easily spot things like the CPU usage has gone through the roof or memory consumptions high And things like that and that's all going to be part of this testing in a box So the qi code there Links to that group So it's quite exciting and if anyone wants to take a look at it afterwards then feel free I think it works not actually plugged it in But yes, that's Yes, that's it for me. Thanks James. So yeah just to recap We started off talking about testing is expensive and it has hidden costs automating testing Functional testing in this way presented us with quite a lot of challenges. We found it quite hard but with the right tooling it can be easier and Obviously all the tooling is open source, so I Think at the end as well that the the great thing with testing in a box It's just worried about why we really wanted to do it With open qa and all of the other tooling obviously I've said before we were on this journey to sort of Figuring out as we were going We everything was ready to go so that would have saved us months really setting up open So that would have saved us a lot of time Okay, and we are out of time. So that's the end of the talk. There any Questions is no time for questions. No. Oh, sorry. I thought we had 50 minutes Okay, well, we're here so you can always ask us please to feel free to ask us any questions