 Hi, I'm Dan. I hope you're having a great day and have a great conference. Now I'm having a pretty great day today. It's really fantastic weather outside. Spring is coming. You can unfortunately not see anything of that here because I had to close my blinds, but let's get started. So I would like to talk today about OpenQA, which is an integration testing framework for operating systems, systems under test in general and appliances. So let's first get started and let me introduce myself. My name is Dan. I'm a software developer working at SUSE as part of the developer engagement program. So I usually develop tools for other developers. And I'm also a package maintainer in Open SUSE and in Fedora as well, where I maintain quite a few packages and I'm also a member of the I3 special interest group where we have recently released and worked on the I3 spin for Fedora 34. One of my big passions is testing. I think we should really invest more in testing and it's one of my, well, it's one of my big interests. That's also why I got into OpenQA and in my opinion, it's a pretty cool project and that's why I like to present it to you. In case you are further interested and would like to stalk me on social media, a few handles are down there on the slides. Well, occasionally post stuff about technology. So let's take a look at today's agenda. So I assume that most of you have no idea what OpenQA is and in that case, this is the talk for you because I'll be covering first what is actually, what is even this OpenQA thing. And I'll give you a brief introduction, a short sales pitch, what it is, what it can do, how it roughly works. And then in the second part, I'm going to talk about more on the side, how you use it. This is going to be all relatively, relatively broad and hand wavy because OpenQA, it can do a whole lot of things. And if I would try to explain it all to you, we'd be sitting here on Monday still and I don't think that you want to do that. One at the end, we'll have a Q&A session and a demo if there's time left. So first let's start out with the elephant in the room, OpenQA, what is this? So in the closest sense, it's a web application. So OpenQA, if you are a tester and user, then for you, OpenQA is mostly a test application. But usually when we refer to OpenQA, we actually mean the whole box, the whole product. And that's a framework for running automatic tests on a SUT, SUT stands for system under test. So this means, this is a framework that's really testing a system and not necessarily a certain, a certain library or a certain program. So the idea behind OpenQA is really, you have some kind of system that you want to test and OpenQA performs these. In the general, generally this system under test will be some kind of virtual machine, but it can also be real hardware. OpenQA then can simulate user input and by user input, I mean stuff like pressing on a keyboard, moving the mouse, clicking the mouse, listening to sound and watching video. So, and by that I mean, watching the video output from your system under test. And then it can do image recognition on the resulting video feed and do some and do comparisons. What with expected output. So this is a, this should give you maybe a rough overview. And before we dive into further details, let me try to be a sales engineer, which I'm not. So probably I won't do a super good job at this, but why should you use OpenQA? So first, you should use it if you want to automate so-called system level testing. So if you have stuff like, I have created a Linux distribution and I want to test that the installer still does the right thing, then OpenQA can do this stuff for you. And in a general sense, anything that does user-centric testing. So as I said, imagine you're your installer. If you have, I don't know when's the last time that you've maybe installed a Linux distribution or Windows or whatever operating system, the installer part is relatively boring to do. And I have never really been part of QA, but I assume that doing QA on installers every time before a new release is pretty dull since you just launched the thing and then you click next, next, next, select the hard drive, overwrite everything, wait five minutes for the packages to install, yada, yada, yada, takes a lot of time. It's really boring. OpenQA can do this stuff for you. So if you have some kind of, if you have something like an installer or something that runs long and that can be OpenQA can automate this stuff for you, when should you use it? So another reason why you should use it, it can run your tests on bare metal. So in case you want to ensure that your system under test still works with the newest snapshot, it can test this really on real hardware. So if you are, let's say, you're building a spin of Fedora or of OpenSUSA or of Gen2, Debian, Windows, Android, and you want to make sure that this still works on your latest kernel version, works on the Raspberry Pi or on a Jetson Nano or whatever. OpenQA can push your newest image onto this hardware with a few, with some tweaking from your site and then run all your tests on that. It also supports for you if you are maybe not a QA person but a release engineer, someone from Release Engineering or the Release Manager. OpenQA supports extensive test labeling and automated bug tracking. So you can attach labels and you can attach bugs to specific failing tests and that allows you as a release engineer, you see, you get your whole overview of all the tests belonging to a single build and you will see, okay, this test is failing and there's a bug for that. And then you as a release engineer can quickly decide, okay, is this a blocker bug or can I tolerate that and still release? And then you can also review those automatically and so on. And finally, this thing is really battle tested. So this is not something that's been quickly hacked together. That might have been the case 10 years ago but it's extensively used by OpenSUSA. It's essentially the thing that keeps OpenSUSA tumbling and not stumbling all the time. It's been, it's used by Fedora's QA. It's used as far as I know, it's also used by Red Hat. It's used by SUSA internally to test Slas and in probably a whole lot more places. There's also some integration to the Linux test project. So there's a few, there's actually also a test project that uses, that tests the Linux, that tests the OpenSUSA Linux kernel and automatically reports that. So with that, I hope I got a few of you hooked. So let's first take a look at the architecture of OpenQA. So OpenQA is, looks roughly like this. You have the web application which consists of a webpage. That's what you interact with as a user and a API. That's for your command line clients. And then, so this is a REST API that you can directly interact with or you just use the command line scripts. And then what OpenQA does is, if it gets a, if a new test should be started, it dispatches this to a so-called worker. A worker is effectively just a process. It can be, the worker can run on the machine that hosts this, that hosts OpenQA itself. It can be a separate machine. So this really depends on your setup and your needs. But the worker itself, again, it's not, it's not a, it's a relatively dumb thing. The main, the really interesting part is done by OS auto-inst. OS auto-inst is a, is the, I would say that's the heart and soul of OpenQA really. So it does all the heavy lifting. So, and now if you come to an actual test, what happens is the worker connects to the system under test, which in most cases is just a QEMO virtual machine and then it launches OS auto-inst. That connects via a serial line to the system under test and sends it commands. So usually you just start out by sending it key presses and once you have a GUI up and running, you can also start sending mouse events. And in many, many tests, you usually upgrade the connection to SSH and establish a VNC connection. And then you can start grabbing the video output, but you don't have to do that. So OpenQA, while we'll be mentioning image recognition and so on quite frequently, you don't necessarily have to do that. So OpenQA can also run just scripts on your system under test. And there are also tests in for, for instance, for OpenSUSE, I think OpenSUSE micro has tests that never use any image, that never test any GUI. Well, and then you got the video feed and that's matched via OpenCV. So how does this thing now look in practice? So let's take a brief look at the web UI. This is just a screenshot from the web UI. So what you see here is the so-called, is the installation test group. And this is from OpenQA.OpenSUSE.org. And these are individual tests that install OpenSUSE Tumbleweed. So, but I think it would be hard to read in the actual presentation, but this is just for the bootloader, the yes, welcome screen, select repository, select installation mode and so on and so on. And so what OpenQA here does is it gets instructed to do the installation of OpenSUSE Tumbleweed. So this is how the web UI would look like. And the cool thing is, as I said, OpenQA grabs the video feed out of that. And the nice thing is, you can actually look at the resulting video. And so I've just grabbed 30 seconds from that. And what you can see here is OpenQA, it launches, it boots from an ISO, then it adds a whole bunch of additional boot options to be able to actually run, waits for the installer to launch. Selects all the specific options that were set up, set up users and so on, selects everything the way it should be and launches the installation. And so this is really all that you currently see here running and that you just saw, that's all automatically set up by OpenQA. So no human had ever touched this. And it's also quite probable that no human actually looked at the output of this as long as the test was successful. So really stuff like this, for instance, like installers, that's really where OpenQA shines. I'd now like to tackle a bit the part of the image recognition, just to give you a rough overview what you can do with this, with the image recognition part. And so the thing here is, as I said, this is not a mandatory feature that you have to use, you don't have to use it, but if you want to test installers, it's pretty convenient. And so how this roughly works is you do this image recognition by a so-called needle, since you know you search for the needle in the haystack. And a needle is that's just two files. You have a PNG file that is just a screenshot that you wanna check against. And then a JSON file that defines which areas in the screenshot should be checked. So that means I have here an image of such a needle. So the image is the screenshot and the not dark areas are what should be actually checked. So the important part is, why don't you check the whole screenshot? That's relatively simple. In most cases, there are changing parts on your screen. Just take a look at your desktop and watch it for 30 seconds and observe how many parts of it will change, at least your system clock will change. Maybe occasionally you change your desktop background. And also in most cases, you are not really interested that certain parts of the screen look exactly like this. So for instance, in this case, this is a screenshot from the Fedora server installer. But the thing here is, so this is your installation summary and here the test writer decided that they just want to check that these icons are present here. Now the thing is, the interesting part here is just, I wanna check that these icons show up somewhere on the screen. I don't care that this is the Fedora server installer. It could be maybe also the Fedora workstation installer. And so that's why you just use specific areas. OpenQA will then use OpenCV and some additional image preprocessing to match to find these areas on your screen. And it also does this image recognition relatively seldomly. So it does it only about once per second, sometimes faster, sometimes slower. So you can't really rely on this being super fast. Let's go into some more advanced features. So what OpenQA can do it, it can produce test artifacts. That means in generally that means you can run your installer. Your installer will install on a disk and you can tell OpenQA, hey, grab this disk and store it. And then you can use this created disk later on. This comes really handy in defining test dependencies and scheduling tests depending on other tests. Since you can use this for stuff like, I have an installation test that runs through the installation of my Linux distro or of Windows or Mac OS or anything else. And then I have a bunch of other dependent tests that use the created virtual machine disk image and run tests on that. And the big advantage of this is your tests of your user space, they really run on an installed disk image that's been created by the installer and not by something else. So you effectively really test the stuff that a user would get. I also previously covered this part that you can add tags to your specific tests. You can attach bugs. So these labels are also automatically carried over if you restart tests. So this is something for release engineering and release managers so that they can get a better overview. You can group tests into test groups which is that share certain variables. And then there's a plethora of back ends that you can use. So for instance, in most cases, you will be using virtual machines but you can also use run your tests really on bare metal. You can use IPMI for connecting to some remote servers. You can also use this weird thing X3270 which well weird is maybe you might be wondering what that is that's a special console for S390. So if you want to test mainframes, you can do that. So with that, I'd like to show you a bit of the test API. And so please don't run away. Yes, OpenQA is written in Perl. The test API is thus also using Perl. It's not that bad as Perl's reputation is. So as you can see, this just looks like most other programming languages. And the test API is in most cases pretty intuitive. So in this case, you say, hey, start this program in this case Firefox, send the key old age that will just open the help menu. And then you do assert screen. Assert screen tells OpenQA here is a search for a needle with this tag on the screen for by default 30 seconds. So that means your needles can have one or more tags and there can be many needles with the same tag. And so you can just, and so OpenQA will now try to find something that matches that the context menu is open. Then just another key A and that will open the about Firefox. And so you assert that that shows up as well. And since this is just a really short end example, we do another thing of the test API and that is send this key combination. So close the window until we match a needle that's called desktop. So effectively just hammer old four until we are back at desktop. So let's get to you, let's get to use OpenQA. First we should think about when should you use OpenQA? Well, and as I've outlined multiple times you should use OpenQA if you want to automate the boring stuff. So your QA person will probably hate you if they have to test a snapshot of your installer every single day. Use OpenQA for that. If you have tests that require some specific hardware for instance a Raspberry Pi, use OpenQA for that. It can do this stuff automatically for you. You don't have to have a QA person fiddle with SD cards every single day. If you have appliances, so if you build something on top of a Linux distribution and you want to be sure that your appliance installer still works and that the application that you ship that they still work, then OpenQA is a good fit for you. And also in the unlikely case that you have a whole ton of hardware or money to buy cloud VMs but you don't really have a whole lot of testers, then use OpenQA to automate this stuff. But I would also like to say there are of course cases when you should not use OpenQA. One of them is if you are testing some kind of appliance application that will have quick changing stuff. So if you really need quick reactions, OpenQA is not for you. Don't test your new video game with that. As I said, you will have this image recognition part is done roughly once per second, sometimes even less frequently. So if you really need to react in under less than a second, forget about it. Sometimes even two seconds are not enough. If your test should be really, really fast. So if you expect your whole test suit to run in under a minute, I'm sorry, but OpenQA will probably not be able to do this since it's really testing a real system. It's designed to test a system under test. And these things, once virtual machines are involved, things are not really super fast. And also if you have really exotic hardware. So if you have something that just doesn't have a serial line, then it's going to be pretty hard. So let's get started. I'd now outline a few very general steps, how you, what you could do to get started with OpenQA. So my recommendation is find an existing, find a project that already uses it, preferably that uses it extensively and take a look around their QA department and how they and get essentially get your feed wet there. So if you are in, if you are Fedora contributor, go to Fedora QA. If you have an open suzer in the open suzer community open, open suzer is a heavy user of OpenQA. Once you've got your feed wet, set up a local instance. There's a link in the slides, how to do that. That will help you to run the tests locally, which makes things a whole lot simpler. Read the documentation. The official documentation is very extensive. It covers all the parts. It's in a few places a bit unstructured and hard to follow. And if you'd like a more guided introduction, you can watch a few tutorials that are up on YouTube. Bear in mind thought these are recordings of live streams. So they take a while, but they are really designed to guide you from the get go if you have nearly no experience with OpenQA itself. So recommended steps to actually get started and get going with OpenQA. So my recommendation is the first start out by modifying these needles that I use to mess with, to do the image recognition. And that's for the simple reason this is for most tests. That's the heart and soul of OpenQA. And so, and there's a few details that you should know about and most of them are really best learned by doing. So first start out by messing with needles. And this is especially really simple to do. You don't have to install anything on your machine. You can do that from the web UI. Then take a look at existing tests. So if your favorite Linux distribution is already using this then take a look which tests are failing and see if you can fix them, if you can improve them. So first modify single tests. Then once you're done with that, write your own tests. So think about what are certain areas of this OpenQA instance that are not yet covered. For example, you have a certain application that's not tested by OpenQA. And you would like to get it tested as well then write a test for that. Once you're done with that, take a look at the documentation, find out what so-called test groups and test suits are and create your own test group and learn how to schedule it. And this is probably already. So this once you're at this stage, you are already a pretty advanced user. And if you've got to this part, then maybe you're now at the stage where you want to start from scratch because you want to test a different project. And I must admit, this is unfortunately the part where there might be dragons since this really depends on the appliance that you want to test. And there's unfortunately no definite guide how to do that. But if you want to do tests with OpenQA that don't require OpenQA scheduler, there's been recently development that allows you to really just run ISO2 video inside a container. And so if you want to do just the image recognition part of OpenQA and don't really need the scheduler and all the parts around that, getting started with that is really simple. Just go to the link that's in the slides for the container-based setup. Otherwise, I'd say go to GitHub, grab the minimal example and start from take a look at that. It's really brief. Then assuming you followed the previous parts, schedule a few tests in the main entry point called main.pm. Hopefully you already learned a bit about job groups and test suits. If not, now is the time to start with that. So learn that part, try to really grok it and write your own job groups and test suits for your appliance. Yeah, then magic happens. And in the end, you profit. So sorry that this part is a bit hand wavy but this really depends on your specific needs. I'd now like to, since we're getting towards the end, I'd like to showcase one example and that's what OpenQA can do and that's bare metal testing of the Raspberry Pi. This has been done by Guillaume Gardé from ARM who has set this up for OpenSuser tumbleweed. And so you can see a schematic overview of how the setup work and how the setup is. So you have the OpenQA worker which is just some machine under, I think in Guillaume's basement and then your Raspberry Pi that you want to test. Now, the thing is you want to test, you need to give the Raspberry Pi a new image every time there's a new snapshot of OpenSuser tumbleweed. And the problem with that is you create, you need to create this, you need to get the Pi to boot from this new snapshot. Now, I think the Pi is sometimes, there is the option to boot from USB but this is buggy and hasn't really worked too well. So you rely on SD cards. Now you can't really put them into your computer and then run to the Pi, plug that in since that's not really automated. Unfortunately, there's a nice piece of hardware called the USB SD Max which is a multiplexer for SD card. So it's as effectively you have an SD card in two PCs at once and switch between them. So how this works is you have the worker first switches the SD card to itself, flashes the newest image onto the card, switches to the Pi, then the worker has a serial connection to communicate with the Pi. There's a ethernet connection so that they can talk to each other in practice so they can talk to each other over line and have a more and have a faster connection than just a serial line. And then very important thing, the worker can also connect the power state of the Pi. So really, if you are testing stuff, stuff can go wrong and you really, really want to be able, if all goes wrong and you can't shut down the system under test, you want to be able to cut its power. That's also why it's not so super simple to test to test cell phones and laptops with OpenQA because if you cut their power, they can survive for anything between a few hours and a few days. So, okay, so now to the test setup. So you just flash a new image, switch with the SD-Max, give the Pi some, give the Pi power, then the worker connects via the serial line, uses OS order in for that, and then all the tests are scheduled. So they, so first only the serial line is used and then you use SSH for that launch VNC and then do your usual jazz. Future additions to that would be also to test the HDMI output and use some capture cards. But unfortunately this is not yet there, but there's been people who reportedly have done this already. Now, all of this looks in practice like this. So this is Guillaume's pretty nice setup. What you can see here is the USB SD-Max, he has multiple Raspberry Pis in there. These are, as far as I remember, his OpenQA workers and you can see the LAN connection. And if you want to see a less professional setup, this is mine under my desk with just one single Pi and a whole, and a lot more messy mess around that. Well, and with that, a call for action. If OpenQA can help you, please use it. In my opinion, it's a really great project. It can make your life so much easier. Your QA people will love you because it makes their job much easier and they can really focus on the hard parts. So if you think it might be a fit, get in touch with us. You can reach contributors and users of OpenQA on IRC, on FreeNode, in OpenSuserFactory and in Fedora QA. And I'd like to also give you a few links. So in case you're further interested, you can find the source code of OpenQA in the OSOtterInst organization on GitHub. There's a link to the OpenQA homepage, the instance of OpenSuser of Fedora. And you can find the slides on GitHub there if you want a clickable version. The obligatory legal slide with all the copyrights. And with that, I would like to thank you for your attention and I'm open for questions. Right, thank you, Dan, that was a great presentation. We have a couple of questions. Well, we have one for now. What operation systems does OpenQA support? Any operating system. So it's mainly used for Linux. That's where it's main users are Linux distribution. So Fedora, OpenSuser, but you can use it to test Windows as well. So actually the OpenSuser OpenQA instance has a test that installs Windows, installs WSL, installs WSL, loads the OpenSuser WSL image and then tests that the WSL image actually works. So it's really operating system agnostic. OpenQA itself runs, I think, only on Linux, but it can test anything else. So as long as you get a serial line to the system and in case of a VM that's really trivial, you can test BSD, you could test Mac OS, you could test, in theory, also Android. Right, and I have a question. So does it integrate with Selenium? No, not at all. Test, okay. It has, I wouldn't know what it has to do with Selenium. I mean, Selenium is a test framework for webpages. Yeah, yeah, that's the use case that I was thinking of to use the image recognition in testing UI of webpages. You could do that, but that's not going to be, that's not really the best use case for that, since the thing is for, I mean, I'm not extremely familiar with Selenium, but as far as I know, Selenium really interacts with the DOM of your web page, but OpenQA, it expects a video feed. So you will have an additional piece in between and that's your web browser. So you could, of course, launch or create yourself an image with Firefox and Chromium and all other browsers that you care about until OpenQA to navigate to your web page and then do image recognition on that part, but you will have the additional troublemaker, which is the web browser. And since web browsers render stuff differently, it might be problematic. So then suddenly the image recognition, it might work in certain cases, but it will probably be more work than actually help.