 Can you guys hear me in the back? Awesome. So good afternoon all. Thank you for attending this talk, and thank you for turning out in a large number, really encouraging. My name is Yash Mankad. I'm a software engineer in the visualization team for Red Hat. I work out of the Westford office. And my co-president today is Lukash. He's also a software engineer in the visualization team. He works remote for the Brno office. And today, we are going to be talking about Avocado and Jenkins test automation and continuous integration. So what are we going to learn today? I'm going to cover the development process followed by the liberal and QMU-KVM communities and the problems faced by the development team in a way to support the process. We are going to talk in particular about the patch handling of patches, the handoffs between the development and the quality engineering groups, and some of the issues with the test automation efforts within the team. We are then going to discuss how we are using Jenkins to address some of those issues. And the approach we took in integrating the test automation efforts within the CI setup within the development process and the challenges we faced in doing the same. I then have a small demo where I'm going to show you how all of these things work together. Lukash will then take over and he's going to be talking about the basics of Avocado and the developments that have been made over the last few months. And even he has a small demo lined up for you guys. So let's begin. Let's begin by discussing a general overview of the development process followed by several open source projects, but Libert and QMU-KVM in particular. So say you are a developer who's found a bug. What do you usually do? If you know how to fix a bug, you write a patch and you send a mail to an upstream mailing list. The mailing list then sort of moves the patch to a patch queue, which in simple terms is just a text file containing all the list of patches that have come in, say, in a given time frame. The patches are then moved from the patch queue and they are applied to the source code and a build is run. Now the reason for running the build is that we want to make sure that any changes introduced by your patch or the developer's patch does not break any existing features in the code. If the build is successful, it is then moved on to quality engineering for further testing. Well, seems simple enough, then why are we changing it? Let's look at some of the issues in the same workflow. Now the developers need to follow a very strict guideline when they are submitting patches to a mailing list. These might be in the form of running diffs or writing extensive descriptions or taking part in code reviews. And a similar set of guidelines applies to patch queues. Now if these guidelines are not followed, then they might result in either build failure or faulty builds, which adds more time to the process. Now on top of this, the mailing list is usually administered by a group of maintainers selected by the community. And in most of the cases, it is the maintainer's job to manually move the patches from the mailing list to a patch queue, apply the patches, and then run builds. This manual intervention in the development process adds further delay to an already sort of a slow process. Another major drawback of the process is the handoff between development and QE. Now in most major organizations, development and QE are treated as two completely separate functional groups. So they have their own schedules, their own testing guidelines, own deadlines to submit stuff, which results in the fact that when a developer has submitted a patch, it might take weeks for QE to test it and get back with the results, which means by the time the developer has the QE results, he has long moved on to work on something else. All of this results in really slow turnaround times on tests. Moving on from QE to test automation. Now the QE team that supports our development group has a testing framework that is based on the autotest framework. For people in the room who do not know what autotest is, it's an open-source testing framework for the kernel. However, autotest is no longer maintained upstream, which means that if something breaks within the QE testing framework, there is no way to support all the wide variety of tools that they are currently running on top of the framework. So these were most of the issues, the major issues. So how did we approach the solution? We wanted to reduce the turnaround times and get faster results back in the hands of the developers. So we wrote a script that automates the initial stages of the development process once a patch is submitted. This script parses the mailing list. It gets the patches into a patch queue, applies the patches, runs a build. And if the builds break, the developers can go and have a look at the builds, build logs to see what went wrong. Was it an issue in the patch or was it an issue somewhere in the workflow? Now this script is very similar to FamZang's patch queue, which is an upstream tool which is available for QMU patch testing the link for it can be seen on the slide. We then moved our focus to the handoff between development and QE. And that's where Jenkins comes in. As you've seen in several talks yesterday and probably today as well, Jenkins is a very versatile tool. So the way that we have it set up is that we have a Jenkins master slave that is responsible for getting the latest RPMs, installing the repos and running tests on top of it. So given the script that we already have, in theory, the way that our CI is set up is that the script takes care of patch handling. It gets the patches, applies the patches and runs the build which triggers the CI to install the latest version, run the test, hand it off to QE and generate all sorts of results and reports. So where does Avocado fit in? Because I haven't talked about it at all. So I know Lucas is going to cover more about it in detail, but to put it simply Avocado is based on Autotest and Autotest is the framework that is used by our QE team. So if we use a testing tool that is used by our QE team in the development process, two things happen. The first thing is that if the test passes, the developers have get very good reliable feedback and they can have enough confidence in their patch or their code that if it passes the initial stages of the same testing framework, it might go on all the way and pass QE as well. And the second thing is that it takes some of the burden off of QE's hands. Now as most of you in the room might know, QE has three different tiers of testing. There's the unit and the sanity test, there's the functional and the acceptance test and there's the regression tests. So if as a development team, we can even perform just the sanity test at our end, it takes at least tens of hours of testing time from QE and well, they have more time to run more tests. So as of now, all of our Jenkins slave, they are only x86 based, but we would eventually like to include PowerPC and ARM as well. So now, I would actually like to show you the scripts that are responsible for generating our Jenkins jobs. So this is a very small script that we run on our master, which is responsible for creating the jobs. Now, we see that we have a very small script that we run on our master, which is responsible for creating the jobs. We switch to the user Jenkins and we run a script called create our core jobs and we pass it, let's look at the only four files that four major files that we pass it. We pass it a list of tests that we wanted to run. This is the name of the tab that will be created in Jenkins and we pass it to Git repose. So now let's look at the test list that we pass it. Now, I'm just gonna scroll through it quickly and you can glance at the names of the file. So there is CPU, there's destroy, there's all the VM tests, there's migrate, then there's network, netflow, nodes, pools. So the way that it works is that this is the main script that takes care of everything. I know it's a little large, just bear with me here. I'm not gonna explain the entire script but just the major part. Now, this is the main XML part that creates the master job which triggers all the other jobs. And if, so this is responsible for things like when the test will, when the jobs will be triggered. So this is very similar to a cron tab when you pass it a time of when you want the test to run. So I'm gonna switch to this view where I have the main job open. So this master job is basically responsible to create a job that's called word test upstream and all the fields that you can see here are filled in by the XML. So you can see here, this is the, this trigger, see this is a trigger tag which basically populates this field over here. Similarly, we have fields below which populate this list of tests to run which is generated again using this list. Then, oh, sorry. Then the build job, I'm just gonna highlight a few things has the Git plugins and it is responsible for things like removing any old installations of LibWord and getting the new RPMs and installing the latest versions. So this is the job that is created. So it has two Git repos where it fetches the latest builds from and as you can see, the first shell removes the any old installations of LibWord and the next install the latest LibWord. Yeah, sorry. Yes, so that's a good question and these scripts existed before we started using Jenkins for all of our products. So the LibWord project had a CI which was already using the scripts and we wanted to just integrate our efforts into the same. So we did not want to remove everything and then start a new. So we just modify the scripts to have our Cardo as well and this is sort of what I'm covering right now. So the test job basically takes care of that. If you look at the command tag that runs the Avocado jobs. So here in this place, we were running auto test and word test before we sort of migrated to Avocado. So that's how it was, yeah, XMLs. So eventually we are gonna move to Jenkins or Builder but this is how it was set up. So we've just edited it right now. I know, I agree with you, that's true. So yeah, oops, sorry. All right, so I'm gonna now move on to the few of the challenges that we faced in our team. As I said, the LibWord project already had an existing CI which needed to be upgraded and integrated with Avocado without affecting any features that were already there. And so we decided to stick with the XML scripts that were there and the same CI supports both upstream and downstream for now. So we will move to Jenkins or Builder in the future. I can't really say when but it is in the pipeline and the script that is used, it had to be modified for the setup for the KVMCI because that project operates in a slightly different way upstream and downstream as compared to LibWord. And the migration from Autodesk to Avocado has been a tricky one because Avocado is developing really quickly and we had to sort of catch up with it. So that's taking time as well. And I would now like to pass it on to Lukash who's gonna talk about Avocado. Okay, I hope we hear me. So why Avocado? Well, because for us it was like the only possible choice for virtualization as it inherited all VIRTEST tests which contains like 40,000 QEMU and almost 60,000 LibWord tests. It has support for three major CPU architectures, lots of versions of operating systems and like almost 900 hardware variants like different kinds of hardware which gives you this humongous number of theoretical combination you can run. I'm sorry. So that's basically why we use that but why you should maybe consider using Avocado too or what else is there. So basically it talks to Jenkins. You already saw the results from Yash. But what else? One thing is that unlike AutoTest, it's quite simple to set up because we provide for Federa and RHEL, we provide YAM repositories. We also supply support Ubuntu and you can download, it's a Python project so you can basically just download the sources and install it on your system. So it's just as simple as YAM install Avocado and hopefully it downloads. And now I should be able to run Avocado without any previous knowledge. So Avocado, oh, you don't see this one, yeah. Avocado run, let's try running something like I'm here for a first time. So let's try to run something and voila, it works. So no need to set anything. So it's basically as simple that even our developers, not QA, but even some developers are running it or displaying with it because it gives them some additional features like basically the tests. But we hope to give them other things too. You'll see that later. As you can see, you can run any arbitrary binary. How it works, let's try something harder. Yeah, it works basically the way that if it returns zero then it passed, if there's none zero, it failed. This is when you use any arbitrary file. We also support native test format which is inherited from a unit test, Python unit test. So if you're familiar with Python unit tests, you probably know how to write tests for us. And you get some additional features. I'll show you some of them soon. So let's extend this script. Let's try something more complex like echo red hat. Maybe it'll work. I also have Avocado example tests installed so I can simply run some tests. I think that's enough, right? So you can see different stages. Oh, we also have support for errors, but I don't think you need to see it. So you can see even just by running this you see the results immediately nice and shiny with some overall results. And there is a link to some HTTP page. So let's have a look at that. Oh, I can't have a look at that because I'm inside VM, sorry. So let me run it again on my computer. So there are some results. This is basically not for QA or continuous integration. This is basically for developers or people like you who just want to run tests. You see again the results. You can even see the reason of why the test failed or passed over here. And you have the links to each test debug log. So let's have a look at the red hat. What it dot did was basically, yes, I'm running echo command and it logs the standard output. Pretty simple, right? So what else do you have? And this is actually quite important. I run this job today. But do I know what kernel I run? Yes, I know it. But do I know it in a week, in a month? When I, for example, discovered that these results were quite important for me, but I forgot to log what was I running. So what we do is we log some system informations before and after each job, optionally even before each job, each test. And I can have a look at what is my kernel or what packages do I have installed and everything. It's configurable quite simply and it helps to identify problems later. If I take a look at the results itself without the HTML, you can see that there is a HTML output. There is an ID file, which is important concept, because each test in Avocado gets this ID. We do have support for our Avocado server part, where you can push and pull from results. So an easy way to share results and do some data mining on the server because you run the test on your machine, but you want to gather the results somewhere. Again, it's optional. You get the overall job log, which collects lots of information and data. Then you have job reply directory, which looks like all the pieces needed to rerun this job later, including configuration and parameters. Then you have JSON and XML results, which are basically human readable results, or not human, I'm sorry, machine readable results. And the SysInfo, as I said before, we have pre-post and profile, which are basically any code which run throughout the execution, like profilers to see, for example, free memory during the execution. Then we have the test results structure, where you get, again, some files which are given, and it's the debug log, output and standard error of all executed binaries. So if you run some binaries inside your test, it runs them and logs the standard output and standard error. You can see redhead here. And we do have a whiteboard concept, which is basically for you. You just write there anything you want. It's used by performance team to store data for later data mining and some post-processes. So this is very quickly the result structure, which is quite important because it's given to you and you can rely on it. You'll keep it in since all the time and you can build scripts on top of them. But let's have a look at the features of Avocado itself. So we do support some sub-commands. I can't cover everything here. And it's not even intended. But let's have at least what it can provide to you. I mean, yes, you can run it in CI, but the thing is, if you run it in CI and you discover a problem, you probably want to run it on your computer. And there are some handy things we came up with. So basically the usage is simple. You just add some positional arguments and it runs them as tests. We do have some dry run, optional arguments like dry run, if you just want to see what parameters will be used. Job timeout to avoid hangs. We have some output formats like silent, which is not that much useful, but you can produce, for example, Jason run Avocado from your scripts and parse the output from standard output and react on it. We do have output check, record and check support because we saw many programs, many like tests only, okay, run this and check whether the standard output change or standard error. So you get a native support for that. You just run a check record output and the next time it always checks whether the output changed. As I told you before, we have various types of tests, like the simple test or the native test. We also have external runner, which is again a great concept if you already have some binary which calls your tests. This was actually designed for KVM unit test, which is a script which by default runs lots of tests and gives you true, false, whether some of them failed or passed. Okay, this is like overall result, but if you run the same script and an argument with a test name or ID, it runs just a single job. So what you can do with Avocado is provide this external runner and then all arguments become arguments of that binary, which means you run these tests individually and you get these results individually, even in the CI or anywhere else you may want to with some additional options. Multiplexer, this is a key concept and if anything you want to take from this presentation is that you should take a look at the multiplexer. As a way to specify parameters to your tests, you may know like metrics of tests or sparse metrics in some way, like Jenkins has it or other frameworks, but I haven't seen something like this. It comes actually from Cartesian config. Now we created even better way with the multiplexer. We don't just specify axis and filters, but we specify a tree structure and say that you want to multiplex these children's and then you create another, like it's recursive so you can create another layer. So you define the filters by creating the structure and it's actually really human readable. So check this out on our pages and either reuse our code. We are very happy to share with you or use Avocado. Another thing not for CI, but probably for developers is that you can easily execute tests on a remote machine, which means it SSH to that machine, it copies the test from your local copy there, runs them and reports results locally to your computer, including liver support so we can actually warm up with the machine and run it for you. And another neat concept, again, this is something we want to add to developers so they use the same tool as QA is wrapper and GDB support. What I mean by that is that you run a test which executes a binary and it interacts with this binary, but you know that there may be some problem in some areas so we just say, okay, run the test, but this time run this binary, for example, QMKVM inside GDB server and set these breakpoints or we can actually supply a script so set like various number of breakpoints and it runs the test, it interacts with the QMU and once the breakpoint is reached, it notifies you on the screen and gives you command how to access the GDB and you can interact with it, you can do your work, you can modify the code, you modify the variables or see the variables and return back while the test is running, which means it's still interacting with your machine. So this is pretty neat feature and last feature I want to talk about is job reply and that's basically what we expect, you just specify the job ID, the thing is that if you specify that like in a month and like all configuration changed and everything changed, it still remembers everything, including parameters and configuration and you can of course specify just, okay, I just want to replay that but replay only the parts which failed so I can see, I don't need to execute everything or you can replay everything but change the parameters of course. That's very briefly about avocado, there are like more supports like in-test support for GDB without the need to like do manually the pipes and everything, so we can meet actually afterwards if you want to, if you're interested. Anyway, now it's time for questions, so Yash or me, yes, if you talk about this, yeah. We explored the possibilities to run it on Windows if you talk about this, there are like three or four modules which does not work so you can't run it but it's not that hard to make it work, it is just that we don't need it right now but if that changes, we can actually make it work. Yeah, that in the background, in the back, yeah. Not right now, we had a request for something like that but we don't support it right now. The way we solve, I solved it for example, I'm running my small CI is that I use the pipelines in Jenkins Builder, like not Jenkins Builder but in Jenkins, sorry. Yeah, and there was another question, yes. Our expectation is that you can do actually whatever you want the thing is if you write it in PY unit test or just simple test and anything else, you don't get all the benefits so you can't actually use for example a GTB run binary because we don't know when you run this but you can use it as a runner or you can use it as a tool so you can use our tests. So eventually you should move to our format and we provide for example, even like wrapper for services so you don't need to care whether you have system D or init five or whether you have RPMs or Upt, yum or Upt. So yes, eventually you may want to invest the time and learn about our native format. The point is this is inspired from VIRTEST of course and well over the years it was hard to set the VIRTEST but then Lukas wrote a small script which was just okay, run this and it run the VIRTEST and ever since then we got some contribution from actual developers even like I had slides for different conference and I put 13 most active members of contributors of Avocado VT but it's inherited from VIRTEST and there is Eduardo who is developer and not QE or any advisor and Paolo also like contributes a lot and others like because it's very easy to start with that was our point, it has to be easy to start with so not only QE uses it. A lot of Autodesk users showed up and their main concern was that with Autodesk you had to deploy the server and then the client and then you had to run the test which was the issue. So with Avocado you can just do yum install Avocado and run the test and that's what is all about this because if you're a developer you don't need a big setup like QA you can just do yum install and run your test and the test is supported by QE so when it passes here, it passes here so it's just making testing easier for developers. Yeah, let me just boot a virtual machine that's as simple as this. Oh, okay, it's not as simple as that, nevermind. Oh yeah, sorry, that's Libert issue. It changed my permissions. Yes, it changed my permissions, sorry. But I can show you, for example, a GDB support. I run a double free binary which maybe does something nasty based on the name. Like feel free to ask questions, I'm just showing this. So this is how actually the GDB feature works like, it breaks and I can go there, I can list the code and you can see, okay, it does double free. Oh yeah, so we can see that it actually does double free so I can actually skip the next three and exit it normally. Oh, sorry, then I need to disconnect and it proceeds and it errors because normally it should fail, right? But the binary did not fail, that's the point actually. If I did not skip the second free, it would have passed. What do you mean by that? I mean, I shown you the results here and if I go upper, then there are our results. Yes, in hard disk locally and then you have avocado push pool where you can push them to the center of avocado server or basically we say it's easier to send it by email if he wants to or report in Baxilla. We left this up to the users because we do have the server sub part where you can push it, pull it, but apart from that, we believe that there are other software like you can use whatever you want, right? It just files. I mean, we're running job because, I mean, job is the, but what gives you the list? Like I can run like, win through, oh, run. Okay, I specify them as arguments. Yes, okay, well, I do it in Jenkins, for example. Developers or, okay, I think I know what we mean. So when I look at these results, you want to know what was executed. So you look at the job log and there is the command line and you see that I run PowerPC64 boot, for example. Yeah, probably I'm missing something, sorry. So any other questions? Yes? Are you going to have a stable version of the Dora? I'm glad that that's all tracking a buffer or whatever. That's a good point. Yes, we planned that, definitely. The thing is that right now it's, you know, it's not the early stage, but it's still quite, it's still not really 1.0. Let me put it this way. I mean, I'm running it for more than half a year in CI without me to change anything. If you keep it since the master branch, it usually works. And we release everything three to four weeks, a new version. And again, if you keep the same version in since and you don't use plugin, all the plugin and newer version of the runner, it should work. We improved the CI, like Autotest was broken from time to time, we improved the CI. So it should not happen that often. But it's still not 1.0 version. Yes. Okay, so if you want to meet us, we can meet outside of this room.