 All right folks, I guess we can get started. It's nearly the top of the hour now So thanks y'all for coming. Just gonna start my timer here. So I know where I'm up to Yeah, so a little bit about me. I'm Adam Williamson I'm a senior principal software quality engineer, which means I test stuff. They keep making the job titles more and more fancy I've been a red hat since 2009. So it's been quite a long time Always working on QA. So I've kind of seen the the evolution that I'm gonna talk about in this talk from Very basic manual testing all the way up to what we do right now, which we're gonna gonna talk about I just want to do a thing. I normally do which is kind of like an audience Census because I like to know who I'm talking to and what you're here for so Who runs linux on the desktop doesn't have to be fedora just people who run Linux Okay, most of you that's good. I like that Who knows how to make a linux distribution who's worked on one knows how that works. Okay, pretty good advanced Class that's good because it means I don't have to spend 15 minutes explaining great Yeah, and for Questions if anyone has questions during the talk, you know, if you have a quick question Go ahead and put a hand up and I can take it in flight if you're gonna Keep talking for 30 seconds and I have to answer for five minutes. Maybe hold it to the end and we'll hope we have enough time Great. So So, yeah, I will just cover this briefly for the folks who don't know about it and it's kind of a preamble to the talk So a lot of how we test an operating system Involves how we make an operating system. So how do we make fedora? Very very simplified this is leaving out a lot of stuff We build packages which are you know all the bits of a bit of software that you might need to have on your system And this is done by a system called Koji and then We bundle packages into things called updates Which may contain one or may contain more than one package And we run them through a system called Bodhi which Combines those updates and provides a web UI where you know feedback can happen And then we bundle the packages that have been through Koji and Bodhi into things called composers Which is you know, we do these daily more or less which are Everything like their repositories with all of the package in them and the images you can download and install on your own system So when you go and grab a fedora image, it has been produced as part of a compose And the system that handles composers is called Pungi There's a little difference which will come up against later between development and stable Here so we have Gauhide, which is our development branch of fedora and things happen a little differently there But I'll take that as we go along Something I'm gonna kind of skip over a little bit during the talk But to be fair I'll mention is that there's a lot of inputs to a fedora You know building an entire fedora compose that aren't packages There's things like the configuration of the compose system itself There's things like the lists that make up what packages are in what group and I'll mention this again later But that's kind of an area Which we don't test so much right now and that's like an area I'm looking at improving on for the future. So we'll come up against that again later So that's how we build an operating system, how do we test it? Basically all those levels that I mentioned briefly about how we build an operating system testing can and does happen at all of them So the packaging in fedora these days I believe in most distributions Packages are in our under source control. They're in a git repository So your spec the spec file, which is like the recipe for building the package and all the sources lives in a git repository And this git repository is behind like a forge interface, you know something like github or git lab The one we use is called pager, but that doesn't really matter. The point is that provides an integration point where the Testing can happen and feedback can reach the package at that point at the point where they put the package into source control There can be testing So you you know you can have a pull request for a package and testing can happen at that point We can have testing at the level of the build not testing at the level of the individual change to the package repository But when you do an official build through koji, you know signals fly and there can be testing at that point We can do testing at the level of the update So when you create an update, which is you know a bundle of updates and an intent that this set of packages Should go here at this time testing can happen at that point And finally we can do testing at the level of the compose So when a compose is the whole thing is spit out Then we can take that entire thing and we run tests on the compose itself so This is kind of going from small to large. I guess you can see right There's it makes sense to do different tests at different levels And if anyone's familiar with the concepts of you know different types of testing, you know unit testing integration testing end-to-end testing We're probably going to be doing you know unit style testing at the level of disk it And then all the way down on compose testing what we're doing there is really like functional testing And I'll go into the details of that later, but that's you know, that's part of the the theory here so The what as I said, I'm the team lead for fedora qa So it's part of my job to kind of figure What testing should we do when like what should we do with the results? How much testing should we do? um, and this is like a constant challenge because The fundamental challenge for anyone doing this like not just as a fedora, but any other linux distribution Even any other operating system is operating systems are huge You know, it's a gigantic thing that you can do thousands and thousands of different things on and you can install it on thousands and thousands of different configurations And I have a team of about eight people and we release fedora every six months So this is it's a difficult circle to square. You're never going to be able to test everything so as we're going through the talk this is like a constant challenge in the back of my mind is like Focusing our resources on Providing, you know the most testing the most useful testing we can within the the constraints we're operating it And of course this makes automation incredibly important for us because As I said back when I came on board around 2009 We didn't do any automated testing all the testing we did was manual and That obviously means you're covering a tiny tiny tiny sliver of what you can do So the more we can automate it's it's a huge force multiplier for us Which will become clear as we go along So we're always looking to automate things because it's it's hugely important to us so automated test systems we have those now I'll try not to get too deep into the background, but So maybe the first three or four years. I was working on fedora. It was 100 percent manual testing and It it was kind of hell like my life was waking up and running a vm eight times in a row and doing a very slightly different Install test on it and then going to bed and doing the same thing the next day So this you know it we could feel that it wasn't efficient and it wasn't any fun for us So we've always had this strong motivation to to automate things Um and a long time ago. We had this project called auto qa, which was supposed to You know solve this problem and that never quite got there in the end but instead we Develop these two systems which i'm going to introduce you to as we go along So the first one i'm going to talk about is open qa I should say off the top open qa is not developed by red hat or by fedora Open carry was originally developed by sues. I don't know if we have any sues folks here But they're around at the conference. They're great. You should go talk to them So the way this happened is While fedora was still kind of officially working on auto qa one of the sues folks As part of a sues hack week spun up open qa and had it testing fedora Just as a kind of a proof of concept that hey this test system we have can test other things So one of the folks from my team saw this and said hey, this is actually we could use this Why don't we use this so we took it and we ran with it and now it's one of our key test systems So open qa back on that chart from a couple of slides ago Open qa focuses on update testing and compose testing. It's kind of down the bottom of the chart It's very much about functional testing open qa The way we use it it's developed more capabilities now But it's original core and the way we still mainly use it in fedora. It's kind of like an extremely stupid intern Um That's sitting in front of a real computer because to use open qa you spin up a virtual machine And you tell open qa to type things Look at the screen and click on things technically what it's doing in the background is it's inputting over vnc and the serial console But really the way it looks is you tell it Okay, look for the certain thing on the screen and click on it or type something into a console and expect a certain result So this is really good for testing a distribution because this is the kind of thing we have to do a lot of So If you think about automated testing depending on how familiar you are with it If you look at it from the perspective of the small software project, you're often going to be running, you know through a very code linked test framework something like pie test which is It doesn't work like this at all and those kind that kind of testing framework isn't great If what you want to do is test that your operating system boots Or test that your operating system shows a desktop and your installer works. This is the perfect system for that So that's what we use open qa for um The weakness of open qa is that it isn't Um a traditional kind of automated test framework you can make it be one And we occasionally do that but it's kind of ugly and it's very inefficient because it's always spinning up virtual machines um And it's it's very slow if you just if really all you want to do is you're going to program and see what the output from it is You wouldn't use open qa for that What is on my next slide here? Yeah, so what do we actually do with open qa? um So we have this thing in fedora called the release validation test suite Which is the big list of tests that we Look at to decide whether fedora is ready to release So we've got one of these composers I mentioned before we're thinking about making it the final fedora 38 release or whatever And we say well is it good enough and the way we do that is we have this list of you know a couple hundred tests That are required to pass And in the old days we had to run all of those tests manually for every single composer We would run those tests and you know put the results in a wiki page Open qa now does about 75 of those tests So it does things like run an install on butterfs run an install on ext4 run an install where you take an existing File system and make it smaller so you can put fedora next to it run a dual boot install with another You know another instance of fedora It tests things like Various gnome desktop applications. Are they working it tests whether you can log out from the system log back into the system All of these very kind of boring Basic functionality of the system tests it does those We also use it for some kind of more complicated tests of more advanced features We have free ipa, which is It's kind of like active directory, which is a feature that Is a key feature of fedora server and that is The thing with testing free ipa is you need a little farm of two or three machines You need a server you need a replica server and you need a client And we also test like a it's web interface So open qa turns out to be quite good at that because it's it's capable of running like a little farm of vms That all talk to each other that are all on the same network segment and because it has the graphical testing capabilities We can load up the web UI and make sure that works So it's really good for that And we also use it for testing, you know, there's database server work. Can you enroll against it? Uh, yeah, so it's it's it's pretty comprehensive the test that it covers Um So at some point we initially we were at open qa exclusively for compose testing So one of the big advances we made Several years ago at this point is at some point I realized, okay We could take like a subset of these tests and we could run them on updates So those are the little bundles of package builds that are expected to go stable And I thought, you know, this would be kind of fun This was almost like a little sideline at first and that's become one of the most important thing open qa does We now run about a third of the total test suite on every critical path update for any branch of fedora So critical path is this concept of basically important packages There's a you know, there's a list of packages that are considered really important And those and any dependencies of them are critical path So any update to any of those packages gets, you know, a subset of open qa tests run against it And if any of those tests fails, then the update cannot go stable. So this is one of the First points really where we had automated tests automatically preventing broken stuff getting into the distribution um The that testing also Does a thing where it it kind of Builds a mini compose I taught open qa to Replicate what the release engineering process does to build a live image and to build a network installer image And these days to also to build a silver blue, which is our um our immutable distribution to build a silver blue installer image So it builds those Which makes sure that we haven't the update doesn't break building those images Which is very important and then it tests that they install and the basic functionality works on them so in this way we Rarely get major failures in the compose tests anymore before we started doing the update testing We would often get a new compose and you know Half the test would fail and these days that very rarely happens because We caught the failure on the update before it got into the compose it never got in so the composers are usually You might get you know 30 failures on something that isn't covered by the update tests But we rarely get a completely broken compose anymore Um, so the scale of our open qa deployment we have two instances We have a production and a staging deployment This is all run on hardware as well. It's a fun thing about open qa The server end of it is running on a vm, but all the actual tests Running on these hardware machines, which is fun to maintain Right now we can run over 100 simultaneous tests in the system across three architectures x86 64 a arch 64 and power pc 64 Believe it or not And so far we've run over three million tests. So it's it's it's pretty pretty good for you know a tiny little distribution project Um, I keep track of all the bugs that have been caught by open qa with a little whiteboard tag in the fedora bugzilla And it's over hundreds at this point. It's I think for 500 and they're always important bugs if they're coming from open qa So it's it's been pretty significant uh So, uh, just a little bit about, you know, how open qa really worked in distribution Um, you'll this part is kind of a contrast to the system. We'll talk about next open qa is like It's not this again. This is a difference from how you'd use a ci system maybe in an upstream software project where The developer might need to sort of you send your pull request and if it fails Then you've got to figure out why your pull request failed the test and fixed it In open qa, it's more like my life now instead of waking up and running seven tests manually is more like I wake up And I look at the open qa front page and if there are any failures I go and look at them And I try and figure out why it's broken and either I fix it myself or I submit a bug report to whoever broke it so With open qa, it's it's kind of this is a service that the qa team provides to the people who are actually building the distribution We test it and if it's broken will help you figure out why Um So these are some con this is mainly for people who are actually no fedora people But this is the way you can get in touch with us And the upstream development site for it And if you're more interested in the specifics of how open qa works There's a page at the bottom there these slides are all available on the conference website If you want to check any of these resources later So the other major system that we use to test fedora is called fedora ci And hence we get to the original title of this talk which involved ci Uh, so a slight sort of digression on ci I I I have trouble with the term ci because a few years ago when it was you know the buzzy thing There was a big fad at red hat for ci and people were constantly coming to us and saying so How about ci for fedora and we're like do you understand what that means? Like you think about the ci workflow on the simple software project where yeah, you know You send the pull request and some tests get run on it and Then you try and do that at the scale of the distribution where you have thousands of packages And you have all of this plumbing in the background, which anytime any of that changes It can prevent the the output working and building a fedora compose a full one takes six hours So we can't really do that every time anyone commits anything to To a package repository. So this is why the original title of my talk was that we're not really there yet Because doing ci full on ci for a linux distribution is incredibly difficult and I don't believe anyone is there yet We do have this system called fedora ci which If we go back to the the list of you know levels where you can do testing fedora ci is much more focused on the package level of testing And the even the repository level of testing, you know the the disk it repositories for the packages And fedora ci is is it looks much more like the kind of ci system you may be familiar with It in the background it's running on zool and a bunch of other stuff But the way it's really expecting to run is yeah, you make a change to a package It will run tests on it and it will communicate to you the results of that So it runs mainly tests that are defined In package repositories, which is different from open qa, which has its own centralized set of predefined tests So anyone who maintains a fedora package can define some tests that should be run on it Within the the package the repository for that package and they can say when those tests should be run and they can also Choose for their for updates to that package to be gated on those tests So this is much more of a self-service kind of system Yeah, it Does a whole test metadata format called tmt which you know, it's very comprehensive and Again different from open qa Which kind of does all of its own provisioning and setup and so on for running the tests in tmt Your in fedora ci your your test description sort of tells the system how to do all of this It tells it you can tell it how many you know systems you want to provision Well that containers usually you can tell it, you know what version of whatever you want to be on there what Stuff needs to be pre-installed before your test runs and how your test is actually going to run How it's going to report But there are defaults for all of these things so you don't actually have to define everything from scratch Testing farm is kind of a concept Which is trying to unify a lot of the there's a lot of moving parts behind the implementation of fedora ci And we're trying to unify those across rel sento stream fedora And packet which is packet as a system for if you maintain both the upstream application and the fedora package for it It tries to make your life easier by letting you keep the the spec file the recipe for the package in the upstream distribution And you can integrate the whole process of building a package Testing it landing it in the downstream distribution You can have all of that integrated into your upstream development workflow Which is kind of a cool thing if you if you're in that position And we're trying to kind of integrate all of these things because in you know in the background We kind of had a lot of different instances of zoo and other bits that make up a ci system Which we're essentially doing the same thing and we're trying to kind of consolidate them all into one Which would also be nice because then you could use the same you know test definition across all of these branches of red hat related products um Fedora ci also has this thing that it Kind of inherited from auto qa where it runs a small amount of generic tests on all package builds So things like tends to be like rpm sanity tests like are the dependencies okay? Are there any obvious errors in the package? It runs those on all builds and files the reports for those And it does this for any pull requests that anyone files to a fedora package So not all packages actually use the pull request workflow Because some pack a lot of packages are maintained by one person who doesn't really want to run through a forge for everything But some packages are collaboratively maintained and they use pull requests a lot So if you do that, then you get free testing with all of your pull requests So yeah, that's kind of The reason we have both fedora ci and open qa is that they do really different things basically Fedora ci is lower level more developer aimed at kind of developers helping themselves with their own workflow Open qa is much more where we the qa team come in and say is your stuff working? And if not, please fix it And you could make Open qa do a lot of the stuff fedora ci does but it would be more Much less efficient and there's really no reason to and you could write your own You know look at the screen and type stuff engine for fedora ci But why would you do that when someone's already made open qa? So we've just kind of naturally evolved to this state where we have both things There are Yeah, so we can cover that here there are kind of integration points between the systems like when you're doing testing like this It's great to test things But it's useless to test things if you don't do anything with the results This is something that i've learned like i've i've encountered test systems where They run 2,000 tests a day and 500 of them fail and no one ever does anything about it And what what's the point of that it's not getting you anywhere So i'm very kind of keen on i like test systems where Everything passes and if there's a failure Somebody really cares and fixes it right away like i never like when i look at open qa and i see a single fail test That that doesn't make me happy So you need the results to be They need to be very clear to the people who are able to take action on those results So as i mentioned for me personally i use the open qa web UI all the time That's how i look at test results, but if you're a fedora packageer It's not really useful to you. You don't know how to navigate it. That's not a natural entry point to you So for that reason we have Whole systems to kind of deliver results to other places um Obviously the best thing you can do with results Ideally is not let things that fail through which we call gating Um, and this is definitely something we've innovated a lot on over the years When we first Built open qa and fedora ci everything was completely advisory And in the early days i kind of had this idea that we would gait the composers So if the composer fedora failed a certain amount of tests or certain important tests We wouldn't sink it out to mirrors and an interesting thing that's happened over time is that we've kind of reached a point Where we don't need to do that because instead we gated the updates So the bad changes and as i mentioned never make it to the composers and the composers are never bad enough That we don't need to sink them out. So we kind of discovered over time that the update Was the perfect, you know gating point for keeping bad stuff out um The contrast to gating is validation testing Which is where you kind of take a thing and you you let it be built and then you manually test it and decide Whether it's good enough to ship Yeah, so The really great thing that we found with gating as we've gotten increasingly into it is that you if you don't do it then something that's broken gets in and Everything is broken after that So if you let an update through which breaks a certain thing then every time you test after that that thing is Going to fail It doesn't matter what you're testing It doesn't matter if the thing you're testing broke that and it wasn't the thing you're testing that broke that But you're still going to get a test failure If you test early and you gate then you catch those tests and you catch those failures right away You keep them out and they don't pollute your other tests your other results later So that's something that we've been kind of focusing a lot on So the difficulties with gating um A capacity I mentioned earlier we test critical path updates in open qa Why don't we test all updates because we don't have the capacity for it? Honestly, that is why I'm all of these tests are running on Machines that have basically been handed down to me from the build system. None of them are in warranty anymore This is as much testing as we have the capacity to do So that's obviously, you know, that's a limitation that we have to deal with Reliability you can't gate on a test that's not reliable people hate it If people are submitting updates and we're saying no, you can't push your update out because this test failed and people are like, well There's nothing wrong. The test was just broken. You can't do that You need your test to be extremely reliable if you want to gate on them um and then As I mentioned earlier, we have this problem where a lot of there are a lot of inputs to producing a fedora compose and sometimes the test will fail not because of a problem in the thing that's being tested But because someone made a bad change to something we don't test So occasionally we have a situation where the test that builds a workstation image fails because somebody went and changed The recipe for building a workstation image, which right now we can't suck into the test system So the first failure we see is just for whatever package happens to be built after that And then we have to I basically have to go back and say, okay This doesn't look like this package introduced it. It must be this other change that introduced it. So that's something we're constantly having to deal with Yeah, so these are the ways that we Communicate results. So this is the one that folks who work on upstream projects are most likely to be familiar with um, so this kit This is the you know, the fedora source control for packages system The screenshots may be not the easiest to see here, but you can see this is like a pretty GitHub style thing This is a pull request for a package and we actually have two systems feeding back into it here. Um We got fedora ci which has said, okay scratch build Success diskit test success zool success and that just means All the tests that fedora ci ran on this pull request passed. So this is You know a very developer focused workflow that is aimed at package maintainers who use the source control for their packages And are comfortable with getting their feedback in this in this kind of way Um, so yeah, it's very efficient. It's a familiar workflow for packages Um, and it's this is very early, which is the nice thing about it Like if you catch a failure here, it never even reaches an update. It never gets anywhere close to the compose process and We're running tests on pull requests. So the maintainer shouldn't know exactly what change broke things and can fix it right away There's a major limitation with this. Um, so When you're making a distribution sometimes, you know, two things need to be changed at once Say you're bumping a library to a major new version You know, you need to rebuild a library and then rebuild six dependent packages against it If you're just doing a pull request that's Say to rebuild the dependent package or the pull request to change the library Ideally you would be able to kind of test all of those related changes together Right now we don't have a way to do that in fedora ci on the back end Zulu is actually quite good at doing that But we haven't hooked it up all the way through the system so that there's a way for packages to flag Multiple changes to different packages as related to each other Uh So the next kind of integration point we have for results is bodi. This is the update system I mentioned earlier. So when you create an update in fedora, which again is a little bundle of packages and An intent to put it in a certain fedora branch Um, it goes through the system called bodi bodi has a web UI And we have set things up so that on the page for your update You can see all the automated tests results related to it This sucks in results from both fedora ci and open qa and theoretically Any other test system right now? We don't really have any other systems reporting But it's designed in such a way that any other automated test system could be stood up and report its results and they would show up here um Obviously one of the key things with bodi is that this is where we can test related changes together So this is how you group related changes in fedora You create an update which has all of those related packages together So when we test updates we test all of the packages in the update together and the report the result We report is related to the update as a whole. So that's one of the key reasons for testing at this level um We've done quite a lot of work recently on improving the display results here This again is kind of fedora specific So mainly for fedora folks in the audience But if anyone has been familiar with this before it it's had some glitches It would duplicate results for a while It would take a long time to actually list them all out And within the last couple of months, we've actually set it up so that it tells you when tests are running Not just when you know a pass or a fail But now you know that the tests are running they haven't finished yet, which is kind of neat um One thing that's kind of fun with bodi is that um, we as I mentioned we gate Most we gate most updates on the test results. We do not gate results for rawhide officially rawhide being the fedora development version um for Historical reasons and because i'm scared to turn it on quite honestly But in the last few weeks we've started stealth gating bodi Which means if stealth gating rawhide So if I see that a rawhide update has failed tests And I can't immediately figure out why or fix it then I go to the release engineering team And I say can you untag that update, please? So we're basically manually gating rawhide So we're sort of secretly taking out builds that fail the tests if we're confident that it's a real failure That's going to pollute down the chain And that's actually been super useful in keeping rawhide kind of sane lately Um and has really helped with quality more than I expected. So I really want to turn on the official gating soon And yeah, the craziest result integration point this is probably the funniest slide You're going to see we use the fedora wiki to collect test results This is the actually the original zygote of fedora testing I didn't have time to include this slide, but if you go all the way back to fedora 6 There is a very embryonic version of this wiki page Which is just a table with like 10 tests in it and that's where fedora qa started and you know 32 releases later. We still kind of have it We still have wiki pages which are tables full of tests with results in them The reason this exists is because it's actually the best way we've figured out to combine manual and automated test results So we have this crazy system where open qa tests are composed All of its results actually get edited into these wiki tables And then for the 25 of tests that aren't automated that are manually tested People can just go and edit the wiki page and put their own results in manually So if you again, it's a little hard to see the slide from here But um as the caption says this shows some results from open qa Some from a small system called rel val and some from an unreliable biological organism The ones which don't have a little robot logo next to them are from squishy humans who test things for us sometimes So that's where our composed level test results get collated. They get collated in the wiki Yeah, i'm gonna go over this quickly because it's very detailed and we don't have that much time left But there's a lot of stuff going on behind the scenes to make all this work Uh a key thing is fedora messaging Which is an amqp-based message bus. So often I say, okay like these results show up here And then you know when an update is created. We test it. How does that work? That works through fedora messaging when an update is created Bodie sends out a message an open qa listens for that message and says hey, I've got to test this update now So that's a lot of the stuff that goes on here even things like reporting to the wiki run through the messaging Results db is it's a vestige of auto qa funnily enough It's a kind of it's a system we use for results storage Which is literally just a key value results store with a web api around it So you can send it a message saying hey, I tested this thing and the result was blar And it will store that and then other things can query the results out of it Both open qa and fedora ci send results there. So that's how bodie can pull results from both systems It's just looks them up in results db. It doesn't care which system reported it Waver db is a similar system for storing waivers, which is Occasionally we do still have test failures, which are not real failures And you really need to be able to say no this test failure is garbage. I need this update to go out There's a button you compress which says wave failures And that creates a waiver which goes in the waiver db system. So that's how that works And green wave is um the system that provides the gating decisions which basically bodie Goes to green wave and says hey, I want to know whether I should push this update stable And green wave looks up all the results in results db. It looks up its policy And it says yes or no a lot of these things are hangovers from a microservices based design We had several years ago which never got completely built But that's why we have weird little things like green wave, which some people wish would go away So yeah, we're getting to about the right time for this this is the the last time I did this talk This was everyone's favorite slide if you've been sitting there zoning out for 30 minutes or looking at cat pictures But you still want to pretend you came to this talk. Here is the whole talk in one slide So take pictures now That's what we do in fedora We test updates and composers in open qa We test package builds in fedora ci We do not gate composers because we don't really need to We manually review the results and when we're deciding whether to release the distribution We kind of we have a formula for these are the tests that need to pass Bodie, which is the update system is where we gate We we collect all the results from automated tests and we decide whether the update goes stable or not and we do pull request tests in disk it Cache I didn't mention I should have taken that bullet point off. That's just another system We have we have very small systems which do very very small kind of things that I consider to be automated testing And caches one that checks whether a package builds from source or not. That's what fdbfs is So the future is it ci yet as I mentioned? No, it is not ci yet It's not going to be ci for a long time If someone can figure out how to do a fedora compose in 10 minutes Please let me know and then we can get somewhere with actually being ci until that happens It's not going to be ci But we're doing, you know, I feel like we've come a long way like from 2009 we had, you know, 40 manual test cases on the wiki which we ran 15 times each per cycle These days we have, you know, 100 Over 150 automated tests which we run every day on multiple branches. We have, you know, 50 60 update tests Which we're running every 10 minutes We're gating failures. We've come a pretty long way But things I want to do even further in the future We're always building more tests for open qa I mean right now My colleague who works on open qa with me is building more and more tests for different applications in gnome Right now I'm reviewing tests for, you know, the disks application and stuff like this We definitely want to officially gate raw height updates I always have one or two things I want to do before I hit that switch Which I think is mainly a mental thing in my head that doesn't want to do it But I do want to do that As I said in the past, we really wanted to gate composers I would almost cross that one off right now because we decided it doesn't really seem to be necessary anymore And more architecture coverage We do test on power, but I only have limited resources, so we don't run all the tests We don't test on s390 because no one's willing to give me a mainframe for some reason The ar 64 tests are going on extremely old broken boxes and fail a lot I would like slightly less old slightly less broken boxes Which I literally I got one yesterday, which I'm very excited about but I need three more And possibly move it to the cloud That would be the solution for the resource things as long as someone's willing to pay the bill But it's it's quite a lot of work to To actually run the tests in the cloud. So I've been looking at that for a long time And of course the more you scale up you you don't just need resources to run more tests You need to monitor all the results if you add more testing You still need to make sure someone is checking all the failures and doing something about them, right? So you always have to keep that in mind Plans for the fedora ci system There's a lot of as I said, there's a lot of consolidation going on behind the scenes Which isn't super interesting outside of red hat, but it is interesting to the maintainers They're they kind of have this thing they're worried that people aren't taking as much advantage of fedora ci as they could be doing The the integration point for the tests being disk it is really nice in theory But this doesn't help the people who don't use it who just maintain the tests with local get pushes themselves and never make pull requests So they're looking at ways they can They can kind of improve on that Um, and yeah, so that's kind of their their focus right now I should mention that I mainly work on open qa fedora ci is maintained by a different team And miro hongkok. Sorry miro slaw vad kirti Contributed most of the slides and information about fedora ci to this talk, which I'm very grateful for And other plans Yeah, so I've mentioned this a couple of times But these are these are the things that I talk about that are kind of inputs to how we build fedora that we don't really test Very well at the moment So kickstart a kickstart You may be familiar with it as the way to automate a fedora or rel installation We actually use kickstarts to build the live images So there's a repository filled with you know kickstarts that define and disk images arm disk images as well And anytime someone changes one of those Ideally I would like to you know rebuild the image make sure it builds, but we don't have that hooked up right now I would like to do that comps similarly comps is where we define package groups, you know These are the packages in the workstation package group. These are the packages in the server package group Every if someone changes one of those definitions it can have you know interesting consequences You take a package out that people expected to be there and things start failing We don't Have we don't have things in place to test that stuff works when you change comps I would like to have that Pungy Pungy is the the tool that does builds composes So if you change the definition of a compose obviously that can break all kinds of things Again, we would like to test that workstation os tree config is slightly misnamed That's where all the definitions of silver blues seracea kinoid all the immutable Versions of fedora that have sprung up in the last few years live Again, we don't test that changes to that don't break actually building the image So this is something I want to focus on in the next year or two is onboarding testing for all of these things Yeah, and I think I'm about at my time. Is that right or do we have time for q&a? Maybe quick ones does anyone have questions there is a mic here if you want to ask a question Right next to the projector So if you can grab grab the mic and I think you may have to turn it on there's a switch on the bottom Hello Okay, uh, so you talked about open qa testing and how It would be good to be able to test gi apps or like no maps. Yes So I was wondering But how is that going to be done? Is it just going to be done over the frame buffer or is it going to be done over like a remote display or like a vnc or rdp? Yeah, uh, it does do I should have explained this better I this is one of these things when you're giving talks you realize what you should have done But it does do that already that is the strength of open qa as I it runs a vm and it it takes screenshots over vnc And it inputs mouse clicks over vnc and it inputs typing over vnc. So we actually this is the key Strength of open qa is graphical testing of you know arbitrary graphical stuff that appears on the screen So all of the test of the installer are using the fedora graphical installer and the test system is literally clicking through it like a human would So that's how that works What I want to add on in future is just more tests like we're gradually trying to cover all of the core gnome tests So that we know every you know core application works every day But yeah, the the technical capability is there already. That's what open qa is good at Thank you. You're welcome. I would show we're running open qa test except I don't have network access for some reason So I can't do that Do we have other questions? Yep Okay So the the kind of tests that open qa doing are notoriously Sensitive to minor changes in ui and timing differences. How how big a problem has that been and How extensive is your testing with open qa and it has this problem really limited the the scope of how much testing you're doing with it That's a great question. Um, it's it absolutely is a limitation and it's One of the more frustrating parts of working with it So the way open open qa has various ways of coping with this open qa doesn't match on the entire screen Um, an open qa match is called a needle And a needle basically defines like a small area. It's looking for like a very small Well, it can be as big as you like, but we normally make them small So it only needs like there can be multiple areas as well So for a match it can be like two areas of the screen Um, and it only needs those specific areas to match and it also has like a fuzz factor It does some image processing it kind of flattens the colors a little Which helps with very tiny changes and you can set a threshold for the match It's using like a an image library I can't remember which one with a matching algorithm and it gives you a score So you can say oh as long as it matches to better than 92 I think the default is like 94 so you can set that floor to you know 80 or 85 or 90 for each needle and open qa also has a really good workflow for updating the needles Has a thing called developer mode. So if a test is failing you can run it in a kind of interactive mode Every time a match fails it will pause and let you go into the needle editor, which is in the web UI So you can basically just blow through update all the needles Commit them back to the git repo we keep the test in and it's fixed But yeah, it is it's it's the biggest sort of manual maintenance task is just Yeah, if gtk decides to change slightly how gray something is I get to update 100 needles the next morning So that is that is a fun part of it, but open qa handles it about as well as it could do I would say Hello um Something I'm very curious about is you mentioned 75 percent of the tests are automated. Yep in your I would say the vision roadmap. Yeah, what technical limitation is Limiting us to reach the last 25 apart from not having hardware Also a great question. Yeah, some of them are not some of them are just kind of philosophical like as the person who Has to basically say is fedora good to go or not There are some I want a human to look at it Like there are some tests which we just say a human has to do this So like making sure you can take an image burn it on a usb stick plug it into a computer boot it up and install it I want a human to do that every cycle So some of them are just by policy basically are off limits to automation A lot of the rest are Hardware tests so the way you open qa does technically have the capability to run on bare hardware systems You can use fan third it can use if you use like a an enterprise class thing which has like a management console um, it can actually inject, you know Inputs and commands over the management console. So that's one way it can test bare hardware There's also like a little raspberry pi based thing you can use We've not hooked any of that up for our open qa instance yet. Sue's is using that for theirs So any tests where we're trying to make sure something works on bare metal like a fedora user installing on their laptop would use Those are not subject to automation right now. So thing we have things like testing that a printer works And we kind of yeah, so that's the main constraint is just things that are Not possible to automate with the capabilities of the system where it is right now I can totally understand. Um, this is something that we have done. We have thrown ourselves against the wall as well And there is a host involved where the host controls the duty. Yeah, and it does the testing. So Yeah, happy to talk more on All right any more Okay Just real quick. Do you guys do any performance testing as Also a great question. Um, no, that is one of the things where resource constraints. It's like that's one of the things We also don't do any security testing on the qa team There are teams within red hat who do security evaluations and they do test stuff on fedora But as the qa team, I basically cut it off and say Make sure the thing works open qa in a sense Implies performance testing because there are timeouts on everything So if the performance is really terrible the tests are going to fail because the timeout will be hit But we don't do anything like is it 5% faster or 5% slower than yesterday We just don't have the capability don't have the capacity if I had 10 more 10 more people on my team Maybe I would do that Yeah, unfortunately Confident All right, thanks a lot to everyone for coming out. Hope that was useful