 Right. Thank you for your patience, everyone. Hello. So welcome to the afternoon session. And hi, Adam. He's here to present Fedora, CI and automated testing and where we are at it. Right. Hello. Hello. Okay. We're working. How's everybody doing today? Hello, Austin. Okay. I guess we'll get started. So, yeah, I am Adam Williamson. I'm a senior principal software quality engineer at Red Hat. It is, and I believe we keep changing our job title to be closer to software engineer in the hopes that software engineers will respect us and invite us to their parties. So, yeah, that's me. More importantly, I'm the sort of the team lead for Fedora QA at Red Hat. So I'm when Fedora breaks, it's my responsibility. There are going to be no funny slide, no funny pictures in this slideshow. I'm sorry, I'm not good at funny pictures just in case I don't want to disappoint you as we go through. I want to thank Miroslav Vadkirti, who is from the Fedora CI team, who contributed some of the content on Fedora CI you're going to see as we go along. I'm going to be Red Hat in 2009, so I've been working on this stuff for a long time. Okay, before we get started, I just want to kind of do a thing. Who here is a Fedora person? Understands how we make Fedora, how it's put together. Matthew, put your hand up. Well, I mean, understand is such a large word, right? It is a very loaded term. Who has no idea how you would put together a Linux distribution, how that would work? Okay, and some people are kind of in the middle. I'm just trying to, because I've written the deck so I can just dive right into Fedora stuff, or we can take it slower and I'll go somewhere in the middle in that case. I also just wanted to say if you have questions, generally hold them till the end. But if you're completely lost and you want to be following along and there's a question you could ask that might help you understand, do go ahead and put your hand up and I'll see if I can deal with it. Also, if I talk too fast, please let me know, because that's the thing I tend to do. All right, let's dive in. So yeah, orientation, this is the section that is a guide for people who don't really know how we put together Fedora. So if you know that already, feel free to look at funny cat pictures or whatever you want to do. So how do we make an operating system? Well, we take things that exist in the world, code bases, bits of software and we build them. So building them means we turn them into a thing you could put on a computer that would do useful stuff. Then we take those bills and those package bills are handled by a system of code. So I'm defining it, we can use it later. And then it's calledites, which is maybe one package bill on its own useful with another package that works properly and the update is where you bundle the unit that should work contiguously that we can add and that it ought to work. It's also handled by a system called Bodi. And it composes when you take all these little pages, these little nuggets and you turn them into more useful things. If I give you 3,000 packages, you can't turn it into a computer that's usable. So instead, we give you images that you can install from. We give you repositories that you can download, upload updates from. And the system making those is a compose process and the tool that handles composes is called punji. So these are all things that are kind of come back into it later. Just brief notes there, the package, the sources for the package builds these days are in source control. They're in a system similar to GitHub, which we're going to talk about later. And there are some differences between stable and development composes but they're probably not too important. So testing, this is how our build process works but we want to make sure things we build work. That's a good goal. We didn't used to be so hot on that. So all those levels from previous slide, we can do testing. You can test within control, you can run tests on. You can test builds, you can test inline or you can build after it's done. You can test an update to make sure that it's contiguous like it says it is and test an entire compose. These are all things we've done for a long time. The idea of compose validation testing is that we make sure to compose meets the criteria that we've decided a fedora should compose validation testing. So we test at all of these levels is the important thing to take away from this slide. So the background to automated testing is manual testing in the old days, as it says on my next slide, and this is around the time I joined Red Hat 2009 for quite a long time. We did all of this testing manually except for some very limited exceptions in the first bullet point there. So I spent years on just sitting up virtual machines on a desktop back. Sending up another virtual machine is going to vary slightly. This is how we used to do it. So for builds, the only thing we really didn't do when you're compiling the software, you can add a step which runs some kind of checks. It's called a check in an RPM file. That's better than you're really limited on what you can test. What we normally do at that stage is just re-run the software's upstream unit tests. We can't really test that it works in fedora just because of the environment that it runs in. Update testing was completely manual. So I'm going to introduce the system for this a bit later, but particularly you could... There's a website you can go to and you can see what updates are currently under testing. You can check if you have them installed and you can provide feedback. You can say, does this work for me? Does this not work for me? This is really good. And the game would focus on important dates and we were trying to test them, but aside from that, it was, you know, people coming in and if they happen to be running up, they could just... So it was very difficult. I have my installed system. I installed the test update system working, which is good to know. My system didn't keep working, also good to know. We wouldn't test whether it broke the process of doing a fresh install or the process of doing a compose, as I talked about earlier, because that was very difficult. It's difficult to do manually. You don't want to run a fresh install every day just to see if an update works or it's quite hard to build a live image or an installer image to see if it works. So I would very occasionally do that manually, but basically we never did it and they would break those things and that was a problem. Finally, for compose testing, this was kind of the biggest focus of our manual testing. This was where we made sure Fedora worked. When I came in at the end of the process, which is this is a legacy way of doing it, right? The waterfall style, you build the thing and then press the thing. So we would come along kind of it and then we would take a composed Fedora that we were going to release and we would run it through, you know, those 100 something manual tests. And if it didn't work, we would send it back and say fix it. This was like the best coverage we had and we did actually do a pretty good job over most of the time, but it has a lot of limitations. It's time intensive and boring. As I said, humans just don't test every compose. We didn't have that bandwidth. So we would test 10, 15 poses early. So we do a compose every day of things. If we had hundreds of feet, we could test but we didn't have the limitations to do manual testing. So enter automated test systems. We have a long history in Fedora with automated test systems because we've been around a long time. So in the old days, it was mainly my team that was really interested in automating some of the things and solving some of the problems. We had some systems. There was one called Taskatron, which previously auto way, which we worked on for a long time, which kind of fed into Fedora CI in the end. But that didn't do a lot of testing. We did a lot of work on it, but it didn't do a lot of tests. And then finally around 2015 or so, some colleagues of mine working with some folks at SUSE got really, really tired of all the manual compose testing and we adopted a system built by SUSE called OpenQA for automating some of the compose testing and we're going to dive into that later. A few years after that, a system called Fedora CI came to be, which is one of our current systems, and this was more of a planned top-down, well, I wouldn't say top-down, but it wasn't just a bunch of QA guys getting mad and building a thing. It was more of a planned idea of like, we're going to take CI, which is the big buzzword right now, and we're going to apply it to the distribution development process. So we have Fedora CI and I'll dive into that some more as well. And we have some other things which I'll cover, but there are things that were sort of developed as itch scratches like by individuals, or there's a bigger, larger project within Red Hat or somewhere else that contributes to Fedora, which got extended to cover Fedora. So we'll cover those too. Because Fedora is an old project, all of this automation tended to get built around our existing processes. So the processes I've just introduced, we kind of wedge our automation in around those. Our basic process of building Fedora hasn't changed a lot for, you know, 13 years in big picture and small picture it has, but the overall workflow has been very similar. So let's start out with OpenQA. OpenQA is a really interesting test system, as I say, originally developed by the folks at SUSE, so we and I collaborate with them on it. It's a really productive process. OpenQA is kind of like, it's like a robot intern. And as OpenQA was developed by the same kind of QA people at SUSE who got really mad of doing tests manually, so they kind of automated this out. So if you're running, you know, 50 installable tests yourself in a virtual machine, that gets so tiring. So what they did was made a robot to do it themselves, and it works very much like a human. OpenQA tends to work. It's been expanded more back end. Like our core workflow is it runs a virtual machine, it boots usually an ISO or something in that virtual machine, and then it clicks on things and types things in and it looks at the screen. It takes a VNC connection, which is, you know, just a VNC connection. And it can type, it can actually both event, move the mouse to the event. It looks at the screen, it takes screenshots, which is kind of neat every seconds, and it also takes a video, which is great for reviewing the testers all the time. And to decide whether it's a test has passed or failed, it can sort of say, oh, does this area of the screen look like it should? That's kind of the main mechanism. It can also run scripts or just console commands and check the exit code, which is kind of necessary because sometimes you need to test the thing that you run at a console, and it's really stupid trying to match a zero or one. So we built in a way to get that out of the serial console. But the clever thing it does is the screenshot matching stuff. So what OpenQA is really good at is those kinds of things, the kinds of high level end-to-end tests that QA engineers used to run, where we take a thing and say, can we run an install of the operating system? That's quite a... If you're looking at it from the other end, that's like a long way down the chain. That's a pretty big advanced test, and it's really neat to have an automated system So what OpenQA tests in Fedora is we have a standard set of tests on all composes. So this is the composer from that early slide. Any time we do a composer of Fedora, we do a nightly composer of the development release, or we do a nightly composer of something like Fedora IoT, it gets a signal and it runs its standard set of tests on those. It has a subset of those tests, which it runs on updates. So that's the update concept from the early slide. When there's a new critical path update, it gets a signal and it runs like a subset of the standard tests on that update. The reason why... So a critical path update is a concept we have, which is basically just like this is an important update. We have a list of packages during... And if a package from that list is in the update, the update is a critical path package, a critical path update, I'm sorry. And the reason we limit to only testing is just resource constraints. We don't have the hardware to test every single update because that's a very big set. So OpenQA, the strength of OpenQA is that the thing it does is a really useful thing, and they're really on any other test systems that do that. There are similar things, kind of, but OpenQA is really good at it, and that's why we like it. The weaknesses of it are that it's... It's not the best place to do what developers might be familiar with and more comfortable with testing where, you know, the GitHub workflow where you send a pull request and some tests execute in the container and get some results back. You wouldn't want to implement that in OpenQA. It doesn't make sense to do that. So for those kinds of testing... Oh, sorry, I have OpenQA test coverage next. I haven't rehearsed this talk. OpenQA test coverage. This is just kind of a summary of, like, this is how much we have testing in OpenQA right now, and this is kind of good for the Fedora folks because we've been expanding this out gradually for years. We started the system in 2015 and we just keep adding on to it and adding on to it. So right now, we cover 75% of the release validation tests. Those hundreds of tests that we used to have to do manually three-quarters of them are now done by the robot, which is fantastic. We also added some new tests just because they were useful and easy to implement in OpenQA. And it also covers Silblue, which is not release blocking, but it's important to Fedora, so we wanted to cover that. The kinds of tests it does, it does a lot of install tests. It tests upgrades, so it can start from, you know, an install of Fedora 35, run the entire upgrade process to Fedora 36, run the entire upgrade process to Fedora 35, and say, did that work? Which is a great thing to have as well. It obviously tests the critical desktop and applications, that's what it's good at, and that's a really nice thing to have automated. We also use it for testing free IPA, which is, it's a Red Hat technology that's kind of like Active Directory. It's an enterprise, you know, user management kind of deal. We test that in OpenQA because to test that, you need multiple machines to train to each other. OpenQA is actually really good at that, and Fedora CI isn't yet, so we test it in OpenQA. The same thing for database server, and then critical functionalities of Fedora server, which is the other reason we want to test them. So it can spin up one machine as a database server, spin other machines, client talk to the database server and make sure that all works, and that's all done from scratch as a regular person would do it. An old school regular person would do it. So for updates, yeah, there's a subset of the release validation tests that we run on updates. Plus there's some neat new, fairly new tests, which it didn't used to do a couple of years ago, and this goes to the thing about making sure we don't break composes. Remember how I said manual update testing? It's really difficult to check that we don't break composes. With a robot, we can do that. So what I made openQA is if you feed it an update, it can build like a Fedora network install image or a live image, exactly the same way as we do it in the compose process, and then it can boot that image and check that installing it works. So now we have some confidence that updates aren't breaking the compose process. And this is kind of nice for stable releases. It's not very important, but for development releases, it's really important because now we have a point where we can stop updates breaking the compose, and that has been a big pain point for the last couple of years. Just to give you an idea of what openQA achieves, because a test system that doesn't keep anything is useless. It files a bug reports. I was trying to get them more deep and it's a dramatic file. I file most of them. But we filed, you know, hundreds of bug reports of things that are as distance of over 200 in red Godzilla, but a lot of upstream, or we have two instances. We have a, you know, a test instance and a production instance between them they do. I was trying to add this up and I'm tired, but we do simultaneous which is small scale for, you know, maybe if you're used to giant systems, it's pretty scale, but this is all run on regular hardware. I'll cover that later. And we've run over 3 million tests since 2015, which again, is a big fancy CI guy. That's not a lot, but for me, I'm kind of impressed for a system that started out. It was running under somebody's desk in Brno, you know, just on someone's, just on Skunkworks server. Now it's kind of a real thing, so I'm pretty pleased with that. OpenQA resources, by resources I just mean how do you learn more about this system, right? So a thing about OpenQA is that it's not very interactive in a way. People come to me and say, hey, can I run my own test in OpenQA? Well, kind of, but you probably don't want to. It's quite hard. OpenQA is written in pearl, which is a stumbling block. So we kind of run OpenQA as a sort of old school a service the QA team provides, right? We maintain the core set of tests. We monitor the results. I monitor the results. We investigate failures and we file bugs. OpenQA as a Fedora developer is likely going to be, you get a bug report where I think you found this and I will do the work for you of figuring out what the detectable means in contact of your application being broken. But if you are interested, we do have, you know, if OpenQA does something and you don't understand what it's doing and you're mad, you can come and talk to us. You can contact us on our mailing list. These links are a little hard to read. I'm sorry about that. I'm bad at impress and I didn't know how to change the color. But that's our mailing list address. And on Fedora Chat, which is the current shiny Fedora Chat system, we have a room called Fedora QA. That's where we are. There's an upstream site for OpenQA, which is easy to remember. It's open.qa and that's, you know, maintained by the upstream project with all the upstream documentation. And we have a downstream page on the wiki for things related to our downstream implementation of it. So now we go on to Fedora CI, which is the other major test automation system in Fedora right now. OpenQA and Fedora CI are the big dogs, kind of. So again, I want to acknowledge Miroslav and Kurti, who pretty much wrote this whole slide and gave me all the material. Any errors are my own, not his. And I'm sorry, but I'll do my best. I know there's a lot on this slide. I'm going to go through it. Don't be scared. So where Fedora CI came from, as I said, it was more of a planned effort. There was kind of a fad within Red Hat years ago for CI. CI is the big, new, exciting trend. We're going to CI all the things. So the way this got communicated down to Fedora was we think there should be CI for Fedora. Like, you know, everything you change in Fedora, we should find out if it's broken right away. And, you know, the way it does the project on GitHub and we were like, you know, that's kind of hard, right? That's a big thing. That's a big challenge. But that's where it came from. So, you know, someone put together a plan for how to do at least some of this in Fedora. So what Fedora CI kind of is, it's one of those tricky things where it's kind of a more in this instance, where it's used to run tests in a way that's more conventional to a developer on, you know, Fedora packages, on Fedora package spec files, things like that. So, Fedora CI, unlike OpenQA it doesn't have, like, a curated set of tests that the Fedora CI team writes and runs. It's a small one, but that's kind of a legacy thing. I'll explain it in a minute. The idea of Fedora CI is that you can kind of write your own tests as a packager. So as a packager, you can, this is, you know, contribute tests. This is where you do it. So you can, if there's something that it would be useful to you as a packager where when your package build is done, or there's a pull request for your package, a build gets, and then it gets tested somehow in, like, an environment where other packages are available and so on, and a network is available. Fedora CI is kind of the place to do it. And there is a standard format which is called TMT in which you can write not only a test, but a lot of stuff that goes around it. So that's the second bullet point here. It has, you can specify, like, how the tests get discovered. You can specify how the test environment is provisioned and how it's prepared, which is, you know, what resources do you get? What's the, what's the start state of the test? Like, maybe you need to install some packages, maybe you need to pull something from somewhere else, not an official package. How does the test get executed? Which means you can use other test frameworks. Like, you can kind of run any really standard test framework within Fedora CI, you know, whatever makes sense for your package you can use. And reporting, there's a kind of a base expectation of how results get reported, but as long as your test, you know, sends the system back, the result in the format it understands, it doesn't care how that happens. So it gives you a lot of flexibility. But there are also kind of sensible defaults for all of these things. So if you want to do something fairly simple, it's quite easy to do that. You don't have to get far into the weeds. It'll give you a sensible environment to just execute some scripts by default, for instance. So, yeah, TMT is the current shiny for how you specify all of this. And you can look, I'm not going to explain that because it would take forever, but you can look it up and sort it out. There's an old thing, and the takeaway here for Fedora people is TMT is the new shiny, old, do not write new things in STI. Testing farm is kind of a, it's like a as all this has been developed it's become clear that we have a lot of when I say we, I'm now talking about the Red Hat Fedora sentos conglomerate. We have a lot of things that are kind of overlapping kind of similar. Like all of the, there were test systems for REL, for sento stream, for Fedora there's this thing called packet. There are a few other things I'll mention later which they shared a lot on the back end. They were all kind of running inside the same Jenkins deployment and testing farm is kind of the idea of let's view all of these things as one thing. Let's put them together, let's rationalize them, let's treat them as a unit, not as a vague series of overlapping blobs. So, TMT is part of that effort and the idea is that it'll make it easier for you to kind of unify your testing so that with limited work you can kind of test running in all of these different environments, which would be kind of cool. The idea of packet is to integrate upstream and downstream development. So, if you're the owner of something both upstream, as in, you know, the original project stream in Fedora or REL or something, you can instead of having to choose those two different things, the idea of packet is to allow you to kind of work on them in a unified way and testing farm backs that. So, what does CI really do in our context of, you know, what is getting tested in Fedora? So, it runs those tests on all official package builds and also pull requests. So, this calls back to the first, the early slide about the different levels. Fedora CI is kind of a lower level thing in OpenQA in general. It doesn't run its tests on updates and composes, it runs its tests on package builds and pull requests. So, you're getting your tests run more frequently and earlier in the process. The trade-off for that is that you can't, you know, you can't really test does it work with this other package or does it sort of, you know, affect the process because you're not at that point in the process. So, they're complementary, you know, they're not in competition to each other. It's also kind of a more developer-driven workflow than OpenQA is, you know, you can write the tests and contribute them and you're getting the feedback in your natural developer workflow. You're not getting the report. You're getting, you know, something we'll see this later about. You're getting feedback in your diskit or whatever. So, yeah, again, just to make the point they're complementary, you can, you could make OpenQA do a lot of this, but that will be dumb and slow and not as efficient. And you could re-implement all of the shiny screenshot-matching things OpenQA do with Fedora CI, but why would you do that because OpenQA is already written. So, we have two systems and it works out nicely. So, yeah, CI runs on all package builds, also pull requests, which is kind of a big deal. And CI, the standard test that I kind of mentioned earlier, this is a hangover from that old system I mentioned called Taskatron. The one practical thing Taskatron really did was it ran this suite of standard tests on all package builds. So, installability is, can this package actually be installed? Does that work? Or does it just grow in error? RPM inspect is kind of like a correctness check for packages. And RPM depth lint is kind of checking the dependencies of the package are correct. So, sorry, I didn't explain that in the initial thing. One of the key things of packages is package is what other packages need to be installed for it to work. Package build, beyond that, it runs whatever tests the package specifies in its own, you know, configuration. So, others, others are like the other test systems. The funny thing about these, they're not all intended to be QA systems. Some of them are kind of QA systems by accident. So, Cache is a good example of that. I hope I'm saying that correctly. Cache is, it was kind of developed as an itch scratch. The idea of Cache is it will, if you're a package developer, you can sign your package up for Cache and it will just constantly rebuild it, more or less. So, the nice thing about that is when somebody else changes something and your package stops building, which will happen, Cache will tell you about it and you will find out when it happened and from that it's much easier to find out why. So, this was kind of a convenience thing for package maintainers, but it's effectively QA as well, because it's super useful for us to know, hey, when did this package stop working, stop building, and why. So, I kind of count that as a QA tool, even though it wasn't intended as one. This auto test is a really little thing, but it's kind of cool, I wanted to call it out. We have, so our validation, remember our compose validation concept, where we run a bunch of tests to make sure a compose is releasable. Some of those require odd hardware configurations, like my favorite example, which is almost a meme at this point, is that we require that it works on serial attached SCSI disk drives, which nobody has. I don't have any, Matt, you don't have any serial attached SCSI disk drives, do you? Well, yeah, there you go. So, this was constant, okay, then you're signed up to do validation testing, congratulations. But this was a constant pain point, and we had a few other things, like fiber channel over ethernet, there were these exotic technologies that open QA doesn't have, but RCI doesn't have, nothing has. Why aren't you testing things, Matt? So, what we do have at Red Hat is we have this internal thing called Beaker, which is basically a big hardware farm, which has all of these crazy technologies in it. It's kind of an old thing, but it's useful if you need to do this. So my colleague Lily, who is on our team, she's great, she wrote this thing called Fedora Release Auto Test, which kind of, it does, similar to open QA, it notices when there's a new compose, and when there is, it sort of generates the tests to send into Beaker, to test that it pulls on that hardware, and then it pulls those results out, and puts them into, you know, public Fedora record tracking, so that's how we can test on that hardware now, which is great, because we don't have to run around trying to find the one person who has this hardware to test it. So that's a neat thing. RelVal is a tiny little tool that does the setting up the compose validation testing, but it also happens to do image size checks, just because that was the sensible place to put it so we are checking image sizes automatically now, that's kind of the takeaway here, which used to be a big pain point, when you release the distribution you kind of want your images to be small enough to fit on a USB stick, or a DVD, maybe, if people remember what DVDs were, the little shiny things, yeah, CDs, we still target CD size for some images, I think, but anyway, there's size limits for the images and we want to make sure they meet them and we want to make sure that when that stops being the case, we find out as fast as possible, so having that automated is nice. Fedora-Coro SCI is kind of a big thing, but Dusty's not here to talk about it anymore, so I'm just going to call it out here. You may have heard about Fedora-Coro SCI at this conference, you may not. It's I don't know how you would not have heard about Fedora-Coro SCI at this conference. It's kind of a special flavor, Fedora, which I won't go into all the details on, but it has its own really shiny system, which is what we use to validate that the Fedora-Coro SCI product meets its requirements and that is like, that's just kind of a cool thing that sits there running and I don't have to care about it, so I love that, but the takeaway is Fedora-Coro SCI has really good testing of its own. These last two, I'm going to kind of skate over a little bit because these are kind of planned to be sucked into Fedora-Core SCI and they're part of that whole testing farm effort that I mentioned earlier, but Zool-based SCI was, well, is a setup for using a neat CI tool called Zool to run tests on, you know, pull requests at the forge level, which we'll go into more later, and pack it, as I mentioned, is for integrating upstream and downstream development, so it makes it easier if you're the maintainer of something upstream and downstream. Okay, so this one, it's just a little introduction slide, but I'm going to talk for quite a long time here, so get familiar with it. As you can tell from the previous slides, we test a lot. For Fedora people, I hope you noticed that we test more than we used to. Some of those things I talked about testing, we did not test before. My pet hate is automated tests that get run and nobody cares what happens. You've probably seen a test system like this. You've seen a project where it has CI and the CI always fails and nobody cares. I hate that. I am very, very upset. To me, this is an important thing. Testing is useless unless you have clear, visible, manageable delivery of the results. The results have to go somewhere where you can see them. It has to be clear and you have to be able to investigate. Test system is failed. It doesn't give you the tools you need to figure out why it is a test system. Talking about how that works, in Fedora we have kind of different doing this and probably other people do too, but in Fedora we do. If you go back to the open QA slide, way back here. This is over here is the open QA web UI. I forgot to do a little sidebar here. Here you see it has a lot of green. If you've seen me frantically typing on my laptop most of this week, at this week there was a lot more red and I have spent my entire week turning the red into green and it's been fun. Sorry, I thought someone had a comment. This interface is kind of for me and the people on the QA team. I would not expect Fedora packages to go and look at the front page of open QA. For me it's great because I can say this update has failed tests, I better go and look at that. This little icon means I already looked at it and if we can't clear that red but it doesn't matter. This is a great overview for me but it's not where a package will get that test result. I'm going to talk to you about where they do get their test results. The other really important thing about result delivery and why I say reaction here is you need to do something about the test results. For people to do something about it manually, the clear delivery is important but the really great thing we want to do is go gating. What that basically means is to catch when something is broken and stop it there. We don't want the broken thing to propagate. We don't want it to get beyond that point in the process and that's called gating. Say we're going to talk about testing updates. If we have automated tests for updates, that's great and if we let maintainers see the failure, that's great. If we still don't see the failure, that's great. That's not so great. What we really want to do is make it whether test is that important, it wasn't important or it was just so hard we didn't want to do it. In the past we kind of focused on the composed validation but the way software development is so fast these days we want to be releasing new things all the time. We want to be shipping CoroS builds all the time. These days we want our development releases to be usable. I'm running a presentation on Roarhide which is our super leading edge development version. I run Roarhide most of the time which would not have been possible in 2009. You would not have wanted to do that but these days we want to make that possible and gating is really important for that. There's a problem with not gating just in the context of doing automated testing as well which is something I fight with on the regular and it's why I had so much trouble with all that red. If you don't gait then a failure will initially come in through the one component that broke the thing but then the tests for every other component you test after that will fail the same way and this will make you sad. If you don't gait what you have to do is identify why it broke and fix it as soon as possible which I spend a lot of my time doing. I wear a lot of hats at Fedora so these days I also fix a lot of bugs just because I want tests to turn back green as soon as possible. It's so much nicer if you wait because they're sensitive and you don't spend your life trying to get a C code you don't really understand why it's broke. So the difficulties of gating are I would say capacity you need the system, people get mad if you're gating on a test that takes eight hours to run. People will accept gating in test runs 15 minutes after they submitted the thing. They're not happy if the thing doesn't get it in hours so you need capacity to do that. Reliability, tests aren't reliable if sometimes they fail not because there was anything wrong with the component but just because the test is flaky that's a problem for gating. You either have to make sure that doesn't happen be super vigilant when it's not a problem anymore. And the other thing we struggle with a lot is what I would call a change introduced in non-covered system. So I simplify the distribution development process quite a lot you will probably not be surprised to hear there are a lot of other things that feed into the process and we do not test all of them yet. I'll come back to this at the end of the talk. So the problem with that is the completely different system or different source of information that you don't test and in that case you're gating the first thing that fails in your test system will not actually be the thing that broke the test. So having gone over all that places where test builds get handle fedora so I'm going to do this git this git is basically github there's a slang term a jargon term forge which is any system that kind of works like github so this git is built on a system called pager but that's not important it's basically like github so if you maintain a package you can set it up so that you send in changes via pull requests and if you do that these days your pull request will get a bunch of comments from automated systems so the one headline zoo is coming in from the zoo based CI system and these two down here fedora CI scratch build fedora CI diskit test are coming from fedora CI proper they are planning to kind of merge these in but this is a place where you can get your results and this happens right now that's just a some pull request I pulled as an example that's live that's all a real thing so in this system you get test and there's other things that kind of part of fedora CI and you can configure to have gating on these like your package can set thank you your package can set configuration that says I want pull request to be gated on the results of some or all of these tests and that would mean the pull request cannot be merged in until those tests pass so that's you know that's one of our key requirements there the requirement that it's kind of visible and click that links back to click on this link and you can pull out the artifacts and figure out why things are failing so the advantages of using this system as an integration point is that it's it's quite efficient and convenient for developers it's part of their developer workflow so that's kind of the advantage and that's why fedora CI tends to feed in here because that all goes together it's low level developer focus tests this is the natural place to get those results the other major integration point is Bodhi which if you remember all the way back to the start of the talk is the system that handles updates so this is where usually open QA results feed in here it does actually include CI results as well any fedora CI result that's for any of the packages in the update because CI test packages will be considered as a result that's relevant to the update and it will be pulled into Bodhi so yeah and Bodhi is all for all of these except for why for critical update any of them feel acceptable that's branched which is our next development release but not for all of these packages. So if you want to get your updates on CI tests you can do it in Bodhi and there's because no process is ever perfect although this makes me sad if your update is being gated on a test failure that you think or know is bogus there's a button you can press which says ignore this test failure I want my package to go out anyway if you do that and you're wrong I will be so please be careful with that button Neil I know and that's why I'm sad all the time Neil so you'll notice up here I say any other compliance system that's just because this is built in a fairly generic way and technically other systems could get results to appear here you just have to publish results to a certain system in a certain way so for Bodhi is actually really good at gating these days we've done a lot of work on it for the test result display it has some limitations if you are a package or you may not love this view some of the problems with it when there is a failure it's not always obvious which failure is important this little thing up here where you probably can't see it but it says all required tests past twice there's a very good reason for that but it's still stupid and I would like to fix it we're aware of these things we really want this to be a really good workflow we try and improve them I'm not a very good javascript front-end developer so it takes me more time than I would like to fix some of these things but we are trying if you have feedback on it please bring it to us if you want to help and you're a good javascript front-end developer please do that would be great and our final integration point is the fedora wiki yes we use the wiki for integrating fedora test results I know this is crazy and this is just kind of because of history when we were doing the validation testing all manually the way is where we recorded all the results and some of that testing is still manual we need a place to combine automated and manual results and it turns out it's really stupid wiki is kind of a good way to do this so it's a little hard to see on the screenshot but some of the results here have a little robot icon next to them and they're reported as having from a user called coconut those are automated results which have been automatically forwarded to the wiki most of them come from wiki way but the media kit ISO size one comes from rel val as I mentioned a few slides back and that all just integrates in nicely and then just down here somewhere yeah we have results from moonlight that's a result from an unreliable biological system otherwise known as a human so yeah we we integrate you know the manual and the automated tests in the wiki because it works oh god that's an old story because the idea of open qa that we would never have to go under the manual test and we could all go and lie on a beach somewhere drinking Dakar is and it was called project coconut so that's why lots of coconut I'm going to blow through this part really fast because I'm short on time and it's not very important but how all this works behind the scenes there is a lot of technology in this is a messaging bus which how all those signals pass between the systems thank you five minutes that's both so yeah publishes messages and listens for messages all of these old things publish messages and messages are great because you don't have to sit there hitting the I every 30 seconds to find out something happened the thing that did the thing can just tell you it did the thing and before we had messages for composers I had a really horrible thing I maintained which just sat there hitting a website to see if a compose had shown up it was awful but I have to maintain that anymore results db is this is the thing from the old system called task atron which turned out to take over the world which we were really surprised by it was the one component of task atron that's still around and everyone loves it red hat uses it it's everywhere results db is a really simple key value based result storage system it's literally yes please greenway coming up next slide uh coming up don't worry so db is basically just a database with a very simple web API around it you hit it some keys and values it stores them you ask it for a result it send them back to you but everyone it's great the message bus which is the important thing so what that means I'll get in a minute so you can get results from it directly you can for messages there's lots of ways you can get things out of it greenway the thing that makes gating decisions um so years ago a lot of stuff got built we had a division called factor you know that instead of building one thing you build 50,000 things that do it there are a lot of these so this was the most part of a micro service design about half of which got implemented so we have this tiny little thing called greenway which just does gating um you hit green wave and you say hey is this thing gated and it says yes or no that's greenway um waiver db is the thing that handles the button that says ignore these results um it's a similarly tiny micro service thing and it just stores a list of wave and a waiver is basically a record that says this test result for this update was bogus ignore it um and yeah so bode greenwave uses it as an input to the gate decision so when greenwave is deciding it's something's gated to check for the waivers bode does it's updating is that it listens to waiver db for new waivers and it listens to results db for new results and any time it sees a result or a waiver which may be relevant to the update it updates the gating decision that process was buggy for quite a long time i fixed the most bug in it last week so if you've noticed as a fedora package that your gating decision is weird and don't get updated fast enough or get updated too much that should be better now if it's still not better come and yell at me about it this part can be super quick now so i like when i come to conferences i don't pay attention to everything um and i like to provide services to other people like me so if you've been looking at cat pictures for the last 40 minutes but you still want to say you came to this talk here it is in ginsleed updates are tested by the request is tested by fedora ci composes are not gated the result reviewed updates are gated in bode pull requests are gated in disk it and cosh a checks for fails to build fdbs fdbs is fails to build from source means the package doesn't build that's all of the automated testing right there just take a picture of this slide and you are done what about fti it's not uh the future yeah i'm pretty short on time but open qa plans we want to build we want to make more tests um so we want to cover all of the gnome applications and my colleague lucas who is the other main person who works on open qa is doing this right now so we've always implemented a lot we test the text editor we test weather we test clocks he's writing more every week i really want to get raw height updates which is a huge step to take and it will make some packages mad but i think it's the right thing to do and it'll make my life easier so i'm probably going to get mad and do it one friday afternoon so look out for that coming soon okay um you have got yourself a deal mr miller we want to get gating composes has been on the wish list forever and it would actually be less important if we get to draw height updates so that way it has a question mark next to it but one of the old ideas for this whole thing was that instead of publishing any compose that succeeded even if it was really broken even if the open qa tests for it are like 90 percent red if the tests were all really bad and it was completely broken we would just not publish it so the compose would exist but when you hit the public mirrors for raw height or 37 or whatever you would not get that content because we know that content is bad yeah i kind of think we should just go to gating updates instead because it turns out when you get the updates the composes are nearly always fine so we may not think about that hard but it's there on the list i want to cover more arches we kind of cover arch 64 but i don't have much hardware so it's a really small set of tests power we only do in staging because again i have a lot of hardware we don't have as many as 3.9 so it would be nice to cover them all more on the list i'm trying to go STs right now which we kind of cool my new best friend we're going to do this if we move it to the cloud i have infinite resources and i can test all the things all the time so i would love to do that yeah i've talked to some people about it we can do it we're totally going to do it fedora ci plans again these are all from Miroslav so thanks to him for that yeah this whole thing to kind of blend in all the bits the kind of similar we want to do that and we want he wants to sort of instead of having the standard tests come out in Bode which he doesn't think is a good workflow for those it's not integrated with the developer workflow he wants to focus more on that stuff going to the pull request workflow that's about it other plans there's a lot of other stuff we could do yes it's green wave dot fedora project dot org but the web UI is very boring i think it just gives you an api overview green wave you interact with entirely through the api i will talk to you after the talk about why you're interested in green wave because it's really kind of a behind the scenes thing but yeah it's green wave dot fedora project dot org i'll give you more information after really oh yeah you have to go to the api group yeah i'll talk to you after the talk so yeah that was my talk thank you i want to thank miraslav again for the ci stuff i want to thank mo for doing this template and do you have any questions this is the q and a section of the talk note q is q and a we get a lot of people coming into fedora qa asking questions about things and it's like thank you everybody thank you thank you thank you how many packages have tmt format tests in this that is an excellent question and it's why i wish mirro was here because i have no idea but the fedora ci team i wanted to get someone from fedora ci to co-present with me and i sent a message to team said hey who's going to be at defconf Boston they said nobody so i am doing it all i will try and find out for it it's like more seriousness i think there aren't very many people are aware of this and something we could maybe have yeah so i just realized i forgot to do the resources page for ci i'm sorry about that there is a docs page docs.fadoraproject.org documentation about where you're supposed to go so that is a great takeaway tmt is a thing the whole fedora ci system has gone better than it was if you're a packager and you've looked at it before and you thought ehh now is a good time to give it another look there's these shiny new capabilities and the new metadata format take another look at it maybe you'll like it more and we would definitely like to have more packages onboarded to the testing next question i'm sorry i've forgotten your name it's a black shirt i'm terrible with names Paul thank you this is not a green wave question i'm just curious in wondering if you're ever going to test on risk 5 because i've been hearing a lot of stuff in the limelight about risk 5 and how that is getting how that's a new architecture that people are working on sure kind of the same answer that other people have given you to this question we don't do it because we don't have any hardware i don't believe fedora ci has any risk 5 hardware i definitely don't have any for open qa for open qa i like arch coverage if people are willing to give me enough hardware that we can put in a fedora data center and it needs to be able to run virtual machines and you know it needs to be kind of server level because we need people who aren't at the data center to be able to turn it on and off and update it yeah so we need that when we have that we can do the testing and we can answer a lot because this is kind of the requirement for any decent scale test system is enterprise management if you don't have that it's not going to get done yeah that would be nice so i'm somewhat familiar with how the test suite works for for fedora coro s and so my question is are there plans to adopt some of the some of the philosophies and methodologies there for fedora c i that is a great question i can't speak authoritatively for fedora c i in general for fedora i would say that would be cool the kind of stumbling block i have with fedora coro s c i is it's not very public it all kind i don't know where it all happens to me it's kind of a black box that coro s i mean just get built and tests happen somewhere i trust that the test is an external person i don't have a lot of visibility into them if all of that was more public and the test results that would make it a lot easier to get excited about applying those outside of door press david yeah you mentioned blender did i mention blender did listen to talk about that you mentioned beaker the qa lab that's something that from the cloud perspective we have some modifications to the dc i system that make it possible to bypass the beaker requirement so we might want to talk about that later and i wondered if there were plans for that circa software factory no it's not it's i wanted to pull that into the talk because it's a neat little system that nobody knows about and i kind of want to give lily a bit of props for it but it's it's kind of a smaller thing and it really is just for those like six lines on the matrix of weird hardware that we want to fill in there was a plan a long time ago to have a fedora beaker instance and really push on this and make it a big part of testing and that never really happened so to me beaker is like it's a little thing we use for this little purpose and it works so it's not a huge part of my plans but i'm always interested in cool ideas yeah okay yeah it's like one of those little dirty secrets that people at red hat would kind of like beaker to not be there anymore but it's so useful you can't make it go away it's like the wiki the info team has been trying to kill the wiki forever and me and like three other people are like no we need the wiki leave us the wiki and beaker is kind of like that it hangs around but yeah i'm not i'm not a super expert on it i'm not responsible for maintaining it thank god there are probably better people to talk to uh more questions nope alrighty then thank you all for coming i really appreciate all you're coming out it's the end of the conference it's saturday everybody's tired so thanks a lot for coming out and i hope you enjoyed it thanks a lot adam and for presenting the most photographs like um uh everyone is trivial you all there