 Well, my name is Tim Flink. I work for Red Hat. My job there is working on Fedora QA, and a lot of what I've been working on for as long as I've been at Red Hat is around automated testing. So I'm going to be talking about automated build checks and what is happening with them or where we're at and where we're going inside of or in Fedora. To be more specific, what I'm looking to talk about is a little bit about, you know, what exactly are build checks? What have we been doing? What is the glorious future? And I plan to leave quite a bit of time at the end for questions. So I expect my presentation to be, I've timed it at about 16 minutes or so, 16, 18 minutes. So there will be certainly time at the end for questions and a little bit of a demo. So build checks. To start off with, it's a term that certainly I use, checks are a subset of tests. And the important differentiator being that a check is something that doesn't require a intelligent human being to run it. It is, you know, you put something in, you get something out, and most of the time, close to all the time, you're going to get, you know, pass, fail, or, you know, some form of undefined bad behavior, whatever that ends up being. But the important thing is that when I'm talking about a check is that it's something a machine can run. It's something that doesn't have to be supervised and something that can be run at a reasonably large scale. When I talk about build checks, these are things that are not specific to a single package. So, you know, system D has a test suite, the kernel has its test suite, most software packages have some form of test suite, whether we're currently running them in Fedora or not. So you know, these build checks can run, you know, generically on just about anything that comes out of Koji. You know, examples would be RPM inspect that David Cantrell gave a talk about yesterday. RPM depletant and ABI check are all examples of build checks. So what has Fedora been doing in the past? How have we been running these things? For the last, what, six years or so, we've had a system called Taskatron. It's a set of services that work together to run automated tasks, hence the name. You know, it works with Buildbot. We have a custom runner. The store results and results DB, you know, as much as, and that's actually where results DB came from, is that it was originally part of Taskatron, but now it's gone on to do bigger things. And there's some other support bits. All the communication was restful, Jason, and it was designed to change. You know, runs and disposable virtual machines, the tasks have to be executable locally. So if a developer wants to run things before they submit a build, if they want to triage a failure, everything had to be as close as possible to production on whoever system with minimal setup. But enough about the past, which this really is. We stopped any new development of Taskatron early last year, and we are in the process of figuring out what the date is for sunset and end of life. So the system as a whole is going to be going away. But we have the Glorious Future to look forward to, and that Glorious Future is FedorCI. FedorCI is made up of Jenkins running on OpenShift. So Jenkins is responsible for triggering the jobs. It's responsible for viewing the results. It's responsible for all that kind of stuff. And OpenShift runs Jenkins and then provides the containers to be more specific, the pods and Kubernetes that are delegated to execute these tasks. The results are emitted as messages. Currently it's FedMessage. We're in the process of moving to footer or messaging, which either way the results is the same. There are other messages and there are other mechanisms that will take those messages, put the data into results DB, and make it into a format that can be used for gating decisions. So just some details on how it's been put together. All of the new checks are written as Jenkins pipelines using JobDSL. The idea of using JobDSL is to get rid of a lot of the boilerplate, so we don't just have a whole bunch of Jenkins files. Because it's pipelines, it relies a lot on library functions that are in Groovy and there's one pipeline per job so that you can go look at the Jenkins master and say, I want to see the RPM Inspect runs. I want to see that kind of stuff, which is an organizational thing. Those I suspect are not going to be interested in seeing all of the RPM Inspect runs that have happened in the last couple of days. They're going to want links to the direct job that's related to their package. But the interfaces as far as the notifications that come through, the things that show up in Bode, those are going to remain pretty much this as they are now. The new checks, they're in a single-git repository. It is, in my mind at least, reasonably easy to modify. There's some check we want to add in the future. It's filling out a template, changing a couple lines of code and putting that into a pull request. And then that in short order becomes part of the Jenkins instance. What I currently have done is we got the OpenShift project. Everything's running in the CentOS CI infrastructure. The Jenkins master is there. The framework for the jobs are there. And we have RPM Inspect running against single-build. What I mean by single-build, and I won't go into too much detail because this was the whole point of David's talk yesterday, RPM Inspect can run against a single-build and just do simple compliance checks. Or in its more powerful mode, you can run it against a build and the previous build. So you can see what has changed, have any files been added, deleted, that kind of stuff. But that's not working yet. The stuff that's left, there's some refactoring that needs to happen. The thing coming up thus far is Ben just trying to figure out how things are changing because there's a bunch of stuff behind the scenes that has changed in terms of messaging, in terms of how all these systems interact. So there is more code reuse that could happen. And once we figure out the details of what messages need to be admitted, what's being listened for, who's listening for, where, there'll be less repetition of code. So you need to finish meeting messages. And at the end, we need to start porting the, or finish porting the jobs and migrating away from Taskatron so we can pave the way to start turning that off. Timeline, I expect to be done with having figured out all the message emission stuff and getting everything ready to give useful feedback for the gating process at the, I'm being conservative by the end of October. I expect it's going to be before then, but I'm choosing to be conservative in my estimates and then have all the things that we're currently running in Taskatron ported to Fedora CI by the end of the year. So short version, Taskatron's going away. Fedora CI is the future, and the transition will be done by, at least the porting work will be done by the end of this year. I probably should have put these slides in a different order. So just to show some of what exists, this is the public Jenkins master. The first job is just, this is the job DSL bit that then automatically generates all of the other jobs that run the actual checks. So right now it is running, let's see, do I dare refresh? I probably shouldn't have done that. So it had to be reset this morning for, I managed to break it while preparing for this. It's not in production yet. It won't happen once we are in production. And just simple stuff, I mean it's basically, this is what it's doing. It runs RPM inspect. And there are log files, you basically get an output of the things that RPM inspect has run, the various license checks. And this has a lot of sub-packages. So that's why this file is particularly long. And then the other bits, but the details of this is the RPM inspect. Come on, load. All the files used to deploy everything. It's all up on Pagger. I'm not going to go into the OpenShift configurations and the Jenkins bits because that seems out of scope for what I'm talking about, but it is all up. And if someone wanted to reproduce it, you are welcome to do so. More time. I've uploaded these slides to thesked.org. So you'll be able to find these links if you are looking for them. This is, don't click on the slide. This is the Jenkins master. This is the Pagger repo. Questions and whatnot about proposals for future things we could do. Head over to Fedora CI. And that's honestly it. Does anyone have questions? So I said, oh, at the moment, we don't have a clean way for people to trigger things on their own packages. As I understand it, that is planned for Fedora CI, but that does not exist at this moment. No. Can I also trigger on co-provails? It could happen. There are no plans for it, but there's nothing preventing that from happening. Other questions? All right. So I severely misjudged the number of questions. I was expecting more questions, hence why this is done so quickly. Well, I guess we all get 20 minutes left. All right, get 20 minutes back. Thank you all for coming, and please let me know if you have questions. Either find me here, email, on IRC. So, yeah, thank you.