 structure teams, and people on Fedora CI Special Interest Group. So the outline of the talk for today is, first I'm going to go through a Fedora height gating overview and remind what it is and how it works. Then I will explain how Fedora CI Special Interest Group is present in this picture and what we are doing. Then we'll cover how your tests to the gate we have and how to add maybe the entire CI system to the gate if you happen to have one. So let's start with the first item, the Rohite gating. Rohite gating is not a new concept but it was implemented last year but I'm going to provide some brief overview. So first of all for those who don't know what Fedora Rohite, Fedora Rohite is the snapshot for the latest updates of all RPM packages in Fedora. So this is our shared development space where we deliver updates first and then these packages go through stabilization phase and through beta and final release stages and then they go into the branched named stable Fedora release. So the important feature of Fedora Rohite is that it's a shared development space. So even though the packages which we put into Fedora Rohite have less quality requirements, there are development snapshots and there are development packages that are not targeting the regular typical workstation. Since it is a shared development space it still has a certain quality requirement which is basically that development of one package shouldn't block development of all the other components in Fedora Rohite. And since we have thousands of packages, this is a very tricky question, how to make it so that one package doesn't interrupt development of other components and these things can go and integrate nicely in this shared space. The second aspect of this, why we started to look into Rohite gating is that catching errors at alpha beta final checkpoints during the release cycle is actually too late. It is very hard to find at the beta release that we have a list of critical issues which prevent distro from being released and then we need under time pressure very fast resolve it somehow which may cause huge dependency chains and a lot of trouble. So short and targeted feedback loops are much better in testing and provide much better testing development experience because as soon if you test the change the moment you create it, you have much smaller scope of a change and much easier debugging procedure. So Rohite gating supposed to address these two pain points. How it does that? So by introducing the concept of the gate, I find that many people think about the gate as like doors open and closing. I find this association with airport gate is closer to what we actually are doing here. Even though, yeah. Yeah, so the gate is not a door, it's a place where you sit with your package until all the locations are passing and only once all the required checks are passing the package can proceed through the gate and land in Fedora or height. So this gate is implemented in CodeGear via on top of CodeGear tags and targets. And if you're interested in exact workflow, how this works, you can read the docs in the change proposal. So this was implemented last year and announced in February at DEF CON, I think, or maybe even early at Flock. And more to that, we have now two kinds of gate. We have single package gate where each change goes through the gate alone. And we have a multi-package gate where if you want to update several packages at once, for example, we have a very tight dependency on each other, then you can group those packages in one unit and test and work with this change as a unit of work in these packages from the group will pass through the gate together. So the gating framework is there, it works and it's nicely implemented in bot here and supported in the UI. But the question which we have now is like, who runs the tests? We have a gate, where are the checks and who creates them? And this is where CI systems come into play. So this is the small overview of how the gating process works together with the CI system. So we have bot here service, bot here is a database of all Fedora updates and it's our main instrument and main tool to work with the updates. Every time you create a new update in Fedora, both in Rohide or in stable branches, you create associated bot here update entry in bot here, which describes what this update is all about. Then bot here sends the message into Fedora messaging bus where it says that new update is coming, new update was created, that's it. So once message lands in Fedora messaging bus, we have multiple CI systems listening to the message bus, listening to specifically to this type of a message. And whenever a CI system recognizes the message about a new update, it triggers its own logic to trigger the test which are relevant for this update. Once CI system does its magic, it sends again a message to Fedora messaging bus with the test result. This test result can be passed, the test can pass, the test can fail, test can also be end within infrastructure error which is not, which means that we haven't achieved, we haven't gotten the result of the test. So result is unknown where it was some error on the way. All those messages land in Fedora messaging bus and then there is another database which is called resultsDB which collects all those messages from all the CI systems, puts them into one storage and you can query resultsDB and to learn all the tests relevant to this particular update, for example, and their statuses. This resultsDB then is used by Bodhi to make a decision on whether or not to let the package go through the gate. But from the purpose of this talk, the important part is here is that we have a CI system which role is in the role of CI system is to react on messages about a new updates coming, do the magic and then send the message as outcome, whether positive or negative. Everything else is managed by the gating system. So the only thing which is needed to produce the test is here is to create an update, to trigger on update and to send message with the result. And one important part about this picture and which I try to highlight here is that this is a distributed system. You can have as many CI systems working in this workflow as you wish as soon as you meet in them. So we can add CI systems independently and implemented through various tools. One of the CI systems which default Fedora CI system and this is what Fedora CI Special Interest Group maintains and develops right now. So we create and maintain the engine which runs the test for Fedora Rohite gating infrastructure. So you may be seeing this already in BOTHE. This is the screenshot of actually my update I made a week ago for Fedora Rohite. If you go on any page about the package update in BOTHE we can go to Automated Test Results tab and you will see test results there. So you see tests has a much longer names than I put in my picture, but they always have a prefix which is a ID of the CI system and then they have some name which describes what this test is all about. So to give the overview of what Fedora CI resources we have currently all of this test execution is implemented on top of Jenkins pipelines. We have one old Jenkins instance which is deployed to the CentOS cluster to be cluster provided to us by CentOS project. It runs this quadrability or zero functional test. It has no integration with Fedora account system and it has several other problems on the way. So basically it's on the road to be deprecated. To replace this old Jenkins we have also two more Jenkins instances like high availability and all that. So we have two different Jenkins master instances. One is deployed on top of the Kubernetes cluster in Amazon cloud and one is deployed again on top of the OpenShift cluster provided by CentOS but now it's upgraded to OpenShift for cluster which is new and which has more features interesting for us. So from a point of resources we have access to the cloud on Amazon which means we have a lot of resources there and we have access to OpenShift for cloud of resources from CentOS. The amount of resources there is limited but also quite a lot. Now what we do with all these resources this is the list of the tests which we are currently running. One of the most interesting tests is the Disgit test so-called. This is what you saw from the previous screenshot. In essentially this test scenario it runs whatever it is described in the test folder in the repository of the package. We take what's described there and we run this scenario in x86 virtual machine. So it's not a container it's a full featured virtual machine with Fedora installed in there and we just run scenario inside and then we get the outcome of this run and post the message to the message bus about the outcome. There are some links on how to do that so you can read for more details from there. The other test which we're running is RPM Inspect. This is a static analysis tool. It analyzes a spec file on most common issues and errors sometimes it's licensing checks sometimes it's ABI symbols. It works not just on the RPM spec but on binary RPM as well. And it also compares a new update, updated RPM with a previous version of the same RPM. So it gives you in, it highlights when file chain, there are some important file changes like you get in new binary with set UID bit which is potentially security threat. So this is the tool. This the tool itself is on GitHub so for more information you can go there. So RPM Inspect is our main example of generic test. It runs on every package not on per package basis. Should we use RPM Inspect instead of RPM Lint? I think they're not completely overlapping. So RPM Lint is more about the form of the RPM spec while RPM Inspect also goes deeper. So for example, it does the Anubin checks and things like that. So I'm not familiar with the details in both tools. So maybe it's a good question to ask to David Contrell who is the main maintainer of RPM Inspect and he is very open for feedback. So you can go to the GitHub repo and file an issue with this question. Definitely something to consider but I'm just not sure about the details here. The other two tests which we are preparing currently you may meet them already in the bot interface but we are still working progress is RPM Dep Lint and installability checks. RPM Dep Lint check verifies the dependencies of a package of a new package as satisfied in the current state of the repo. It's quite simple but it catches some interesting errors sometimes. An installability check is a verification but the package you've just built you can install then you can remove it. You can also upgrade it from the previous version and then downgrade the upgraded one back to the old one. So these tests are actively in progress. So we're working on them right now. There are more ideas and tests in consideration but like as usual resources are a problem. People resources mostly and definitely there are a lot of ways how to improve this part. If you're interested in learning more about this test then we have Fedora CI Special Interest Group. It's in Fedora Project Wiki. We have Fedora CI channel on IRC and we have also my list which is called CI. Now I promise to explain how you can run your tests actually and this is where it gets more interesting. So the question to you as the audience is like do you have ideas for the tests which we might run as a generic test for all Fedora packages? So first of all like if you have idea how you would like to test one specific package then the easiest way to add such a test is to contribute this test to the this git repository of a package into that this git test workflow. Obviously to make it work you need to first talk to package maintainer and see if package maintainer agrees with the idea of a test then you contribute it to the repository via pull request. We recommend to use pull request for such work because on pull request we run a preview of a gating test and we post results of the gating test of the this git test in the pull request interface. So you kind of know before you merge if the test which you contributed actually works. If you have an idea of a generic test which might be I don't know checking build requires and not just requires like our PM Deplin does or checking consistent consistent version in between some packages, whatever logic you feel like you like Fedora to have enforced then if this check is simple enough to be run in a container environment which means it's not exactly that simple because we run virtual machines in container environment as well with no problem. So if you have a test which can be run as a script in virtualized, containerized to virtual machine environment then we now have resources and possibility to implement it as a part of Fedora CI generic checks. So to implement such a test you can just go directly to Fedora CI SIG what we would need to make it work is the test script itself which would be run on certain container image. Once we have that, we add the Jenkins Roper around this script and around this container so it becomes a Jenkins pipeline. You can create your Jenkins pipelines on your own if you would like, just take a look into the doc and clone, for example, RPM-deplined repositories and do the same thing but for your test. Or you can come to Fedora CI SIG and we can work on that together definitely. So one of the highlights of the current state of the things in Fedora CI is that we are refactoring our Jenkins setup and these new Jenkinses now give us more flexibility in how we work with Jenkins pipelines. We are now connected to Fedora account system so you can retrieve your jobs just by using regular Fedora account but also you have a more simple approach to writing the pipelines itself. So we decoupled a lot of things and now writing pipeline usually means specify the pipeline metadata, specify the container in which the pipeline should be running and specify the script which the pipeline should be running and the rest of it is the library functions which will be just like triggering and sending messages when it's all done. So we're open to such work and we're welcome all ideas which come from the community to this area. Now the other part about how to add your tests to Fedora Rohite gating is how you actually can add your CI system. This use case is more interesting for people who maintain, let's say, their own hardware labs or certain specific environments which you cannot share and you cannot reimplement on top of basic OpenShift cluster. If you have a dedicated hardware sitting in a lab somewhere in a dark room and you want to test in advance that let's say update of Fedora Kernel or update of Fedora LibWirth doesn't break your staff on that particular hardware. What you can do is you can maintain your own CI environment, CI system and then you can onboard this CI system to the Rohite gating workflow because as I showed before the only job of the CI system is read the message from the Fedora messaging bus and send the message with the result. Of course there will be logical assumptions on CI system so we can vote in Fedora packages. So if you vote with the test result but don't provide logs we can do anything about this. So such a vote is probably useless thus if you want to have a CI system voting in Fedora it can be a private CI system but it needs to publish some of the debugging login information in a way that Fedora community can read that and can understand what's going on. Of course additionally to that you need to talk to people which you're supposed to test. So they approve and agree with the concept of the testing and they will not just ignore the test results you're posting and you have some technical things you need to do like choose a name for CI system the ID which will be the prefix of all your test cases you need to add some triggering mechanism and to request the right to send messages back to Fedora Message Bus. So reading Fedora Message Bus is open for everyone sending to Fedora Message Bus is restricted so once you set up a system so it reads the messages, posts the test results somewhere you have a history of actions done by the CI system you can request then the access to Fedora Message Bus and then all the test results start to appear in body. We have a more detailed documentation on that so based on the interest in the audience I can answer some questions let me know if you want to talk more about it. One important note about all of this setup is that we do separate the part of running a test and providing the test result and the part of making this test result blocking the update of a package. It can be done in totally independent way. You can add tests, you can run them they will be visible in the interface will be reported but then unless package maintainer explicitly agrees to block package updates on this test results this test will not block in the package from landing in Fedora Orhite. So it will not be considered as a required check in the gate. Configuration of the gate is a separate thing it lives in a separate configuration file in the dis gate repository. And we have a talk about that. I wonder, yeah. And basically every package has a possibility to write down for his or her package that this is the list of tests which should be blocking for my package update. And this will add this enforcement part to the whole process. Currently in Fedora we don't do any global enforcement on any level. So all the tests we run in Fedora CI are not providing feedback but we leave it to maintainer to choose if this test will be blocking or not. Some of the maintainers already chosen that dis gate test or RPM impact test to be blocking but not all of them. And yeah, not as much as I would love to see people are still are not quite familiar with the system. So by refactoring and by cleaning up some documentation pieces and we code and opening it up for contributions we expect to get more traction and more understanding from the community on what's going on there. And yeah, one note here is like even the very, very simple here which looks like an absolutely obvious thing actually can be very important. So as a Red Hat OSCI person we tried this sometimes and we saw that tests make a difference and you don't need to build too much logic at the beginning. You just can start with something smaller and grow into a larger set of tests once you are evolving in that area. So to be honest, this was faster than I expected. So I wonder at which level the audience is and do you have more questions on Fedora CI on Rohite gating or on specific implementation details feel free to ask. So I'll check how long CI system have to either report results or be skipped. We don't have any limits right now. As I said, we have gating framework, we have CI systems pushing results to message bus and results to be aggregating them and this system, it needs to learn certain corner cases, needs to add certain limits but it needs to grow out of the usage. So basically right now we don't see requests and problems with long test cases. That's why we don't introduce limits. Any result that is available before the next Rohite compose. So let's put it this way, you create an update. This is your package of version 1.2.0. We run tests for this particular update. And so we send a test message which says package X of version 1.2.0 past this RPM deplete test. So this result will be valid just for this package. And this means that this package can land into Rohite. Once the package lands in Rohite, result is not going anywhere. It still is the entry in results DB but it's attached to that specific build which you tested. If you create a new package update of the same package and you bump version and it's now 1.2.0-3, then you get a new test run, a new test result specific for this build. Previous test result is not valid for this build anymore. Does it explain the question or not really? Okay, good. About how long CI system run? Question. We have an example of certain packages. Of course our favorite packages in on CI are GCC, kernel, TechLive, and a couple more. So I think GCC, test cases run for more than 10 hours right now. So we originally had a default limit of four hours per test for these good tests. But we added the feature into the configuration option so that limits can be extended to whatever time you wish. Basically our idea of this time limit is that if package maintainer is okay with waiting, we don't mind running this test for that long. So we try to give more possibilities to package maintainer to navigate this part while it works for package specific tests. Of course for global tests where we consider eventually adding installability check for example as a global policy, then in that conversation we will need to figure out the top limit which we must as Fedora CI system provide before we actually enable this as a global blocking test. So it's definitely not going to be just our decision to one day to enable the test and everyone is blocked. This will be discussion and conversation about limits about reliability and all of these parts. And one more thing I forget to mention is that I talk about like blocking and non-blocking tests but if we look at the implementation there's actually no blocking tests at all. So every blocking test, even if it's really a blocking test which package maintainer decided to run for this package and block on that, every test result can be waived. Because again we want to keep the control on the maintainer side. We expect that maintainers in certain cases where we just cannot predict some corner cases and what's happening with the test in this particular combination of packages, we give the freedom of making a final decision to the maintainer. So what will happen in such case is that you add your test as a blocking test and you configure your repository, your package updates to block on results of say RPM de-plint. Then you create an update, it fails the RPM de-plint and you see that RPM de-plint is just not getting the right information because this package is slightly different from what we expected and there is a huge explanation why it still should go into Fiduro-Rohain. Then in this case you can say I'm going to ignore this test result for now and let this package in, I will deal with this test and this package later if needed. But I know what's happening and I know that it's okay and I know that package can go forward. So even blocking tests are not completely blocking in our setup. Any other questions? If not, then again we welcome everyone at our Fidora CI RSE channel, MyLineList and Fidora CI Special Interest Group. We are open to ideas, we are open for contributions, our code is on GitHub right now under Fidora CI organization. So if you're interested in Jenkins stuff and in things like that, DevOps managing Kubernetes clusters come talk, we have interesting conversations about it. And yeah, we're open for adding new tasks to us and we're open to help you to make your CI systems testing Fidora updates. I think there could be a lot of hidden benefit for Fidora from having such environments. Thanks everyone and see you on all other sessions, including social and antisocial and whatever it is. See you!