 Hello everybody, thanks for being here, we're going to present an alternative CICD workflow that we worked on for Fedora packaging and from there we'll start a discussion with you to see what we can do to improve and continue from there. So I am Mathieu Uain, this is Fabien Boucher, we are engineers at Red Hat and basically we live and breathe Zool there because we contribute to the Zool project upstream, we contribute code, we maintain instances for Red Hat, for other projects there and we also help people get bootstrapped with Zool. So now you might have a question about what Zool actually, because you might have not heard about it, because it's a relatively new player in the CICD ecosystem, but it's gaining a lot of traction because it has lots of very good features and actually talking about those features will take a whole different presentation. So we're just going to focus on the features that are really interesting for us right now. So basically Zool is a CICD system that was developed by and for the OpenStack community and it is well adapted to the pull request workflow. So first because it supports code review systems that implement the pull request workflow or something similar such as Garrett, GitHub, Pagure et Zool GitLab. And also it is event based, so that means that it's going to trigger job and builds for different events during the life cycle of the pull request. So that is especially interesting for testing pre-merge and post-merge. And also it is natively able to make cross-compository testing. So that means that you can test ahead of time before pull requests are merged. So you can test all of those pull requests together without having to merge them beforehand. So that's also interesting for packaging workflow because packages are interdependent. And finally, another interesting feature is the fact that Zool has an approach, as a CIS code approach. And by that I mean that jobs and configuration are written in Yammer so it's pretty simple and straightforward to understand. And the jobs in particular are written in Uncivil, those are Uncivil playbooks. So that means that while it's easy to maintain, it's easy to version and to develop and also you have access to the whole Uncivil ecosystem for your jobs and your tests. So all those features make Zool quite a good candidate for RPM packaging, testing and CD on that. And actually we talk, it's not out of nowhere, we talk from experience because we are implemented not one but twice RPM packaging workflows with Zool. So we did it once with the RDO community, so to package OpenStack for CentOS and we did it also for ourselves for the project of a factory which is packaging productivity tools like Zool actually and other things for CentOS as well. So we know what we're talking about because we've done it twice and we were thinking of maybe doing it three times for Fedora because there's no reason why we shouldn't share the goods with other fine people of Fedora. So the idea would be, the benefits for Fedora would be first a more familiar public request base workflow for newcomers because most open source contributors are used to this workflow so that would help, that would make the entry cost less steep for newcomers. Also it would open more possibilities for automation at every step of the pre-request lifecycle because as I said Zool is events based and reacts to events on the pre-request lifecycle. Also even if more experience packages were more used to pushing their changes directly the more experience packages could still be using this alternative workflow to trigger tests on demand, on their changes and get feedback early and fast. Also this system allows contributors to test dependencies together as I said before, so before they are published so you don't have to merge them, to build them and potentially break everything because you introduced a broken package in a raw height for example and that's a huge gain of time and also it's much better for the community because you don't break anything beforehand. If you break something, it's before, so that's okay. The body is impacted. Also, so the fact that most of the CI can be defined with YAML files and jobs are on simple playbooks should also lower the entry cost to contribute to the CI which means that the CI could be everybody's business which is ideal actually. Everybody should be concerned by the CI and make sure that it's working correctly. And finally we believe that the workflow should allow for easier integration with the CI of other third parties like other Fedora flavors and other Fedora vendors. Let's get a bit technical and talk quickly about how Zool and Pagure so the code review system for Fedora packages interact. So first Pagure emits events on its event stream that Zool will listen to. And on Zool you define some pipelines and if those events match trigger conditions for a given pipeline on the repository on the repository for which the protocol was emitted then you will trigger a set of jobs that are going to be executed by Zool. And on the other side, once those jobs are executed by Zool Zool will emits, will uphold the Pagure API to notify Pagure that the jobs are finished with a given result, failure, success, etc. So there is an example of a pipeline for Zool as it is defined. So basically you will tell Zool which triggers to listen to so we want to listen to a request event on Pagure for example and this pipeline check is going to be triggered every time we have an open request or a check on the request or if there is a comment on the request that matches this regular expression or the word we check in the comment. So we also want the request to be open so we don't want to act on a code that is already merged and then you can define what you want to do once the jobs are executed so at the beginning of the jobs you are going to notify Pagure to tell that the jobs are starting and then on success and failure you will add a message called success or failure. So you might wonder where this file which was Yaml file as you can see is leaving so everything is listed on Pagure.io we have defined three repositories to set up this CI so the first one where this configuration leaves is Fedora Zool Jobs Config so this repository defines which jobs and which pipelines are going to be executed for the diverse repositories. It also defines some jobs that need height and security context for example when you publish with Koji so you need extra security for that and it's a special repository for this. Then you have two repositories that act as job libraries so the first one is all these four jobs which is a generic one that gives you some unsymbol jobs and unsymbol roles that are already defined in order to interact with Koji with Copper what else with Mock, RPM inspect and so on and then Fedora Zool Jobs which is more in-arrays from those Zool Jobs but in a more specific context for Fedora also we have computing resources that are hosted by our team and we also provide maintenance of the images that are used and so on So Fabien, if you will So now let's talk about the jobs we want to implement such a flexible major workflow So first RPL scratch build This is a job that builds SRPM from the pre-request content then trigger Koji scratch build then retrieves the artifact from Koji so the artifact are RPMs and the build log Secondly, the RPM build this is a job that runs a regular Koji build Then RPM Inter and RPM-RPM Inspect are jobs that run respectively the RPM Mint and RPM Inspect command And finally, RPM tests that run the embedded functional tests on the test channel ML playbook that is contained in this guide So each jobs are meant to run at a specific phase of the pre-request life cycle So that's why we would project template that can be seen as recipes that associate jobs to pipeline So on the top right you can see the build and Mint project pipeline definition and at the bottom you can see how we associate a list of template to a repository So we wrote the following recipes So the build template that in the check pipeline will run the Koji scratch build then retrieve the artifact and expose the artifact to the child jobs Then the lint template that will run the lintter jobs so RPM Mint and RPM Inspect by using as a context the RPM built by the parent jobs So you can notice the use of dependency the dependencies keyword in the lint project pipeline definition so it means that RPM scratch build is the parent jobs and RPM intter and RPM Inspect will weigh up their child jobs and they will weigh for RPM scratch build to finish before they start Then the test pipeline that run in the test the test template that run in the check pipeline the functional test contained in the this guide by using as a context the RPM built by the parent jobs And finally the publish the publish template that through the gate and promote pipeline will merge the pull request and run the regular Koji job Koji build So now let's see the workflow we have built around the event pull request created, updated or rechecked So let's say when a packageer create a pull request Zool will get notified about this event and due to that event it will trigger the check pipeline In the check pipeline Zool will run the RPM scratch build that will perform the Koji scratch build it will retrieve the artifact from Koji then at the end of the job Zool will store this artifact so the RPM in a public storage because RPM intter and RPM test jobs have been defined with dependency on RPM scratch build Zool will start them in parallel so each jobs will fetch the artifact from the public storage and they will execute the rest of their task for instance RPM intter jobs will run the linter command the RPM int command on the RPM fetch from the store Once it's done Zool will store the result and it will return to the pull request as a command a link to each jobs result page and it will also return a CI status so success or failure in the pull request The second part of the workflow is around pull request approval and package publication So let's say packageer want to approve a pull request so it will do that by adding the gated flag so this is the tag on the pull request and if the pull request CI status is already success Zool will start the gate pipeline in the gate pipeline it will start the no-op jobs this is a special job that do nothing but always success and due to that success Zool will merge the pull request automatically on behalf of the packageer On the pull request it's merge Zool will get notified about this event so it will receive the pull request merge event from Pagure then it will start the promote pipeline in the promote pipeline Zool will run the coji regular build to publish the package on coji then as usual result are stored and returned back So now let's time now it's time to talk about dependencies so how we manage RPM runtime and build dependencies so dependencies between pull request can be defined using the dependency keyword followed by the URL of the dependent pull request so this keyword can appear multiple time in the pull request initial message to define multiple dependencies and this is really nice in the packaging context dependencies between pull request can be defined across different repositories and it is even better because it works also across different platforms like Pagure and GitHub so given a dependency graph Zool prépare job workspace with all dependent sources check out at the right version the source are ready to be used and validated and if dependencies are exposed by the dependencies Zool will expose dependencies to jobs as well so thanks to this feature we can write jobs able to under dependencies easily so for instance we can write a job that under RPM build dependencies so it can be a job that inject dependent RPM in a mock build route to expose in the mock route the dependent RPM to have the build as expected and this is a radio sum for instance when you need to bump a package when you need to validate the bump of a package that requires a bump on a dependent library et lastly regarding the runtime dependencies the RPM test jobs already under dependencies by preparing a YUM repository to make a YUM repository by preparing a YUM repository definition file to make the dependent RPM available during the jobs so no time to do a quick demo so I will create a pull request so I already prepared a branch on my fork of the Python Gear packaging so I will create a pull request so maybe it's going to be quite long so we are going to move on something else so this is this is a pull request that has been that have followed the full workflow so from the proposal to the publication so someone proposed a pull request and we see that run the jobs and report the job result as a command on the pull request it seems that everything passed except one job but this is a job with a non voting attribute so it does not count for the final result so here it is a success and we see that Zool reported a success in the CI status so it seems that the package was ok with the pull request and the update so to approve the pull request it just added the target gate it to the pull request and thanks to that Zool catch that event run the gate pipeline and merge the pull request on behalf of the package ok it's going to be a bit difficult to do the demo so Walfabien works on this so all the bricks are there believe us it's just a network program so we tested the workflow and it's pretty strong context and this works here we're just going to retry this so so the idea now is that we're going to report your no we can continue so I want to show also a bit the UI of Zool so this is I will be just able to show the result page not the artifacts so we see some information about the result we have also access to the artifact that have been produced by the job so this is the result of a run of RPM scratch reed so we can access I don't know if you are familiar with ARA but it provides nice report on cible playbook run so out of the run of that job we can click on the ARA report and show all the details so this is really handy for debugging a job you can you have also access to the repo artifact so this is a repository RPM repository that contains the package that have been built by the job ok so here the pull request has been un lid but I won't be able to show more because the network is not great but we see at least that Zool under the event and he is running the job at the moment ok so that's great so we see the RPM scratch reed is running we see the child jobs waiting for the result of the build we have also access to the live console I can quickly show so the use of dependencies so let's say this pull request so you can notice the usage of the dependence keyword followed by a dependent pull request so this is a pull request initial message so Zool under that and we can see this actually here so Zool prépare un Ansible inventory for each job and the Ansible inventory is exposed to jobs so each old jobs can use the variable defined in the inventory and here you can see that we have this dependent pull request that has been added the information of that dependent pull request has been added to the inventory and if we click here you can see you have access to the repository data the valid repository with the repo data and the RPM and we I don't think we have time to show that but in that in that pull request there is a dependencies to a pull request that define for the CI configuration that define a new jobs and we can see that the job by defining the dependencies so this is a new job, myLinter Zool run it for that pull request but this is only self-contained for that pull request so we can see first it was a failure so I had to fix my job my other pull request but after I reject the job passed so it's really handy for CI operator because now I'm confident as a CI operator that I can merge this pull request without breaking the CI the rest of the CI so as we said everything is ready you can play with it yourself and try it for yourself and give us some feedback cause we're really interested in knowing what you're going to do so we'll be at the Fedora CI and you can request as well and we'll be taking questions with you then yeah, as far as I know it's fantastic, it's a great job thank you second question are you currently or are you planning to publish some results maybe the messages for these results yes, it's completely possible and even in a nice way with a zool job so on a test job love job after all the leaders test, yaml and test board test, the current thing I have there and I run them champions and it's not always working is it compatible but I just keep running it both ways is it the same thing which is works so I'm going to run it now it's just me yes so I can offer for this yeah anyway it's one of the way for yes yes it's in parallel, you can play with it if you like it we can promote it something but it's in parallel, it doesn't block anything you don't even have to use it all information are in the project wikipage so you have the link again we have stickers let's talk tools c'est possible c'est possible c'est possible