 So today, the talk is about continuous packaging. So I'm quite like lucky. Just a quick introduction. I work at Radat in the OpenStack group. I was working in OpenStack company by Radat. I've been in the open source world since a long time. I'm feeling old today when I read this, but that's my aim. So I'm the author of a payment. You can blame me for this at some point. So sorry about that. I've been involved in a lot of different things. In Debian, long time ago, in Mandrake Linux. And lately, more OpenStack created an open source project. So I've been doing build system, packaging, CI, a lot of things since too long. A quick survey with a package, as I suppose everybody's package here, with a packaging, some upstream project that is moving very fast, a lot of comics every day. Yeah, some of you. So this talk is to address this kind of project. So where you have a lot of different comics and you have a hard time to catch up and how to cope with these kind of problems. So the talk is part of the four parts introduction. Why we do this. And an explanation of one way to do it. And deep, going deep into how it works under the hood. And what will be the next step in Fedora and what we could do together. So the vision of all this is that the packaging is doing integration. We gather some bits from different parts and assemble them, compile them, package them. And then we tie them. And we are doing integration. And the state of art in integration is continuous integration. But in packaging, we are not doing this. We are doing integration at point in time, at values point in time, not in a continuous way. So the idea was to bring to packaging the state of the art that is continuous integration. That's why the continuous packaging was kept up. So traditional way of doing packaging is that you wait for a new release happening upstream in your project that you are monitoring. Sometimes you are packaging an AC or this candidate just to look at what has changed. And you test your new package. And then you report any problem you have found upstream. You try to fix. You backport the patches. Because the release has already happened. You backport the patches to your current version. And then you can release the package to Fedora. That's basically it. Do you agree on this or not? That's the way you work, is it? No? Yes? I can't even. Oh, do you take weekly snapshots? And how do you work with your upstream projects? On the upstream, really? Yeah, yeah, yeah. That's why I'm asking you. Depends on the speed and the pace of the upstream projects. But usually, that's bad practice in the agile world, the XP world. It's a bad practice to do this big line integration and to try to discover all the things at one point. It's very difficult because you discover a lot of problems. And if you could do it little by little, it's much easier to spot what has changed because less has changed and you can adapt your packaging work to this small change and then continue little by little. Because at the end, when you have a big new version that is coming, you have tons of changes and you have tons of problems, basically. So what do we do? We do continuous packaging to solve this. And we did it for, we do it for three years now. We did it for three years in the open stack group. The idea, the big issues that we had is that we got a huge number of packages that are moving very fast every day like multiple commits in all the projects. The release cycle was six months with fixed dates. And we wanted to be ready at the end of the month, of the sixth cycle. So last release, it was 20,000 commits on all the projects that we were monitoring. It was a lot of changes and everything can be broken by any of these changes. And we had also a big issue is that we were developing installers. And installers were leading the packages to be able to function. So we had a chicken and egg problem. Because the installers needed to consume the last bit, but it was not packaged, so they cannot really start to work until the end of the cycle to test the last release. So it was difficult. So what do we do? We created something called Dolorean. We don't use Dolorean as a name, because it could be trademarked. So that's the area around that I will say Dolorean. So we switched to continuous integration mindset. So it means that each time there is a commit in the upstream project, we build a package. And this package is not only built, it's tested to be installed. We run the unit test during the build phase. And at the end, we create a repository that can be consumed by the CI systems and by anybody to do installation. So it could be the installer, it could be the CI system just to do some tests. And the good point for packages is that the system is in a reactive mode. You don't need to bother about it. It will build automatically these new releases of packages. And when something goes wrong, that's the only point where you are alerted. At the beginning, we were alerted by an email. So something goes wrong in your package. I don't know if the balance sheet has changed or something is broken in the unit test in the package build. You get notified. And now we went one step further. It's that we are creating a new gearing review that is representing the failure. And you just have to amend the gearing review to have the thing fixed. And then the loop starts again with your fix in your spec. If it's a fix upstream, you fix it upstream. And then when the new change is coming back, you have your fix augmented and built automatically. So it's a stream of packages that are coming. And what happens to the gearing review? No, you have to close it. You just discarded it and that's it. So you can take a look at this URL. You will see a status report with all the packages that are managed by Dora. So with the idea of all this atomic CI pipeline, the idea was to try to see if we could apply the same thing to Fedora and see if we could have the same system to monitor what is happening upstream. So what I did is I started an experimental $hand server for Fedora. And I picked two fast-moving targets. So I picked Ansible and SystemD, very different. And to see what was the benefit to do it. So in Ansible, they are moving very fast. A lot of commits every day. You have some issues that they don't track in their CI because they don't do release every day. And when we do packaging, we take the bits that are done like it was a package that is being created as a released target and they don't spot this kind of thing. For example, there were multiple issues with unit tests. For example, one of them was doing some network connection during the unit test. And we spotted this very easily. And we had also some issues with unit tests that were depending on some Python dependencies that were not installed and that were not documented. So we catch all this and join this with this build. And we reported it upstream and fixed it upstream. And then it was fixed. And we found also a release issue. They were doing some seemling magic. And when the table was created, the seemling disappeared. So we could not, at the end, use the table and the unit tests were failing. Systemally, we didn't find any issue with unit tests or whatever, but we changed the global system. So the spec file was not working anymore. So we spotted it and we created a new spec file. We adapted the spec file to code release at the right time and that was easy to track and not wait at the end for the new system to release. So that's how it looks. I took the screenshot at the moment where the system was started to fail. So you see a failing commit. And for each line, that's a commit upstream. So you have a direct link. You can take a look at the commit. And that's the timeline of what has been built. And you have also a repository that is available. And the build.joc, it's a mock output. And you can take a look and inspect if you need to inspect what happened. You don't really need to operate on the server. And? This report generated by the tool of the name of the tool? The Romin? Which Romin? I mean, the UIR is the VIR of Romin. The name of the tool? Yeah, the Romin is the name of the tool, yeah. But you can put it on your server, but just choose this one. But just an experimental server. And you have also a synthetic view, so now you can see what's the status for all the packages that you are tracking in your server. And this way, you can see that it has been 10 days to fix the issue in the system. Please. Sorry, sorry. And where are the builds actually being done? Is that happening in your scotgy? No, it's on a mock server. And it's a local mock instance. Did you restrict the mock server to not do network connection? Yeah, and on the new versions, that's what is going on by default. That's where we catch this issue. And change the version of mock and then, oh. Revolution. So if we go deeper, how it works, it's we have a file to describe the server settings like the URL where the file are located on the disk. We have a standard mock config file where we describe what is needed for our mock. And we will have it in an example. We have a YAML file, that's the important bit where we describe all the different packages and where this git is located. Where the git repository for the upstream project is located. And the branches we are using because some projects are we have bunch in the name, so we need to cooperate this. So how it works at a high level. Doloran is fetching all the repositories that are listed in this YAML file, extracting the upstream and the disk git. Then it's building a package using mock on these repositories. And then it's creating on success. Successful build and install with mock. It's build a repository with all the packages inside. And it's saving space by using series to avoid copying around all the files and things. Doing some magic once we've seen this. The versions of the RPMs track the versions of the upstream products. I think it's a magic. Yeah, yes. Yes, that's what we're trying to do. That's not always easy to guess the next version of a project. Some are well-defined and some are not so easy to guess. So sometimes we work with git to try to fetch the last tag and to increment it. And we have some tricks to try to copy these projects. That's black magic. Watching the git rebuilding and creating a repo. It sounds very similar to a copper. Actually, a copper might even be able to do that already. No, I think the idea will be more to couple copper with this at some point. There was the graph for that. The git rebuild on copper. That would just pull on the project and would be on copper. But that wasn't part of copper. So it would be good. Yeah, yeah. I'd say the good thing, when you have multiple packages like we have in OpenStack, is that we can do some kind of bisect or so after on the CI job. So we have a bisect, but on a set of git repositories. So we have 200 repositories. And we can find the one that is breaking the CI, for example. That's just for in-user testing. If you have somebody, say you're not talking about OpenStack. Say you're talking about some in-user application that you don't expect somebody using it to want to rebuild it to test. You have a continuous list of packages that you can tell them, try this one. Try this one to find whether the problem happened. Which is actually very useful. A lot of people do that with the kernel now. It's just that we can't really build the kernel more than once every day or so. With the kernel, we have some. I tried it a bit with the kernel, but the spec file is a bit difficult to work with. But it can be adapted. The thing is that we added also some stuff to build only on tags. So we could build all the RCE, all the other. And that could be another solution to build less often. That's possible with the Lorentz. The other way to say, so that relies on the file that you have seen as in the test? Not really, no. But it relies on that we can generate a tarball like Optimus doing during its release. Yeah, but then. And the fact that it says it's a way for you to test it, but it still builds if you don't know if it works. Yes, not after. The idea is to be used by the CI system. But it doesn't rely on the unit test. If we have unit tests in the spec file, we run them during build, but there is no unit test. That's not a big deal. We will test it later on by the CI system. And yes, the idea after what we do in OpenStack, we could do it also in general, is that we are doing some run in the CI system. And we are promoting this particular hash to be the one that has been tested by the CI system and can be used by human after, for test days, or for whatever you want to do with your test apps. So just a quick look at the configuration file. The first one is where you put your data. Where do you extract your Git repositories? The script here is just a local machine. The base URL, that's what you see. That's what is exported and we did it internally to generate the files, the HTML files. And then we have a bunch of variables. What is interesting is being maxed with files. We have some logic inside the Lorent that is catching some standard errors from infrastructure like a network connection or something. And it retries the build. Here, it's retries. If you have a mirror server that is not answering and these kind of errors that are not really related to the build and it will tag this build as a need to be retried. And we store everything in a database and we are working on some API to expose this to be able to do nice dashboards. And that's working for us. And for back to master, it's just a... If you cannot find a point, it's spring back to master. The log configuration file, it could be more familiar with this. It just installed a minimal base system with the RPM build and just what is needed to build the package. Just an extract and nothing really fancy here. What is more interesting is the file to describe the actual packages and the actual repositories. So at the beginning, we describe the release we are targeting. That's more symbolic. There's no real meaning in the names, in fact. We do nothing with FC27, for example. And after, what is interesting is that the package config and the package default are things that you don't need to replicate as they use by all the definition of your packages. And you can be specific or not. Here we define the maintenance and we are too to work on this. And so we just define it for all the packages. But usually we define a maintainer per package because that's more of a way to do it for a set of maintenance. And you define where the project are located. Yeah, it happens that we are both under GitHub. So we just specify templates that is GitHub. That could be a look at multiple package configs. So package default is the default problem. And then we have the two packages that are described here. So systemd and systemd is using master. By default, we prefixed with rpm- to see that the project is using the master branch. And for example, in uncivil, we are using devol as the master branch. So we are using rpm-dvol. It's just a convention. We can do whatever we want. It's easier to have something regular. Any question on this? So how it works in details after in the Lorraine, where the black magic is happening. So it's extracting the data here, which we saw before. So here, the data here is raw-aid data. So we extract in raw-aid data. We obtain repository under the name of the project. So it will be uncivil. It will be systemd. And then we extract the diskit repo. Or we update the diskit repo because it doesn't extract. It doesn't clone every time. We just update it under uncivil underscore distro, or systemd underscore distro. And you have your spec file and the usual thing on your diskit. And then it's switched to the right git commit for both. But we are targeting the build. And it's creating a table that's where the black magic is happening because it needs to guess what is the number of the release that will be the next, in fact, that will be the next known version. I assume each project has a different command that you used to step forward. Is that defined anywhere? No. Up to now, we were able to do it in a generic way. So we tried to do it like this. If at some point we cannot, we will make exception and we'll have something in the diskit that will describe how to extract the next version. Up to now, it was not needed. And then we substitute the tarbol name and the version of the spec and the release number in the spec file. The release number, we use the Fedora naming scheme for Perilis. So it's zero dot the SHA1 for the upstream. Plus the short SHA1 for the diskit dot 1 to other release. This way, if you update to the real one after it will be to the one release, it will be OK. Now there is a timestamp. There's a timestamp, and then the SHA1, and then the short SHA1. And after that, it creates a source RPM from the tarbol and builds and installs using Mock just to test using Mock or the dependencies of the resulting RPM and ROK. And then, on success, it creates metadata for RPM repository that is exposed for STTP that you can reuse anywhere. And on failure, it's like I said before, sending an email or opening a carry trade. So it's not probably good to use the other repository. So if you have package A depending on package B and you have already updated package B, is we really in package A going to use the latest version that's available in package A? Or is it going to be the one that the RAM builds? It will be the previous one that the RAM builds. And adding the Fedora. The idea is to, if we have circular dependency, when we bootstrap the RAM, we can have trouble. And we have also some ordering options to bootstrap a repository when it will compute the order of the required and then build them in the right order the first time. And then we let it build on the history of the Git codes. And then on this, it's organized like this. And you can take a look if you want on the website. You have a project where the Git repository is located. This Git repository is under score distro. And then under the repository, you have all the generated files. So it can be the logs. And then build.log is the main one you have to look because that's the output of Mock. And we will see what happens when something goes wrong. And you have all the RPL and the repo data. It's like a Git path. So if you have a lot of packages to build and it's taking a long time because of compilation or unit test, you can have some trouble to build on each code. So what we do in this case is that on each one, we take only the last code of a branch. And we rebuild only this branch. Only this code. And this way, each time, if it's taking a long time but there is not a lot of code, you will have nearly on each code. If it's taking a long time but you have a lot of code, you will have a big holes in your commit history. But you are able to keep with the pace of the project. I had to do it on my VM for Ansible because it was very, very fast. And then we have some script to do some dissect. If you really want to know which code has broken your build and not the exact code, you can look at the history of your site, using git log and see, okay, that's this one, that's broken something. But that's a reduced number of logs you need to look at. It's not the full history of overrids, so it's still better. So next step, so I want to get feedback from the federal community, if it's useful, if you think it could be useful. So that's why I'm presenting here. Could be to put an official server with a reduced number of packages to just test if it's useful for our people. Involve the official packages because up to now we did it ourselves and just to start and to see if it brings value. We haven't discussed a lot with the Ansible or with system D maintainers, for example. So we need to go one step further. Use the git, the real git repository because up to now that was our disk git because we wanted to test but it was not the official one. And we'll be able to use Pagure as a way to collaborate on this. Another idea was to create a plugin for module dv, md, for module md format, this way we can describe. I don't know if it's possible to look because that's this file that I showed, that's the YAML file that is describing all the packages of our repository. We could use a module dv because that's a set of packages that could be also a good way to not reinvent another format and to reuse information. I saw that there is already this git that can be put in this file, but I don't know if we can put the upstream git repository in this file. We need to discuss. I see it's possible. And then the idea, as always, like in OpenStack, is to connect the CI system on these repositories and see if something is wrong in advance and try to fix upstream before release with a CI test. Up to now, that's just unit test. And we do it in OpenStack, we can do it in GitHub also. It's to be even shorter the feedback loop, but we need more and more resources to do it. That's the whole back. And the idea is to catch issues before the armors. So to be able to do that, we need to run this packaging test and this integration test because at the end of this integration test in advance on each proposed change. So it could be a great review in OpenStack or GitHub or a bigger change. And we can try to build the package and see if something is broken, if something is broken, try to say, OK, guys, we need to fix something, or we need to fix the package. There's two ways to do that. So we can add a web hook to be aware that a new change is happening in the metadata project. If it's too much, there is an alternative is to download all the proposed change, metadata, and try to filter in some clever way. Usually it's more, is there a new debanancy, is there a new file, this kind of thing? We can just trigger on this kind of change, not on a code change in an existing module, for example. So that's one way to do it. So the big advantage of continuous packaging is to align the two cycles of packaging and development. And this way you can fix things without needing a patch that you will have to carry on the next version. And that's really changing the way we work. We work the two communities together, and then you fix things at the right time. Because when you discover something and you have to see the upstream project and see, if something is working, the release is finished, so I'm coming now, and it should be alerted before. And the nice thing also is spreading the load of this discovery along the cycle of the upstream project, not at one point in time. But that's a lot. And the big thing that is very good in terms of release management, and that's what we do in OpenStack, is that we can release the OpenStack for the RDO version one day after the release of the OpenStack project. So we'll release all the 275 packages one day after. Because we know everything is ready. We just need to pass the CI to rebuild men and pass the CI. And then we can release. There is no surprise, because everything has been rebuilt and tested all the time. And at the end of the release, we can release. Yeah, it's easy in OpenStack, because that's a coordinated release, we have released all the packages at the same time. But if we had to do all the packaging at this time, it would take around one month to do all this and to be sure that everything is done. Yeah, we can do it on one day. That's a big improvement. So let's put continuous integration in Fedora, and do continuous packaging and see what we can do together. Perhaps with the core of the system that is moving fast, see what we can do in the kernel with things that are moving fast, and let's work with the questions. Yes. I did have a look just quickly, and you've offered. It doesn't support adding a stack file in a different place than the sources that are coming from right now. It does support the output, so in theory, you could hook it up to a GitHub change and trigger a RDO. But that GitHub RDO, right now, we need to contain the stack file. I don't think it would be a huge change for them to add a different source set to be able to say, oh, the stack file is going to come in a different place and just want to do this in pre-pro, basically. It feels to me like a lot of the functionality I'm building, you're basically replicating opera, and it just seems like a waste. It's going to go into production, and you're going to try and do this. It means you're going to have to have a lot of built resources, and you were saying, right, that even you're here and you're going to keep up with the ensemble changes and that's just one project. So it feels like a more, I don't know, just better utilization of resources to just put it into what, okay, we have the official Koji build system, and then we have this sort of playground theme being offered, where people can either build their own projects or do the CI, hide the bills, and we can explore resources into it instead of just creating another system that's built. Like, I don't know, it feels like that might be good. The thing is that it's already existing, it's already working. You have to start somewhere. Yeah, yeah, sorry. And I was not at the original creation of these tools, I cannot speak for them, but. Maybe they saw some of the proper folks, and. Yeah, but I don't know if Copper can do, can manage a set of packages, because that's what is important here. You have a set of packages that are current, that you need to build at the same time, in fact. You need to build a repository for all these packages on every commit. Yes, you can, I would suggest you have a little bit of a little bit of a proper, because you can add multiple packages and different sources in a single repo and create a repo. I'm not saying it's gonna, for example, you were talking on the bisects, and I didn't find what are specific at which bit. And the reports, and there might be things that are not exactly the same, or might be just out of the scope of Copper, but in total, that's more than likely, I would say. But at least as far as building those, that's certainly in scope, I would say. I don't wanna speak for Copper's edge, but. They already have builds that can do different type of SCM, IPI, and regions. This is just like a different type of SCM build. Yeah, but that's not something that is really, that's really on a grid code, so. Yeah, they can certainly, they already have those. It's just, right now, that thing would need to contain both the spec and the sources. That's the only difference. If you have the spec file together with the sources, you can actually do that. That's not the case here, because that's the only way to fit all the packages that are in stock. I don't know, but, again, just in that one source then. Right, okay, next time. You have a question? So, I guess, so you talked to one head about applying to look at a set of packages, and he also talked about getting a friend to get every commit to your advice. I guess I'm curious about how that, those things are at, when you have, I mean, I guess, there's different kinds of stories in every package, but we can go independently for the way we depend on it. There are two levels, in fact. If you have a build issue, we do a git beset, and we find the issue in your story, where the package is broken. But if you have a test, a CI integration test, and all the packages have been built, we can do a beset on all the repositories that have been built in time, and find the ones, which package has been, has broken the CI test, in fact. So, you're basically affecting on the build as well? Yeah, yes, on the build history, but at the end, that's very close to the git history because we are building according to the git history. I mean, it seems, probably, if you considered like a large, like your kind of build, I don't know, on every package, included on it, that's probably... Yeah, that's why there are some resource issues that you need to balance and find the right way to do it. But, like I said, it's possible to do it in head only, and we can do that. And after, we can still beset and find which repository was working and which one was not working, because that's the same thing for CI. We cannot test in CI. All the repositories, we will choose the last one, try to test it. And then, if it's broken, we need to go back and find the ones that is in the middle but that is not broken, that's beset again. If you were going to do this in Fedora, you would need some way to get the results of the builds in front of those making, right? So, this is kind of a unique situation because it's actually a test process that happens before any content is actually entered in Fedora release stream, right? Yes, if you update it today, it's going to be broken and we should fix it. So, I don't think we have a good model for how that interaction would work with Fedora maintainers. Like, would you file a spectacular new Godzilla, and say, you know, the spec file in this game, Fedora disk game is out of date, relative to the upstream code, and you should properly look at it. Yeah, I don't know about some of the discussion. We could do something like that. I don't know if we'd be really accepted. Because whether or not something is broken at this moment in time is a different question of whether or not, like, a Fedora maintainer wants to pull in the upstream changes into the distro at this point in time. Yeah, so, and it's interesting, I just wanted to think about it. If you have something to put forward, it's a dedicated class model, it's an abstract game. If you update every package, is it going to be broken? I think we all know that the answer to that is always yes. So, and this is who we really deal with. I don't care if we're a good on containers, but good on containers is a, we don't have to build a good on containers thing where they build the head of every package within the scope of the good on desktop. And they are, I mean, it's what if the project was made, one is that back end people, they're gardening that saying it's broken, so they're not fixing it. In fact, there has to be some kind of a panic saying, okay, this particular project, logical, it's broken. We're going to bring it back to the way it is there, in order to get the other releases. That's what we do also in our stack because sometimes something is broken, we pin it to a particular company. That's the standard procedure. We introduce a patch in the packaging that is fixing, that's the patch that is in progress to fix the issue in the packaging. And while the patch is not merged, we carry it, and when it's merged, we remove it. That's another way to fix the issue. In terms of resources, this is really about, that's not probably one of the things built from source, which I think is probably, I'm sure, what about the packaging stuff? Working your mind to figure might want to look at some of the options around like build your caching and maybe a Ccache plugin, because that could reduce the amount of building that's actually happening and then it could just be the packaging stuff. Right now, it's only a small VM I was using, but we have plans to do parallel grids and to get used to OpenStack cluster because we were so difficult to make use of that. That's working for us on this also. The difficulty is when we build in parallel, we can lose the real history. In fact, the history could be wrong, in fact, according to a grid commit, but sometimes it's not so big deal. To be fair though, you know, build from source is, yeah, most back and most on-streams test that, but not on the system of 3.9 million. Yeah. PPC 64, both Indian citizens inside. And that's where most of my problems come from. They test, but they don't test on everything that we expect to shift on, which is, of course, even copper, I don't think I'll be there. But, oh, it does. The system is specifically about finding problems with the packaging. Yes, but it could be on a different architecture. Finding any problem that causes it to value copper. Yeah, exactly. Copper has E386 degrees to 66 degrees. Yeah, it's not enough. Yeah, but after that's the thing, we need computing resources to build anyway. There's sort of two very related problems. One, which is how would you CI on what, how do we, they always discredit and gather and test that if it works. And see if it works. Any question if you took all the time files of this N2 upstream? So it's a little bit different. Why would you, how it works? How does that integrate in the N2 would be the same type of connect across the upstream of the turbine as it would across the version? We have this to reuse the test, the test framework and all the things that are being put in place on these repositories that should do the same result than the one that is already released and we could have a meaningful result should have a more question between you and your lunch, so I don't want to. Okay, that's fine. All good, so thank you for your time. Thank you.