 Hello everyone, my name is Karolina Sorma, I work at Red Hat as package maintainer. I do a lot of package updates in my daily work, most notably Python Sphinx, which is quite popular documentation tool, which has got over 1,000 dependencies. And every time I want to bring a new update of Python Sphinx to Fedora, it means that I'd like to be pretty sure that this update will not break anything. Lumiere has provided a lot of context just in the previous talk, why it's important. So if you haven't been here, check out, make sure to check out later the Lumiere stock. And the process I'm using in my daily work, I'd like to describe here today for you. This talk is going to present a very practical approach. I'm not going anywhere deep in my explanations, there is no time for that. It's going to be applicable immediately for Python packages and for packages that are built from our languages, maybe with some tweaks. I'm not very familiar with anything else. But we can talk about it later in the social event. The projected audience for this talk is me two years ago when I started my journey as a package maintainer. And I really needed a lot of context in order to be able to do my job. So how does this typically look like? There is an upstream project developer, someone who creates a really nice library. They create a new version. Me as a package maintainer, somehow get to know that there is a new version available that it will be really nice to bring it to Fedora, Rohide, because this is where the newest, freshest, hottest things should go to. It's the developmental version of Fedora, really good place to learn such updates. And that's it, right? I package it, I push it to Rohide, build it, I'm done. Well, this is the place where a decision can be made. I can either do that and do it just like push it to Rohide, I build it locally, it builds, great. Or I can just step for a moment and think, maybe I could leverage Fedora quality tools that are available in the ecosystem. And what would be that quality tools? In a very simplified manner, and please don't quote me on that because I have no idea how the Linux distribution is built, really. Distribution like Fedora consists of many RPM packages. And in the ideal state, those packages are buildable and installable. Unfortunately, that's not the case for all of them, but we aim for that. And in order to ensure that the update I'm bringing to Fedora is buildable and installable, I've got some local tools that I can use, which is mainly Mock, or I can build my update in container, or I can leverage Fedora CI. Fedora CI does scratch builds and runs any tests that you have with your package in the Fedora dist kit. And there is Zool CI, which does everything Fedora CI does plus much more. So if you're going to leverage Fedora CI, go for Zool CI. With the only disadvantage that it will take some more of your time and it would be best if you just opened a PR pull request to your own package in order to get any failures before you bring them to the distribution. But it's not only about the particular RPM packages. When we create the distribution, the whole point of creating a downstream distribution is that the packages actually communicate with each other. They play with each other nicely, which means I can build my package on top of the other Fedora packages, and I can use my package to build some other Fedora packages. Also, again, Lumiere has got a great graph showing exactly that. And for that, for checking the integration, we've got a process we called impact check. In copper language, it's called mass rebuild, and you can think of it as an integration test pretty much while the CI is more of a unit test. It takes care of the health of one package. Impact check makes sure that no other package stops being installable, sorry, stop being buildable when you bring your update to Fedora Rohite. In copper language, copper is a build system. It builds packages. Its main usage is not how we are using it. We are creating a lot of copper repositories with throwaway builds. And in copper manner, what I want, what I really want to have is this last build status succeeded. This is the only thing I care. So the process looks like that. I create a new isolated Fedora, virtual Fedora. I bring my updated, just updated package to that copper repository, build it, and then take all of the other packages that depend on my package and build them. If they all build, I'm happy. Probably my update doesn't break anything. If there is something that failed, and I really don't like that red journal there, HTTP, sorry, then I should investigate and take a look at it. So in order to show you this process, I took Python Rich library. It's a really nice library to build text-to-user interfaces. And the authors of that library and the maintainers of that library in Fedora didn't do me anything bad, so they don't deserve what I did to their package. I just removed the emoji module. I just prepared a downstream-only patch which removes one of the modules the package has. It's a silly update. It's just something I did downstream. The upstream developers wouldn't probably agree with me, but it's not so far from the truth. Many times it happens that when you have a major version coming from upstream, it contains a lot of removals or renames and some incompatible changes. So it can and it does happen. And I can still have some packages in Fedora that rely on this emoji module to be available to build and to run. So I'd like to know whether there are any and which one and what happens there. I've got 32 packages that require Python Free Reach on build time and 22 packages requiring Python Free Reach on runtime. COPRA is a build system. I don't have any means to check the runtime thing unless those packages have upstream tests. If they run the upstream tests, they should pull the runtime dependencies in the build time and then that would increase the visibility for me. So the question I'm left with is how many of those packages, those build time, those packages that have Python Reach on build time will just stop to build. So the greatest catastrophe that can happen is 32 failed packages. To start, I assume that each one of you has already done the Fedora packaging tutorial and has got installed the basic packages. In this process, I will describe, I will use tools that come from Fedora repository, some additional ones. Fedora Packager contains PKG name, very nice tool to get just the package name from the whole RPM name string. Fedora repository contains repositories I can query. I don't want to install packages to my system from Rohite. It wouldn't be a good idea, but I want to query through the RPMs and the metadata. There is a nice binary COPRA CLI to enable me interaction through the command line so that I don't have to click on the web interface of COPRA and Morita's parallel comes with a binary parallel to speed things up. If I want to send 1,000 builds to COPRA, I don't want to wait consecutively one after one. The really great tool that comes with DNF is Repo Query. It's complex and it can do a lot of things and it probably deserves its own talk, so I'm not going to go into Repo Query and what it can do. The basic commands to query the repositories is just as follow. With the first command, you will list all the RPMs, all the build RPMs that are in the Rohite repository and all the source RPMs that are in Rohite-source repository together. Dash queue is an option to suppress the metadata check which is not something I'm interested in, so all the commands have this optional dash queue. For a released Fedora, the query is a bit different, so I just put it there for documentation purposes, but I will only continue with Rohite. So first question, I need to know what gets built from my package. I already told you that I just destroyed half of the Python reach package, but I'd like to know what is the build RPM, what is the name, the other packages will depend on. In Python case, the convention is that the source RPM are not versioned, I have Python dash reach, this is the name of my Fedora component, but what gets built is Python free dash reach and this is the name I'm going to use in the following commands. There can be multiple build RPMs coming from a single source RPM and in such case, if you'd like to do an impact check, you should probably repeat all the following commands for each of the built RPMs that come from your source RPM. Any other packages can depend on any of the built RPMs that come from your. I will not begin to that too much, build time and runtime dependencies. For the Python world, the same invocation, if I want to know what Python reach requires in build time, I need to query repository with the option dash dash requires and provided the source RPM name, there can be the full name of the RPM and what I get is the full list of build time dependencies. If I'm interested in the runtime dependencies, which for the purposes of the stock, I'm not, I would invoke the same command with the build, with the name of the built RPM. It's significantly smaller, but also I can see that the runtime dependencies are listed in the build time dependencies. So my package pulls them and probably tests with them. So I'm fine here. Back to the build time dependencies. I want to know what other packages require my library Python reach to build. I already told you that the answer to life universe and everything is 32, but how did I know that? There is the commands, of course, like for everything, the repo query, that will tell me. There is the option dash dash what requires, which takes my built RPM as an argument recursively so that I can get the full impact, the full picture of what would get broken if I really push to the update to the repositories. And I'm interested in only in the source RPMs, which are the ones that expose build time dependencies. Those are the ones I can check in copper. We've, from all of the results, I will type them to the PKG name tool and this will tell me only the package names. So if I run the same command, you can see that it gives me something. And if I make my bash count, that is 32. This is what I want to build on top of my update in order to assess the sanity of my changes. Knowing that, I can go to copper. Anyone can use copper, you only need to have fast account, federal account system, and API token, which you can easily get from the copper federainfracloud.org API, slash API page. Once you're set, we need to do those five steps I mentioned at the beginning. So we need to create a new copper repository, add and build our package with the update, add and build all the other packages. First of all, I did all of those steps just before the talk so that I don't clutter our time typing those beautiful commands in my terminal. But we will go to the live demo with the results. So first of all, I define a handy variable only so that I can reuse the commands every time I do a new impact check. And I do them a lot. Creating a new repository means that I need to invoke copper binary with create argument. It takes another argument name of the copper I want to create. I want to create it for Rohite. I don't care about architecture really because Python packages are mostly no arc, but it may be different for your case. There are different routes. There are also not only you can also build RPMs for other systems, not only for Federa using copper, so it's really universal. The important thing is that I'm adding a new repository to my copper repo. It's not only looking at Federa-Rohite, but also to Koji. Why is that? There's quite often a little bit difference between what's in the Rohite repo and what's in Koji. Packages change all the time. You can start working on it and the state in which the repository will be 20 minutes later will be completely different. So sometimes it's like some changes sit in the Federa repository, but the maintainer deliberately or by mistake forgot to build it. So we are grabbing the latest changes, the latest packages, the latest versions from Koji in order to test on the hottest, freshest, newest. And because this is a throwaway repository, I'm not going to keep it forever. For small updates, 14 days is just enough, so I'm deleting after 14 days or 30. Only big updates require more time. Adding the package. I already conveniently destroyed my package and sent it to my fork of Federa-Rohite's git, so it's there sitting in the branch devconf. I can add that package, I can tell copper that I want to add it from the source control management system. So for that, I have an argument at package SCM. And again, to which copper, what's the URL where my package sits, what is the branch name where I pushed the update, and last but not least, the name of the package I want to build, which is Python Rich in my case. I didn't have to go to my fork of the package, I could only build my local source RPM, but then I would not, in order, it's easier for me to switch any automation I would like in order, if I wanted to tweak with my package and push new changes to my fork, then copper has the ability to pick them automatically. When I have added my package to copper, I should build it. For that, I have a command copper build package. Where and what? And now, I should wait. This is important step. I should wait for my package to build successfully and only start pushing new packages for that when it's ready. If I do it too rapidly, and the package will not be yet built in copper, the copper will have no problem with going for my Python Rich from Koji repository or Rohite repository, and I will not test at all what I wanted to test. So provided that my package actually built and it built successfully, if it didn't build successfully, then I need to repeat it so long as it actually builds, otherwise I will not test anything again. I can use the utility code parallel to speed things up. And another great feature of copper is that it can allow me to add packages directly from this kit, and if I don't specify which this kit doesn't mean it means federal this kit. So I'm adding packages that are sitting in federal this kit, all those 32 packages to my previously defined copper. I want to switch on the webhook rebuild. It's really nice piece of automation I just mentioned before I didn't call it like that. Anytime a package maintainer, any other package maintainer does a change to their package while I'm doing the impact check, while my repository is on in copper, the copper will automatically know that there was a change of state in the Rohite repository and will automatically rebuild the newer version for me. So I don't have to think that oh, I did my impact check two weeks ago and it's probably still now because the packages move so fast. My copper will pick it up. And another argument, dash, dash, name, another option, I need to provide to the command all of the packages that require my package on build time, which is basically again the output of the command you've already seen piped to sort and unique just to be sure that there are no duplicates. There shouldn't be, but it doesn't hurt. Last step of the process, building the depending packages. Again, using parallel to speed things up, I can send to parallel the command build package and it looks very similar with some new options. So no wait, dash, dash, no wait is an option not to clutter my terminal output. I will not have to wait for each build to conclude in order to be another one to be sent. And the factor of niceness, background. Copper has got different queues. There is a normal queue for normal packages and there is a queue that gets populated in the background. So when someone has got a priority work, they should probably use the main queue, but impact checks are rarely that important. So it's only nice for the other users of copper not to clutter all of the available workers with my 1,000 packages that I will just remove when I see that they built successfully. So background is really nice option, please use it. And again, I need to tell which packages I want to build. So this is my magical invocation. When this all happened, my biggest question is what failed? This is what I really want to know. Maybe nothing, that would be great. Maybe something, that's not so great. With copper, I've got another handy API to interact. There is a monitor. Each copper has got information about its state. How did it go? If I just run copper monitor like that, the output will be provided in JSON, which is not so nice to interact with from my command line. So I can tell that I want my report as a text row and I only care about fields state, whether the build succeeded and the name of the package that succeeded or not. And in this case, I'm only interested in what failed. If I do that, I hope it will work. Yes, there are four packages that failed in my copper. That's not too bad. I could go through the build logs of all four of them, but it can easily happen that I have 10 or 20 or 60 packages that failed. My Python interpreter colleagues would have thousands of packages that failed. It's not that nice to go through the build logs. What we often do, we use that command to create another control copper. And I have the habit of naming it completely the same with the dash control in order to speed up my typing kung fu. So with that control copper, I would rebuild only the packages that failed in the first copper to look for false negatives. This could look pretty much like that. So provided that I got one copy repository with failed packages, I created a new one, control repository, which rebuilt only those failed packages. And from those, I'm only interested in those packages that actually succeeded. This is my difference. What failed in the first one and what succeeded in the second one. If the list is long, then I can use another Linux utility to compare the both results. In my case, it will not be that bad because I've already done my copper control just before the talk. And now I see that there is only one package out of those four that actually succeeded without in the copper, which did not contain my update. So this is my problem. This is my culprit. This is something I need to really look at. That's the process. And now we come to results evaluation. And this is for now a manual step. All of the steps I showed you can be somehow automated. You can write batch scripts. You can even script opening a browser with logs from the command line. It all depends on the workflow, which you are used to. Each one of us in our team does it a little bit differently. So we don't even have one process for all of us. But we have automation to some degree. In my case, I will just go to copper for the moment. I'll show you the most mundane thing that can happen. So I'm in my DEF CON prep, which is the repository I was doing all the work before the talk. I will find packages. It was Python Typer that I'm interested in. Oh, it actually failed. Not that I didn't believe the CLI, but it did. There's a lot of builds and they all failed. That's not a good sign. It's not probably just a flaky package. You can see that my build was running in the background. And there are the logs. There is the root. This is the Rohite build, which failed. And there are the build logs that I'm interested in. What gets open is a really nice build log. Everyone who built a package knows them by heart. And I'm of course interested in whatever is at the end of the build log. And I see that some tests failed. And when I start looking at what failed, it's like emoji has no attribute replace. You remember what my update was about? Oops, I've got it. So this is something I would not know if I just pushed my update to Rohite and be happy about it. This is an actual problem that I would introduce as a package maintainer who didn't care about all the rest of the ecosystem. But right now I'm looking at it and the real maintainer of the package Python Typer doesn't know about it. They have no idea that this is going to happen. How would they? They are not going to actively go through all the packages their package depend on. So when I'm looking at that at this point, I'm the most competent person to actually do something about that. And the bare minimum would be talk to the maintainers of Python Typer and tell them heads up. Hey, I'm going to bring this update and this is going to break your package. You can do it by any means possible, maybe opening a Paxilla ticket. If there are more packages than one, if there are 10, 20 broken packages and you don't know exactly what to do to help all of them be again buildable, you can write to Fedora Devo and say, okay, I'm bringing this update. Those are packages that are going to fail. This is some guidance I can provide but please in a week I'd like to merge it and maybe you should take a look at it. If I'd have 60 packages that fail with my after checking my impact check, that could be a good material for a Fedora change. So increase the visibility of the issue. And if I'm preparing a really big update, then most of the time I've read the change logs and I know or I can find, I'm in a better position to find a migration path. So I can help from just telling the maintainers, reporting it upstream, fixing the issue upstream downstream, increasing visibility to providing some guidance. I can pick from a big variety of activities in order to help make all of it smoothly. Seeing that we are going to an end, coming to an end, I just wanted to tell you about one big copper gocha. Some packages are quite aggressive when it comes to dependency version constraints. And they say that this package only will work with my library which has a version lower than 14, for example. My update doesn't bring such version bump. But if it did, I would expect copper to say, okay, I have Python free reach 15 built in copper. This package requires Python free reach lower than 14. So we are not meeting the condition, so the build fails. That is a very reasonable expectation, except in copper it would not work, it would be false. Copper has got no problem with skipping my just build version with update and going to Koji or Rohide and take the version of the package of the dependency that it needs. And in order to mitigate that, there is this beautiful shell snippet that will go through all the dependencies, all the packages that require my library and check whether they require my library with the lower than character. And if they do, it will print them out. And in my case, Python reach, I will not show it, it's a bit slow. But actually, there are two packages that require Python free reach lower than 14. And that would be, again, my big candidate to reach out to the maintainers of their respective packages and tell them, hey, your packages will stop being buildable, installable and usable when my update reaches Rohide. There are some further resources. This is a piece of a tribal knowledge that my team has assembled. We are trying to write it down and prepare it for further generations. So the impact checks are described in our poor request review guide. Also, copper has got pretty usable documentation about the master builds. And this presentation with all the code snippets can be found on GitHub. This is it. Thank you very much for your attention. So if I have any idea how we could speed up things when it comes to the browsing build logs, it is yet to be done. Actual build logs, the, I will not. Okay, so the URL is, okay, so the URL is divisible. I only need my copper name, the build number, the package name. So I can get all of it from copper monitor. I can automate creating this URL where the logs sit. Then I would need to download all of these logs and somehow assess how much of this output I need to take and maybe provide some automation of results, some aggregation of results, whether a certain string is found so that I don't have to just go through 10,000 logs, but in some nicer place. So yeah, it's something that can happen, but it didn't yet. Maybe my colleagues have some ideas, some AI engine could. So yeah, maybe this is what will solve all of our problems. The issue is that the outputs of the failed logs can be really weird and different. Sometimes you need to grab more context. It's mood to go through the logs of packages that don't build for half a year because probably their failure is totally unrelated to my update. It's definitely doable to some extent. I don't know to which and none of us had actually much time and cycles to do it. So maybe this is waiting for some really brief community of folk who would just come and solve this problem for us. I'm sorry, I hear you really badly so I didn't even grab the core of the question. I'm sorry, I still didn't get the question. Maybe we can talk about it later when I can actually hear you. Thank you.