 My name is Miro Hranchok. I am one of the Python maintainers in Fedora and in this talk, I will try to tell you the story about how we updated Python to 3.9 in Fedora. Fedora 33 is the first not really, but we can say it's the first Fedora with Python 3.9 and about the not really I will get to that. First of all, how new versions in Fedora even work? What's going on? So before Python 3.9, there was a new Python version approximately every year and a half. It was not quite often the very same time. So the new Python in a new Fedora, it was kind of random. We could expect something and usually free Fedora releases would have the same Python version. Since very recently, new with Python 3.9, there is a new Python version coming out each year. That's for two Fedora releases. So every second Fedora release, we will be updating Python if everything works out. The Python schedule was actually adapted too much to Fedora schedule. So now we know that the new Python version is released in October, which is usually also when the Fedora is released and Python developers upstream make sure that we can make it and we can work together to update it in Fedora early. It's important to note that the development of a new Python version takes the whole year. So when a new version is released, immediately work happens for the new release. And this is also true for Fedora because we could either integrate new version really soon and get involved in upstream development, be it Python or a lot of other software written Python that we have in Fedora. Also, it means that we can get our hands dirty because it usually involves a lot of patching and making sure things work together. Basically integration. This goes really well with the Fedora foundation first because we really try to be first. Ben says it's really great that upstream projects align their schedules to ours. I know it has been really, really nice working with Python upstream to get this done. And it worked perfectly in Fedora 33 and I hope it will work perfectly later as well. Fedora actually doesn't only offer one Python version or two as it used to be. Currently we have a long list of Python versions. This is just so you have an idea. This is mostly intended for Python developers. So if you code in Python and you need to make sure that your software works with multiple Python versions, Fedora is your best platform of choice. We have multiple old Pythons, even some new Pythons. We really care about developers who use Fedora as their work machine, but they need to deploy for REL7 or even Debian and stuff like that. We also have different Python implementations like MicroPython, Chython or PyPy. But what's important is that one of the Python versions is the main one. We call it internally the main Python because default Python was a problematic term. And what this means is that in one Fedora version you only have one main Python version. For example, for Fedora 29 to 31 it was Python 37. This is only now relevant on 31, which will soon die, hopefully soon. And for Fedora 32 this was Python 38, which was originally planned for Fedora 31, but we didn't make it. It was before the schedules were aligned. So now when they are aligned it should usually work out that there will be two Fedora releases with one main Python version. For Fedora 33 and 34 that's 39, which this talk is going to be about. For the next two we plan to do three then. There is actually already approved Fedora change for Fedora 35, which as usually is the very first Fedora change for the new version. People now care about 34. We are way ahead of you. We are already planning for 35. What's different on this main Python version is that there is a stack of packages. Fedora has thousands of packages that use this Python version and it has zero packages that use the alternative versions except some launchers and test runners that allow you to test your software on multiple Python versions. Fedora itself runs on this. So whether you install packages, whether you install Fedora, whether you actually make packages for Fedora, you are using Python and you are using the main Python version. It's fairly critical and this is the Python you get when you run Python 3 command or recently or not so recently when you just run the Python command, you will get the main Python version, which is 39 in Fedora 33. We realize that while many developers use all the Python versions that they care about, the majority of our users will only ever use the main Python version. So we try to make sure everything works smoothly and that there are no issues. If you look at the some statistics, I usually wanted to show you some images, plots and graphs, but I didn't have time to do that. It's really complicated for me. So I prefer text. This is some historical data. I didn't want to go too deep into history. There was a problem because there were usually Python 2 packages before. Now we have Python 3 packages. But you can see that the number of packages in Fedora is growing steadily. The numbers in here are how many percent of packages is more than the previous release. So basically it's around 10% each release. And for five releases difference, it has been 68%, which is quite a lot. And it's approximately 10% of all the Fedora packages, which I would say is one of the greatest stack in Fedora when it comes to the number of packages. I didn't do any checks, how many Perl packages or Node.js packages there are, but the 10% I would say is quite a lot. When we have these packages, the thousands of packages, they require a specific Python version. For those of you who know packaging, the RPM packages have free environment on Python X, Y, which is the version that was used to build the package. Many Python libraries installed these specific directories that are like versioned. So you can have multiple Python stacks if you want to. Can I do that? Yeah, I can do it from here. So if you have an application that embeds Python, or if you have a cache file for the specific Python version, it's tied to the XY version that would be 38 before and 39 now. So this means that every time we actually upgrade Python, we need to rebuild around 4,000 packages in Fedora, and not only rebuild them, but we need to rebuild them in the proper order. Because in order to build A, you need B, and in order to build B, you need C and C and D, you know that dependency hell. And so when we build this, we need to maintain some clever ordering. And obviously packages don't always build. So sometimes you try to build something in order to build the next package, but you need to stop and go fix something. What's important is that Python is not backwards compatible. It's mostly is, but not completely. And when we upgrade from 38 to 39, it should be considered a major upgrade. There were other maintainers in Fedora who would ask, why is this so broken on a minor upgrade? It's important to know that Python versioning does not follow semantic versioning, because it predates it. So this upgrade is pretty major. So what do we do in our team, which is either the Python special interest group in Fedora or the Python maintenance team in Red Hat? Mostly it has been me who worked on Python free nine, but there were other people as well. And I will thank them specifically later. But what happened in November, and this was November 2019, which is a year ago, that there was the first alpha release of Python. And what that means for Fedora is that we packaged the new package called Python free nine for testing purposes. So people could install it, like you can now install Python free 10, and developers could run their software with the very first alpha. There was an article in Fedora magazine to promote this, if I remember correctly, maybe later. What happens is that we take the Python free eight package, we fork it, and we adapt some macros, like variables in the definition of the package. For example, there is like what's the base version? It's free eight. So we edit it and say free nine. And we rebase our Fedora specific patches on top. We use git to create our patches. So it's fairly easy. And we remove some old craft that's no longer relevant to a new Python version. What happens often is that there are some patches of features that were developed in Fedora first in coordination with Python upstream. And we can ditch them out because they have been incorporated into the new Python version. What's interesting is that this new Python package for Python free nine can be built in two ways. Either you build it as an alternative version, or you build it as a main Python version. It has a beacon for those of you who know packaging. So the same spec file can be built twice in different environments. And we don't need to care what Python version is the main when we create this. Since Fedora 33, thank you, Neil. There is some explanation in the chat. Honestly, I can't read all the chat. It looks like a lot of hellos and stuff. So if you have questions, don't be afraid to just pick up or something. Since Fedora 33, there is no more Python free source package or component that used to be there before. We only have Python free eight, nine, 10 components and one of them builds the main Python free package from that. What we do next is that we create a copper repository. Copper is a place where anybody from the community can build their own packages and users can then possibly use the repositories to install additional software to their Fedora machines. Or Fedora packages can use it as a playground when they can test new stuff, new features and make sure everything is well integrated together. We use this copper repository to do several things. I will talk about them a little bit more detailed. One is that we bootstrap the initial package set. This means that we build the packages in proper order which sometimes means we need to build something without tests and without documentation first in order to use that software later to build something else which is used to test the software there. So you need to bootstrap it. That's the term. What's interesting to mention is that we only use one architecture in copper to make it faster, which will obviously kick us back later. But at this point in November, a year ago, we only built on x8664. Once we bootstrap the initial set of packages, we try rebuilding everything else. We just shuffle it and then shake it and try it again and again until it builds. And obviously, there are build failures. And build failures could be critical. Either the package has missing dependency, so you need something else built first, which is usually solved by more bootstrapping or by shaking it harder and trying more. But obviously, there are also incompatibilities because, as I've told, Python 3.9 is not backwards compatible with 3.8. We need to report them in bugzilla for the maintainers and figure out what to do about them. There are also build failures for unrelated reasons. Unfortunately, that still happens a lot. You can't build a package because it doesn't work with new glibc or whatever, but it doesn't have to do anything with Python. But you still need to rebuild it in order to test it with Python. What's worse, sometimes you even have build failures that only happen in copper and not somewhere else, so you need to work around them. What happens in this copper repo later in the process is that we must rebuild three and a half thousand packages, which are all the packages that require the specific version of Python. You can see here there are four thousand and something total, but only three and a half actually need to be rebuilt. We need to keep up with Rohit. Fedora-Rohit is the next development version of Fedora. At this point, this has not been free yet. It was 32 still. Rohit is a very, very moving target, so you need to have some automation in place, otherwise you would never keep up with what changes in Rohit. We'll show later a little bit more about that. First, we start with bootstrapping the initial package set. We are using a RPM list builder, which is a cool open source tooling written by a software collections team. Basically, you have a YAML recipe with all the packages you want to build. It reminds a module file a little bit. Then you can have the beacon snail explained before like disable tests first and stuff like that. What happens with Python 3.9 is that we took the YAML from the 3.8 bootstrapping and we adapted. Unfortunately, and interestingly, Rohit is so moving target that the data from 3.8 is garbage. You need to really rewrite it. There are new dependencies, new dependency loops, new bootstrapping conditionals, and you might as well start from scratch, I would say. This is mostly automated, but it needs a human operator who would jump in and change the data a little bit when it doesn't work and then continue from there. We initially bootstrap about 500 critical packages. Critical packages could either be critical for Python itself. For those of you who are familiar with Python, it's set up to spit, bypass, swing, siten. Those are the packages that many of the other packages will need in order to build. There are also packages that are important for Fedora and without them, Fedora would not work. It's just RPM, DNF, Anaconda, and stuff. Also, we try to rebuild packages that are used to build packages like meta packages, so Koji, Bodhi, Fedpackage, and stuff like that, and also packages that are important for the Python ecosystem and have complicated bootstrapping, which includes stuff like SciPy or Jupyter Notebook. There are other packages that we consider important that are in the set, like C-Leng or GDB. Anaconda is written in Python, yes. This is not 500 critical packages that we would know. It's like 20, 30 critical packages plus their build dependencies. And together, this forms a cluster of approximately 500 packages, which is quite a lot, but it's still, if you compare it with the rest of the packages, you still have 3,000 packages more that we don't deal with one by one, but we build them all together. When this is done, this is how the YAML looks like, by the way. You have packages. First, you need to build Python. You disable some stuff to make it faster or to make it work. Then you need to build GDB and stuff like that, and then you build Python again. It has a lot of comments for the operator, that would be me, like you need to skip a certain test when it happens in copper, because copper has some different file system or whatever. I don't know what the difference really is. It takes a couple weeks to make this happen. And then when you are done, you build everything else. At this point, you need to first figure out what is the full package list. We use DNF repo query to check what packages depend on the specific Python version in Fedora. We get their names and then we put them in the copper. We disable RPM list builder at this point and use integrated copper automation. You can define packages in copper and make them use the Fedora sources. This is very cool because it supports hooks and it works out of the box. We set all the packages to use the sources from Fedora, and every time any package updates a package in Fedora, pushes something into the source git, the repository, the git, copper triggers a rebuild. You don't have to care row height getting too far away because everything something happens in row height, it also happens in the copper, and it's all automated. Sometimes it's a little bit broken, so it's not 100% reliable, but it's a huge help. What's also a cool feature is that if packages use pull requests, every pull request triggers a build as well. It's in an isolated repository, so it doesn't break all the other packages, but you can check immediately in our copper whether your new version fixes the problem with Python 3.9, for example. Obviously, we can also rebuild packages manually, which we do at a lot of times because when it doesn't build, you just try it again. Sometimes it's easier to build something five times before you even look at the build log and see the actual failure. It's in the cloud. We build packages in waves, so basically you just put the 3,000 packages in, go something different, and then you see, oh, 1,000 packages built successfully, so let's put the remaining 2,000 in and see what builds. Because every time you build something, it may be a missing piece to build something else. You basically push these waves until it helps. When it stops being helpful, you need to go and figure out whether there are no more dependency loops to be bootstrapped or stuff like that. What's important is that we triage and report failures. We use bugzilla for this, and I'll talk about that a little bit in detail. Also, we need to keep the package list up to date. As you've seen before, about 10% of packages are added every Fedora release, so we need to monitor this and add new packages to the copper repo every time something is packaged, which happens every week. Fedora packages, package, Python, they are crazy about it. They keep adding stuff, which is great. When we triage the failures, we have some computer help for this. First of all, there was a problem with copper API, which should be fixed now, so we couldn't use that. We wrote a little script that looks at this page and downloads the HTML source of it, and with regular expressions, it parses out whether it succeeded or whether it failed, which is obviously wrong and it cannot work, but it does. When we find the failure, we check the length of the build log, which is like all the stuff that the package outputs when it's being built. If the build log is short, we assume that's a dependency issue and we just try later. When the build log is long, it's usually an actual build failure and we need to investigate the reason. And when there is some reason, we need to report it to the package maintainer. Either it's unrelated, and fortunately, now we have policies in Fedora that helps us escalate those issues, and we block the appropriate bugs, or it's just Python 3.9 incompatibility, and we need a bug. We are at alpha stage, so it might be a Python bug, and we do this to discover bugs in software before it actually impacts anybody. We figure out what's going on, why there is incompatibility, and we offer help to the maintainers, but we ask them to go to their upstreams and fix their packages themselves, because it's hundreds and hundreds of failures, and it possibly can't be done just by the Python people. So I would like to thank all the package maintainers who actually are responsive and go and try to fix the issue and ask questions and we help them figure out what's going on, but they do have a lifting. They go to their upstreams and they fix it there. What's interesting is that often packages don't build, but they are not broken, but their dependencies are broken, but the dependencies have no tests, so they just build and ship even when they are broken. Unfortunately, a lot of packages in Fedora still don't have any tests in them, but it makes the build faster, so it also helps a little bit, but it's dangerous. This is a screenshot from Buxilla. It's like 800 something. Some of them are duplicates, some of them are not a bug, but most of them are actually fixed bugs of some failures. It's really tedious and all of this has been triaged mostly by me, and you just figure out what's going on, why it doesn't build, you try to explain it, and then you offer help to the maintainer if they need it, and sometimes you actually end up fixing it if it blocks you. In December, we are still at the beginning, a second alpha is released, so we update the Fedora package, which is only used for testing purposes for developers. What's nice is that when we update the Fedora package, it immediately gets rebuilt in the copper as the main Python package, and then we need to do some basic checks because at alpha level, there is no guaranteed binary compatibility, and sometimes packages that use C code to have Python extension modules, they start to sac fault with a new Python version at this point, so you need to do a partial reboot strap on the packages that are actually shipping some compiled code. There were also some problems in copper, so when you build one package again from the same source, it has the same version and same release, and copper ends up installing pretty much a random build when it needs it. Usually it's the older build, so it doesn't work, so we actually needed to write a script to go and prune the old builds from copper, which turned out to be problematic because every time we deleted old builds, like 3000 builds at once, it triggered 3000 jobs on the copper back end to regenerate the repositories, and I was contacted by copper maintainers and they said, no, don't do this, don't do this, we need to figure out what to do, and at the end, I was deleting builds by emailing people to do that. Now it has been fixed and there is an IPI to delete builds in bulk without this problem. What happens with the new version is that we just must rebuild everything again to see if there are any new failures or possibly fixes. It took about three days to rebuild everything. We run background jobs not to steal copper from all the other community members, and there have been some performance issues if the package is really simple, doesn't take long to build, it usually takes one to two minutes to start the build route and mock and whatever is happening in there, and then the repository needs to be recreated, which took eight minutes and couldn't be run in parallel. So there was a huge backlog of builds that needed to create the repository, and it didn't scale well. This has been changed in copper since as well, so now if there is a new creator repo job, the old ones are cancelled. In January, there was a new alpha and everything happens again. This is basically the same slide with a different title. Nothing new happening. What happened in February was more interesting. There was Fedora32 branching and Rohite became finally Fedora33. Up until now, some people don't consider Fedora3, bugzillas, very urgent. I get that. So now it started to be a little bit better. From March to May, I needed to stay at home for obvious reasons, so I was building packages. There were new alphas, new mastery builds, new stuff, everything happened all over again. Two minutes warning, I'm very slow, sorry about it. An important release was the first beta. There were no new features, so we could finally go into Fedora proper, and this is a basically point where we realize whether we do it or we postpone it to a new release. Now we need to announce, announce, announce, and in for everybody what's going on, and we build stuff in Koji's site deck. I don't want to go into details. I don't have time for that. When we reproduce the bootstrap sequence, mostly build everything in it and have some new architectures to worry about. Then we need to merge the site deck back, which could be either too soon or too late, so we try to find a compromise, and at the end, it was open for 10 days. There were 100 failures still, what we needed to go through and some collisions with other site decks. What happened later is that there were new beta releases that actually happened in Fedora, not in copper, but in Fedora proper, and we needed to be very sure not to break anything. The short story is that it didn't, so that was lucky. With Python 3.8, some of the betas did break stuff. Very important mouth was August. There was a Fedora 33 beta freeze and the first release candidate of Python, so we managed to ship Fedora 33 beta with the first release candidate, and there was only a dozen packages that failed to build. There was no problem whatsoever related to Python discovered during beta, so everything worked quite fine. Finally, with the final release in September, October, we updated to the second release candidate and then we pushed the final release one day after the final freeze, through a freeze exception. There were some packages that we needed to obsolete in order to unblock the upgrade path, which worked quite well. Most importantly, not Fedora, Python 3.10.0 alpha one was released, so back at square one, and we are starting over for a new year. We will be doing this forever, I guess. That's it. Thank you for listening. I hope I didn't rush it too much at the end. If you need details, I will be here and we can chat.