 If you don't know me, my name is Miro. If you know me, my name is also Miro, so it's easy. I'm one of many people who maintain Python and Python ecosystem in Fedora. Those are my colleagues, Lumiere and Peter, and also Thomas Schroeder at the back, he checks it from behind. And there are a lot of community people who maintain Python packages in Fedora. And we have this big issue. And up until recently, the biggest issue was Python 3. A couple of years ago, we flipped our thinking and now the problem is Python 2, because we just tried to get rid of it. And we have plenty of reasons to do that, and this is not mostly about reasons, but about the execution, about how we're going to do it. If you don't know it, the short story is that we're going to remove Python 2, and that's it. There are, of course, many exceptions, and we still try to do it in less intrusive way than just get it out of the distro and be done with it. Even if somebody, I think somebody proposes just to do that, just retire it and then let it all burn down. I'm going to talk about what was happening in Fedora recently, and when I saw recently, I think it's five years or something like that. And what's happening in Fedora 31 and what's going to happen in Fedora 32. This is no news, mostly except for the 32 part where we actually have something here that we want to announce and then document it later. So that's not on the web yet. So let's start with the Python 3 transition. Do you remember when Python 3 was added to Fedora? It was actually in Fedora 13, and it was introduced as an extra thingy you can have and this crazy version of, they called it by 3K, if I remember, and it was the new thingy. And it was even before we updated to Python 27, which is the craziest version of Python that now existed since Fedora 14.2, whatnot. And there were some cool changes. In Fedora 23, we said, let's make Python 3 the default and our idea was that we're going to switch everything and then throw away Python too soon. And it was really naive. And it turned out there are so many projects that depend on Python 2 somehow. So we start cutting the stuff and switching it over. And there we switched from YAM to DNF and to Python 3 at the same time. And we switched Anaconda and we switched a lot of stuff. I believe, but I'm not sure that this was the first workstation version without Python 2 installed when you install workstation. We had to keep it in the server. And I think you might know a lot about that. It was because of Samba, which has, what do they have? They have parallel program that generates SWIC files that generates Python bindings in Python 2. So you actually have to rewrite the parallel program if you want to port it. So we didn't make it there. But currently Fedora server is no longer even a thing, but Samba is ported already. Most important changes happened in last Fedora release. We kind of, it looks like we want to change something every Fedora release to make it a little bit harder for everybody. So we need to keep track. The general idea is getting rid of Python 2 and moving to Python 3. But there were some obstacles, like people calling things Python and didn't really care. And it picked number two because upstream recommended that if you call Python, it should be Python 2. And we started doing a lot of stuff. One of them was renaming everything from Python to Python 2. Then we moved the user in Python to an extra package. So all the packages that just depend on Python doesn't see this by default in the build route and it broke a lot of things. And we somehow fixed them except like 40 packages that just ignored all our advice and they just pulled in the user in Python thing. And now they are broken, definitely, anyway. We deprecated Python 2 in Fedora 3, which technically means you can't add new packages in Fedora that depend on it. And people tend to ignore such rules. But we have monitoring, so every time somebody does this, we go and we do, no, let me put it down. Anyway, the latest change for Fedora 3.1 that was already implemented at Rohit is that Python means Python 3. So if you type Python, it gives you Python 3. We chose previously against upstream recommendation but we changed the upstream recommendation and now we can do that, so we did it. All of the packages that ignored the you should not use user in Python as Python 2 are now broken, but it's like 20 of them and they failed to build from source anyway, so we retired all of those yesterday. You might have noticed on the Fedora Devil list that like 700 packages are gone because they failed to build. One of them was get text, so everything was broken. That's back, but there's gonna be some mess. What's happening in Fedora 3.1 is the mass removals. We already started this one Fedora ago. We continue and we try to be a little harsher. So the general idea is that if your package is only a Python 2 module, like something you would normally pip install requests or pip install, what do people pip install? Happiness. And then you have a Python 2 package with the module and nothing depends on it. Then why would you need to maintain that at Fedora? There was a certain period of time where people just said, packaged everything. Like, we need to have more packages so they took random Java stuff and random Python stuff and they just packaged it and now it's causing a lot of trouble because the contributors are gone and the dependency graph is huge. What we are doing is that we open Baxilas for leaf packages with modules and when people don't respond, we just kill it with fire. They usually don't respond. During those things, I opened several thousands of Baxilas for several thousand packages and the main issue here was originally that the maintainers just, they don't give a damn. You open Baxilas, they don't respond and officially you need to go to the non-responsive maintainer procedure, which is a great pain if you have to do it for one. But imagine trying to do 200 non-responsive procedures at once, so that was almost impossible. So then we invented our own mechanisms into the federal changes, get it approved by FPC and FESCO, get a rubber stamp and now we can do stuff like, wait a week and if there is no response, we just remove the package. If, can I somehow show you the graph? Is it here? So I believe somewhere around here we start dropping very, very, yeah, I will explain the cars. So the green one in here, probably somewhere in here where we didn't track it back then, are the packages that are Python-free only? The yellow ones are the ones that have Python-2 and Python-free package, but nothing depends on the Python-2 part. These ones have both packages, but they are dependent and those are something that's Python-2 only and the red ones that basically a dead zone because those packages that depend on other Python-2 packages are also Python-2 and this is not so important. So here we started dropping Python-2 packages. We are slowly moving up. Now the numbers are in here. So we have 70% Python-free only packages for the Python packages and 81 at least have Python-free but might not be only Python-free. What we do is that we also look at the graph which is huge, really, really huge and we try to identify places where we can break the graph and say this dependency over here is not really needed and then you can cut a lot of other packages down. Unfortunately, we still have plenty of stuff that was written five years ago by Fedora Infra or something like that as an application that runs on Rails 7 but it's also packaged into Fedora and it has a lot of dependencies and it's really getting slow to say this is an application that you run on Rails 7 and you don't need it packaged in Rohite in order to run it in Fedora Infra. I don't know whether they have an internal rule to package everything or what was the original motivation. We are slowly getting rid of them mostly through feltable from sources and missing dependencies, bugzillas and stuff like that. We started, what is it? Two weeks ago, three weeks ago, Lumiere started filing automatic bugzillas. If you have a Python 2 package, you might already got the bugzilla that says your package uses Python 2. We now consider that as a bug. So if your package depends on Python 2, it's a bug and you need to fix it. We try to ask for information because when you have hundreds of packages, it's really hard to see, is this important? Everything is important at some point to somebody but there are packages that are not really important to anybody else but the packageer. One of my favorites was something that converts macro media flash files to X files or something that extracts sounds from WAF files and stuff like that and it's packaged. There is no check section of course in the RPM so it builds forever because it has no dependencies and it was last updated in Fedora 12 but it's still there but then if you try to query it and then nobody replies that you know that it's probably the dead package anyway. We try to ask the project, the maintainers what is the transition plan for Python 3 and upstream and try to pitch in, give some advices for Fedora. Ultimately, the goal is to get rid of all those packages because I don't care if we have gimp in the distribution from my Python maintainer perspective but on the other hand as a Fedora ambassador and somebody who likes Fedora, I think removing ruthlessly stuff that people actually use like Chromium is not such a good idea. So we try to message the thing that you need a Fesco exception in order to continue to ship the software if it's running on Python 2. So there is a way to keep it. If it's important, it has a high impact and it usually helps if there is a plan. So for example, Mercurial has a plan. They already ready for Python 3 but all the extensions are not working. So this is just an idea. We might say, okay, it makes sense to keep Mercurial on Python 2 otherwise it's gonna be broken for the users and we give you the exception or not. It depends on Fesco and just me. The problematic packages are not the ones that people actually use like Chromium and whatnot. It's the packages that release engineering and infrastructure is using because there's no movement. It appears dead and then you attempt to remove it and then somebody jumps on you and said, no, if you remove this, we can't do composes anymore. I'm like, okay, but the package doesn't install and you do composes. How is that even possible? And then there is no reply. And basically what happens is that you open a ticket for opening a package and then somebody appears there and says, oh no, we can't do that and everything is gone. So explicitly for Fedora 32 change for the retirement, they said, there are no implicitly important packages. If you consider your package important, you need to get an explicit exception. I'll gladly retire Kernel if they depend on Python 2 and they didn't get an exception. That's their job to do. So now the important part about what's going to happen. Before I go there, somebody asked on DevL, why are we not just keeping it? Why are we doing all this madness if we can just keep things as they are? And the problem is not that the Python interpreter itself is going to be end of life in what, five months? Because we could handle that somehow. We will eventually keep it there anyway. But it's the many upstream projects that release new versions that are not Python 2 compatible. So if you wanna update NumPy, you can't because you have Python 2 NumPy and Python 3 NumPy in the same package and the new version doesn't support Python 2. So the only reason of the thing to do is you create a new package called Python 2 NumPy. You keep it on the old version but you update the original NumPy. And now imagine you need to do 500 new packages just in order to make it happen and nobody would care about the Python 2 packages at all. And then they will get orphaned and then you will get this email by some person that says orphaned packages are all going to be retired and it's gonna be the Java mess all over again, except it's gonna be a Python mess. And as Python maintainers, we have very limited resources in order to support this legacy group of craft. And Fedora is the leading or bleeding edge and not some enterprise thing where we need to support all those old things. We have another change for Fedora 32 and that's updating Python 3 to 38. It got postponed from Fedora 31. And we know that as soon as we start removing Python 2 bits, a lot of packages will fail to build because they try to build Python 2 and Python 3 at the same time. So the idea is that we don't do anything until 38 is ready. Mid-August, which is right now, something important happens Fedora branches, that's Tuesday after vlog. We're gonna do the 38 transition as fast as possible. There's another 1000s bucks where nobody responded so that's gonna be fun. We will do some split-offs from PIP and setup tools so we can easily update them separately, but it's still not breaking anything. For all the other packages, everything looks the same. Mid-September, which is a month later, we will package the Python 27 package which is the interpreter with all the stuff. There are no more sub-packages, no debug builds. It's just a legacy package so you can use to develop your stuff. And we will retire Python 2 from the distribution mid-September. However, use of in Python 2 stays and Python ABI provides stays in the Python 27 package so everything still keeps resolving unless you depend on Python 2 debug which are two packages in Fedora and they can do whatever they want. But there is no more Python 2 to develop. So as long as you try to rebuild your package, without any changes, it will immediately fail to build and you need to do some change. Obviously, people will just remove the devil bit, change it to seven and call it a day, but they're gonna end up very bad anyway soon. When you realize your package doesn't build because you have a dependency on Python 2 devil, the obvious way is to remove the Python 2 bits and if you can't do that, to remove the package and if you can't do that or you think that would break Fedora for everybody, you file a Fesco exception and you say, hello, I maintain this tool for, I don't know, managing my Lamas again. And it's written in Python 2 and if we drop it from Fedora, hundreds and hundreds of Lama shepherds will be set and they will keep running Ubuntu. So we need to provide this and then Fesco sits and say, yeah, this makes sense or no, it doesn't make any sense. And we continue to cut leaves. So as long as you remove your package because you say this doesn't make any sense, I realize that a lot of people are actually scared of getting exceptions. So they open a package review and you say, this depends on Python 2, you need to ask for a Fesco exception and the person is like, Fesco exception, I can't do that. And it serves a purpose because they never ask it and we are happy, but if you actually want to get an exception, it's really easy. You just need to explain why is it important and Fesco doesn't usually bite unless you are a troll or something like that. How does the exception look like? Most importantly, except reasons and stuff like that, you need to realize what are you trying to preserve? There are packages that only need the interpreter. Those will get the exception easily. It's just you, you just need the interpreter, the interpreter is staying anyway. If you have some security implications during that, we can help you with that. But the problem is a lot of packages will depend on other packages like setup tools or stuff like that. And then you need to get exception for the entire tree of your dependencies, otherwise we would just remove your dependencies so your exception would be useless. Or you can bundle everything. You can bundle everything or you can flat back everything and just remove it from the packages and then we don't care. So if you want to bundle stuff, you flat back and get it out of Fedora. There are obvious ways how to bundle stuff. You can always discuss this with us through bugzilla or in the exception request. Mostly the idea is that as soon as somebody needs spite and to set up tools and request an exception for it, I immediately hand over the package to them and I'm done. I will probably keep being a co-maintenor so I can coordinate stuff but we are not updating this anymore and if you need it, well, you need to care about it. Obviously when you request something and you say I need an exception for Python requests and then there is an existing maintainer of Python requests and he needs to agree to keep the Python 2 bit or it's cutting out to a separate package and it's yours. Maintaining Python 2 software will be very hard and that's the point because we just want to get rid of it but if you need to get help to get it working, we are here to help. We may appear like drop everything, drop everything because if you have thousands of packages that's the only approach that works but if you have a single package that needs Python 2 for a reason, it's okay to approach us and ask for help. How do I remove this part of the package or something like that? We will gladly do that. What's most important of this talk is this flag day. It's not actually a day, it says mid-November. We might artificially call it the 15th of November but it's not important. Mid-November, if you don't have an exception, your package is done. I will just go and retire everything that depends on Python 2. If I see, obviously, that is just one sub-package that can easily be removed, I will try to do that but if it's not obvious, I will just retire the package or some other proven package available. At this point, everything already had like the first heads up mid-September so two months to realize your package is broken and to ask for an exception. You already got at this point a bugzilla report from Lumiere or for somebody else that says your package depends on Python 2, you need to start figuring out what to do. We remove everything that has runtime dependency on Python 2 and we remove everything that build requires Python 2 to develop still because that doesn't even build so no point of keeping that. We will not do more complicated queries so you can still somehow trick us. So for example, yeah, this is not written here. For example, if your package depends on user bin Python 2 during build time, it's forbidden but we will probably not automatically retire it yet but when everything is done, we start looking for this thing. What is the problem during those removals is that there are packages who go really crazy. Like, obvious is I need an if conditional for Python 2 bit and it needs to say a Fedora version and we say, no, there is no Fedora version when you stop building this, you stop building this but there is nothing that requires it anymore and they are like, okay, and they flip the Fedora version to 32 from 31 or something like that. So it's gonna happen that after the mastery build, some packages are gone just because somebody assumed something and currently if some Python 2 package is gone and it has dependencies, it's still a bug and you can report it to the maintainer and say, you remove a package, you didn't communicate it, it broke stuff, put it back and we will be sad because there will be one more package that depends on Python 2 but nothing will break. In Fedora 32, if your package is gone, the dependency, because the maintainer went crazy and removed stuff without communicating, it's no longer his problem but yours. So as long as your dependency is gone and you really need it, then you need to file an exception for yourself because otherwise you don't need it and you need to file an exception for the missing dependency and you need to package it for your own. We hope nobody will want to do that and that people will just stop doing this altogether. In mid of February, which is what, one, two, three months after November, there is the beta freeze. So we have three months between the flag day and the beta freeze to clean all the mess and make it work again and probably if we remove stuff, something will break. I expect the compose will stop composing and the images will stop imaging and stuff like that and hopefully through this period of time, we can fix things and obviously if you didn't get an exception before, you can still get it. So usually the packages only wake up when you actually retire their packages. You open box the last, you send emails and nobody gives a damn. Then you retire the package and then it's like, what did you just do? The package was mine. So we expect that in mid November, when we start removing stuff, people will wake up and then you can still get the exception later and reintroduce the package if you get it. Later, after a branch, we probably start worrying about Python 2 packages that are hidden and they don't have runtime dependency and they just need the interpreter to build or something and it's much easier to get an exception and say, hey, I only need the Python interpreter to build and it's important because otherwise package doesn't build but yeah, we probably just keep that if we know your package needs. That's all I had in my slides. Most important thing is this link. It's very long for you to remember but it's just finalizing Fedora switch to Python 3. It's just like a huge meta change that has all the changes that are happening and inside you will find all the links that describe this and of course, if you don't like what I am saying, if you want to keep Python 2 forever, you are always welcome to take the package or to propose a better solution. We are open to discussions, but we are very, very stubborn about not keeping it if it means we need to maintain it ourselves. Is there any important information I totally forgot to say? Okay, so if you have questions for us about this or anything else that relates to Python and Fedora, shoot. Yeah? So you have more possibilities, some of them are crazy, like maintaining a fork forever. There are also packages that still have downstream only Python free patches in Fedora but it's not maintainable. So basically, if you really want to keep this, you would need to get an exception, but if you don't want to keep it, just retire it. I mean, is it something important? No, and the question is simple or the answer is simple. If it's not important, get rid of it and do it now even in Fedora 3.1 or... Python 2.7 probably is a big warning or something. That's probably the little bit of information that I completely missed is that the purpose of this change is not to get rid of Python 2.0 for the users. We still think that Fedora is the best choice for a Python developer as their workstation and some developers, poor souls, actually need to support Python 2.0 applications because they are deployed in RHEL or whatever. So if you run Fedora and you need to test your application on Python 2.0, obviously this is still a supported use case. For such developer, the only issue is that you can only pip install packages into virtual environment that are Python packages. So you don't get DNF bindings for Python 2.0 because you can't install it from pip, but you already don't get this now. So there is not much changing on that front. As a side note, we also have Python 2.6 package for the exact same reason and it has been orphaned because the Compat Open SSL dependency is also being orphaned. So if you actually need to develop for Python 2.6, that's RHEL 6.0, you would need to worry about that, but unless you don't, I hope nobody needs to develop for Python 2.6, it should be fine. Okay, if that's all, thank you for coming and have a nice retirement of Python 2.0. What do you think that the answer for a bug that you software developers have is it might be good content for retirement, just do it and you will get rid of all notification email I will send you in the next days, so. So the quickest way to get rid of my emails. As a side note, we have two Python related hackfests tomorrow, one is about the Fedora Loves Python thingy that we have but didn't move anywhere for a couple of years now, so we are looking for people with interesting ideas like marketing and design and stuff like that, because we usually just meet as engineers and try to promote Fedora, but we don't know how. So we would like to gather with more broad group of people. I'm actually not coming because I have to head home tomorrow, today, but the guys will be there. And after lunch, there's a hackfest, more like doing actual coding about the PyProject macros, which is a new way how you can build Python packages upstream and we are doing RP macros for it, so you can use it in spec files and the purpose is it should be all automagical so you don't need to maintain. It should be automagical, no magic. No magic in there, it's just, okay. So the magical bit is that the spec file will always look the same no matter what. So then you can't, you can completely forget about the spec file and you can generate it on the fly or something like that. And the tool that called pip to RPM that actually did it before can be obsolete it completely. So if you wanna help get this done, talk about the API of this, come afternoon tomorrow.