 All right, hello Python fans, if you're not a Python fan, I will, then welcome. You will not be bored for the first few minutes. So I will be talking about Python packaging in Fedora. Maybe you have seen this logo Fedora of Python, so I'll try to tell you what that means exactly. First thing I'll be talking about is Python 3. So Python is a pretty old language, older than Java, and some years ago, it turns out that the old baggage in the language was accumulating so much that they needed to do a new version with some backwards incompatibilities. That was Python 3.0, which was released eight years ago already. It's been with us longer than, for example, in NJS, so it's a pretty long time, but the world was different then. Of course, the evolution has gone on. Python 3.5 was released last year, and a lot of the work between Python 3.0 and Python 3.5 was adding features to make it more compatible with the Python 2.0 line. So if you tried porting something to Python 3.0 or 3.1 and realized that it's a very tough job, then you might want to try again because it's gotten easier. And another important date, 2020 is coming up when Python 2.0 will be officially unsupported upstream. We'll see how that works out. Of course, unsupported upstream means that somebody can still pick it up and support it further, but it won't be me, at least in Fedora. So by 2020, we should have everything ported to Python 3.0 and running on the new version. If you look at the dates, it's 12 years, which for supporting a piece of software is pretty long time. So to track the effort in Fedora, since we need to get done by 2020, we've created a website called the portingdb. If you have internet, you can look at it, it's at fedoraportingdb.xyz. Unfortunately, I don't have internet, I was counting on that, so I can't show you, but it's this colorful little dashboard that shows you. Because I came in about 20 minutes earlier and didn't check everything out. Maybe I can give you a local version. So you saw the URL before, this is my local version. I'll show you what all the graphs mean, basically green means good, the other colors mean we're not there yet. So as you can see, we're about halfway done. I'll talk more about it here. Here's a graph from the dashboard showing how the effort evolved over time. So this is... I will be talking about that. So as you can see, the green part keeps getting bigger and that's the proportion of packages that are ported to Python 3, which means either they're Python 3 only in Fedora or they have both Python 2 and Python 3 versions. Below that, there's this little gray part that's packages that won't be ported, but already have some kind of alternative in Fedora. So for example, Python 2 would be in the gray part because that won't be ported to Python 3, but it has a better alternative already. Then we have this yellow, which is supposedly things people are working on, but people marking something as working as being worked on doesn't really work well in Inkinsource, so I don't rely on that too much. This blue part is probably the most important for now. It's packages that are already ported upstream, so the project has a Python 3-compatible version, but it's not the package in Fedora yet. I'll be talking more about that later. The gray and red parts are things that are not ported upstream yet. The red ones are things that are blocked by some dependency that's not ported yet, and the gray ones work and start on those right now if people want to. So there's still quite a lot of things Python 2 only upstream, but we're almost halfway there upstream, so half the things in Fedora, which means they have some standard of quality and some people care about them. So half of those are ported upstream. However, we're missing about 130 packages to port them in Fedora. This is a job for Fedora packages, and nobody else can do that. We'd like if everybody chipped in and tried to help us port. If you go to the porting DB again, you'll get a guide to how to do that. If you port a Python package to Python 3 in Fedora, you'll get a badge. If you port more of them, you'll get more badges. It's quite fun, actually, maybe not the first time, but learning all the new guidelines. If you do one or three packages, it becomes quite easy. And of course, if you maintain something in Fedora, it's better to learn the new guidelines and port it to Python 3. Now I'll go back to the things that are not ported even upstream, which is a bit of a problem because they tend to be sometimes things that might not make it into 2020. One category of that is Fedora infrastructure. There has been a lot of work on that, actually, recently, but still it's work that only Fedora people can do. Nobody else will probably work on Kochi or Bode. So if you want to help, that's an area to work on. Another big area is desktop toolkits. GTK 2 will never support Python 3. They move to a new mechanism called Geoject Introspection in GTK 3. So every package needs to be ported to GTK 3, and then it will presumably get Python 3 support automatically in some way. But porting from GTK 2 to 3 is difficult at times, and it's something that Python people can't really help with. So I'm a bit worried that this magical timeline we have will not be met by the GTK people. Also WX widgets, the Python bindings there are a bit slow currently, although I think the work has been done there as well. There's the sugar desktop where there are just talks about porting to Python 3 and GTK 3, but they really lack manpower, so if you like desktops for kids, that's a good way to contribute. And then, of course, there are big apps where the programmers aren't really Python people, because if you have a package like Django or NumPy or something like that, the programmers know Python, and they know that Python 3 is a better language, so they have more incentives to port. But if you develop something like GIMP, which only has Python bindings and lets other people use Python, you might not have that much incentive to go and do the work. And then it goes for SAMBA, which I'm currently working on. Also the developers don't really care that much. We'll see how that develops once the deadline draws closer, but those are the big problems upstream. All right. By the way, if you have any questions, I'm happy to take them during the talk. Yeah, lack of documentation in GDK 3, personally, whenever I wanted to find some kind of example, specifically a Python example for GDK, either 2 or 3 was pretty hidden, but I don't know, I haven't really read on that one, so documentation might be a problem. So let's move from Python 3 to another thing that we're doing in Fedora. If you're a packager, you should know what a packager should do. You should make sure that the software doesn't show too many ads and doesn't track people, stuff like that. You should make sure it integrates with the rest of the system, doesn't conflict with any other packages. You should check that the license is all checked out. And this is why we package stuff. This is why we make a distribution. This is why we curate the set of packages that are available to people. Unfortunately, if you want to be a packager, you also need to read a lot of documentation, most of which doesn't really apply to you. And you also need to know how to write spec files, which honestly is literally ancient black magic. And that is a problem. That is why we don't have enough packagers because people don't just go, yeah, I'll make a package in Fedora and then they get 2.4 and 5 and they say, maybe I'll leave that to somebody else. I mean, if you just want to start, the spec file isn't that hard. You just copy it from somewhere, put in the values that are relevant to your package. Yes, if you read the documentation carefully, you may find that there's a tool that gives you a template. Good point. You need to read the guidelines and that's what beginners don't do, unfortunately. But yeah, there's a template, but most people still copy. Unfortunately, what happens when you copy stuff, you also copy over things that somebody added for old versions of non-Fedora systems and things that were useful five years ago and nobody understands that anymore. I do because it's my job, but not a lot of people understand this. Can you just delete this? Probably. What will break? Yeah, that's not the most efficient way here. Or you can just try deleting it and see if it works. I mean, if you're an experienced packageer, you know where to go and ask for help, but again, if you're just starting out, it's pretty hard. Now I'll compare that to Python's native setup, which is setup.py. It again starts out pretty, pretty easy. It's just Python syntax. If you want to make something more complicated, then you just write some Python, which you presumably already know because you're making a Python application. Surprise developers are doing this instead of making packages for Fedora. Another problem with RPM and with the spec files is that they were made for an era where when software was distributed by putting a tarble somewhere on the Internet and then the packageer had to figure out what has to be done to install it and build it and integrate it and all that stuff. Now we're at least, we're Python at least, trying to have some kind of structured metadata and a standard way to install the software on all kinds of system and make it integrate as well as it can be done within the language. Of course, it's not perfect, but there are efforts to do that. So we have structured metadata for every package, for almost every Python package, and we have a standard place to put those packages, which is PyPI. So at least some of the things that packages do could be automated. And that's where bit2rpm comes in, which is a package that you install. You run it on a package on PyPI and it gives you a spec file and it's magic and it's perfect. Yeah, it's still in development. So it might not give you a fully functioning package. It will most likely not give you a package that will make it through review, at least if you get a good reviewer. But for starting out, this is probably the way to go, this is probably better than a template. We should also focus on this one, if it becomes a full functional, nobody has to become a problem. It's just a fast process to make a lot of packages in the short term. It's going to be much more reliable to focus on the other packages, but if it's finished, it's important, right? I don't think I got the question. Is it a question? It's like a suggestion, I'm just trying to say, if it's going to be better to finish this package, become full functional, about to give the spec file, we actually need it with a full training plan. Well, it's probably never going to be fully functional. Yes, it's probably never going to be fully functional. Also, another problem is that this metadata is not complete in all cases. The good thing there is that it's open source so we can actually go in and say PIP to RPM, it's this kind of metadata added to whatever Python uses. Instead of writing spec files, you can apply some part of whatever for the spec file generated by this tool to make it actually work. Yes. Rather than focusing on fixing spec files in your file. This way you can automate it. This is what we are trying to do in Cooper. Yep. We've already heard that. We've tried to rebuild the whole IPI, and we used to put PIP to RPM. And we were finding common errors and problems, and we reported to PIP to RPM maintainer Michael Cyprian. So we were fixing that. So we are trying to focus at least my work right now to find the common patterns, and not all those 70,000 modules will be able to package, but if we fix some small issue, it will get us 100 of working packages. So we are trying to work on that. Probably never 100%, right? Okay. But we will probably give you 100,000 packages, which somehow work, which maybe for a lot of people, enough. 80% is really good. Right. So now when we have the metadata and the common place where the packages are, we can do exactly that. Why exactly are we doing that? One reason is to test PIP to RPM, giving an algorithm all possible inputs it needs to work on is a pretty good test suite, I think. I think in the future we should also use it to run upstream tests. When I was at Python, people were complaining that if you have, for example, a Django and a Django plugin, then testing if one still works when the other is upgraded isn't really something the tooling around GitHub can do. And I think that the distribution is conceptually a pretty good place for this kind of integration tests. So combining this with Corsche or something like that to detect these changes, rewriting the tests should be a good service, not only to Fedora but for the whole community. And of course, this all has to work first. So it's a bit of a future plan. And we can provide a repository so if you want to get packages straight from PyPI but not use wheels but use RPMs, you'll be able to do that. Although you'll lose all the things that a human package does, of course. Might not be the best idea to do it. Now when you have a repository, you can theoretically write a DNF plugin that will emulate what the Python installer does and install Fedora packages from the requirements.dxt file. To do this, you have to solve one more problem which is mapping the names they use on PyPI to Fedora which differ in lots of weird ways. They do because the distribution name or the PyPI name is not always actually the name of the project itself. And Fedora prefers to have the name of the project. And there are historical incompatibilities between the names. So it's better to keep them separate. However, we can add a virtual provides to every Python package that says I'm providing this Python distribution so when you do that DNF pip install, it can look at that and install this virtual provide. And the good news there is that's already done in Fedora 25. So if you have a package in Fedora 25 and it uses standard Python installation it should have that provides already in there. If it doesn't, then it's an RPM bug. Yes, thanks. So things are looking better. Now I'd like to talk a bit about packaging Python itself rather than libraries for Python. So we did this thing called system Python which is something you'll probably already have on your computer. It lives and lives except. And it came out of an effort to minimize the size of the minimal install. If you look at the Python standard library, a lot of this is unneeded stuff. There are of course tests which you don't need at runtime. There's live 2 to 3, which is development only. There's idle live, which is graphically, you probably don't need that in the cloud. And some of this, like the graphical stuff in the tests are already separated. So you don't get those in a minimal install. But things like installation tools and development tools are still in there. So what we did is split them out. And when you install system Python, you get just a subset of the Python standard library. So you use this binary in your shebang. You do some kind of macro magic in the spec file to remove the normal Python dependencies. And then you'll not drag in all of the standard library with your package. This is probably useful only to stuff like DNF and cloud in it at the moment. And we're trying to talk to them to switch to system Python so we can actually get the size reduction. In the future, this might be used to isolate system tools from changes that the user does to the system. For example, if you do, if you upgrade some kind of system library using Pip, which you shouldn't do, but people still do it, this might be useful to isolate stuff like DNF so that it doesn't break. But again, that's that's future. And it's probably what the name system Python implies more than more than reduced standard library. But so far we have we have this. Yeah, so some technical details. An illustration of how it works when you install system Python, you only get important stuff when you install something that needs the whole Python. You get all of that plus the development and installation tools documentation. Does that make sense? Are we going to have libraries built for the system Python? Excuse me? Are we going to have libraries built for the system Python? Like, you know, libraries built for Python. No, it's a proper subset of Python. It's just a subset. So if your program doesn't need these these development and whatever is not there, then you can just switch to system Python. It can't need anything else. It can't need anything else and you need to depend on the whole Python. But the problem is the test, splitting the standard library is something that Python core developers don't really anticipate. Fedora already does it. Like, it splits out the test module because it's giant. It splits out the graphical stuff because that would bring out the extra dependencies. So, but that's something that Fedora and Debian and all the distros, this is something that is not that well supported on the screen. So the testing for it is much harder. So I wouldn't recommend actually switching to system Python for a random package. Did you say supported upstream or discouraged upstream? My understanding was they weren't very happy about the idea of doing that. Yes, there was a pretty long discussion about doing that. Imagine reading the upstream discussion. I did. It was long and rambling. One argument in our favor is that Python already, already if you build Python without some dependencies, then the corresponding modules will just be missing. And they'll, yeah, so Python already does something like this. So, yeah, and about it being supported, well, Python is community supported. So if Debian does it, and Fedora does it, and Arch does it, then it's pretty much supported because there are Python, core developers in all these communities. Do those distros do it the same way? The test and graphics, I think they, well, there's not many ways to do that. I don't mean to derail, I'm just curious. Debian has something called, I think, minimal Python. Now, that's something similar to this. We specifically chose not to do it the same as they do because there's some problems with that. OpenWRT also has some minimal Python and full Python. Excuse me? OpenWRT packages also has some services, but minimal, normal, and full Python package, like a distributed three-section thing. So my plan for kind of long-term future is to actually make a supported way to do this upstream so you can choose what pieces of the standard library you want and have some good error reporting for the case where you should install something extra. The current way is pretty bad, but we're doing what we want. If we keep it to these two packages and test them and help them with migrating, it should serve as a good experiment about how this can be done as well. So at least that, hopefully, it will give us some actual benefits as well. Besides, reduction is pretty big for the cloud image. Right. Another part of my talk is about Python 3.6, which is the upcoming version of Python. From the features point of view, it's not done yet, but for the obvious programmer, I guess there are not that many changes. There are format strings, so there's another way to format strings in Python. I think it's the third now. You can just write a string and use variables in it, and you don't have to repeat the variable name so many times. So it's pretty useful once you get used to it, so hopefully it will entice more people to switch. Excuse me? 25 years. 25 years and three iterations of string interpolation. To get to normal string interpolation, we're used to it. It still looks pretty magic on Python to most people, but it's pretty useful. I totally disagree with the concept of Pythonic or Xenophyte. It's a tool. Yeah. It's a tool and practicality beats purity, so there we go. Another thing is the... You might be familiar with bytecode. Python compiles source code to bytecode, then interprets that. Bytecode is gone. It's replaced by wordcode, byte byte code, but better. It brings a pretty big speedup. There are also planned speedups for dictionary access and optimizations, but it's not clear if they'll actually make it to 500. But expect some pretty big speedups, like 10% percent. And if you're making a library like Django ORM or SQL Alchemy or some kind of magic class creation thing, there are some pretty useful tools there. I'll not go into details, but read the release notes. People that are really deep into Python were pretty happy with this, so I recommend you reading that. So how does Python 3.6 align with Fedora? The Alpha Freeze was a few months before Fedora. Alpha Freeze, pretty good. The problem is Python's Alpha Phase is a lot longer than Fedora's, so the better freezes were a lot closer together, and the first release candidate will unfortunately come two months after Fedora's final freeze, so you're not getting Python 3.6 in Fedora 25. There will be a copper repository if you want that. If you want to live on the bleeding edge, then you do what you do when you want to live on the bleeding edge and use Rohe. We'll try to get 3.6 into Rohe as soon as possible, as soon as... I mean, we might go with even the RC one, the release candidate, to get as much testing because it's... Another hood, it's a breaking change, this bytecode versus word code, so it should get as much testing as possible. That'll require a rebuild of every package I've seen. Yes. Well, one thing to note, already in Fedora 25 we had a rebuild of every Python package, even though there was no mass rebuild for the whole Fedora, to get the virtual requirements. Fedora 25 was branched a few days ago, so you can build a 5.6 into Rohe right now. I would rather put the release candidate there because before the release candidate, it's not feature-complete and it's broken a lot. Okay. What's the date of the Swedish expanel? I think it's written at the same date as the one I've got last month. Oh, right. You're correct. Yeah. Some of them would like to release it by the end of the year because they're... Yes, yes, the RC date is wrong. I'm sorry. Yeah. I will fix that on the slides I put online, or you can search Google for the correct date. So, that's where we stand. No, Python 3.5 for Fedora 25 and 3.6 and 2.6 as soon as possible. Now, I have been talking a lot about we do and stuff, so I'd like to explain a bit about who is we. Unfortunately, I have a bit of bad news because we is, in most cases, the Python maintenance team at Red Hat. Now, here I'd like to give a bit of internal history. That was this Python maintenance team. I'm going to give you a little bit of internal history. I'd like to give a bit of internal history. That was this Python maintenance team. And I joined it. I was supposed to work upstream, like on Soundva and C-Python and not touch Fedora too much, so I was happily doing that. Then a person went to another team, a new person was hired, and more people left, and more people were hired, and suddenly I was the most experienced full-timer in the team. And this happened in a pretty short amount of time, so we had problems with knowledge transfer and fighting fires and stuff like that. And we skipped on some things, and one of those things, which was probably our biggest mistake, was communicating our plans with the community. So we did a bunch of changes, like an individual contributor would, getting permission from Fesco and doing all the processes, but not really communicated with the Python guys in Fedora. This history shouldn't be an excuse. It's supposed to be more like an explanation. We made a mistake, and I apologize for that. Hopefully, we're well. It'll be my goal to make this better, so that the next time I give a presentation like this, I'll be able to say with a little more honesty that's where Fedora's Python special interest group. If you'd like what we do, you should join the conversations on Python develop or in the Fedora Python IRC. You should check out what's on GitHub on Fedora Python. And if you like it a lot, then please join us and help make Python and Fedora better. We can jump by, actually, as we debate, which is currently kind of updated, but we are also in the process for updating everything. Yeah, you're going to see hopefully a week after. Yeah, after clock ends. Yeah, after clock ends. Right. That went faster than I rehearsed. Questions. Questions, comments, anything. Yes? Could you go back to the slide where I already did on in the slide where you talked about running upstream tests for Python packages? So, what's your take on whether to include tests in the, maybe the wheel or the source distribution or just in the source repo, let's say, Git? So, this is a long-term plan, right? So first, Copper needs to work for most packages and Git to RPM needs to work reliably. And running the tests will probably require some changes to the metadata. So, we need to figure out a way for upstreams to say, these are our tests, this is how they're packaged, this is how you get them, this is how you run them. And if we can get that in the setup pie, then we can use it in Fedora, and of course everybody else can use it. I'm speaking with the tests. The tech section. The problem here is that currently, if you have a random project, Git to RPM is run on it, it tries set up by test. Right. Because that's the only thing that it can try. And most of the upstream projects, like in Python packages, they just don't enable this. They just use Snow's test or PyTest or whatever. Well, that's one problem. Another problem is that they don't package the tests. They don't raise the tests. If we have them on GitHub, they don't include them in the sort of, you have to go, that's the human package. In my experience, there was no standard way to run any tests. I don't know why. There is a standard setup PyTest. They don't follow that. I did a lot of work from Perl module packaging before, which seems to have solved this problem about 20 years ago. Everything is standardized and it's all been automatically done. Perl has the good fortune of being around when distributions were still cared about by developers. Nowadays, people just put stuff on by PI and people are still with Bit. So there's not as much initiative for this to happen. But we need to do this to lower the work that needs to be done. Because it's also a problem of dependencies. It has to bring other dependencies than, I don't know. So yeah, I think there is also a standardized way to do this. In Setup Py, you can specify this requirement. Yeah, that's something Setup Py does, but RPM doesn't. I have a question about those dependencies. Because when I go to Python for Perl, the guys are saying, well, don't use the system. I'm going to use Bit because this is the way to go. Those dependencies are currently running in Python. There are RPMs. The system has some different set of dependencies and versions. How to make sure that those two match? Because there are just a new menu, right? A new dependency, new versions. We're defining the option project, the requirements text, and test requirements is all the system, and those dependencies are taken from each other. So how to make this system audited? So the problem is here that in Setup Py, you can have dependencies. But if you install the RPM, you also have... I mean, the way to do that now is to install the package name and not give it the dependencies. You have to be sure that the dependencies actually match. And that's why we have a version in this version we'll provide. So here, if you install some Python package with dependencies specified, it will either install it or tell you such a version is not found. In which case you have to use PIP to get the version that's not in PIP. This is on the DNS side. You can work on the PIP side to make sure that whatever is found in DNS will be inspired to the PIP. That is already there. So the problem is that if you're working on the system itself, if you do Setup Py, and install something that needs a new version, you will upgrade the system package and break all the system tools. So you shouldn't use Setup it ever. Just don't. It breaks stuff. It doesn't work in Fedora. Or you can use a virtual environment, which is completely isolated and doesn't use the system installed packages. So you can set it up too, but nobody does that. It's not the default. That is the solution. So make your virtual environment use system packages. It is possible. And then PIP will use them if they have correct versions or install new versions into just the virtual environment. And the plan is to... This is the problem also. I'm using it and then whatever is not found, I need to use the code to build it. But then the name space is different because I cannot use the same name space on the environment. I need to translate. And then it's not a code because it's different. Mixing PIP and RPM is difficult. And that's why we're doing this to bring you closer together. If you have a specific problem, then maybe talk to us after the talk. So I have a question about IPhone 3 and Apple. There's been IPhone 3 or 4 packages that have been brought to Apple, which are available, but there are very few libraries built for these packages. I've been building a lot of them because I'm working on Mainnet. It's a IPhone 3 only package. So I've been working... I've been changing a lot of specs files to build for IPhone 3 or 4. It's not very hard to change, but I wonder what should I do with those packages? I've had them in my private record. Should I put them somewhere in Cal Poly? Or should I just change the spec files in these gits and ask the maintainers to change? It's very simple to change. It's just a macro that replaces... Ask the maintainers. If somebody maintains a package in Apple, they... What's the problem with your running the packages that are maintained in Apple? So you want to maintain them, right? I... They're not already there, then you should request a branch and then maintain it. Okay, yeah, sure. That's the best way. So it's okay to have these new IPhone 3 or 4 macros in Apple? Oh, yeah, of course. Yeah, no, we make sure that everything works all the way down as far as we can make it even into Apple 6, some of the stuff. Obviously, the Python 3 is there, so... That's just a trick. If the Python 2 package is supplied by Rails or CentOS, you have to exclude the Python 2 package from building. Right, and that's such a thing. It's not that complicated. Just like to exclude it on the Python section. Right. It's just a trick. You can do it, though. Okay, so... I just remembered, the owner has a week to deny you if they don't want it, otherwise it happens to him. So, you know... Well, that's always... I'm not a robot, but I needed Pelikan to generate some static websites. So, as a newbie, I typed Pelikan and then I told me, if you want to use Pelikan, you have to install Pelikan to... Pelikan. So, I said, it's too bad. I have Pelikan in Python 3. How do I use it? And I looked for a few minutes and I didn't find... How to... When I'm typing Pelikan, how to run it with the Pelikan 3 Pelikan? Because the user bin... Right, so the standard... Right, so the standard way is to type Pelikan 3 or Pelikan.dash3. One of those is in the package guidelines. I can never remember which ones. But some package predates this guideline, so you might want to run... Install Python 3 Pelikan first. And then run the magic command R-P-M-Q-R-E list of files for Python 3 Pelikan. And maybe you can grip... So in this case, it's not named according to the guidelines. Oh, and it's not a boss. Well, who owns that package? That's true. So, file a box. You shall file a package and pass it on immediately. Yeah, so it... This is horrible. Yeah, I got it. Yeah, so with all this... Normally I shouldn't only have Pelikan... It should be Python 3.dash. No, no, no, it should write it. Is there a dash? Yeah, yeah, yeah. So it should be like this. Okay. So this is a program of packaging, because I tried to type Pelikan something but it just didn't exist. Also another thing is, what does this tool do? Where is it for? I don't know. It generates... It generates a lot of data. Do you carry people around in Python 3? That's the important part. You do, because it has Python configuration. Yeah, okay. Then it actually cares. This one actually cares. And the... The contents to keep Pelikan is marked down. And I put in some written script to produce this markdown after I made an HTML file and the word website structure. So I learned which... I have the basics in Python, so I learned Python 3. So I was confused. Python 3 plus Python 2 didn't know what to do. Well, in 2020 we're dropping Python 2 and there will be only Python 3 and no more confusion. And I think we're out of time, so more questions after this. Thank you for your attention. I... Thank you very much. Thank you very much. Thank you very much. Thank you very much. Does anyone have an internet connection as it's in an IRC-type file? Yeah, my Wi-Fi... I'll get you a sticker. And don't forget the stickers. Yeah. So just... Are you... You can... I've got online through the hotel. You can... You can try it. You're close to that? Close to that?