 Okay, so hello, thank you for everyone for attending. I'm, my name is Piotr Herofsky. I'm with Matias and Scott who is not here are maintaining Python defaults package, which is responsible for a list of, the main purpose of this package is to list the Python versions that we currently support. And I'm responsible for helpers we use in Debian to package Python stuff. And this is above, so it's about discussion various Python related things. I prepared some on the Gobi, you can join it, edit it, that would be great if you can add comments or notes from what we talked here. But we can talk about anything Debian and Python related. So it's just a proposal, but please feel free to ask any questions you have in mind. So the first topic is actually answered yesterday by release team. They, on the slides put Python 3.5 only as supported Python 3 versions. So I guess we don't have to talk about it. Unless you really want to support Python 3, but 3.6, but I actually think that supporting only Python 3.5 is a good idea. Not only because by, as Scott I think mentioned, by supporting two versions we will increase the size of our packages quite a lot, but also final version of Python 3.6 will be released really late and I'm not that convinced. We will be able to update all our packages to a state that we will be happy with in the stable release. So we can, how many Debian and Python modules packaging team members are here? Can you raise your hand? So just a few, so we probably should skip the second topic because it's really team specific and move to another one. One that I already mentioned last year and we still didn't, we still are not there is Python 3 package depending on DH Python. Python 3 is a package that depends on currently default Python 3 version, so right now it depends on Python 3.5 and it's a special package that allows user to install always the latest version of Python. So you don't have to know which Python 3 version is currently the default, you just install Python 3. And this package used to have some helper tools that were used during, for example, package build. Another package that supported Python 3 had to at least Python 3 to get DH Python 3 helper, for example. And right now all these helpers were moved to another source package called DH Python. And this new package contains all the tools, DH Python 2, DH Python 3, DH PyPy, so and few other tools that are used during build. And because they were, shipped in Python 3, we had to add DH Python dependency in Python 3 package until all these packages still needed until they add DH Python to build the pens. And I just checked there are over 1,000 packages that still don't do it. So in case of Python 3, it's not that a big deal because well, we still, if somebody installs Python 3, DH Python is installed as well. And it's not that big, it's just a few hundred kilobytes. So it's not a big deal, but in case of Python 2, it's kind of different story because we couldn't do that. We couldn't do the same trick to just depend on different package because Python, we kind of depend on Python 3 in Python 2 interpreters. If user wants to install interpreter in version 2, we would have to force him to install Python 3 as well and we don't want to do that. So this problem is kind of important because in Python 2, we still had to ship the older version of DH Python 2 helper. So the fix for that is really simple. All packages that build the pent on use during build, one of these tools have to build the pent on DH Python that they still do not do. And my question is what can we do to improve this situation? I prepared a patch for LinkedIn that detects that, but the first version of that patch was a bit stripped before applied by LinkedIn maintainers. So I created another one. It's still unfortunately not applied and that's why LinkedIn still doesn't warn about most of these cases. But the question is can we do something about it? I mean, not only the LinkedIn thing will be fixed really soon, I guess, but what can we do to encourage maintainers to add new dependency to in Debian Control 5? Or can we do some kind of mass back-squashing party or like with it with Python support or, and what are your ideas? Because this is a buff and I have problems and I want you to help me fix them. Or maybe we can talk about something completely else. It's a buff, sorry? That is to Python 3. Port everything to Python 3? Yeah, that's a goal, yes. But I think even, actually that's something I should talk about because during this DEF CON, the most question I had was how long will we support Python 2? And my answer to that is we have no plans to, and I think Mattia will agree with me. Can you take my mic? So the idea is that 2020 is I think three years after the stretch release and if you want to have a long-term support release of stretch, we won't get, well, that much more upstream support for Python 2. Okay, there always will be somebody who will update that, but Python upstream made it very clear that they want to move to Python 3 now. Maybe it's a bit ambitious to drop Python 2 for stretch, but it's good enough, or Python 3 is good enough to so that we can, well, convert all our small scripts and things which we use in packaging, or in small packages to move to Python 3 and that should start now and we shouldn't delay that until after stretch. Yeah, so all the Debian infrastructure will, if not in stretch, it's probably too late for stretch, but in stretch, in Buster, we'll use Python 3, but it doesn't mean we will remove Python 2.7 from Debian, it will remain there because there are lots of companies that still use it, so even if we internally in Debian don't use it and even if every package, I mean, every application is ported to Python 3, we will still not remove Python 2.7 for at least Buster and probably Buster plus one. We can maintain it then, we're good. Who's volunteering to do the back porting to Python 2.7 for all the security fixes? AppStream will not fix it, will stop fixing it in four years and we will probably not have manpower to do it ourselves, so. That might just be a question for the LTS team as well, I mean, they might want to say, starting from release date plus two years, no Python 2.0 thing will be supported, security is supported, and if they do a public announcement about that, if you still want to run Python 2.7 to ish on stretch, then you're on your own. Another solution would be probably to support only the interpreter and to remove all the libraries. Does it make sense? Honestly, I don't think we can do that now. There are still some big applications which are not yet ported to Python 3. So one big user is OpenStack, they are preparing for Python 3, but I'm not sure they will be ready for the next release. Another one is Samba. There's some work done porting the bindings, but Samba itself is not yet ported to Python 3, and upstream doesn't want to support both 2 and 3 at the same time. There are some net applications for the desktop which are not yet ported, but I mean, it's a good goal for Buster, but I think it's too late for stretch. My personal opinion, even in Buster, we will have Python 2, but I still encourage everyone to port to Python 3. So on the other hand, for Buster, why? I mean, we could also say we remove Python 2.7 at the release date from testing, and then the ecosystem will follow. I mean, we're also leading. I'm sure everything in Debian for Buster can use Python 3 only, but there are companies that have very large applications and infrastructure that they simply don't have manpower right now or will to port to Python 3, and they probably would be fine even with the interpreter that simply compiles in Debian Buster or Buster Plus 1, even if they don't have security support. So it's not that easy to move it just from testing because then we have to change all the packaging not to build Python 2 modules anymore. And I'd rather support this than, well, going to each package and try to remove it. So I might say that in Ubuntu we are currently, well, moving out our build dependencies out of main into a universe. So it's a non-supported Ubuntu section, and what I'm trying to do there is to get rid of Python 2 in main. So that we say we still need to build stuff, but we don't rely on running stuff. And that might be a sign to say, well, now we can run stuff with Python 3. And I intend to use that to make a proposal for Debian when we can consider dropping Python 2. Would it be feasible to use a different Python 2 implementation like PyPy, which I'm sure will be continued, or continue to be supported past 2020? PyPy can't currently share a package, a module path with CPython because the PyC files are different. In Python 3 that is less of a problem in Python 2, it's a big problem. So for Python 2 we are currently adding PyPy for binary packages, and that's a lot of work to add to all. We can try to figure out something that we'll use, I mean, to use the existing Python full packages, not PyPy packages, but I assume it will be a lot of work as well. Well, as an alternative migration path, if Python, CPython 2 is end of life, and you can't port everything to Python 3, switching to PyPy 2 gives you a way to still run your Python 2. whatever code without having to rewrite a lot of it, just as a potential idea. Ziko, I have a mic for you. Yeah, that doesn't work because PyPy at some point will move to Python 3 sooner or later, and they are already working on it, and PyPy is already batches available for it. So it may work short time, long time, it doesn't work. Where are we? Yes, I assume PyPy is gonna continue to be maintained for quite a long time still, if nothing else, then because PyPy 3 is written in Python 2, and I don't see that changing anytime soon. PyPy 3 isn't really particularly useful for Debian yet, because it doesn't support 3.4 yet, and well, it doesn't, yeah, lots of the Debian packages that support Python 3 require newer versions of Python 3 than PyPy itself is. We can package it. We cannot use the main advantage of PyPy 3, so it is sharing these packages, because these packages can contain code that requires Python 3.5 or later. That's why I haven't bothered with PyPy 3 recently. I've got so half done the thing on my laptop somewhere that is almost there, but I don't really see the point of it yet. So I'd be very much if after freeze, after stretch is released, we start to aggressively remove Python 2 libraries when they don't have dependencies. What's your guys' view on that? When packages which slowly move to Python 3 and we can remove Python modules support for Python 2, I think we should start that work just after freeze is released. We already encourage containers to, if given source package supports both, then to create, at least for new packages, to create only Python 3 version of the package, but add Python 2 support only if it's really needed. So if some kind of application needs that as a dependency or other library needs that, but that's our recommendation for new packages. And shall we delete productivity on already existing packages, remove them from stretch? I would say yes, but you have to start from the bottom so that you don't have any dependencies. So what we probably could do is identify all Python modules which are not available in the Python 3 version, check if they are portable. And if they are not portable, well, just drop them now. And that will get rid of some, how are they called, Python GDK 2 stuff, and possible dependencies of that. Some of that stuff is really useful to our users probably, because it's really hard to build from yourself. You can't just stick it in a virtual and run, set up the py install. Well, port it to Python 3. I mean, how many packages are these? Applications, yes. It's people's personal scripts as well, and that's the harder part. It's actually trying, isn't there some way of actually getting Python 2 to sort of put out warnings or something when you actually run in the applications and this thing really needs to be upgraded? So the Fedora guys had a, well, big database made for packages which still use Python 2. And then, yeah, they are showing their progress, how they are converting to Python 3. So maybe it's time for somebody to do that for Debian as well. And maybe we should disallow uploading new packages on new Python modules, which package for Python 2 only, or even new applications. So we don't accumulate more stuff to port. Yeah, but unless, until somebody is doing such an analysis, how many packages need porting, I think we are still speculating. But about removing Python 2 binary packages, I would not, I just want to add that I would not remove them in stretch and start doing that in Buster and not sooner. So that we can, maybe we can add something to the release note of stretch that if you are using Python, please note that it's a last version of Python where we fully support Python 2 and we will start beginning to remove libraries or something like that so that users are aware that they support Python 3. Well, or we could ask FTP master not to accept new packages. Yeah, but I mean about users who don't know about packages they just need libraries. So we need to communicate that to them that we are starting to fade out Python 2 and they should do that as well. Companies, users, so maybe a note in stretch release notes would be a good idea. If you guys are going to remove the Python 2 binary packages, is there a point in keeping the interpreter around? We will definitely not remove all of them in Buster, for example, so interpreter we have to stay. And in Buster plus one, I don't know, we can check if the PyPy idea is good enough if PyPy is good enough to replace Cpython or maybe it will be better to ship even without security support Python 2 for now. But it's a Buster plus one or plus two. I think it's fairly safe to assume that there's going to be security support available for a Cpython from somewhere because all this Python 2.x code is not going to suddenly die in three years time. It's going to be around for decades. I hope it's only one day. We can hope. Another thing I listed, unless somebody has... Another thing I listed is a problem that I'm aware of. But so far nobody really needed that. In Python 3, there's currently no way I mean for helpers and in the interpreter itself as well. No way to specify minimum required Python 3 version. So if there's a library that requires Python 3.6 and we have 3.5 and 3.6 as supported, which is not the case now and will not be in stretch but maybe in Buster, there's no way to describe... And in Python 3, libraries share. I mean packages share the same, this packages directory. And there were some ideas upstream, how to solve that with subdirectories or headers in files, but it's not fixed upstream. So I didn't try to... Solve it in any way in helpers as well, hoping that upstream will fix it. And right now we support only one Python 3 version, so it's not an issue. But maybe you have some ideas how we can prepare for future. But I guess it's more a question to upstream. We have X Python 3 version, don't we, for our source packages? Yes, and it tells our helpers which interpreter versions use to build a package. But once they build it, it's installed. For example, if user has Python 3.6 installed even now, even though we don't support it, things can break. For example, if user installs something locally, all the tools that are used during upgrades will try to byte compile these files for Python 3.5 as well. And if given file requires Python 3.6, it will fail. We will have the same problem with Python 3.3 sharing those directories as well. And we decided to ignore that one for now because we haven't seen it actually be an issue yet because Python 3.6 doesn't exist yet. Well, in Debian. But it's possible even now that users install it. And we are not prepared for that. Discussion happens on Gabi? Yes. Should I read it? Then we would be suffered. Maybe one point related to transition. It might be obvious, but I think one thing that really helps for these type of things that are otherwise never-ending is having metrics and making sure we actually measure how big the problem is and how much work is left until we can actually remove. We might do that through Lintian, which has now measuring purposes, so we can have graphs that measure across the whole archive, dependencies and stuff. Python 2 or removal? Yeah. Okay, don't work in the void. Just make sure you have a graph that says you're done or you're not done. Don't be silly. Anybody have any problems with the helpful tools I wrote? Is there something I can automate? Did you have to override DH Auto, install or build or something? I didn't have to. I think the usual problems are people struggle to find documentation on it. I know I suck at this, but I try to answer every question I get. I try to reflect that in the Wiki or in Man Page. But if anybody wants to help me writing some documentation, I'm more than happy to answer any questions. I'm more than happy to answer any questions, even if you don't want to write documentation. And I think I did answer at least on IRC. I try to answer at least. But I know that I'm not a great documentation writer, so that's something a newcomer could probably do because the best way probably to write documentation is to write it when you don't know something and you have to dig for answers. I'm more than happy to provide answers as long as I know them. And then somebody has to at least write it down in a way that everybody else can understand it and not in a way I write it. There are probably too many Wiki pages already and sometimes it's a problem. We have our documentation is there but it's in different places and it's maybe not easy to find. I try to keep PyBuild, for example, examples only on the Wiki page that is mentioned in PyBuild's man page, so it's all linked, but I don't know. I still keep hearing complaints about documentations, but I would like to hear more about specific, what is missing? What do you want me to add? I can add that documentation myself, but I need to know what is missing. There's a question. On the documentation, it's less that I don't necessarily know how to do something. I think I'm doing it, it works, but I think it's pretty ugly and I'm not sure what the solution would be, but if you perhaps document some best practices or I feel like I'm hard coding a bunch of stuff and yeah, it works. If you have to do it, then I probably did something wrong in the tools because they should automate everything and if you have to hard-code things and so on, please contact me and I will try to figure out if I can help you somehow by showing how you can automate this or by improving tools to automate that. What would be the best approach for that? They're typically not bugs because I can't really file a bug saying, I think my packaging is ugly and assign it to DHPy. Do you see what I mean? It's a bit... Being me or on IRC and point me to Debian rules file and we will talk about it. For me, those problems are usually around tests. It's just some packages are really hard to test with PyBuilds. The test suite... Are the tests because tests do not run or they fail? Because the test suite is rubbish and it requires you to be in a particular directory and have a totally insane import... Yes. Yes, I added quite a few hacks like changing directory to the build directory, copying test files over there and so on and there are probably more needed but I need to know about that because for packages that I maintain I had to add hacks like that and I add them via improving tools. So I knew about these hacks and I had suffered for them but I need to know what else is missing, what things you need and you can report bugs or just ping me and show me what you needed to run these tests and we can find a way to automate that so that you don't have to do it anymore. And then ideally get to a point where we can automate auto-PKG tests from them as well. But not in PyBuilds. I think I can improve the situation in that area but in the PyPI tool, the one that creates the main directory from scratch I can add support for auto-PKG tests over there but it's only for new packages. I don't really think I can do anything about it in PyBuild because it's PyBuilds used during build and auto-PKG tests is mostly about creating Debian tests control file I think, right? So it's not a place where PyBuilds should do anything. For some test cases it could share the same configuration but it depends on the test suite being very nice to begin with. There is a new Python module in auto-PKG tests. Somebody from our team created it but I'm not very familiar with it. It automates auto-PKG. Yeah, that was Andres Novi. He added support auto-Deph which is a support package that creates, if your package doesn't have Debian tests control it creates one for you based on metadata on the package. So what you want to do there is probably have a extract a tool to run tests from PyBuild and then have that in a separate binary package that you can use as a test dependency. It's something you can run in source directory and it will create Debian tests control file? So that part is partially done by Andres so right now it just does a simple import to make sure that you can actually load the test but when you have that other tool that actually runs the test it's something that auto-package tests will call. Auto-PKG test has a mechanism of auto-tests so a particular type of package can have a single test that works for all of them and this essentially works for all Python modules by doing a simple import. How can we enable it? It's enabled already. Yeah, but it's just an import test. It doesn't actually run any tests. So what we need to do is to improve the auto-generation of the auto-package tests to include a call to the tool that actually runs the tests. Yes, and they have a well standard mechanism calling stuff. I don't like it if we introduce auto-package tests without then doing something with it because usually even if an auto-package test fails nobody cares about it and sorry to say Ubuntu cares about them and I'm doing the whole QA for the Ruby team in Debian or for the Python team so it would be nice if the release team could well honour results of auto-package tests and then it would make sense to turn them on even in a very simple way. So is somebody from the release team here who could comment on that? No? Just making sure it's not me that asked for it. It's someone from the Python team that started doing it. So I know that person because he's been doing some open-stock packages with me. What we do in the team is that we do have tests at build time and we also run the auto-package test after the package is built. That's probably one of the ways to make sure that these tests are in order. It's like you run them after the package is built so that you make sure it's okay to upload the package this way in the archive. That's something that PyBuild can do. And by the way, hello Andres. Actually we're running now out of time so I think we need to close the session because we have an attention session after here so thank you for the BUF and for the fruitful discussions.