 So I think we should get started. So this next talk is Piotr Ozorowski, who is a member of the Python application packaging team and the Debian Python modules team. So welcome. He's going to talk about packaging Python modules and applications and how we can do that more and better for Debian. So everybody, welcome, Piotr. I'm from Poland. Today I will try to convince you that packaging Python applications and modules are not that scary after all. I maintain or sponsor over 100 packages. And recently I was added to the Python defaults uploaders. So you can blame me for all the recent bugs in this package. I introduced some already and fixed all of them, please, the ones I know. And today I will briefly talk about what you have to do before invoking Python helpers about recent changes in Python policy, because we introduced some for Python 3. We want to treat Python 2 and Python 3 as two completely different languages. I will try to convince you to use SDDep. It's a tool written by Andrew Stroh. It creates really nice templates for Python packages, Debian Python source pack and binary packages. And later I will talk about two new helper tools I wrote recently, DH Python 2 and DH Python 3. Originally I wanted to just improve DH Python, but later when we decide to split Python 2 and Python 3 packages, the stack is completely separate now. I decided to introduce two new helper tools. One of them will work with files for Python 2 and DH Python 3 will handle files for Python 3. And at the end of the talk I will talk a little bit about our teams, Python applications packaging team and Debian Python modules team. I will point you to the page where you can join us and talk a little bit about our teams. So first, SDDep. It's a really nice, as I said before, it's a really nice tool that doesn't have much popcorn yet, but it's worth to start with it when you package a new module or application. It does something similar like DHmake, but it's focused on Python packages. At least the ones which use setup tools or these new tools. So you can install it, it's packaged already in testing and unstable. You can install this binary package and it will provide, among others, these free commands, PyPy install, PyToDsc and an extension to these two tools. Actually, a lot of upstreams recommend sudo is install and package name to install their stuff, but we really don't like it because it creates a lot of mess in the directories that are not supposed to be used by users, especially in Python 2.5 and older versions. Things changed a little bit in Python 2.6, but we still don't recommend to use at least this sudo part so that it will not be installed system-wide. Instead, please recommend to use PyPy install and then what usually you pass to easy install. So what PyPy install do, it downloads the tariff from PyPy, so far the same what easy install does, but instead of installing the files to user local Python site packages, it creates a binary package and installs it so that you can now take advantage of all things you can do with Debian packages, like an install, for example, because with easy install, you cannot really uninstall files. There is a command, but if you, for example, use easy install to install pylons, pylons will download a lot of dependencies, and when you will try to uninstall it using this stuff, it will leave all the files in your file system, and it's really hard to clean after easy install. So just recommend to users to use PyPy install if there's no Debian package for something they want to use. If you want to start to create a source package, it's really good to start with PyPy to DSC. This command will unpack this tarbal and create Debian source package with Debian directory, control file, and so on. You still cannot use this source package and upload it directly to Debian repositories because, for example, you have to fill in Debian copyrights, but it's really close to what you will get at the end, so it's much better than DH make. And if you already have an unpacked tarbal with the sources and with the setup py file, you can just invoke the last command, and it will create Debian directory and all the files inside that normally PyPy to DSC creates. And recently we introduced some changes in policy. As I said before, we want to separate Python 2 and Python 3 so that the interpreter itself will be available as Python 3, not Python. Python will still point you to Python 2, the default version of Python 2. And we decided to name all the binary packages with Python 3 prefix. So instead of Python 4, when your package provides Python 5 for Python 3, you should name it Python 3, where foo is the thing you type after import something into import the module. Please remember about this because a lot of people choose to name their binary package after what upstream calls their module, their complete package, and not after the module itself. When your package provides more than one module, then it's okay to use it, but our policy suggests to use the module name in the binary package. And there's another tool, PyVersions, which helps the helpers, CDBS and other, and DH7 to list versions supported by source package. It uses it, for example, to build extensions for all Python versions currently supported by Debian. I will tell a little bit later how to feed these tools in your source package. One of the ways to feed it is XS Python version. It's used by all helpers' tools. And this field will be used only for Python 2 versions. If you want to describe the same way versions, Python 3 supported versions, this uses Python 3 version. Note that there's no S in this field. We decided recently that it's not really useful. While working on Python 2.6 transitions, notice that XS is not really useful. And they were exported to source files only to help and XB Python version only to help release managers. But as Jakub noticed, it's not really helpful. So for Python 3 we will not use it anymore. It has only X now. So it will be in control file, but it will not be exported to any changes or DSC files or binary packages. So what to do before invoking helpers themselves ? First you have to describe supported versions by your source package. I will tell about how to do it in each helper in a few minutes. Py versions, as I said before, will read this data and return a subset of supported versions supported by the source package. Then you have to, if your package provides an extension, you have to build the band on Python all to get all the header files for all Python versions. If your package provides Python files and uses Python helpers, you have to depend on one of these packages. These are virtual packages that depend on currently supported default Python version. Python dev depends also not only on the default one, but also on headers for default Python version. Python all will get you all Python interpreter packages currently supported by Debian. And Python all in the back will get you the same for the back and for interpreters compiled with the back support. Then you have to, once you already know which Python versions your source package supports, you have to install files to standard Python locations. So for example for Python 2.5 to user-leap Python 2.5 site packages, or to user-local-leap Python 2.5, most helpers will move the files to the right locations. So even if you install to user-local, they will be moved to the right location, but try to set the prefix, correct prefix or new flag install available only in Python introduced by Debian, install layout depth. This is even more important in Python 2.6, because in Python 2.6, we decided to change the default prefix to user-local. And another big change is that by default, Python 2.6 will not only install to user-local, so easy install will not mess with our files, but also Python 2.6 will look for files in these packages instead of site packages. This is introduced because when users compiled the local versions of Python, there was a conflict on the SysPath where system provided Python used the same location as Python compiled by users. For example, user-local-leap Python 2.5 site packages is in the system Python's SysPath and in the Python compiled by users. So if, for example, user wanted to install some stuff for his local compilation, they automatically appeared in the system-wide pythons. And to avoid this problem in Python 2.6, system Python will use these packages instead of site packages, and locally compiled Python will still use site packages, so there will be no conflicts anymore. So by default, user-local and these packages in system Python and in local compiled Python, user-local and site packages. And if you want to just use helper tools to move the files to the right location, you can not touch anything and it will work. But if your module hardcodes path at build time, it's better to pass install layout depth to setup py install code so that the right locations will be hardcoded. Or if you want to install, if you are packaging Python applications, you can also use install-leap and install scripts and commands. And, for example, install files to user share package name. This way you can avoid polluting global namespace because Python applications usually provide module names, really common module names, so this way you can easily conflict with other packages. And also install-leap, user share package name provides a way to avoid current keyword in Python central because Python central has a feature to compile py files only to one Python version at the same time. The current one or the current one, current with the minimal version required by Python model. And later you just have to, in devian control, add Python depend to the depend line and that's pretty much all what you have to do for simple models or applications. Next, DH Python. DH Python is the first hardware tool. It doesn't really do much. It fills in the Python depend, so creates dependations on the minimum required Python version. If there are any Python versions hardcoded in Shebangs, it will add it to the Python version required to the depends. It will delete PyC and PyO files that should not end in the binary packages. It will also take care of creating these files at install time and remove them later when the package is removed. Python central is the first of the new helpers which will complicate things a little bit at install time because both Python central and Python support create links to the, in the Python, currently supported Python versions at install time. So they both ship files in user share Python shared and later at install time create sim links in the locations which are recognized by interpreter. So Python central first. To feed it with the, to feed the py versions command I introduced before, you have to add, in the source section of devian control file, you have to add XS Python version with, for example, all or current or minimum required Python version. Actually, we don't, all is already deprecated and current is already deprecated and we do not recommend to use all. Instead, please use the minimum really, really minimum Python version because there are quite some packages that just declared all and maintainer didn't really check it with minimum version and Python support and Python central try to compile, if you put all here, both of them will try to compile py files for all Python versions and, for example, by removing Python 2.4 a few months ago, we noticed that not all, that some new packages still, if Python 2.4 is still installed and maintainer of new packages didn't correctly set the minimum version, Python 2.4 were still compiled and sometimes failed due to, for example, syntax changes. So please try to set the minimum version. Even if you don't know, it's probably better to set it to the minimum version you tested your package with rather than all. Then, if in Python central, you have to provide another field in all binary packages that use, that provide that ship py files, you have to set XB Python version field with exactly this value. Please don't put versions or any other strings in this field because Python versions will be replaced by helper tools with versions actually tested at build time with Python versions that this binary package works with. So, for example, if a binary package provides extensions, it will list versions that package ships, extensions for which, versions for which extensions are shipped. If you use DH, you have to invoke DH with PyCentral switch because, by default, DH is using Python support and in CDBS, all you have to do is to add that Python system PyCentral line at the start of Davian rules file. All packages created in Lenny had back and they are not removing PyC files at upgrades. This will be a problem while Lenny to squeeze upgrades because many packages were converted to Python support and even for those which were not converted, this back will create some mess because even if packages still, even if package still uses Python Central, if the list of files changed from Lenny to squeeze, the PyC files will not be removed and, for example, if module was in a file and now is Python package, so there is a directory with initpy file and all other parts of the module are moved there and Python will have problems recognizing actually it will choose only one of them so if you think one of them should be used, you can be surprised that the other one is actually used by Python so there is a squeeze version of Python, there is a trigger which tries to remove all the files that do not belong to any package so it should not be that of problem anymore but please don't remove your maintainer scripts that clean after Python Central. The next helper is actually the preferred one. Most packages now use Python support so if you don't know what to choose, just choose Python support and it will be the right choice most of the times. You actually have to choose Python Central only if other packages that use the same name space, if your package uses the same name space that other packages and in all other cases just choose Python support. In DH Python, you don't even have to do anything else, it will be used by default. You can use the same field Python Central uses to describe supported versions or you can use Debian Py versions file which has a little bit different syntax. I didn't like it at first. I actually found it really ugly but after implementing DH Python, I started to like it and even used something really similar. Two dot four minus means that this package supports Python two dot four or any later version. This is how you describe a range. Two dot two is the minimum Python version, two dot six is the maximum supported version and two dot six is included in the list. If your package supports every Python version less or equal to two dot seven, just use something like this and if your package uses only one, is compatible only with one, Python version just list it like this. As I said before, Python support is default in DH so you can just use the tiny version of Debian rules available in the helpers examples or at the top of Debian rules align with the Python system by support if you use CDBS. Python support creates the sim link farm in triggers so if your package has to work at install time, for example, if you want to start a demon or actually import some of your modules, you have to invoke update Python modules minus p command so that it will speed up the process where PyC files are created. But use it only if you really have to because this command is rather slow because it has to check all the installed packages and check if the files, PyC files are still needed and so on so it's slow. Use it only if you really have to. Another feature that distinguishes Python support from other helper tools is namespace feature. Python support tries to avoid providing a common package or providing a namespace. Namespace in Python is declared by slash slash underscore underscore pi file and if this file is missing, the Python package will not be recognized by interpreter and Python support creates these files in directories where this file is missing so that two different packages, binary packages that use the same namespace don't have to provide this file. So they will not conflict with each other. You don't have to provide a third package that will just ship this one file or not to provide it in one file and depend on the second package in the first one. Python support will do it for you, create these files and add install time type while update Python modules is invoked. So next are two helper tools written by me recently and both tools are available with interpreter packages, DH Python 2 in Python package and DH Python 3 in Python 3 package so you don't have to depend on any external packages. Also binary packages that use these helper tools don't have to depend on any other packages. All is included in Python. So advantages of DH Python 3. All files are included in the package. That means all sim links are provided by the binary packages. There is no way to fail at install time while creating sim links. You don't have to invoke any tool to create namespaces and so on. Everything is available out of the box. You can start demons in your maintainer scripts. You don't have to wait for PyC files being created because Python, if PyC files are not there, will just use source files and this makes upgrades really friendly and fast. There is one big problem with this approach that makes these tools not really friendly adopted so far. It's the fact that binary architecture independent packages will have to be rebuilt when a new Python version is added to the supported versions. It's easy when your package provides extensions because you have to rebuild the package anyway but all other helper tools, Python support and Python central didn't require a new upload when new version was added to the supported ones. This helper requires this. DH Python 3 will not require it because it uses some different techniques I will talk about later but DH Python 2 unfortunately requires uploads whenever new version is introduced. You don't really have to upload new package if version is removed from the supported ones because PyC files will be removed anyway when you remove our upgraded package but you have to rebuild package when new version is added. This helper handles the back packages so it removes Py files from the back packages and generates dependencies. You don't have to worry much about it like in other tools because in Python central or in Python support you had to do all this stuff manually and additionally in Python support, GDB didn't recognize files in the locations that Python support uses by default because GDB tries to load the back extensions from the same location where runtime files are available so when Python support ships files in one directory and then at install time creates same links in different place, GDB tries to load files from the same location as the runtime by adding user the back first. It's not a problem in DH Python 2 because it uses the same locations at build time in binary package and in install time so GDB will work as expected. Another advantage of this helper is that it's really fast. I made a little change in Python interpreter itself and I can now, for example if you have two cores it will try to generate Py C files at the same time for different versions so if we currently support Python 2.5 and Python 2.6 it will create byte compile files and create this cache files at the same time. You don't even have to wait for it to finish because Py C files are already in the package so it will work out of the box even without Py C files. So how to describe supported versions in Python 2.5? Just use one of the previous versions. There are some differences in the Debian Py versions file actually but it will be, it's a bug on my site. It will be fixed soon. You can, so recommended way is to use XS Python version field, the one which is recognized by all other tools. If you use DH Python you just have to invoke DH with Python 2 attack and it will use DH Python 2 instead default DH Python support. If you, DH Python 2 by default tries to read, tries to do something like one PyDep does, it will read all the metadata provided in X and tries to translate them to Debian dependencies. If your upstream didn't list all the dependencies you can invoke DH Python 2 with something like this. This is what, what upstream should put in requires takes the file but you can also add it here and it will be translated into Debian dependencies. This is actually another advantage of DH Python 2 because you can provide PyDest file which will help DH Python to translate this, this upstream requirements into Debian packages. You can use it like this to only translate Python distribution names to Python packages. You can tell DH Python that this package uses PEP382 standard so that it will apply all the rules. You can add regular expressions that will translate or use both of them. Regular expressions that convert upstream version scheme to Debian one. If you don't want versions to be translated just add version to the PyDest file and DH Python will not try to translate it. You can also use it if you know that your upstream is breaking up your abby from time to time and you can predict it. Something like this happened recently in NumPy and if you create PyDest that looks like this, DH Python will create at this dependency on to every binary package that has NumPy in request.xt or in one of these commands. If you put it in depends it will add to this line to depends, recommends to recommends and so on. If you don't like what maintainer or upstream declared, you can overwrite it with in your package in Debian PyDest overwrites file. You can use in both this overwrites use the same syntax as PyDest files. You can in both cases you can use multiple lines. For example, if your package provides more than one Python distribution. As I said before, we have a separate stack for Python 2 and Python 3. DH Python 2 will ignore all binary packages that start with Python 3 prefix and Python 3, DH Python 3 will ignore all Python packages. Other differences are that user share by shared will not be used at all because in Python 3.2 there are some major changes described in PEP 3.1.4.7 and 3.1.4.9. I will talk a little bit about them in a moment. And to use DH Python in the helper, just add Python 3 to the width. You can use Python, for Python support with Python 2 and DH Python 3 with packages that use Python 3. If you don't like the fact that DH Python 2 requires new uploads every time new version is introduced. By this file are also available in DH Python 3. It's just a single change. You have to add 3 to the file name and DH Python first will install. If you create such file you don't have to worry about installing it. DH Python will install it and to overwrite what upstream or maintainer wanted to put in the pants just use overwrites. I will speed things up a little bit because we have five minutes left. So PEP 3.1.4.7 it creates, it changes Python import mechanism a little bit so that we will be able to use common directory for Python source file and it will create, it will look for PyC files in new directory PyCache. The file will contain Python version or this file, the interpreter will look for. The same directory will contain, for example, files for Swallow or other Python versions. So there will be one directory with common file and PyC files will be created in sub directory called PyCache. But this will work only for source files and only for files that are not changed among Python versions. For extensions there's another PEP that will allow sharing the same directory for different extensions. Extensions cannot be shared so the Python itself will be changed to look for, not only for foo.so but also for C Python version and attack. U means Python was compiled with white Unicode. This way we can have the same location for, for example, the back packages. We will not have to add underscore D and the more Python itself will care about it. So the final files will look like this. There are two more PEPs that will help us. PEP 3.8.4 will describe stable API for extensions. Some API functions will be declared stable and if Python 3, if you will use only this stable set of stable functions and declare that your extension doesn't use the different functions, file name will be used like this and Python interpreter will, will use this file. There's another PEP which will extend PTH files that will help us to deal with source Python files that cannot be shared among Python versions. And about our teams. We package a lot of packages in both teams. We are really friendly. You can join us. We hang on Debian Python IRC channel. We are open to Wuntu contributors. We actually steal a lot of developers from them and later force them to complain about this with supported Python versions in the Wuntu. We have some, our members are among release, no, in FTP master's team and in new maintainers. So when you, if you join us, you will have some extra points in there. Your packages can be processed fast or you can find application measures, probably a little bit easier. Any questions? I'm actually afraid that we probably need to wrap it up. So maybe one question because we need to leave video team time to prepare for the next talk, which is in five minutes. So just to see if I understood correctly. Sorry. Is it possible today to package Python versions or modules for both Python 2 and 3? Right now? Yes. Python 3.1 doesn't use these peps. So it behaves similarly to Python 2.2 versions. So just, but Python support and Python central do not handle Python 3 versions for now. So you just have to use DH Python 3 if you want to provide extensions or files for Python 3. Thank you. So all you have to do is provide a new binary package because Python should not be, Python 3 files should not be in the same binary package and use DH Python 3. If you have an application that uses mixed Pyrex and Python, how do your tools deal with this kind of thing? These files are created at build time. So helpers do not do anything in this direction. You build extensions the normal way and later invoke helpers and helpers will move the files to the right locations. Is that what you ask? More or less what I'll ask you more later. Okay. All right. Thank you, Eastern. Thanks.