 Packaging, turning a project into distribution. What is a project? It is a collection of Python files that usually make up one Python package and metadata about that package. So here's where it starts to get a little confusing. I'm sorry for this, but I didn't make up the terminology. A Python package is, are your source files? They're .py files and the directories that you sit inside and they're what you import and turn into module objects. They have nothing really to do with our talk today about packaging other than that. It's going to be inside of our distribution. So the interpreter is what defines how you lay out your code so it can be imported and what it gets turned into and we don't care what those rules are. We care about how to put it into something that can be shared with other developers and turns out the interpreter doesn't really care how we do that. It doesn't care that we upload it to pipi.org or if we ship it on floppy disks it doesn't care how we get the job done. So packaging is turning project into distribution. So what is a distribution? Distribution is a single file that by itself represents an entire project. So the Python package and at least some of the metadata that was originally in the project. And so it's a single thing that you can upload, you can download and can be installed. And once it's installed, the Python package is available for you to use in your Python code. So that's packaging. So we're gonna talk about who does the packaging? It's the package or it's me, it's you maybe, maybe you want to do packaging or you've started to and you're not quite sure all the choices to make. But we're gonna go on a little story today and we're going to start with just the Python package which is the code and we're gonna transform it into a project. Then we're gonna take that project and build it into a distribution. So it's the packaging part and then we're gonna publish our distribution so that it can actually be used by other developers. And then finally, as little bonus, we'll see how the end user actually consumes the project, even though it doesn't really have anything to with the packaging story for the package or themselves. So this is our package for today. It's some Python files and it's not the simplest package. It's got some sub modules and modules and it all works perfectly. It's a great package that everyone wants to use. So we're not gonna look at it really any further. We're gonna turn it into a project. We're gonna do that by stepping up one level in the directory tree and we've moved out of the EuroPython that is part of the package. That would be the root module you import from Python and we stepped into the EuroPython folder that is the project. So it contains a package but it also contains metadata. And all that metadata is gonna be contained in one file today, pyproject.toml. So it is the pyproject.toml. It is designed to be a single location for you to put all your metadata. So you only have to have one file. It does a pretty good job with this. It's tool agnostic. Any tools can use it and they can share it between different tooling and if depending on the tools you're using besides just packaging but for other things through projects it could very well be the only source of all your metadata, which is nice. But it's not itself a build tool. Pyproject.toml isn't something that you can run and it's not something that can package for you. It's just a place where you put data. So choosing to use pyproject.toml is something you can do and should do for new project X but it's not a choice of how to build your project. So what are we gonna put inside of pyproject.toml? The first thing we're gonna put in there is some build system metadata. So we're going to have to choose a builder because in Python, unfortunately or maybe depending on your opinion, fortunately there's really no way to build a Python distribution these days without a third party dependency. The language itself can't do it for you. You have to have something else installed beyond just Python. And what that is is kind of a choice you'll have to make in the end because there are a lot of different options but whatever you choose, consumers of your package who want to build it, including machines have to know what that choice was. So you have to write it down somewhere. And these tools are given a lot of autonomy about how they structure themselves and what they do. So you have to also write down the entry point into the build system of how it wants itself launched. So these are some of the build back ends that we have in Python, just a few of them. There's many more and there might be more in the future. Anyone can create one and upload it to PyPI as a brand new build backend. But we're going to choose today with our project to use setup tools. It's kind of the workhorse and one of the oldest choices for Python packaging. So if we select setup tools, we'll take this PyProjectTable, which is sort of just a generic start that doesn't have quite enough information to build a project yet. And we'll stick setup tools specific information into it. So we'll put in that our project in order if you want to build it, you have to have not only Python, but you have to have setup tools and you have to have a fairly recent version of it as well. And then once you have setup tools, we're telling mostly machines at this point, but we're also telling humans, if you want to actually launch setup tools, this is the object location in Python where it contains some standard APIs that are used for building Python projects. With this, anyone who first grabs your project, a human or a machine who doesn't know anything about it, they can figure out how to build it without having to know ahead of time what tool you chose, which is nice. So there's another section we're going to add to pyproject.toml, which is project metadata. So this is all the information about your project itself, details that will be attached to it when it gets uploaded in distribution. Some of them are important for packaging, necessary to be accepted by pypi.org or necessary to be found by installers that are looking for your package. Some of it's not necessarily mandatory for packaging, but it's nice information to attach to your project, things that help humans out or maybe searching for your project or discover it and want to know more about it. So we're going to add a new table to the pyproject.toml file, the project table, and it contains several entries here, more than is necessary for bare bones build, but it contains a name inversion, which are pretty mandatory fields to be able to upload to pypi. We've listed our dependencies so that anyone who installs us knows that if you want to use EuroPython, you must also have EPS installed ahead of time. And then some of this other data is not really needed by packages or builders, but is helpful to humans who may come across our project. And so there's many, many more fields. You can put it in the project table. They're all listed at this packaging.python.org, Glink that tells you every field name and what types it can take within the toml file. So back to our projects, we've added metadata. So we have a proper project that has working code and it has correct metadata. We can build this project now. But we said already that having a pyproject.toml doesn't add build commands. It's not executable code, it's configuration. We haven't changed any of our code, so there's nothing to act on yet. We chose build backend, but you're not actually expected to go into other people's pyproject.toml and find the dependencies installed in yourself. There's tools for this. So we're gonna have to look at a new category of tool called the build frontend. And as you might imagine, build frontends are designed to call build backends and interact with each other. And they do this through an API that's been standardized through a few PEPs so that any build frontend can work with any build backend. So we're gonna choose build frontend and install it, but we're not gonna record our choice anywhere in the project. We're not gonna write down pyproject.toml or any other metadata file. And this is because the choice of a build frontend is much more personal than the choice of a build backend. Whoever is the one building the project gets to choose the build frontend. And since they work with all build backends, the author of the project doesn't get to dictate what build frontends can or will be used. They do choose the one and only build backend that's possible to build that project. But the frontend, any one of them could build it. So we'll choose the one and we'll install it, but we're not gonna write it down anywhere. These are some of the build frontends. Again, there's a lot of choice in Python and we have to have some third-party module that above and beyond just the language. So this is actually mostly the same as the list of build backends. There's no reason a tool can't be both. But even though a tool does both sides, this doesn't mean it can't be mixed up with kind of the other ones. They're still all interoperate. But the build backend that we chose, setup tools, a very popular choice for build for backends is not actually also build frontend. It doesn't provide a UI to building. It doesn't work with other backends. So we're going to have to choose a new tool to perform this new task and we'll install and use build as it's just a frontend and doesn't do any other jobs. So we'll install build and once it's in our local Python environment, it provides one command line to call, which is PyProjectBuild. We call it, it doesn't take any options or arguments and it goes ahead and does its thing or report success. It builds our project. So what did it actually do? Well, in our project folder, we still have our package, we still have our metadata, but it's added this new disk folder with two files inside of it. So what's up with that? We said originally that packaging is a process of turning a project into a distribution. That was a bit of a fuzzy definition. Actually, the process of packaging is turning a project into multiple distributions, one or more, in this case, two. So each of those files is a distribution, but we made two of them with one build command. So there's all kinds of different distribution formats, but in Python, we have two primary ones, the wheel and the S disk, and build one ahead and created one of each for us. So if you're packaging Python, you're going to make at least two distributions. There's always just one S disk per version of each project, but there's one or more wheels per version of each project. This is because wheels are binary format, which means that all code inside of them is pre-compiled. The compilation is done by you, the Packager, and then it's cheaper and faster for your users to install, but you must support all platforms, all Python versions, perhaps different Python interpreter types ahead of time. Whereas the S disk is a source distribution, and it's S disk, which means you don't do any compiling, you force your users to compile on their machine at install time. So we made one of each. And then this is the end of the action of packaging, but now we want to publish. We want to actually share our code, be able for it to be discoverable and installed by others. So for this, we will need one more type of tool, which is a publisher. These are some of the publishers you need because you need another third-party dependency to publish in Python. Some of the names that were in the build tool sections are still here. They're doing multiple jobs, building and publishing, but not as many do all these jobs, and the tool we chose to build doesn't also publish. Builder, just build. Twine is for the most general purpose tool for uploading distributions. So we'll install them to our local Python environment and we'll use it to upload the distributions that were just built. So we install twine and it provides this command, twine, and we use the upload sub-command. And these two short commands are enough to get our distribution onto pypi.org in reality. The first command isn't even strictly necessary. It's just good practice to upload to test.pypi.org first. And that is because it is a test namespace of pypi.org where you can make sure that your distributions aren't rejected by the API for some reason and that even when they're accepted, the project page on the website kind of looks like you thought it would render. It has links like you thought it would. All your pictures look nice and pretty. And if you're happy with it, then you can do the real upload, which by default goes to pypi.org. So you don't have to specify that. So one or two commands and you're publishing on pypi.org. And that's kind of the end of the responsibilities of you as a packager. You've made your project into distribution and you've even shared it. So you're kind of done, at least until you wanna go and make another version of your project. But the package isn't quite done being a package. It's gone to pypi.org, but if it just sits there, it hasn't done anyone any good. So to finish out its story, we'll take a very brief talk package managers, which kind of do a lot, but quickly, a package manager takes a distribution file, the thing you made, and it kind of reinflates it from a single file back into potentially a directory of files that you can import into Python. But that's not enough because we pretty much get all our packages from someone else's machine these days. It has to also know how to download them and has to know how to interact with package index APIs to find the thing you asked for. Because really when you install the package manager, you ask for packages, not distribution filenames. So it has to correlate the package you asked for with a version, which is probably a range of versions and find the best fit. And once it's done that, the package exec says, sure, I found that package at this version that you exactly asked for, but that user uploaded eight different distributions. Which one would you like download and install? And the package manager will figure that out and once it's done all that, it then also has to do it for all that packages dependencies and dependencies of dependencies. And only when it's been able to solve and find one distribution file for each of those, does it come back to you and say, yeah, I was successful in installing that thing you asked me to. So it's a lot of work, but in the end, package managers are the thing to download the distribution and unpack it and put it into your environment. So again, we have to choose a tool to do this. Python natively doesn't install packages. The list of tools that I choose from is quite different from the list that we started with back with building. Most tools don't go this far in the path of the package lifecycle, doing all the steps. Poetry I think is the only one I know of that does all of them. Every other tool is kind of dropped off the list, but new ones have come up and our kind of favorite PIP, I think everyone has run a PIP command at some point in the Python journey. So we'll run run right now. PIP install your Python would go and fetch one of the two distributions we just uploaded and we'll install it into our local Python environment. So we could actually import it and use it. That's pretty great. So we've kind of done it, you know, packaging has been accomplished. There was a few steps and a lot of tools to choose from but it wasn't too hard. We transformed our package into a project. So we had to add some metadata. In our case, we could add all the metadata in one file that fit on one slide and it was enough to build our project. But doing so, we had to select a backend and we chose set up tools. And then we stuck that information in the build system table. If you chose a different backend, you would have different requirements and a different build backend entry point and you would have to find this from the documentation of your chosen tool. Otherwise, it would look very similar. And after we have a project, we want to build distributions from it, at least two that represent our project. And then we want, so we needed a new tool for that. You wouldn't always depending on which backend you originally chose. So we had to also install build locally and then we could run that very simple command. After that, we had our distributions but we wanted to share them. So we needed publishing tool. And again, depending on what build system you're using, you may be able to stick with the same tool or in our case, we went to twine and we ran two twine commands to get our distributions onto pipi.org. And then to round it off bonus story, we saw how packages very quickly are downloaded from a packaging decks and then installed into your local system. So it reinflates the distribution and some of the metadata onto your Python path so you can import it. And that's a quick journey of how you package and Python at least right now. Now, can you take the screen right now? Though I can't hear anything. Okay, can you not hear us? Yes, I can hear you. All right, so good. I think if you're clapping might be that you, we have to clap louder next time. And if there are any questions, oh, please come into the front to the mic. And yeah, he's gone. But maybe you can queue up already if you have any. Jeremiah, are you still there? Yes, good. I'm still here. Hi, thank you for nice, like overview of like tools for various steps. And at each step, you choose one of the tools. Should we take it as a recommendation that I don't know, you set up tools and not something else or like was it like just like the simple solution? So yeah, I mean, all these tools have trade offs which is why, you know, they all survive. There's some competition in the packaging space. I would say the tool that I laid out are very common, especially because I think, you know, as little as four years ago, and that's kind of little in the history of Python, they were your only choices, set up tools and PIP and to find at least kind of predate all the others by years in most cases. So they're very popular choices and I go with them for some cases. So yes, it's a bit of a recommendation. It's not my workflow for every project that I work on, but some of them it is. So I would say it's a very common and easy to find answers if you stick with those tools. Thanks. Nobody else is in the queue. Can I have one more? Of course. What do you think about Hedge? I think it's because I noticed that it's a new like recommendation at like Python packaging side, but I don't have experience with that. I don't have experience either. Yeah, I haven't used it in any project I work on. So I'm not the best person to ask it. I think it has been around a long time, but I've never actually seen it used either in any of my projects or any project I've directly sort of cloned and built myself. So I don't, I know that some people recommend but I don't know who's using it. There must be several people. Thanks. Yeah. Okay, does anyone else have a question? Otherwise I have a question for you. We in our organization, we still use setup.py and I still have the benefit of like calling a function and setup.py mainly to get the version from GitHub, for example. Is this, is there a way to support this also through the Tomo file? Or how can I get, for example, a version for distribution from GitHub? How do you do this? You can use pyproject.tomo and keep your setup.py around. They can actually coexist and they can do so, you know, very well where you can put all your static stuff in the pyproject.tomo and then if you need to run one or two functions, you can just keep all the dynamic stuff in setup.py. So it works very well, but the recommendation is to get off setup.py even if you're using setup tools. It's best if you can to just use setup.cfg. If you need something that's not supported by the Tomo yet, but if you need to do something dynamic, there's not, none of the other build systems as far as I know, let you run arbitrary code. I'm sure some of them do. I guess my recommendation if you're looking to get away from it is there are tools that do things like figure out your version from a Git tag or from Python code or something like this, like version here is one example. And you can include those other dependencies in your build system requirements. So you can put things in there that aren't just the exact build backend. So if you found some module that was able to go to GitHub on your behalf and retrieve it and somehow insert the version into your distribution, you could add that to your dynamic requirements and somehow hook it into your build backend. There are ways of doing that. And then so it's not you putting arbitrary code in a set up UI, it's you using a build package which technically is still running arbitrary code but it's sort of like a hook in your build system. All right, we have a remote question. Perfect. Let's come on. All right, let's see if that works. All right, thank you for the talk. I have a question regarding the competition that you mentioned. So on the one hand, you have the pipe UI set up tools, totally ecosystem. And on the other hand, you have the Kondaforge member. How would you compare those ecosystems? Thank you. Right, so you're kind of veering off from Python packaging into more of general packaging or OS level packaging. Kanda and Anaconda are very Python focused but part of the reason that you would switch them is because they're pure Python or like pip implementation is generally because you need things that aren't just Python code or you need to link dynamically into languages that aren't just Python. And so I mean, that's kind of advantage is that they natively handle things that aren't just Python or C code that is directly being turned into Python extension modules. And you also have that. So kind of Anaconda is sort of halfway, maybe even a little closer to like an OS distribution like your app, do you yum? And so sometimes you need that, but yeah, don't go there if you just have pure Python or things are just working great in pip. But I guess do if you're doing something that requires packaging in languages that Python doesn't understand how to unpack and pack. All right, the next one. Hello, thank you for the talk. My question was a little bit towards like the setup.py question but more focused towards C extensions. What if we do need to do some compilation step? What options do we have other than the setup.py? So I've only built extension tools using a setup.py. So I think you can do it with jessaserv.cfg I'm not certain about that. But that's definitely a reason to choose setup tools is all of its long support of building Python extension modules it's pretty good at. Several of the options I showed are actually restricted to only pure Python packages and they don't even try to build C code let alone Python extensions or dynamically linked C code. If you want your Scython, like that's a different kind of extension module that's not pure Python. So it's definitely a reason to take a second look at setup tools if you aren't already using it. But some of the other tools like Nscans can build C code and Meturin is specifically for building Rust code. So if you're trying to make a Rust extension module like setup tools natively doesn't work with Rust. There are setup tools rust cross plugins and now you have multiple build time requirements. You could do it that way. But some of these backends only work with certain languages. Some of them only Python, nothing else. All right, I think we are through and just on time. Thank you for the talk and for the nice Q&A session as well. And let's give him a very loud applause. So...