 Yeah, so I'm going to talk to you about packaging and I'm looking at the slides right now, they're really not that great, so let's... I mean, I don't know if I'm going to keep being a software engineer, seeing as I can't use an HDMI output, but I'll probably be a really good stand-up comedian, so... So, who here has already done packaging in their lives? Okay, so I'm in the right bedroom, so it's basically, people have given up on me, it's basically turning an upstream package into a downstream package, so you take code upstream and you make it available for your distribution. So, it involves a lot of debugging, so the software doesn't just work out of the box, it never happens, and it means you have to send patches to the upstream developers who have to maintain them. So, that's the basic idea of packaging, right? Is it hard? I have things to actually show you, but... Is it hard? Will you say it's hard to package software? Yeah, I would say... Depend... Depend of... Depend of... Depend of software... Depend of software... Okay. The issue is that you have a lot of pieces of software to package today, and they come from very different sources. So, I'll be focusing this talk on public package archives, such as CPAP, or both, or PyPI for Python, and that kind of stuff, so NPM, that kind of thing. And you have many different targets, so you have to package software for TV and Fedora, DX. You might use one operating system at home and a different one at work, and you might want to use the same software, so you might have to do packaging for a different operating system. You can't really automate everything, because debugging cannot really be done by a machine yet, otherwise it could be out of the job. And you often end up in dependency health where you want to package X, and you have to package Y, and you have to update the package Z, or whatever. But some tasks can be automated. Think of a deviant package, so if you don't do deviant packaging, bear with me. It starts with creating a folder called deviant, and then you have to put 27 different files in there, and some files just have 9 in there. Why not? I work at Red Hat, and I'm not great at sales on camera, and I use deviant, so I pick on the deviant that I like it. So I also use GIG, so basically this can be done by a machine, and you have a lot of things that just... What's the name of the package? What's the version of the package? What's the email of the author? And that can easily be automated. You can have a script to this kind of work for you. So over the years people have used scripts, have written and used scripts that grab a package from Python, from Ruby, and turn it into an almost working deviant package of Fedora package. Has anyone ever written or used a tool like that? Yes. Okay. So has anyone written a tool like that? Yeah. Which one? Well, I worked for a different ego system. Okay. Oh, okay. Okay. I'm going to tell you why you are wrong, and keep in mind that I started by doing the same. So this is something I wanted to show you, maybe if I fold up my unknown. So the thing is, you have to imagine that we basically do... There's one tool that maps between a public package archive and a distribution. So if you have M package archive and M distribution, you have N times M source. Okay. You don't need the slides. Maybe you have slides for the line? Yeah. The slides are on my first N page, if you want to follow along. And basically I have a page page. So you have to imagine... You have a huge table full of tools. So basically you want to package Python packages on dvn. You have your tool. You want to package JavaScript packages on Fedora. You have your tool. Why is it terrible? You have... Thank you. I started two minutes ago. Come on. You have one CLI for tools. So if you want to use multiple tools, you have to learn how to use... And you have to remember the command line options. So different CLI, different behaviors. Some tools might run tests. Some tools might create directories. Some tools might not. So you don't really know what to expect. And that's code duplication. So all the tools that read metadata from PyPhi will write how to pass metadata on PyPhi. So that's code duplication. All the tools that you can use on dvn will have code that basically makes your dvn and eco whatever to dvn slash rules and stuff like that. So that does code duplication in two different parts. And you might think, is it bad to have code duplication? Because things are really simple. And if you were in school, teachers would tell you, you do not want to copy-paste code. But in real life, we're talking about how thousands of packages, so it's a huge code duplication. Yeah, but there's only one tool that maps between, say, PyPhi and dvn. And then there's one tool that maps between PyPhi and Fedora to create a package. So, yeah, there's a huge table of... So you have to mention, you have a list of package archives and you have a list of distributions. And that's a huge table that you can see on page 8 of the slides that are not here. So is it really bad to copy-paste a little bit? And so this is where I have code to show you, but I'm going to describe code to you. So PyPhi, just like a lot of package archives, gives you a JSON file that has a lot of metadata in it. And it has the dependencies of your package that really matters. In this JSON, for some packages on PyPhi, you get the list of dependencies. You're really happy with that. It's just one field to read in a JSON file. It's a really easy code. And you're like, well, maybe you can duplicate that. It doesn't really matter. But on some other packages, for reasons that I don't understand, and if someone can explain that to me after the talk because I'm running out of time, I would be glad to understand why. For some packages, the JSON file does not have the list of dependencies in it. So you're not getting it. What you can do is download the Python wheel. So if you don't know what a Python wheel, it doesn't matter. I'm a Python software developer. I'm not sure exactly what it is. It's a type format for Python that is great for many reasons. And in five years, they'll use something different because everything has to keep changing all the time. So no worries. It's basically a zip file technically. So you can unzip that file. And the metadata you're looking for can be found there. So you have to do one thing. If it doesn't work, you can do a second thing. And I'm a really annoying person. So I found other packages where the first method doesn't work, the second method doesn't work. And basically what you can do is try to read the source code and find out what the dependencies are. Good luck because it means actually running code and that's extremely unsafe and that's difficult anyway. So basically, you have to try one thing, try a second thing, try the third thing I haven't thought of. And you can do one step, two steps, three steps. And that's really complex and you have to write unit tests. And you do not want to do this for all the distributions out there. You'd much rather have one single tool method. Hey, yeah, I can get you metadata from PyPI. It does all the crazy work. And then you can use it on Debian. You can use Confedora or whatever you want. So to sum up, what do we really want? We want a unified CLI. You want a tool. You can say, I want to pack it. A package that's available in PyPI and the distribution I use is Debian. It's Fedora. It's open BSD. We want a unified behavior. So we want to be able to know that what's happening is going to happen the exact same way for all distributions. We want something that's modular because we want a module that can read PyPI. We want another module that can read RubyGems, another module that can read the Perl archive and gather metadata. And then we want a module that can write a Debian package, a module that can write a spec file, a module that can write a mech file for open BSD. And we want something simple. So is there such a tool? No. End of talk. So there wasn't. And I decided to write one because as the fine gentleman over there, I started by writing a tool in Geeks that was just mapping from one archive to one distribution in this case, Geeks. And I thought, that's basically nobody is going to reuse the work I've done on PyPI and it was a huge pain in the... Well, so maybe I could share that work with some other people. As we can see on slide 17. So the idea is just what I've explained. So it starts with modules that read metadata provided by your public archive. These modules send all that data to the core of the tool I wrote, which in turn sends it to a bunch of backends that create packages. So we have a few backends. So my tool is called the universal packaging tool because I have no ideas for naming stuff. In short, it's Upt. And I have a circle Upt-pyPI which passes PyPI Upt-cpand that does the same for cpand. And then I have Upt-openBSD that creates an openBSD file and so on and so on. Is that clear without the slides? It's basically... You can think of it as a compiler. But in a compiler the frontends are the languages and the backends are the hardware architectures. And here the frontends are the public package archives and the backends are your distribution. So I wrote this as a proof of concept. I have three backends for cpand.py three frontends for cpand.py and RubyGems. And I have five backends for Fedora, FreeBSD, Geeks, Snicks and OpenBSD. So it works. You just have to use the Upt package command. It's all in the slides that you will be reading in the train or in the plane when you're bought to death and going home. You just give it a dash-dash frontend option that says, oh, I want to package something that's on PyPI. A dash-dash backend option that says, oh, I want to package it for OpenBSD. And it will create all the files you need. I'm really proud of this because, of course, I was supposed to show you a Mac file for OpenBSD and you're like, yeah, you see, it's almost complete and although the packageer can just review it and fix the very little things that could not be automated, but I won't be able to do it and you'll have to believe me, the Mac file is amazing. It's much easier this way. I'm going to just call THD and my cables and all my tops now. And yeah, then I would show you that by replacing OpenBSD with Nix, you would generate something that would work in Nix. That was fantastic, but you won't get to see it. About modularity, what's really interesting is that the modules are distributed in their own git repositories. Why is it great? It means you do not have to ask upstream, that is me, to include your work if you want to create a new language that has its own public archive or if you're creating a new distro. You can just write your own stuff, use the technology you want, as long as it's Python, but you can use the test runners you want, you can use the coding style you want, you can use type hints, you can use Python too if you're crazy. But you have a bit more freedom with regards to technical choices and you can run your own project. And the idea also is that if you're a devian developer, you do not have to agree with Fedora developers and it's the same if you run a pipeline or Ruby chance or anything. So it means you can collaborate with other people without actually having to agree with them, which is probably great because humans are generally not good. I mean, we've been waging wars for thousands of years for stupid reasons. I've given up hopes. So it's easy for small projects, you can be independent. Who sought this guide? So you'll remember me because I'm the guide and I have no slides. But hey, on top of having no slides, this guy made an excellent point. We should ditch all the crazy tools we had and use his tool. Who's like, hey, oh fuck. So, have there been anyone? Thank you. There's a whole interest. Do you know Kuki Cutter? It's a tool that allows you to create skeletons for projects. So there's a Kuki Cutter available that is either a front end or a back end. So you just fill in a few... You have to answer a few questions and create all the codes for it to go. You just have to fill in the parts that seem interesting. So you can do that. I would have talked to you about a different tool that's called FPM, but, you know, it's not so good. You can look it up. Basically it generates binary packages rather than source packages. So you can't use it to submit batches to your distributions. And that's why I don't really like it. Yeah, so future work and then we'll have questions. Something that would really be great if it's people from IPI and RubyGent and Cpan, et cetera. I mean, they all provide the same thing. They provide the JSON file with the name of the package, the version, the name of the author. But in a different format every time which means you have to pass... you have to write a different JSON but basically another full JSON file to metadata files. It would be great if we could all agree on a single format for metadata and have this shared amongst all public archives. Which means if you create a new language, a new archive, you just use the same format and you'd be compatible with magic tools such as the one I've written. I'm not sure it's going to happen though. So if... do you have any people who run IPI, Cpan or stuff in the room? Yeah, so that's not going to happen. That would be great though. I mean, if you've got no questions, you can close your eyes and dream for a few minutes that people have questions. No. If, additionally, all these tools agree on the same package in your tools, that's useless. So I have two users other than me. Yep. I think it's much better without the slides. I've seen some of the slides. You know, you explained it so well. Okay, so I have to repeat this for the... for the... This guy said, can you please stay for another two hours? I would like... I'd love to have lunch with you. Can you come back next year, be on the main track? Yeah, you wanted to ask a serious question about too late. What is your current audience? Debian maintainers, Debian developers that are all the people who have to package stuff and who... I mean, let's be real. When you go to a web page and you manually copy-paste the version of the package, the name of the author, you're not playing with your kids, you're not playing the guitar, you're not doing anything interesting with your life. The interesting part of that packaging is the results and maybe doing some debugging at some point. And so the target audience is also people who use the tools they would have shown you that already exist because I've tried some of them and I'm pretty sure that the... some of them... none of them will give you all the info you need. The thing that's very important is that if someone sends me a path for upt-pyfi, so the tool that passes pyfi, the improvement will be available for all distributions. So that's a way of sharing all this code. And currently, well, you have to update your own tool and Fedora developers that have to update their own. And even if you fix a bug in the generation of the Fedora package, you'd have to put that change in many different tools. Here, by not duplicating code, we make sure that nobody loses too much time. Yep? Yeah, so, I'm maintaining a generator and I do agree with you that it's a great thing for example to be able to share metadata and also the multiple target responses are quite similar. For example, for MPM, I encounter some many weird issues like circular dependencies. They don't translate well to what conventional package management distributions do. And also, for example, how version ranges are resolved. That's also weird. Sometimes in MPM, a version range can be sold to a existing version, but necessarily the latest version. You also thought about dealing with such corner cases. So the question is basically someone is maintaining a similar tool and says, oh, I have issues with MPM because there are circular dependencies and weird stuff, basically. So the thing is, if it's hard, it's going to be hard for all the tools that have to deal with MPM. So the idea is that we do the hard stuff only once. I do not know exactly how it works in MPM because I didn't write an MPM back and I'm allergic to JavaScript. It doesn't know from my doctor. You know you have to know at the North conference when people like you are JavaScript sucks, that's all the rage nowadays in computer programming. I don't know exactly, but I think if you took the time to figure out some of these issues, it would be great to be able to share a code and if it's available to other people because I think you do not hate anyone enough to wish for them to spend time solving the same issues as you on MPM. Yeah, it's quite dirty. He had a few ideas of people he hates. Give me the back. I'm curious what needs to be done to add another new backhand. And the second question is, what would be the idea? How would you imagine updating stuff? Like, even if you don't have a generic format, the distributionist who wants to add the data, who's maintaining that kind of feature, what specific dependence means that you haven't known in the generic format? Like, our biggest problem is we have something like 1,200 work packages. And then, when somebody didn't maintain them for a while, there are 200 of them updated. And we would really like to, let's say, optimize that stack. What distribution are you talking about? Microphones? OK. Yeah, so the question is, there are multiple questions. First one is, how hard it is to add a module, basically? Do you have a laptop in a couple hours? Because we can do that in the corridor, people. That would be awesome. OK. Anyone who also has a hackathon with me, and this. So there's a particular project. And there's, I think, good documentation, as I'm showing you, I think. And you can, so you can create this kind of project. And then, yeah, I kind of doubt why to you, because the naming conventions for packages differ in distributions, depending on whether you use a Python package or a Ruby package. So some of the info that's passed the back ends is, hey, this package comes from Python. And you have to do a little bit of glue to say, yeah, OK, the naming convention will be this. I cannot show you the code right now, but in OpenBSD, basically, I need a few classes. Like, I have a Python class, and it's just a few methods. And most of the work is done in a generic class. So I think we can quickly reduce the duplication, even though it's not magical. I think there is no time for any more questions, but I'm available. If you do not want to see the next talk, if that's lost, what is it about? Is it really interesting? Wouldn't you want to see the next speaker hate me?