 So, hello everyone, my name is Alex Salm, and today we'll discuss about Autodeb, which is a new Debian service or project that tries to produce automatic packages for just about everything. So, first, I'll quickly present the project and the goals, then I'll go over the current state of the project, move on with things that have improved, that we've been contributing other than directly Autodeb, and then we'll go over the roadmap. So, the process of updating a package can be quite simple and repetitive. It's a matter of downloading the new version with use can, refreshing patches, or even removing patches, building the package and uploading. Most of these steps can be automated, and sometimes there's almost no other modifications to do. So, yeah, so in many cases, all there is to do is to update to a new upstream version and things will work fine. So, why don't we update packages faster? Well, at least in my case, it takes time, and one of the big issues is that things can go wrong even after the upload, so build failures on some architectures that I could not really have caught before uploading or breaking reverse dependencies. Technically, you should be able to detect most of these failures before you make the upload. You should be able to build on all architectures, and you should be able to at least test minimally your reverse dependencies. Some teams have tools to do that. There is like, rebuild all the things, which is a tool that allows you to rebuild all reverse dependencies with the DAB, and you can also run a auto-PKG test on all of your reverse dependencies before making an upload. Issues like this, they get caught in unstable, so it's fine. Your package won't migrate, but you'll still have to fix it, and you can slow down other people's work, because your package in unstable will block migration. So it's better if we can detect as much issues as possible, as early as possible, so before you upload to unstable. So the process of back-porting a package suffers some of the same downsides. So it's quite simple and repetitive. Sometimes when you back-port for stable, all you have to do is just build a package and it's done. For a good number of packages, there is absolutely nothing to do. So why don't we back-port more packages? Well, in my case also, it takes time. And then I don't know if there is demand for a back-port for some of my packages. Sometimes I assume there's not, so I just don't do it. I just don't spend the time doing it. And if we back-ported everything, we would end up with a copy of testing, which we don't want to do, right? The back-ports repository is a collection of packages that we know are working well together and that will not break your stable install. And one of the characteristics of the back-ports repository is that they are... So if you upload package A in back-ports and then you upload package B, package B is able to depend on A. So they all depend on each other. And if you back-ported so many more things, then you would pull a lot of things when people install stuff from back-ports. So we want to try to keep it as minimal and as useful as possible for our users. So is there room for improvement here? Well, package maintainers could definitely benefit from more automations. Some of the steps I've described before don't have to be performed by human. They could benefit from more confidence when uploading a new package. So maybe some of us would refrain less from updating a package if we knew it wasn't going to break reverse dependencies. Our users, they could benefit from just having more back-ports in the stable repository. I know I do. As a stable user, I really want more stuff in there. I end up sometimes with a Franken-debian install where I have stuff from unstable or from stable on my stable machine. And then our users could benefit from newer package versions, too. Whether that is in stable or unstable. So what does Ode-deb try to do? We generate and distribute back-ports and packages for new upstream versions automatically. So we have repository where we have a ton of back-ports and a ton of updated packages for unstable that were not built by humans. So they are not in the main archive. And we also want, there's a typo here, we also want to enable Ode-deb and package maintainers to test their packages and their reverse dependencies before every upload, just to give you more insight before you upload and detect more regressions. So I'm doing this in the context of a Google Summer of Code project with Luca as a mentor. Luca has actually been doing archive rebuilds for a while. So it fits into his expertise here. One of our goals also is to give, for now Luca has been doing this and opening bugs pretty much alone, even though some of his scripts are available, I think. But we would like to make it available for every Debian developer. So if you have a GCC upload you want to test, it should not require you asking someone. So we'll go over the current state of the project, what was done and what's coming. The project goal one was to generate and distribute backports and packages for new upstream versions automatically. So how do we do that? Well, we try, for backports, we try to backport any package in testing that has a newer version, that is newer than the one in stable. We find the candidates with the API at api.ftpmasters.debian.org. It's Debian Archive Kit's API. We currently only run job for fixed number of packages. What I mean by that is that we do not detect new candidates. Like there's a button I push and I write like, I ask for a thousand packages and it will try to backport a thousand. I'm still in the testing phase, but eventually what will happen is we'll just detect new candidates and build them as they appear. For your information, there is about 17,000 candidates right now that can be backported. But this particular stat can be a bit fallacy because it includes packages that have unsatisfiable build dependencies in stable. So it's pretty much less than that. Results are added to a repository that you can already access at auto.debian.net.repose.backports. We run Autopeak EJ test on all packages that we build. We consider them as a build failure rate if it fails. We only build on AMD64 for now, but that's definitely not the end goal here. We hope to build on all release architectures. For package upgrades, we attempt to update packages in unstable to a new upstream version whenever it's available. And we can know that because of the watch file. Candidates are found in the universal Debian database. So that's already available publicly. There's an API you can call and you'll see every package that needs updating. Again, we don't detect new candidates automatically. There's a button I have to push to build new packages. For information, there is almost 6,000 candidates. That's very little, I guess probably due to the fact that maybe there are some missing watch files for a good number of our packages, I don't know. Or maybe UDD is missing some things, I don't know. So right now, as of today, you can already install Autodeb backports or package upgrades by using our repositories. With the mix of app preferences, you can install only one package from our repose. It also can be used for Debian package maintainers. So if you have a package that we've already upgraded, you can just pull our package, test it, and then maybe just upload it yourself. So success rate. Based on 1,000 backports and 1,000 package upgrades, we have 45% of backports successful and 31% of package upgrades are also successful. There are reasons for failures that I'd like to go over with with you. So there are a large amount of failures due to the fact that we don't refresh patches yet. So this is a very large amount of failure I haven't counted. But just by opening logs, I think this is the number one error that I see all the time. It's a quick patch, and I'm sure we can improve this with some heuristics, like trying to drop the patch or just trying to refresh the patch. Like I said before, packages with unsatisfiable dependencies are included in the backport failures. So these take one second to build. They don't cost any time on our infrastructure. They take one second to fail, I mean, not to build. And then some failures are due to bugs in the order that infrastructure and code. Mostly for package upgrades, we have a lot of failure in my broken code. So yeah, we can expect significant improvement with little efforts on these stats. So I expect to reach way more than 50% of successful backports. And for package upgrades, it's a bit hard to tell. Now for testing, before every upload, so we were building this infrastructure that builds and tests a large number of packages. So we thought, why not make it available to Debian developers for other purposes? So that's the second goal of the project, is to enable Debian package maintainers to test their package and reverse dependencies before every upload. So how does that work? Well, Autodeb has an upload queue. And we would suggest that you upload there instead of uploading to FTP master. The way it works is it will build the, we only accept source uploads. We will build your package, run autopickage a test, and forward the upload to the FTP master queue if it succeeds. That's what works already. But in the future, we'll rebuild all reverse dependencies and run autopickage a test on all reverse dependencies. So depending on how you uploaded the package, you might be able to tell our queue, well, don't forward the upload if this fails or if this other thing fails. You can configure it when you upload. Like you can ask, don't run autopickage a test, just try to rebuild all reverse dependencies, or you can ask, run autopickage a test. So it's all configurable. So yeah, that's what works right now. So what that does is it provides you with easy access to through testing before every upload. It runs tests that are not often run due to lack of time and resources. The way it works right now is we have a HTTP queue and actually we make, when you upload a package to Autodeb, you have immediate feedback. Unlike the FTP master queue, for example, where you have to wait 10 minutes for an email, we will immediately respond with an upload ID that you can use via our REST API. So you're able to build tools on top of that queue. So if your team has some kind of workflow that would benefit from this, it's easy to build automated systems that use our API. So yeah, and when you make an upload to Autodeb, we will make some immediate checks and you'll get feedback from that. We've had some deep patches, I'll go over later. So for infrastructure, where does that run? This runs on Debian's Amazon Web Services account. It's already available at auto.debin.net. However, well, the thing is I'm still developing this a lot and I don't have like two environments. So I wipe it like every week or don't expect things there to remain where they are. And repost service are also already available at auto.debin.net.reposts. So yeah, we've been improving some other things in the Debian related projects. So for both Deput and Deput HNG, they will now display HTTP error message after failed uploads. So I think the next slide I have an example. So if you upload a package to Autodeb, you can immediately get feedback. So only source uploads are accepted or other kinds of checks. This kind of thing could be implemented in the standard FTP master queue. For example, they take like 10 minutes to send you an email saying that the checksums don't match. You could know that right away. I think the Deput patch was not merged yet but the Deput HNG patch is already in the archive. So the Debian archive kit has an HTTP API that was also improved while working on this. We have new features to search for files in the archive page under checksums and to browse source packages by control field values. Then I've also been improving Go's crypto library. The way Autodeb works right now is you log in with your Salsa account. In the future we will use the, I think there's a student working on JSOC for a replacement for the Debian S single sign-on. It uses the setup that we have right now does not work with OAuth2 but it will in the future. So you log in with your Salsa account, you add your PGP key and then we know that uploads are made by you. We will assign them to you. So we hit some bugs in Go's crypto libraries that were fixed. And now, so what's coming? Well, web interface. So right now the web interface, the autodebian.net is pretty poor. What will improve is we will have pages for every package so that Debian maintainers can refer to them and they will tell information like, here is the last build that we have of your package on this version and we will link to the package. We will provide build logs. Build logs are already available but they are hard to find for specific packages. And eventually I hope that some of these informations can be added to the tracker. So instead of just saying that there is a new upstream version available for your package, we could say there is a new version available for your package and we've built it. It fails and here's the log. So you can already know there are new dependencies and it produces a lot of information that you can use before even trying to update your package. So you can better prioritize your work like that. We will detect new candidates automatically instead of just me pushing the button and to run large jobs. We intend to build on all release architectures. It should be pretty easy considering the resources that we have available. We will rebuild reverse dependencies like I said earlier and run auto-pick-a-test. So we have a REST API like I mentioned before so that you can build tools on top of it. It's pretty undocumented right now so but that will quickly improve in the following weeks. Yeah, so code and bugs should go to Salsa. If you wanna contribute, the project is in GoLang. We also have a public infrastructure code written in Ansible on Salsa and I hang out in Otodedman OFTC. I was hoping to get some ideas or question from you guys. So any comments on the project and what you think we could do. Hello, I have a question. What is if you update to a new upstream version and you have some change lock entry, will it go to the version control system of the package? No, it won't but yeah, so we don't commit on your post-service, right? So we build the package. We will host the source package so you can then download the source package and see what we did so that that's all saved on our system. So there are already teams working on features like I think for the Go team we have something coming up for this that where we open pull requests automatically on Salsa. So for example, if there is a new package version available for a Go library, we will eventually have some tool that opens a pull request on Salsa asking you to merge. So you will have build logs. Eventually, I don't see why we could not do it with Autodeb so it would be very generic and it would work for all teams. We can easily detect that this package is hosted on Salsa and open a pull request. Yeah, but the danger I see is that maintainer just didn't notice that package is just automatic updates and just pulls his Git repository and uploads from this something else. Yeah, of course we would never commit in the master of your work or... Yeah, you will not commit in the master and then but if you don't interact with Git at all so no pull request and nothing, the maintainer will not notice. Yeah, yeah. And so it would be good if there is some signal, hey, there was an upload and you should respect it. I have another question because... But we will never upload to the main archive, right? Okay. That's for sure, I think. Yeah. But why not? Well, I don't know. But they are not verified. Yeah, but if upstream ads like malware in their code we won't even notice. Well, the thing is my next question would be in my GNU app, I showed the system where I can auto-update our packages, which is very trivial and very nice and in principle I could imagine that it goes without attention because the packages are tested on the C-RUN mirror. Yeah. They are working. Yeah. It says nothing then signing and uploading which could be automated in principle. Wonder what the Devin policy says about this. But just we are discussing here. Maybe this is something that can evolve in the future. Like the first step would be like having users test those packages. Then maybe the next step in a release or two would be to move on with something like that. The thing that is unique with our backports repository is that the package there they don't depend on anything. Like they are like one-off packages. They don't depend on each other. So they are very useful for users for this reason. I think it would not be sensible to do it with any package but if you say we have a wide list of packages this is possible. Yeah. I could imagine this. Yeah. I could too. Thank you for your book. Yeah. Good for Todd. So I have a question which is probably maybe somebody here in the audience is going to anticipate it. You're doing all this with source packages. Source packages are pretty nasty. Yeah. Why are you not using Git? What do you mean using Git in Webster? Why are you not fetching the source code from Debian with DeGit and then manipulating it in Git and then building it out of a Git tree? Yeah. So I have a very similar question. I use a Git workflow where it's going to be the same question so you can answer at the same time. Yeah. Of course. Go ahead. So I fetch the upstream tags, merge it into the Debian branch, and generate the table out of Git archive command which I wired in my Debian roles. So I would very much as well that you could do that and send me a pull request that would do the merge from the upstream tag, do the change log version change, and that's it. That would be awesome if you could do that. I think that the number one challenge for this is that everyone has different workflows. So if we use DeGit, then that would be the more standardized way to do it. But for example, your workflow, who says maybe people will have different workflows. The reason that I work with upstream with source packages right now is that because they are the only standard, right? They work for every single package in the archive. If I contribute that to your film, will you accept it? Yeah, for sure. Because that's going to lower a lot the amount of work I have. Yeah, yeah, of course you will eventually save work by contributing to automating your tasks. So yeah, maybe we can support a couple of repository layouts and have some way to detect what layout you're using. But surely Debian could benefit from more standardized Git layouts. Right, I mean, that was my question really is why you're not fetching with DeGit because that is standardized. Yeah, yeah. And then you don't have to deal with like will and patch and stuff. I agree. We also have to keep in mind that our patches don't bring a lot of value. I think that the more, the most of the value that we produce is the information, right? The fact that your package builds and the fact that you can already try it, like there's a dev you can install. But our pull request is simply going to be an update of your new upstream sources. And they are like, at least with my workflows, that's just one command to generate. Like import or use can, that's it. So I'm not sure how much energy we should put on making better pull requests like that because they don't really have value. But we couldn't imagine that, for example, some teams could contribute better heuristics to update their packages. So for example, if I'm the go team, I can write some exceptional ghost script that runs when we try to upgrade that tries to add new dependencies to the package and smarter things. As soon as we start doing smarter things like that, then our pull requests will have value. OK, thank you. Thanks a lot for this work. It seems very interesting, especially the part that would give pull requests in Salsa. I was also thinking the other way around, could it be possible that when someone sends a pull request for my package in Salsa that it would automatically then make a webhook call to the auto packaging and then run the test and say if it comes back OK or not? Yeah, and we could even, so you can already do this. Well, what I mean by that is that we have this rest API that you can use and you could write a script on GitLab CI that does that, that calls our service or whatever. So for sure, I should probably write an example of this and have it working for you guys. So basically, you won't have to put that depot yourself, but it would happen automatically without wrapping the thing into a WM package first. Sorry, can you repeat? So basically, so that you wouldn't need to depot that package manually, but it would come automatically from the Salsa. Yeah, we should definitely do that. And we would also host the, so for every package that we build, we create a one package repository so you can even have users install them quite simply if they are non-technical. So it makes, maybe this can also be used for non-technical users that try to contribute to Debian, like just pick up a pull request on Salsa, download the package from there, try it and give feedback to the maintainer. Yeah, I can see that working. Sounds awesome. Thanks. Any more questions, comments? I think we're done. Feel free to ask questions around if you see me in DevCon.