 Good morning, everyone. Welcome to this talk about Debian infrastructure. Andi Bart has been with the project for quite some time. He started his work in QA and very quickly moved on to release work, and that was in the Sarge and Woody timeframe. And after concluding his work on release, he sort of retreated back into infrastructure, and that's where he's been active ever since. And in his talk, he's going to be talking about sort of connecting the dots, the individual bits and pieces of Debian infrastructure have been introduced in various talks in the past, and this one is meant to trace the path of a newly created package through the archive system. So thank you for your for the start of the opening of this talk. As I'm just telling or speaking about what does happen with a package. I mean, you upload a package and it's available from the middle, but something happens in between. We've had a lot of talks about now what exactly does FTP master scripts do, what does the release team's things do? There's a build the one build both every time, but also what housing is that glued together is something we don't speak so often about. When I talked about that yesterday with nieces and well, just a lot of pisces and pearl, which of course is true in some kind, but we can take a deeper look. So it's really packages gets uploaded there somehow accepted installed auto build a push to mirrors, visible on packages in QA, Debian org, migrated testing, sometimes hope for not to offer a box needs to be filed and finally released as stable. So now, okay, I see it looks like the last thing has an issue. Let's check if I can. I don't get I don't get get that fixed. It seems so I will speak anyways, because I know what what should be on the slides slides will publish later on. Then these days, more and more uploads gets more and more packages uploaded via SSH to SSH upload Debian org. So really it's the SCP. If you upload it to upload, then it's forwarded from there to FTP master. On FTP master, it's forwarded from a script called QDemon from the upload directory to Q unchecked. Very, very traditionally, you could just upload directly into FTP master that was three 2003 until it was restricted. Then, of course, what also works, which also quite a few people still do is anonymous FTP to FTP master. Then you start directly on the line three. It's QDemon will take your package upload it to Q uncheck with FTP master scripts will get up also from for uploading. There was a time when intercontinental internet was really slow. I don't think that all of us could remember that, but at that time, we had multiple upload posts, just a different continent so you can just upload locally on your continent and get it forwarded on the slowing later on. So then, yeah, package gets installed. So your package is uploaded, then it's moved from Q unchecked to or it used to be moved from Q unchecked to Q accepted. At that point, you get this mail, your package is accepted. And once per day, it's installed as the installer used to be once per day. As I said, I'm looking a bit on what used to be, what is now. We had an incoming that was an ugly hack for build demons. They could build on seven experimental for it, but yeah, better don't ask how it was, how this code looked like. And the best thing of the setup used to be, you could just change something in between on the archive during the day, and then yeah, your dean store would fail because it changed between the time you upload your package and the dean store runs. We had an enhancement multiple dean stores per day that was a great progress for those of us who can remember that time was only one dean store. Now, if a package is installed, it should end up in the pool director. So if you look in the FTP master master's mirror or any FTP mirror, it's this director, it's a pool director, the packages end up in the pool directories. Now, the package install, it now means since not too long time, the install is fixed. So if a package gets accepted, it just directly pushed into the pool, which is of course much better than it used to be before, because you don't have this thing, which is like, which makes it easy to break things. And also, any package is now available from incoming debit org. So this is now something you can just use. In previous times, it wasn't, yeah, it was basically an queue where I directly, the packages were copied into and it's in a custom app FTP R trifles run over it. Now it's just in, let's say, a certain view on the FTP master's database. So if we say, if we now say database, yes, what do we have an FTP master? There's a PostgreSQL database in the back, the packages are in, and then there are just links from where they are installed to the packages. So you don't have the package multiple times. You have the package once, and if it's in two threads and three threads, so it's unstable and in testing at the same time, it's just one package linked multiple times. And if the package is not used for some time, like one and a half day, it gets deleted from the pool, which also means if you accidentally delete a package from unstable, you could just put it back because it's not deleted yet. It's just a link missing. So, yeah, but not always packages get installed. Sometimes they get not installed for some reasons. Of course, it could be a new package and it ends up in Q and U. So the directors are really named like that. They are named Q and U, Q is ejected, Q unchecked, Q, whatever. So packages are sometimes ejected because, for example, they are an older version for assembly having currently in the suite, so they are ejected, Q is ejected. Sometimes packages even don't get that far because the QDemon, this is the thing which picks up the package from the FTP upload and puts it in the FTP master directory, says, well, yes, this doesn't have any GPG signature. We can't install it anyway. We just delete the changes file. So that is when you even don't get the mail, usually if you upload a package, you get first a mail handled by the upload queue and then a mail is now installed. So the upload queue mail is a QDemon and it's installed as FTP master scripts. And yeah, if QDemon says, this doesn't look like a legitimate changes file, you don't get the first mail and nothing happens from FTP master scripts point of view. Also, the QDemon is a nice thing which can, where you can send the decode, cut commandos commands to delete wrong uploads. So also, that works via the QDemon. And somewhere in the Q unchecked and FTP upload queue, there is a delayed queue so that you could upload a package saying, please install this within 14 days, which also, by the way, also started as a hack because in previous times, you could just upload your files to FTP master, set up a ground end to saying, move the files in the directory. This was no longer possible after it was restricted. So we needed some queues to do it now in a better way. And yes, we have policy queues. So if you upload for stable or for stable post updates, you don't get into the queue accepted. You have put in a special holding queue, whereas on the release team can say, yes, this package is approved. So, yeah. So packages are installed. So what happens if a package is installed? Of course, you get a mail saying, yes, your package has been accepted. Then the bugs, as there are mails going out to the bugs that you mentioned in the changes file. So mails to bugs, mails to maintainers, to mails to packages, QA, dev and org, where you could also subscribe to mails. So if you're interested in a foreign package, you can just subscribe to say, please send me all the mails that go to the package maintainer and you get them as well. Then if you upload a package to unstable or experimental, this is a build from queue accepted, which then is used to be built from queue accepted and of incoming, dev and org, which means it's as a fast auto build suite. So these are the suites which are just built as soon as you upload it. Then now packages are pushed into a special incoming suite. So if you go on FTP master or on Quartionella and say duck LS, some newly uploaded package, you see it's not just an unstable, you see also it's an unstable accepted so that you can just get it from incoming dev and org. Incoming now, these days, not on FTP master anymore, it's load balanced via the static dev and org network to multiple hosts. So it's possible that you could take packages from servers out overloading FTP master, which was one of the reasons why incoming was restricted earlier to only very few users, which the build demons are. Then FTP master database is now right ahead locked to mirror FTP master. Even if you don't have access to the FTP master host, you can just look up in the database, which is just the same as a live database except that you can't write. But you can't write anyways. I mean, I can't write ISO, so yeah, only FTP master script. And yeah, what also happens? There are some special files created, for example, for the list scripts, for the bug scripts, so like writing out urgency. So there are a lot of tiny things that are happening. If you want to really know the details, a copy of the scripts is available. And at least I forgot so much every time, so if I want to know it, I always look up the source code, which means the shell scripts. So yeah, now our packages are installed in FTP master, and now they need to be available to the world. So what happens? Yeah, the installs, which runs no more than one time per day, triggers a mirror push. Mirror push means it is a shell script with SSH to the mirrors and starts a false commenter. And this false commenter then errsinks the mirror in a special way to hopefully not break it too much while doing the errsink. And then if the errsink is fully done, it starts subsequent mirrors. So FTP master starts the tier one mirrors, the tier one mirrors starts the tier two mirrors, and so it goes down. If you have ever looked into the FTP mirror, there's not only the disks in the pool director, but also FTP directorly, which contains an FTP blowcheck trace, the timestamps. So how it's a level from FTP master to your mirrors that you're using. These timestamps are also written by the push scripts. Some of you have some very, very special mirrors. That's packages that be an org, which is not a mirror, of course, but which is just triggered. One build is triggered, bugs master is triggered, so because bugs wants to know which new versions have appeared. If you looked on a bug report, you could see bugs are not just opened and closed, but bugs are reported for a version. They are closed with a version. So bugs knows now, oh yes, this version is a child of that version. So if a bug relative to a version could be, the bug is not related to the version, or it's reported for the version, or it's fixed for the version. And to know that, of course, bugs master wants to know what's happening, and as people want to know it soon, so bugs master is also triggered by a summary. Oh yeah, as it pushes sometimes, or more and more often, they are not directly starting the shell script from the SSHL-stricted comments, but SSHL-stricted comment is just touching a file, and the Gronchop is later on picking it up, which has advantages in the sense that the network is not involved anymore in its execution, but just in triggering its execution. So yeah, if you have looked at these things running in live, which as Vanderbilt maintainer, I had done before, sometimes, oh yeah, now it's triggered, but it doesn't run, yeah, of course not. It's waiting for the Gronchop to run. So you need to know that, otherwise you might get crazy what strange things are happening there. And of course infrastructure is very well, let's say, has lots of experience built into. So yeah, now packages and suites, as you might have noticed, I quite often use the word suite, not distribution, so usually if you speak about unstable or testing, a lot of people say distribution, scripts usually prefer the word suite. So I say suite. So we have stable, testing, unstable, we have suite names, we have code names, stable versus chassis. And the great thing is we have aliases, so chassis is the same as stable. These aliases are in, I know, offhand, directly a couple of files and databases versus is coded in, which is always a great fun at release time, because then you need, so basically if we do a release, what does it mean? It means we have lots of place where you can put files to say, now don't execute the script at all, ignore triggers, whatever, just don't do it. And then so we just touch the stop files, fix different aliases, for example create new database links, then remove the stop files and then hope that not too much breaks. It's, however, that now doesn't look so, doesn't sound so good, however, it's being, or that's now better from time to time. So it used to be way worse when I started doing QA work, basically we had to shut down the auto builder network for, let's say, about a week because we had hard code, it's the suite names in what is built, so the auto builder believed it's building for stable, so you need to, the names are changed to all the build demons, only after that had been done you could enable the build demons for that architecture again. Now it's way better because the build demons now know that it's building for chassis, so if chassis changes from testing to stable or one time from stable to all stable, it just will continue that way, it just needs to fix up the alias at some time, so yeah, it's getting to be better but it's not, there's still some work to be done. Then the last, what also do we have with suites? We have different kinds of how suites do interact, we have overlay suites like simple post update suites, so you have a stable package and you could just do a binary NMU from stable to proposed updates or from testing to testing proposed updates. We don't do that with stable, very often we do that with testing proposed updates if things start to get wrong and then we have suites with secondaries, so for example unstable has the secondary and experimental, so you build experimental packages from unstable, however you can't, you don't be NMU from unstable to experimental or same from stable to back ports, so there's two different things how the suites are interacted, in FTP master sites not so relevant, that's more relevant distinction on one build site because it's a different way how to mark this package as available because also, of course, one build wants to know what is available to not waste time trying to install packages which you can't install anyways, so now as I said, packages building, one build, I said before some suites like unstable are just fast auto build, which means on every queue a chron unchecked run, say you're done, by the way the chron runs for queue unchecked are done every 15 minutes normally, except when the install is done so that's very often, so fast suites are also built and I think last time I took stats, but now might have changed, we needed half of the time between the two chron runs to update the one build database, so there is some time left for growth, but not too much, so on D install packages one build does some housekeeping tasks, it starts auto build for non-fast auto build suites, however this auto build for non-fast auto build suites on D install really means if you upload a package to one of these suites, it takes at least two D installs till you could have all binaries in your installed, even if the build demands are very fast, because of course if you only do it on D install, yeah, but it does a bit of statistics so you could see how the packages keep up and also we do some kind of filtering for non-free, we used to not build non-free at all, now we build non-free however only packages are non-free which where they maintain explicitly says this could be auto build legally and safe and they mark some header and also mark some header in the packages source package field and so one build says yes I'm filtering those and only taking those packages into consideration which could be safely auto build and all packages are only built against dependencies from main, so if you are packaging non-free or in a concept it still will only use the dependencies from main, so yeah that's the way it's currently set up most due to legal reasons because nobody checks that you could safely use the packages from non-free or concept for auto building but you could use them for main yeah then we go to Buxmaster the Bux really backs the package version matches so unrelated bug you fixed however usually if you go if you go on any bug and say yeah show me this bug you see this you could see the graph and you say yes unstable is affected or unstable is not affected so that's happening indirectly because it says it knows okay from this package these versions are currently unstable it says okay and is at least one of these versions affected an unstable is affected so yeah that's the interaction which is usually which is displayed but internally it's not all are just package versions nothing more and also Buxmaster generates Bux files for the least.debian.org which means when the list gets one to run they want to know which Bux packages are more as buggy as they used to be they take files from from Buxmaster which are specially prepared yeah so now the list.debian.org this generates its packages file from testing unstable and from testing proposed updates it sits in package urgency so if you upload a package with an urgency high then it it notes that until your package has been migrated it keeps all further uploads with the same urgency so say that in a stored it gets bugs in from Buxmaster it updates testing as appropriate as I said only FTP master scripts could write to testing or it could write to the database so there's another script which so which release team could call so they could call a script and say sync this file into the database and then this file is thinking as testing to the database so yeah lots of shell script lots of things to to to get clear interfaces and which which works and yes stable releases and also points that these are somehow special as I said before we touch a lot of files make sure that nothing breaks CD needs to be built but both of them has to do a lot of this with with with holding things by hand however also both things are getting better and better at the time so it's now way better than it used to be and I think the strength or I hope the strength will continue now as it was a quick walk through the infrastructure now have a bit time left for your questions I said you don't need just your your cards so yeah please okay same as always first question is the hardest the workflow have you tried to to put it in a graph is so I I doubt I doubt that it's possible to when trying to or when preparing this talk I have done a small graphics of of the workflow for myself and yeah it was barely readable because it's so there's so many intersections here this happens so there are graphics for what happens in fdp master there are graphics for what happens on build demon was so so for different parts I doubt there's any if someone wants to do it I'm happy to help or to answer questions or whatever but let's say putting things readable in in graphics is not one of my course things thank you the phone is no I know yeah last time I looked at the infrastructure lots of components were sending emails to each other which were then parsed at the recipient and yeah then several actions were triggered by by this for example in case the package is uploaded it sends an email to the back tracker and things like that is there are there any plans to change that and to for example use the fed message system which was developed at fedora to create one place where components interact with each other which is not email I will start with why do components that email to each other because it works so I mean yeah we are changing things so for example currently let's put an example where it's a bit more accurate with an email for example currently the build logs of packages are sent by email whereas the packages are uploaded by ftp and now now all has to do that way now all is is about just to fix it to make it to make it better and also it would be sensible to just upload all the files together and not say we take logs difference and we do other builds results because a log file is also just a build result in fact so yeah I think there are things where we want to go away from email but for example from the bugs I mean the Debian bug tracker is so the main interface of the Debian bug tracker system is email so why not just close bugs per email for other things it might be more appropriate as I said the log file mails I'm not so convinced also at least for from I also get sometimes log file emails per email and so yeah I trusted my mail set up because yeah and for example and see a package log is not so small so special care is to be taken of all involved email demons so yeah otherwise I don't think there's a great master plan of what needs to be changed it has really it depends a lot on the people who are doing the work I mean if you notice things are ugly breaking taking pain making our life life difficult you want to change it if it just works oh great it works we could do we could fix something else okay next one okay seems to be the first talk people are still shy normal questions anyone hello does the system use external tools for example other software like revitem q for example for the q system pardon of course a bit this is the last part um does the old system use external tools that are currently packaged by indivian for example for the q system does it use revitem q for example um well let put it that way I mean this is now centered about packages infrastructure and this one doesn't use it yet other parts have been using it so if you now look at the other side of the package and I just spoke for example about auto billers so I just touched one a bit because how the package how the build team is pick a package of one a build is something else but the but the dsa uses uses some some other messaging system so to signal for example or you have to think up to the account database so yes of course so this was you're only looking at one side of the infrastructure there's another side so if you would if you would if you would now continue with how to build them as interact they're also using lots of ssh but we are going slowly in direction of some other messaging system but I think there are here people present who know better than I do what these a is currently doing so yeah then seems like there are no more questions finally or still not questions or in which in which case in which case I would say thank you thank you for being here and yeah if there are any more questions during the next days please feel free to ask me thanks