 So, hi everyone, I'm going to talk to you about migration to testing and visualization or how do I beat the freeze deadline that was basically my main issue. I didn't intend to work on release team tools, but in the end that's what I did. So I'm going to split this in several parts. The first quick one is just about context. Then I'm going to talk about migration to testing, which is like the core of this talk and how we can visualize what's happening and what's holding back some migration of packages to testing. And then some openings as usual. So, just a show of hands. Who maintains at least one package in Debian? Not quite everyone, but almost. And who keeps an eye on the status of the migration of this package to testing? Or the other way around who doesn't care? Okay, no one. Okay. And who already noticed that the package was blocked from migration because of another package? Okay, still people. And who had troubles understanding why and how many packages were included in this migration problem? Oh, still some people. The other ones, you can stay here. Maybe you'll learn some bits anyway. And I'm not going to talk a lot about my company, but basically I'm doing Debian work as a volunteer, but also as a consultant. And I'm mentioning this because when I'm doing systems administration, I tend to not do manual work, so I automate stuff. And you can do that with Ansible. I know Evolyx people already use it quite a lot. But there's also Puppet that I'm using for the centralization aspect that makes it possible to automate many tasks and keep track of everything happening on some infrastructure. But one of the tiny little issues I'm getting is how do I monitor possible issues with the infrastructure I deployed? And there are many ways to address this problem. You can use a full-blown solution like Foreman, which can be plugged with Puppet, but comes with the HCP server, Pixi server, and many other capabilities. So maybe it's a bit too much. You can switch to Puppet Enterprise, which has some web interface to look at reports and so on. But I don't want to pay some subscription just for that. So on the other end of the spectrum, you can just look at the YAML reports yourself, parse them, and then maybe send some notification. So I started with that. It's not really reasonable to ask people to look at reports themselves. And the middle ground is Puppet Board, which replaces Puppet Dashboard using PuppetDB. PuppetDB will collect many information about your Puppet setups, your nodes, and so on. And Puppet Board will just bid a web interface to PuppetDB. But my new problem is that PuppetDB was not in testing. So that's the starting point of my little journey with release stuff. So what can prevent the migration of a package from Unstable to testing? There are many possible reasons. Maybe it's not old enough. It hasn't spent enough time in Unstable. So maybe there are some underlying bugs that were not discovered yet. It can have new release critical bugs compared to the version in testing. It can come with dependencies that are not satisfied in testing. Maybe some build dependencies that are not satisfied in testing. And many more issues. Over time, we've put some effort into testing packages and not letting users only test packages. So we've got few parts that test installation, upgrade, and removal of packages on some automated fashion. And we've got some continuous integration with CI.dvn.net, which makes it possible to run some automated package tests. So auto-PKG test is something that you can put into your source packages so that they get tested automatically. If there are some regression, your package might be blocked. But if you've got some automated tests, maybe you've got some bonus like you migrate early because it was tested by some infrastructure bits. Anyway, some other possibilities, your package might be blocked because some release manager puts an explicit block on it. So that's basically what the freeze is. We block all packages. And it can be kept out of testing because the Debian Installer Release Manager decided that, oh, I'm working on a Debian Installer release. So I'm going to freeze all packages that might have some impact on the installer. So that's the blog-hew-deb hint, which prevents some more migration. Or it can be many, if not all, of the above. We've got some really not bad packages, but packages that are in a non-releasable state. So they are kept out of testing or maybe even removed from testing if they migrated before. And trying to understand which part is blocking might be a bit tricky. To understand what's happening, we've got the package tracker. So you can use tracker.debian.org slash pkg slash the name of your source package to get some information. But we've got the excuses page as well. That one lives under release.debian.org. It's related to Brittany, which is our tool which automates computations, what can be migrated, which package must go together with another one, and so on. So as we'll see in some screenshots right after that, you might have to follow some links to get more information because some more packages can be involved. So for example, here, relatively easy packages are relevant. As you can see, it's migrating after another one. So maybe that other one might need some other package in turn and so on. It's too young. It just got uploaded. So it has no opportunity to get widely tested. New parts is okay. So it can be installed, upgraded, and removed. But in the end, because it was taken a few days ago, it's blocked because we're in deep freeze anyway. But I'm going to concentrate on the first bit, the migrates after bits. Here we have basically the same information, but on the release team side. So that's the excuse page. As you can imagine, we basically copy paste from one to the other one. And a little less easy case is GitLab, which migrates after many GitLab-something package or Golang packages or Ruby packages. Plus it introduces some new bugs. So there are many reasons not to let it migrate right now. And that's another part of the excuse page for GitLab. So everything in blue is a link. So you can click and move around in the page and get more information about your package. But basically it's a bit cumbersome to keep track in your head that, oh, okay, there's this package. And also that one. Maybe I'm forgetting that one because the names are closed, but that's all. So basically what do we do? How does it scale when you've got too many packages involved? So that's where visualization comes into play. The first solution is pen and paper. I'm no artist. Really no. I can barely read back my own handwriting. And it might be a bit hard to start drawing something and then figure out that, oh, but I need some more place to put this and that in that package and so on. So it basically didn't work at all for me. The second solution is manual graphics, which is a piece of software which makes it possible to describe a graph like I want to draw a graph which is or isn't directed. So with like transition between two edges, the direction is important or not. And you just describe with a specific language, which is called dot, which is also the name of one of the graphics tool. And you run a command that will pass your graph source and generate some picture. So basically you just have to describe your graph and it's going to compute some layout that makes sense for this set of nodes and edges and so on. So that was basically the quickest way to get some information and maybe it can be automated. But let's look at the manual graphics part. The algorithm is really simple. You start with the package you're interested in. So in my case, that was PepeDB. You look at all the packages that are involved. So with the migrants after that we saw earlier, you register a relationship between PepeDB and package one, package two, package three and so on. And then you look at package one and then figure out whether there are some other package involved and you do that recursively. It's really easy except you need to be really focused and not miss any package and not look at the same package twice. Here's what it can look like. So we've got, I'm not going to look too deep into the dot language specification. But basically we've got a directed graph. I'm going to be organizing the layout from left to right. It can be top to bottom or some other stuff. And I want the nodes to have the shape of a box. Because by default that's some kind of ellipsis and it's not really rendering too good. The first step was PepeDB, migrants after this and that and that and that other package. Each link can be given a different color. So that was blue for all of them for unrelated reasons. And then once you've got that first part done, you look at du jour version check closure and then honey SQL closure and so on. But looking at du jour version check closure, oh, it depends on Pepe Labs, HTTP Client Closure, which in turn depends on this other package and that other package, which themselves depend on other packages. So it's really annoying to go and do some really manual work and make sure that you're not forgetting anything. So that was step two or at least part of step two. And when you're all done, I didn't count the number of steps, but there were way too many transitions. So 36 total. I was glad I didn't try to do that by hand because I wouldn't have enough paper anyway. So that's a set of packages that were preventing PepeDB from migrating to testing. So that was all the packages I needed to fix or to look at to get PepeDB possibly in shape before the freeze deadline. Because at some point we don't allow any new package in testing. So basically at this point I had a good idea which packages might be involved in this issue. So I checked release critical bugs, failure to build from source. So basically many closure packages had test failures that were depending on file ordering. So depending on the order in which tests were run, so-called unit test, were depending on some other unit test. And that could work or not work depending on file ordering on the file system. So it was a bit tricky, but basically the same class of bugs. So once you figure out what is happening, you just run them in some specific order which makes it work. And you fix many bugs. And you get some packages reviewed, uploaded, and stuff should be better. But then how do you check where we are at? Like half of the packages in the previous graph were fixed. Some of them might have migrated to testing. But then how do I check what the current situation is? Do I redo the steps before? Like go through all packages again, I don't want to do that. So my next step was, okay, we've got the ball rolling. We've got PuppetDB people, closure people involved. We've got some packages fixed. So I'm going to take a step back and try and automate what I did manually the first time. Once you do something manually, you can understand, especially when it's repetitive, you know exactly what steps must be done, in which order, when do you stop, and so on. So I looked back at some preliminary work I did years ago when I joined the release team in 2013, I guess. At the time, I was looking at the ex-USIS page, the HTML version, and I was passing HTML. So that's a recipe for not read disaster. But you've got to update because your scripts and RegExp and so on. Because the HTML version was not stable, you could have some strings that were edited slightly. So what worked back in the day was not working after a while. And basically that was a really nice idea to get graphs at the time to understand what was going on with packages. But in the long run, we don't want to do that anymore. So it was time to switch to machine-readable things. And fortunately, we've got machine-readable excuses. So instead of looking at HTML files, there's also ex-USIS.yaml file that you can, like, pass with real tools. And they are also saved and archived for later use. So you can look at exactly what happened on the very first Britnay run, on the first of February this year. But also you can look back into previous years and so on. So here it's just an excerpt on the PuppetDB entry back in early February. We see there's an entry with dependencies. So the migrants after with all the packages we've seen before. We've got an ex-USIS item which, oh, very much looks like what's in the HTML page. We see that this package is not a candidate because it's being blocked for whatever reason. It's not going to be considered for migration to testing. We've got some administrative, like, version and so on. And the policy info, we've got the many reasons why this package was not a candidate. The age requirement was okay. It's been in Ansible for more than 100 days, and it only needed five. So that part was really okay. The auto package test was failing, but it has always failed. So that's not a regression. So the verdict is passed. So that was not a blocker. And then we've got some other checks on block, I guess. Is there a block hint or something like that? And build depends. And some other stuff. I'm not going to look into all of them. I'm just going to note that there was a really critical bug anyway. Basically the package was not working at all. So it was not a good idea to try and push it into testing at this point anyway. So based on this YAML file, it's really easy to iterate over all the packages. Maybe limit yourself to one set of package that you want to look specifically into. Or to generate not one graph, but as many graphs as there are clusters. So sets of packages that are depending on each other. And then look at the big picture for... So the big picture is split into many graphs. But you've got a clear picture for all of the set of packages. So I'm going to switch to a browser. So that's a page that was generated with what's in the archive today. One might argue it doesn't really make sense because we're all frozen and so on. But looking back a few months ago, that was looking basically the same way. Because packages that were ready or getting ready on a regular basis were migrating to testing anyway. So it was basically looking the same. So we've got many packages involved and a really, really, really, really big picture. So I'm not going to look at that one specifically. But the idea of this page was to have some kind of a view for, for example, release team members to have a feeling what is going on, what is blocking, what are the big set of packages and so on. It was an easy way to search for a package and get the appropriate graph. I'm going to look at some other, some specific graphs that I picked earlier. The first one is, again, Avril. I said that was a really easy case. At first I thought there was only Erlang Koboy. But in turn there's also Erlang Ranch. So it's really like look-a-look-based or something. But still pretty easy to figure out. You could have clicked through two packages. Here we have an example of some specific topic, which is Android tools. You see that packages can depend on some other package that itself depend on the first one. You can have circular dependencies. On the left side I tried to inject some extra information, which is looking into not only source packages, which we have seen all the time, but also look at the binary packages. Because one of the reasons for packages not to be a candidate for migration is dependencies between binary packages. And sometimes you can have dependencies on package A or package B or package C. And you can also have provides. So a package is provided by another one. So I've tried to work on implementing some binary lookup. So that's why there are gray boxes on some graphs. That one is another specific ecosystem, which is Codi, previously known as XBMC, which is basically a TV station of some sort. You can see that some add-ons package depends on many, many, many packages, which themselves depend on a common package, which is some platform package. That other one outlines the link we can have between different tool chains. So here we have Octave, which is math-oriented, which depends on GCC. And we see some cross-toolchains packages, which also depends on GCC. That one is about Perl packages. So apparently, all of them depend in some way on some HTML parser. That one is for PHP. That one is for Ruby. That one is a bit larger, because apparently, some people don't really care about the release schedule in Debian and upload their package any way to Unstable, even if there is no way they are going to migrate to testing. We try to convince people not to do that. Apparently, we fail with some people. So R is a prime candidate for, hey, look, everyone depends on R-base, and everyone is blocked because of this specific dependency. But we're in freeze anyway, but still. You can have some really big graphs like that, but that one really is really, really, really small when you compare to Askel packages. We have many rebuilds and package name changes and so on. And it's not going to render in my Firefox. It's going to be eating all of them, and so I'm not going to complain about that anymore. That one is for updating your firmware on your machine. We see some signed packages, which depend on the source package. They were originating from a similar package is SHIM, which is the really small part that we try to use for secure boot before we switch to Grub. If you want to read more about signed stuff and so on, I've got a blog article detailing some bits. But basically, the same is happening with Linux, but we don't tend to block Linux for too long, so it already migrated at this point. And if you wonder why Kubernetes didn't make it into testing, that might be one or many of those packages that were not in shape, and maybe Kubernetes itself that was not in shape. But those are the parts we are missing in testing to get Kubernetes. But again, we're frozen right now, so it wouldn't migrate anyway. So getting back to my PuppetDB treasure hunt or something like that, that's a slightly different graph where I took some notes. I ended up editing my .file to mention release-critical bugs, like the depends on the left bottom side, or FTBF faces, so the failure to build from source, or security issues with CVE and so on. So it really would be better to have more information on the graph, not just the set of packages involved, but for each and every one of them get an idea whether their age is OK, whether they are RC bugs, whether they are dependency issues, and so on. But one of my issue right now is how do we get that into the graph, because there are already possibly many packages involved. And I would like very much to have some maybe input during questions or later in the day regarding some kind of image format that would let me have more information, have a bird's-eye view right now, and then make it possible to click on some item, or figure out some color code to figure out which package could be ready with just one day or two of waiting, or maybe with just one bug fix, or something like that. So I was anticipating a little. So more data on the graphs. I didn't mention it, but we could have the hints by release manager, for example, the unblock that we use during freeze time. We can change the age requirements as well. We can say, hey, this package is urgently needed in testing. So we don't care about its actual age. We want it right now. I mentioned block, UDEV, and so on. But basically, every hint we can give Brittany which can impact a package would be welcome on those graphs. And as I said, some way to make that clickable, zoomable, expandable, or foldable. And I need some input, because I don't know anything about visualization anyway. And everything is good, except those graphs are PNG or SVG. But I'm not sure how accessible an SVG with relationships is. So maybe it would be good to have some way of getting the dot language and then transform it into some text, be it in English, maybe localizable, and so on. So we don't need to actually see the graphs to understand what's going on. And on the infrastructure side, I would like to integrate that onto release.demia.org, also in the Brittany area. But also maybe on the tracker pages so that we can click on a given package and then get the appropriate graph generated for this specific package. And as I mentioned, I tried to work on source and binary mapping, but then you need the sources file from the archive to actually get the mapping. So I wanted, especially when looking back at some old exfuse.tml file, automate grabbing the relevant sources file directly from snapshot.demia.org so that we can have the appropriate mapping between source and binary packages at the time. So thank you so much for your attention. You can read more about some stuff I do with my company. And for example, I mentioned the sign packages, the secure boot stuff on my company's blog. And questions are welcome. We've got eight minutes. So I understand that you check build dependencies only. I understand that's enough, OK. So I understand that you check build dependencies only on AMD64. Is that right? The build dependencies for arched specific binaries. So the build dependencies and the build depends arched or checked on all the architectures where the binaries are available. And the build depends in depth are first checked on AMD64. If that doesn't work, they are checked on all the architectures with our binaries. And if that doesn't work on any other ones. So that should find a solution if there is one. OK. So that means if build dependencies are not satisfiable in testing on at least one architecture, then the package will not migrate. But that was implemented fairly recently. So at the beginning of the Buster release cycle, only the arched specific ones were checked. So there were some packages that migrated to testing that were not where the build dependencies were not available. So that problem should be solved now, but it wasn't a year ago. Thank you. Thanks for your great talk and your work in Debian. It could be also interesting to do back port packages to know if a package is in stable. Do you it is possible to have, for example, with a color box to know if a package is in stable? You mean when preparing some package for back ports? Yeah. I guess it's feasible. Basically, the idea is just looking at either excuses or maybe you can basically do the same with the archive itself and just look at dependencies and so on and establish some graph. It's really flexible and really dumb, actually. You might run into issues with version dependencies and provides and so on, but I guess you could just use apt to figure out the dependencies and then render that on some graph. I believe apt even has some dot command or had at some point. So it should be feasible. Just one short comment. You mentioned that you wanted to look up sources on the snapshot that Debian.org for to find out binary mapping, so binary mapping. Actually, what you really want to check is the binary package because sometimes you can have a binary package built by multiple source package and to find out which one, which source is really responsible for it, you have to look up the binary package. But I wanted to keep the talk short. I didn't dwell into the specifics of multi-sources and multi-packages, but yeah. OK. I had a question also. The graph includes all the relationships based on the migrates after. And blocked by. I didn't mention the distinction as well to keep it short. But does this list include all dependencies or only dependencies where there are different versions in testing and stable? It's all packages mentioned in the excuses. So then you would have to ask Evo. That means the graph might expand if other packages get to bloated later. If the excuses file lists either blocked by or migrates after, that means that it can only migrate if the other package migrates as well. If there's a dependency that is already satisfied in testing, then it won't be mentioned. So if it's mentioned in the excuses file, then it's really something that needs to be fixed for it to migrate. Two minutes still. We have just one, two minutes for the latest question. OK. Feel free to grab me for any more questions during the day or tomorrow morning. And again, thank you so much. Thank you, Cyril. Thank you very much.