 Okay, please welcome Michael Bebel, who will be giving us an update about system D during the last release cycle, and also what you can expect during the stretch release cycle. Okay, does this work? Okay, great. So, hello, my name is Michael, and isn't it a nice coincidence that the system D talk was scheduled after K3BC? So, yeah, I've been a package system demaintainer for quite a while now, and today I'd like to give you a short overview of what we did since last year, since our talk last year at DevConf, what topics we worked on, and I think it's mostly three issues we concentrated on. It's polishing the existing package and making sure packages we upload to unstable are always in a releasable state, but also we put focus on cutting our way with excesses dependencies and making our base so as smaller, because that's useful in a lot of cases. And also related to that is getting rid of legacy stuff, mostly related to system five in it. So, since the release of Chessie, we had four stable point releases, and those bugs that were fixed weren't always grave bugs, but smaller bugs and just stuff that annoyed people and documentation updates and stuff like that. So, we use user tags excessively and track bugs that way, and if you have any issues with system D in Chessie, please talk to us, please file bugs, we try to get them fixed in Chessie and keep that updated. For new releases in unstable, we started to work pretty closely with upstream because we wanted upstream has a really rapid release cycle and we wanted to be able to ship new releases pretty quickly and unstable. So, what upstream already has is a pretty elaborate unit test suite, but Martin Pitt worked on auto package, on the auto package test suite. So, we run that on every pull request that's actually hooked up upstream in the GitHub CI. So, basically, every commit that goes into upstream system D is tested on Debian or specifically on Ubuntu, on various architectures. That means we could actually release a package to unstable at any moment, and that's actually what's happened. When there is a new upstream release over the last year, we just shipped it more or less the other day, and it's worked out really well. I mean, we need close interaction with upstream for that to work, and we need to make sure to push up our patches upstream as much as possible, and we worked really hard on that and shrink our Debian-specific patches to a minimum, and so far, we are pretty happy with the workflow that is established out of that. So, trimming down the base OS, what does this mean? Sometimes it's better if you install less stuff, and one of that use cases is, for example, the init package itself. There are use cases where you don't want to have it installed. For example, the init package in a built-in environment is not strictly needed. So, we actually made the init package that was introduced in Chessie non-essential. This means you can app get, app remove init, and system dsv and system d along with it. Actually, that's not really possible because we use this flag important colon yes, which means once you installed a system, the init package, it can't be removed without that d-package warning. That's just a precaution, so users don't accidentally remove it. So, what you typically do is you run a reboot-strap variant build-d or min-base, and what you get is a truth environment without system d installed without the init meta package installed. It's just a really minimal build-the-environment, or truth environment. For that, we also not only had to drop the essential flag, we also made sure that the udef is also optional. So, we dropped the dependency on udef in system d because sometimes you have containers where you don't need udef. Docker containers are a good example for that. So, we still want to ensure, though, when you install the package, the system that udef is installed, so we use priority important for that, that makes sure that a standard installation has udef and init, but as I said, you can remove those packages. There was some fallout from that, for example, packages expecting has been run-leveled to be around, but it was really minor, it worked really well. We also split the package for various cases where we thought that the additional library dependencies weren't justified. We tried to keep splitting up the packages as minimal as possible because we do not like to split it up in 20 packages for every binary and have a coherent base system. But for those cases, system d container is mostly for system d and spawn and system d machine d, that pull in additional library dependencies. Journal remote is a special case where we thought it pulled in curl, curl lip curl, I think. So, we split it out as well. And we actually tried to coordinate the cross distro. At last year, system dconf, I talked to other distro maintainers and wanted to know if they're interested in sharing the same names. So, they actually did split out of the Fedora people, split the package in a similar way like we do in Debian, which is really nice. Something that just landed a few days ago is system d, lip system d shared. So far, the approach that system d used is was to link all the object file statically into all the binaries and use some compiler tricks to remove those symbols that weren't needed. Philippe Sattler worked closely with upstream and turned all this common code into a shared library, private shared library. It's not an official API, but that had a really nice effect. It cut the package size in half. So, the latest upstream, the latest release that landed in Debian a few days ago dropped the package size in half. And he worked upstream first on that. So, we didn't ship that in Debian, but worked upstream and once it was accepted, we pulled that into system d, which is really nice. And upstream was really helpful in that regard. So, what we have in, we also tried to get rid of init scripts in this VR scene. Because under system d, the init scripts that are shipped in the init scripts package aren't really used anymore. So, why should we install it? At the moment, it's pulled in because of its priority, but it is close to possible to remove init scripts. For that to work, we had to move that libinitvar.shell, shell library that is used by a huge amount of init scripts. We moved that to the CSV init.util package. We first considered moving into init system, init system helpers, which is an init system agnostic package, but then we decided it was too CSV init specific and kept it in that package. So, we are on the closing stretch of getting those removed. Those are the user tag bug reports. There are some of them left, but for a stretch, you should be able to just not have init scripts installed. The same is true for this VR scene, which in Jesse and previously shipped update RCD and invoke RCD, those we moved to init system helpers because they are really init system agnostic. There is support for open RC in them. There's support for system D in them. So, those move to init system helpers and it should be possible to remove those as well. At the moment, they're still installed. As said, they still have priority important, so are pulled in by the installer, but that's something we work on and there are still some corner cases we have to fix, but that's on our to-do list. Moving on, I'm not sure if you know what INSERV really is in our old CSV init-based init system. INSERV was a tool that generated all the dependencies between the init scripts and calculated the priorities, made sure that the one in script runs after the other and there are some system facilities defined by LSB which are due to a limitation in INSERV have to be specified via INSERV config files. So, what we did in Debian to support that, make sure that the ordering between the init scripts and the system D service units is correct. We shipped a generator, which amended the generated unit files or service files with drop-in snippets and pull in those targets because system facilities in the CSV init world are mapped to targets in the system D world and it's really a Debian specific patch and we really like to get rid of that and it's really simple to do it in your service file if you already shipped that. You just declare once before a target name and even if you ship a CSV init script in your package and you don't want to migrate to a native service file yet, you can still amend your CSV init script with a drop-in snippet. So, there are not a lot of packages left which need to be fixed. Those are tracked with that user tag if you want to take a look there. The biggest topic is the removal of RCS init scripts. Those are really tricky because the dependencies defined by those init scripts, RCS, maybe I have to quickly talk about what RCS.D is. RCS.D is Debian specific and means those init scripts are started during early boot and they are pretty prone to cause dependency loops under system D because the generator which runs for those init scripts doesn't know the specific dependencies those init scripts have. So, sometimes the dependencies are too strict and for example we have hooks in DHC client if up down which start RCS init scripts and those can lead to deadlocks. So, it's also Debian specific patch we would really like to get rid of and the best solution we have is just ship native service files for your Debian and in that service file you can declare very specifically which dependencies you need, which orderings you need and then this way we can get rid of the patch and we can get rid of all those dependency cycles that happen because of the RCS.D init scripts. Finally, I've taken a look at how well system D is accepted in Debian and what you can see here the yellow line is the number of init scripts we shipped in Weezy, Chassis, Stretch and Sit. The green line is the number of packages we have shipping init scripts so it's pretty close. I mean there's one system five init script per package and what you can see the blue line is the number of system D unit files or service files and the red line is the number of packages with service files. Typically what you have in system D or packages shipping service files that you ship one service file per Debian will not start multiple demons in one service file so that explains why there are that much more service file compared to the number of packages shipping them and what you can also see that the increase was pretty nice over the Chassis release cycle. We over doubled it and we already have 50% coverage of service file in Debian which is really great and we didn't expect that, that people are so eager to work on that so huge thanks to everyone. So I wanted to leave time for questions. So this is where we track our bugs. We use, as I said, we use user tags pretty extensively. If you want to have a look and help us fix those bugs look at either system D specific box or box and packages which are user tagged. If you want to get in touch with us we do have an IRC channel, an OFTC, it's called Debian minus system D. We also have a mailing list. So just come by and talk to us if you want to know more. I will also be around the next couple of days so just grab me in the hallway and if you have any questions specific to system D just talk to me. We are friendly people and if you actually fix a bug, come by. I didn't bring cookies but you get a free hack. So thanks everyone and I left time for a few minutes for questions if there are any. I think we should have five minutes, right? So I'd like to go back on the RCS problem. Yes. So about two years ago when I attempted to port open LC to Debian, I've hit the same problem which is that... Which package did you convert? Open RC. Okay, open RC. So I also found out that the RCS scripts have many dependency loops. So the way we did it was that we have some logic to break these dependency loops in some way or another but I had the strong feeling that the problem was the RCS scripts themselves. Wouldn't it be healthier for Debian if we just attempted to fix these rather than going around the problem? Well, what do we do? Because K3BSD port and her ports still have the issue. Okay, so what we do in system D is actually we don't want to work around those issues anymore but make the package ship native service files. So I'm not sure what you mean and what we should do differently. Do you want to abandon RCS in its scripts altogether? No, I'd like these dependency loops to be fixed in the RCS scripts so that non-Linux ports could still enjoy having a healthy RCS set of in its scripts rather than these that we currently have with dependency loops. Okay, so thing is system D in the early boot sometimes behaves differently. So some of the dependency loops you don't get with Sys5 in it or the Sys5 in it just doesn't care so it doesn't happen. And I'm not sure if we can easily detect those dependency loops with system 5 in it. So I'm not sure if we can, or how we can fix those. We have the code to detect them and they print when you start OpenRC and it says, here's the loop, I'm breaking it. If someone wants to fix the RCS Sys5 in its scripts, I mean all power to them. The solution to system D for system D is really to ship native service files and we can't really fix that with RCS, keeping the RCS D in its scripts. So we will push getting that fixed and it's actually looking pretty good. I mean we are on the closing stretch. Christian Hofstädler who is here worked really furiously the last couple of days so we get that for a stretch for system D at least. And if there's interest for OpenRC working on that, I mean, great, why not? Any other questions? It's network, no it works. Maybe I wasn't paying enough attention but did you mention system D network D? No, I didn't because there are no specific plans for stretch yet. We ship that within the system D package. It's not enabled by default, you can try it. You can use it if you want. It works nicely for different use cases but we don't have any immediate plans to make it, well replace if up down for example for stretch. I think it's too early for that. There's still some missing bits and pieces here and there. For example, there is no hook interface yet and option is not really so eager to add a hook interface so this needs to be worked out first before we can propose that. And are there any other things that are available in system D that should be in the same situation? I mean I do know that Ubuntu is pushing for system D resolve D which is a really nice technology but I'm not sure if we are there yet in Debian. I mean I will just keep looking how it works out in Ubuntu and maybe Ben for Buster we propose the same. Okay, I mean there's also system D boot but it's also optional at the moment. It's for UFI booting, not something we push for Debian 2 to make the default. Yeah, one question. Do you guys plan to update the version for the freeze? We are in 2.30 for some time now. The version is unstable? Yeah. Sure, I mean as soon as there's a new upstream release if you follow the Debian change lock we basically uploaded a new upstream release the other day. That's possible because we build it every day anyway and we test it every day anyway and I actually use a trunk build and it just works, it's really nice. So yeah, before the freeze we will try to get in the new version this. Well, we try to if there is something upstream and which breaks then we probably will defer a new upload but sure, we track new upstream releases pretty closely. More questions. Hi, is there any plan maybe by the security team or somewhere to go through and add some of the security flags to various services? So for example, you can add flags to remove use of home or add second one for, there's a whole bunch and over time it's beneficial to add them to various services. Yeah, that's a good point. Actually one of the benefits that system D provides. I don't know there is a coordinated effort at the moment because it really needs input from the individual maintainers. They know best what their service needs and how you can restrict those. But sure if someone would want to work on that coordinate that maybe put together a wiki page documenting that stuff that would be really awesome. So if that's something you're interested in, get in touch. Okay, yeah, so system D has a lot of flags where you can restrict the capabilities the service has. For example, you can use private temp then your service gets a temp directory which is not the real temp, but a private directory. There are lots of simple switches you can enable in your service file which restricts our service and makes it as likely that open box and in the service affect your system. Okay, questions? Oh yeah, we ran out of time. So we're out of time for this session. So thank you already, thank you. Thanks for coming.