 Okay, so I think we can start. Here we are with Michael and Martin. They will talk us about SystemD in Jesse and how to survive in stretch. So please welcome them warmly. So hello everyone. We are glad to be here. Thanks for coming. This is Micha Ebibel. I'm Martin Pitt and we are the current maintainers of SystemD in Debian and I also maintain it in Ubuntu. So we want to review the challenges that we had with the SystemD migration in Jesse and also take a look at some planned and potential changes for stretch and of course show you how to get involved. So big thanks to Tolle Foghain and Micha E. Stapelberg who did a great job of maintaining SystemD in the past. Unfortunately they don't have time anymore. Also thanks to Marco Dietrich, the former UDF maintainer who is still around for answering questions and giving some advice and also to several contributors like Didi Rosch who really helped us to crack some tough nuts like the FS check integration and to some new contributors like Philippe Sartela who are starting to get involved. But we always need more help. So this time is a rough timeline of bringing SystemD into Debian. So the discussion is about switching to a more modern internet systems really started already in 2007 with upstart at the time of course. SystemD wasn't around yet. And since then we've seen an ever increasing pressure from some upstreams and administrators to use features like LoginD or the Journal. And towards the end of 2013 the pressure finally got high enough to seriously propose a switch. And I hadn't actually been a Debian SystemD maintainer until then so I was following the big technical comedy discussion with great interest especially since I was involved in the boot stack and plumbing stack on the winter side and I was one of the SystemD proponents. So Debian fought with itself for many many months at the time. And we also remember that bad time when like essentially all technical development stopped and we went into these great flame wars and some external participants even proclaimed that this would be the end of all freedom and choice and we all were going to be enslaved and so on. But in this context curators to the technical comedy who despite of all this noise remained a really rational and fact-based research to this. And even though the final decision was very close as we all know it was very well founded. And I think in the end we reached the right decision. And fortunately most of these unfounded complains and runs in the Debian SystemD community wore off by then. And the political discussion essentially died and people realized hey life goes on and my computer still boots and you could all go back to our technical work. But then the real work only started for us because we did the switch only a few months before the Jesse release and this is like a huge transition. So there was work for us to make Jesse releaseable. And Michael will now explain us some details about that. So in the end it actually turned out to be not that big of a deal when the social aspects of the transition died down. So to actually do the transition we already had a package in Weezy which worked of SystemD in Weezy and that worked quite okay. We at a time the decision was made we had a package in Jesse which worked quite okay. But to do the actual switch of the default in its system meant we had to deal with making CSV in it actually replaceable. And that wasn't actually that easy because CSV in it is a so called essential package. That means you can you can just easily remove that package without getting this nice od you really want to remove that package and shoot yourself in the foot and note by the package or app. So what we decided to do was to split up CSV in it and make the old CSV package a transitional package and move the contents of the old package into CSV in the core. And but you really want to ensure that you always have an in its system installed. So we introduced a new meta package an essential meta package called in it and let that new package depend on CSV in it and which triggered the actual migration to to the new default in its system. So after we've done the switch, we actually thought, hey, people can now actually switch between multiple in its systems for the first time really in Debian. That also means maybe if you switch between says is five in it and system D, you want to keep the state between those systems in sync. So you have a service a on there is this is the in it you disable it, you later decide to switch to system D and you I at least I would expect that the service the service date would be transitioned over to the system. So we did quite a lot of effort and work and make it possible to for users to easily switch between those two systems. It also meant additional complexity for us because if you can't rely on the system CTL tools or always being around, you have to adjust your maintainer scripts, for example. And we reimplemented parts of the system, the interfaces in a package called in its system helpers. And if you use the system D, which we created along the way, then then you ensure that your package can can in can be installed under a system five in it and still enable all the services. So that this was one part, but it also one of the deciding factors of actually switching to system D was that console kid and essential component of today's desktop had been deprecated over the years and if you needed a replacement and system D lock in the really provided nice interfaces. And this was one of the reasons why upstreams like gnome decided to switch to lock in the and since we wanted to really support as is being it, we need a solution for that as well. So we introduced a package called system to which was first introduced by in a bunto. We use that in Debian as well, which provides just enough API from from system D to make a system D lock in the run under CSV in it. So you still need parts of system D, but you can run CSV in it even when using a desktop nowadays. So another aspect which we decided early on was if we do the switch from CSV in it to system D, we didn't want to support all the old conflict files as is being it used at C default RCS at C in a tub and stuff like that. The major reason for that was that we would have to carry a patch for that for all eternity. And more importantly, people coming from other districts, people who are used to system D would have a different experience on Debian, and we didn't want that. So for settings, which we thought were a sensible and feasible to do, we did a one time migration and move that over. And still there was other lots of integration work to do. For example, I mean, there's muscle memory, you type at the NAD service start, and we still wanted to do the right thing here. And to create the LSB hook for that. And I think we succeeded, at least I hope so that people actually don't realize that they're using system D actually this happened quite quite often that people are coming to us and say, Hey, I want to use system D and we tell them, Hey, you are using it. So I think we kind of succeeded that. Another aspect of some users aren't aware is that we actually did switch the default over also in upgrades. And that sounds scary at first. But we decided to do that to make new installation and upgraded installations as close as possible. And as this is a scary thing to do, we decided to keep a fallback. So the CSV init, the transitional CSV init package we introduced earlier was repurposed a little bit and it provides a fallback CSV init binary, which can easily be used. And you can select that in your grab boot prompt to boot that in case there would have been a problem on upgrades and you get your old CSV init back. So this shows a back graph of our system D package in Debian. And it starts around August 2014, when the actual switch of the default was made in SID. And as you may know, we released and there was a steady increase until the end of 2014. And as you may know, the actual release of Chessie was in April. And we actually expected there would be a flow of bug reports, but that actually didn't happen. And so we were actually kind of surprised by that. There were some hard bug reports to track down, but in the end, all those issues were fixable. And I regularly also try to follow the Debian user and Debian user German mailing lists and and see if users are having problems and trying to help them and try to understand what kind of problems they have. And so far it's, it has been pretty, well, I think after, after the flamers had died down, Debian use, for example, became usable again to watch. So the most important issue users, at least what I found users are having with system D is that system D is stricter in certain regards. For example, if you have obsolete FSTAP entries, system D opts for correctness instead of this is the unit which usually tries to hide errors. And margin has an example here where this is causing problems. Yeah, so this really helped us a lot to discover bugs, which were years old for my primary example, there is the eCrypt of a setup swap, which promises you to set up an encrypted swap so that if you encrypt your own directory, you wouldn't leak your data to the swap partition. And it turns out years ago, we made some errors there infect several. And this turned out that for years, we were actually installing systems which either didn't have any swap at all, or even worse, which had an unencrypted swap where you inspected an encrypted one. And since neither Upstart or Sys5 in it actually told you about this, we ran for this for years. And now system D will complain and say, hey, you try to configure a swap partition here, but it's not there. So it tells you and gives you the chance to correct that in the emergency system. But yeah, then looks for look, let's look forward to the future a little bit. One rather intrusive change that already landed in stretch is the change of the persistent network interface names. In the old system, we just remembered the original relatively random kernel name as it appeared the first time with a particular MAC address and then created dynamic UDF rules which would assign it the same name on all subsequent plaques or reboots. But there were several problems with that. It was inherently racy. You would sometimes ended up with interface names called rename one. And it was disabled for all the virtual machine interfaces and hence but also in the cloud. And it didn't really work with things like read only rude. And if you wanted to do wanted to create an image, but you then want to roll out to like thousands of identical machines, and then you got unpredictable names again, because the pre-generated rules would of course not match any of those machines except one. So in the new system, we never use the kernel names to avoid this kind of race. And we now have a declarative and rather flexible naming policy with, I think, sensible defaults. And you see how they look like now. And of course it will take some time to get used of those, get used to these. But we survive changing FSTAP to your UIDs and similar things in the past. So it's just a transitional pain. And I believe this is finally a better default. And but of course, if you don't like it, you can tweak the policy or just entirely disable it to just retain the original kernel names. An important thing is that this will not be done for upgrades because the interface names might be hidden in like firewall configuration scripts and so on. There is no way we can transition automatically. So this only applies to new installs. Another new thing is network deep. This is a small and lean service to bring up network interfaces, which was designed from the ground up for a hot, pluggy and virtual server world. So it's configuration is similar in the abstraction level and spirit to IFR down. So you won't have too much trouble with that. But you don't need any extra packages or huge shell scripts to set up bridges and VLANs and bonds. And unlike IFR down, it works perfectly well without plugging. And it specifically is not meant to be a network manager replacement. So a network manager will continue to be around on desktops, particularly it has excellent support for Wi-Fi and dynamic reconfiguration through debas and that stuff, which network D doesn't aim to. You can use it right now in Debian, perfectly well working, but there is no integration with other Debian packages so far. In particular, it does not call IFR.D script and there is no integration with packages like resolve.conf. These two are things which we really want to work on to make the transition easier. So another ongoing effort is the removal of the RCSD in its scripts. These are the ones which are run through early boot. And these are also the ones which cause the most issues because usually those are the ones which generate dependency cycles. So there's a new contributor called Philippe Satter who got interested and we guided him in working on that. And he worked on a lintian check and should you watch out for that lintian check? Should your package be flagged by that? Expect and bug very soon because we intend to do a mass bug filing soonishly. And yeah, some things we might do, some things we might not do for a stretch is KDBus. And KDBus is for you who didn't hear about that yet. KDBus is a Linux kernel implementation of the D-Bus IPC protocol which is general purpose protocol coming from the desktop and there were several attempts over the years to bring that to the kernel. In our view, the real core benefits of having D-Bus in the kernel is having it available during the whole lifetime of the system during early boot and late shutdown. So far, KDBus hasn't been merged yet in the upstream kernel. It might be proposed for for .3. And I would say it has good chances to be merged and we'll see how that goes. At least coming to the user space part, you also need a component which sets up KDBus. Currently, SystemD is the only user space implementation for that. And actually SystemD in testing and SID is already able to set up KDBus. And KDBus is also available as an out-of-tree module. So you could easily build KDBus, test it on your system and it's easily actually so easy to do that we decided to create a DKMS module prepared today on uploaded it. And this laptop actually is running KDBus today. There will be some issues, but if you want to try it, I think we're going to upload it to experimental. Just install the DKMS package and test it and report bugs. So two other interesting issues which we don't have to time to go into our presets, which could be interested for derivatives who don't want a different set of packages to be installed and SystemD boot, which is a very simple boot loader for AFI. And then there is further changes which we, the two of us, don't actually plan working on, but which would nevertheless be interesting projects if someone else is interested and please come and talk to us. One of these would be using SystemD in the INIT-RAMFS. Right now INIT-RAMFS tool uses a kind of a home ground shell-based INIT system and it could be replaced by a SystemD INIT which would ease the handover between INIT-RAMFS and the real operating system and also avoid some race conditions. And you also get nice things like you have the entire boot in the journal and it's much easier to debug INIT-RAMFS and stuff. This could also be influenced by the decision whether we actually keep INIT-RAMFS tools or move to DRACUT. There was a bar for about that on Monday, but at this point in time it's not quite clear yet where we go. And the next thing is the depth helper integration which pretty much affects all the package maintainers who ship unit files. You will be very familiar with the usual trinity of DH-SystemD-Enable and DH-Install-Init and then DH-SystemD-Start which feels a bit clumsy because this was kind of bolted onto the side at the time. But now that SystemD is the primary INIT system I think it's really time to unify all that and simplify it again. Okay, and so thanks for the warning so I'll skip some bits here and finally with some something else which all of which affects all of us is to actually make use of all the shiny features that SystemD offers to us. So using timers and paths and socket activation or security confinement with just two extra lines in a unit file you can actually restrict your service to be much more like a confined container and having great isolation and so far we don't really exploit that a lot. So in case you got interested in any of these topics please come and talk to us. And you can reach us on this mailing list or an IRC. We are there and just just poke us. But you can also just help filing bug reports, working on bug reports and if you want to hack on SystemD itself SystemD is managed in Git on Aliov via Git belt package. And as a closing note if you want there's a buff tomorrow which you can which we try to show you just in case you have problems with SystemD how you can debug those problems. So thanks for your attention and I'm not sure if I have time for one or two questions. So thanks again. You can ask for a couple of questions. Started a little late but yeah should be okay. You mentioned network D. Why should I use that and what is it good for? Do you take that Martijn? And yeah what as I said it is mostly useful to have a very small solution which covers all the server side's cloudy virtual interfaces because they unlike IF up down where everything comes in different packages and doesn't isn't very well integrated the demand basically sorts out all the race conditions correct bring up of nested interfaces and so on. And the important part about network D is that it's event-based. It sees devices coming to go. IF up down is started during boot and then it's done. And so it sits a little bit in between of if up down and up network manager. I would just like to thank you deeply because I don't think many people realize the pressure and the shitstorm we have been through and I'm very grateful of your work. Thank you very very much. Thank you. Thank you. Thank you. Okay well thanks everyone. Oh well. It's one more. Not sure if you still have time but okay. What about system D in DI? I had that on the notes I tried not to do it to not upset Cyril too much. Well it's essentially the same story if it's the shell based home ground init system and in DI you want to think about replacing that with system D. I have no idea whether that would make a lot of sense or bring a lot of improvement. We just didn't look at it into it at all. If anyone's interested in looking into that yeah you're more than welcome. But I think the higher priority should be in a drama fest because that happens all the time while installing just happens once and it's like less visible. I'm not sure whether it is easy to answer that in this auditorium but how did you manage to keep your health when dealing with this upstream? Well actually upstream I find him pretty well to work with. So I mean he has a clear mind he speaks his mind and I think he's honest about what he says and sometimes people don't like that and maybe he comes off as sometimes abrasive but actually we have a pretty good relationship with upstream. And I also must say Luna really learned to become a really great maintainer so he beautifully reviews patches and of course he's very picky and that's good technical excellence helps us all but like he's not discriminating because we come from Debbie and all going to all what not. And now I mean the system is far more than just Leonard by now there are like the core team is like five, six people and by now there's hundreds of contributors who send more than 20, 30 patches. So it's really a big community project these days. Yeah and he actually asks us if there's stuff that is upcoming then he says do you have some import from the Debbie inside? So at least I feel included in the decision process. And we are upstream committers so in theory we could break stuff ourselves but of course we always go through the review process and yeah. So you just talked about removing the S renewable script. I know because I worked a bit on open RC not as much as I wanted but anyway I know that it's full and full and full of hacks that we build up over time. I'm just curious if you want to remove them then what's going to happen to CS5 in it? Well you don't remove them you just add support for system D with native service file you don't remove the RCSD in its scripts when doing that. But we need to have the script by policy so we can't do this. So I'm not sure how we are with time and give the other ones time to prepare so I think we should stop at this point. Right so thanks again for coming and you can meet us outside at the hall. Right, thank you.