 And now, can you now hear me? Does it work? OK, wonderful. So yeah, I will be talking about system B today. By the way, I prefer my talks more of the interactive kind, so meaning that if you have a question, just go and interrupt me. I much prefer that over questions at the end, so that all the questions we can have are right on the topic that we're discussing. And I would also much prefer if you guys would lead the talk into the right directions by your questions instead of just me picking a couple of things that I like to speak about. So yeah, I have a couple of slides prepared here. I will start with an introduction to system D. And hopefully, based on your questions, we can touch particular areas in more detail. So let's jump right in. This slide contains the original description of system D from the system D website. It's a long, long paragraph. This is the first half sentence of it. It's a paragraph with a lot of information on very, very little room. It's not necessarily easy to understand, and that's why we're hopefully going to parse it a little bit so that everybody understands what is meant by this. So the first half sentence reads, system D is a system and SAS manager for Linux. So what does that mean? A system manager probably everybody might have an idea. The system is like this operating system thing. And a system manager, what we mean by that is that it's an inner system, that it manages the system, that it manages the components of the system, meaning that it controls a little bit what processes are being run based on a couple of things. A session manager, on the other hand, is the term session manager. It's probably known to many people like GNOME Session as a session manager, and KDE Session as a session manager. And system D also manages the session. So basically it's a replacement in some ways, or it can be used to augment GNOME Session or KDE Session for Linux. Let's go on. It's compatible with System 5 and LSB Unit Scripts. System 5 and LSB Unit Scripts, probably everybody has known, at least if you ever came in contact, in closer contact with the Linux system, you probably have played around with System 5 Unit Scripts. They are basically these things, ETC, INIT, D, some service, start and stop you can start. We're compatible with that. We're in a system that tries to stay compatible with existing System 5 or LSB Unit Scripts. LSB and System 5 is basically the same thing in this context. Basically System 5 introduced the original contact ideas, and then LSB standardized around this a little bit, and extended the original System 5 specifications, like standardizing exit codes of those Initscripts, and then standardizing the verbs that you can pass to it, and standardizing and commenting a couple of other things. But it's mostly synonymous. System D provides aggressive parallelization capabilities. Parallelization, probably many have quite an idea what that is. What that means in a system, in an INIT system context, is probably not so difficult to understand either. Basically it just means we start everything that we start in parallel. But the word aggressive here. It means that it's probably a little bit beyond what existing Initsystems do. And in a later slide, we'll go a little bit into detail on what precisely this kind of aggressive parallelization means. It uses socket and divas activation for starting services. That's probably one part here of this paragraph that is only understandable after you had a look on the further slides we have. Basically it means that we can start services if something happens on a network socket, or if something happens because somebody required a divas service or something like that. So why this is so very useful and it's so interesting that we put it on this initial paragraph. We'll see a little bit later. But yeah. It offers on-demand starting of demons. I figure most people will also get kind of an idea what that could mean. It could mean, for example, that we start a demon the moment we use it. For example, if we have a demon like Bluetooth, which is responsible for maintaining the Bluetooth hardware, that we then start the demon only if the Bluetooth hardware is actually plugged in. But this is actually very generic. It could mean a lot of other things that if some service requires another service, that we start the service right the moment we actually need it. It's an interesting feature because it does less work, but it's not the core feature. It keeps tracks of processes using Linux C groups. Process is probably, I hope that everybody knows what a process on Linux is, but what does it mean keeps track of it using Linux C groups? Linux C groups is a new kernel interface that has been introduced, I don't know, 10 versions ago. C groups is short for control groups. What that really precisely means, I have a couple of slides about this later. But this is actually very, very useful to not only start and stop services, but also keep precise track about every process that a service spawns. And that can be quite a lot. For example, if Apache starts up, it can start a gazillion of CGI scripts and whatnot. We use Linux C groups to keep track of them. Again, a little bit more detail about that later on. It supports snapshotting and restoring us a system state. Snapshotting, probably people know from context of databases or stuff like that. It basically means that you take a snapshot of the system like it is, you start it away, and later on you can return to it. System B supports that. So you can say, OK, I've started this service and that service and that service and that service, but I don't have started that and that and that and that. When you say, save that state, later on you return to it. This is, for example, useful for quite a few things, like you're administrating your machine, you have a Apache start and everything else. Now you say, oh my god, I need to administrate something and I need to make sure that nobody and nothing does anything to the system at this time. So you change to a single user mode. You do your changes. You can be sure that nobody interferes with you because you're the only one. And then we return to the original state again, as if nothing happened. It maintains mount and outermount points. This is probably something surprising to many people because in its system so far, didn't really do mount and outermount handling. Mount points, probably everybody who ever dealt with Unix has an idea what it is. Automounting, not necessarily. Automounting is a lot like normal mounting, except that instead of actually mounting a file system to some place, you just mount an outermount point to it, which is something like a magic thing that just stays there. And the moment somebody, for the first time, accesses it, is backed by the real file system. In system D, we use that to parallelize boot up. And we also use it to delay boot up certain jobs during boot up so that we don't have to do them right away during boot, but only when they're actually needed. It's not speeding up the boot up. It implements an elaborate transactional dependency based service control logic. That is quite a half sentence there. Transactional, probably everybody heard in the context of databases. Like you have something where a couple of operations, you bind them together, call them transaction, and you either execute them or you don't execute them. But you do not half execute them. And in system D, we have a very weak definition of transaction, but we have one in there. Basically, that means if you start Apache and that pulls in MySQL, then either you end up with both, or you end up with nothing, but you do not end up with Apache half-started and MySQL half-started as well. Dependency-based basically means probably everybody heard that in the context of package managers. You install the MySQL client and pulls out the MySQL server or the other way around, something like that. We have the same thing in system D, that you can say, well, D-Bus requires SysLog. So SysLog is pulled in by D-Bus. And service control logic basically means, yeah, you can control logic, you can control the service with system D, surprise, surprise. It can work as a drop-in replacement for system five-init. System five-init, everybody knows, I hope so, is a classic implementation of an inner system for Linux. It has been used by almost all distributions. Historically, very recently, a couple of distributions changed to an alternative implementation called App Start. We're not going to talk too much about App Start here. But yeah. So this is original paragraph that's on the website. It's quite a lot of information for very little text, but I hope everybody has a rough idea what it actually might be. So on the next slides, we're going to do a couple of these topics mentioned here into a little bit more detail. So let's talk a little bit about init. Init, as I kind of mentioned already, is a special process. It's the first process that is started after the kernel is booted up. It's the init system. It's PID1. And it has magic capabilities. System D installs itself as one implementation for this magic process number one. It's magic for a couple of reasons. It's magic, for example, because if you press Control and I'll delete, this gets forwarded as a special request to PID number one. If a process dies and then all its children will be reparented to this magic process number one. Every single process that is not a process or something else is automatically a child of this magic process. For this reason, it has a couple of additional requirements, additional constraints that it needs to implement. Because basically the entire user space depends on this to be running and to be controlling everything. So yeah, as mentioned, there are a couple of implementations around. The big ones are System 5 init and AppStart and now System D. I like to believe that System D is the most advanced of those three. So by the way, if anybody has any questions to all of this, just ask. Raise your hand, then there's some people here with microphones who'll give you a microphone. If anybody has a question, ask. The next topic we'll touch is parallelization. Parallelization is one of the key things that System D is about. Yeah, as mentioned, probably everybody has a rough idea what that means. It means if you boot up your machine and you start a couple of services, depending on what you're running, it might be, I don't know, up to 50 services or so, that we start them as much as possible in parallel so that whenever the CPU has nothing to do, it can do something else. We have this wonderful graphic here, which tries to explain the way that System D implements parallelization and how traditionally parallelization was implemented or was not at all implemented. To the left, we have this traditional System 5 parallelization. It's basically how most of the distributions five years ago worked, and actually Fedora until Fedora 14 still works. This shows you basically the order in which four services do. We just picked four services for the graphic here. I'll start it. Syslog, D-Bus, Avahi, and Bluetooth. There are a couple of dependencies here between these services, which is why this order is the one that is actually used here. D-Bus uses Syslog. So D-Bus has started second, and Syslog has started first. Avahi uses D-Bus and Syslog. So it's started after those two. Bluetooth also uses Syslog and D-Bus and started after those two. Bluetooth and Avahi, however, they don't have any dependency. Bluetooth does not use Avahi, and Avahi does not use Bluetooth. However, since traditional System 5 bootups like Fedora implemented in the month of 14 were strictly serialized, this meant that still we had to pick an order. We had to start one first and the other one second. In this case, we just picked the alphabetical order because we didn't know any other thing to do. Of course, it could have been started the other way around. There would not have been a problem. Now, a couple of people looked at this and said, oh my god, yeah, well, the ordering between Syslog, D-Bus, and Avahi, we can't do much about. And the one between Syslog, D-Bus, and Bluetooth, neither. But we could do something about the ordering between Avahi and Bluetooth, because it can be started parallel. We should start parallel. And then people came up with this, the middle kind of parallelization. Syslog and D-Bus are still started one after the other. And Avahi and Bluetooth are started afterwards. But Avahi and Bluetooth are started at the same time. This is the traditional parallelization, how Upset works, and how, through the updates, it's a classic System 5 boot process. It's an improvement. I mean, if you look the arrows all together, basically should give you an idea how long this takes to bring up all those false sources. And you notice that the traditional System 5 took like four arrows. And this one just takes three. So it's a little bit faster. But it's not as good as we do it in System 5. Because in System 5, we actually start all four of them completely in parallel. And that is kind of surprising. I mean, how can we do this? Because there is still a dependency between Syslog and D-Bus and between Avahi and D-Bus. How do we actually pull it off that we can actually start to completely in parallel? And this is a technology called socket-based activation. It's something that Apple pioneered in launch D, which is in the core part of the MacOS operating system. They basically looked at these kinds of boot-up graphs and thought, hmm, so if we look at all of this here, why precisely is it actually that Avahi has to wait for D-Bus and Syslog? What is the one thing that Avahi waits for? What is the one thing that D-Bus waits for in Syslog? And they looked at that and looked in all detail. And then they noticed it's about the sockets. It's about the sockets that are created, the Syslog socket dev slash log that is created, that is bound to by Syslog, that D-Bus waits for before D-Bus can start up. Because D-Bus wants to connect to that socket and write messages to it. And then they looked on the other dependencies and said, OK, yeah, so why exactly is it that Avahi has to wait for D-Bus? And again, it's about the sockets. Avahi wants to connect to the D-Bus system socket. It's a socket called slash var slash run slash D-Bus slash system minus cross socket. And they looked at it and said, well, if it's really just about the sockets, can't we somehow do something about that so that we can start things in parallel? If the socket is really everything that it's waited for, wouldn't it be possible to somehow speed that up? And then they came to a solution and the solution they came to is that they pull out the actual socket binding out of the diamonds, do them in one big step in the inner system itself. And then there's a pass these sockets, pre-initialized, pre-bound to the actual services. And that's what they did with this lock. I mean, they don't use D-Bus actually on macros, but they have a couple of other services that work like that. And then they pulled out the socket binding out of that, did that in launch D. So in one big step, all the sockets, that all the services, be it Afunix sockets or Afinit sockets or whatever, they're all created in one big step and then it tied in a loop. That gives really, really fast. And then they start every single service that is supposed to be started at the same time. And the services get the pass through the socket they should be listening on later on. And this is actually really, really nice because suddenly you can start everything in parallel because the sockets are already established. So if D-Bus wants to connect to SysLock, it doesn't have to wait for anything because the listening was already done before SysLock was even started. There's a question. Sorry for that. By the way, I know that I speak very, very fast. I'm sorry for that. If I speak too fast, say something. I'll try to slow down. Hi. The socket-based activation seems to lend itself very well also to work across the network. But from what I understand, you rely on the kernel to kind of resolve the dependencies by queuing. Won't that break if, for instance, one of the servers in your cluster hasn't been booted properly yet? Or will the dependency resolution still work properly in those cases? So the focus of this kind of socket activation for us is mostly AFU next, actually. It's not so much AFInet. So it's a little bit different from the traditional INET stuff, which INETD stuff, which focus on internet sockets. We also cover INET sockets. In the case of cluster stuff, where you have an actual network, you probably need to program your stuff more defensively anyway. So you probably need to continue trying to make the connections. Generally, on the local case, the dependency, if you have cyclic dependency, whatever you have, don't become different through adoption of this scheme. It just means that you create the listening sockets earlier. It doesn't mean that if there is a cyclic dependency or anything like that, they suddenly go away or suddenly more cyclic dependencies get created there. That doesn't really change much in that way. But I would say, if you focus on clusters and stuff like that, you probably should program defensively, should retry, because packets get lost all the time. So does that kind of answer your question? OK. So the socket activation has many, many advantages. One of them, as mentioned, is that we can do this drastic form of parallelization, where we can start every single service at the very same time and then you make the best of the CPU and the IO time available. But there's a couple of other additional, really great advantages. One of them is, suddenly, you do not have to encode any kind of dependencies anymore. Because in one big step, all the sockets are actually established. So whether DBAs use a syslog or not doesn't matter anymore, because it can just connect to the socket and it will be there. And I mean, traditionally, you needed to make sure that DBAs got started after syslog. So you needed to write down somewhere that DBAs require a syslog. But it's not necessary anymore, because all the sockets are created at the very same time and everybody can just connect. So it's a lot simpler for the administrator and for the developer, because they don't need to think anymore about all these dependencies. It has a couple of other advantages, too. For example, we can actually restart stuff without having this service being unavailable for the tiniest bit of time, even. For example, you start syslog. And then syslog crashes, for example, because I don't know why. Syslog's implementations tend to be gigantic beasts nowadays with all kinds of for empty, pricey SSL and whatnot. So they have every reason to crash. So if they crash and use this kind of socket activation, then the socket they are listening on got created by the init system and is there. So if the process goes away, the init system still retains that original socket. And if the init system then notices, oh my god, syslog crashed and was configured to say, OK, if it crashes, then just restart it. Then it will do that and will pass the original socket again to syslog. And however, this is still the original socket. So every message that got queued into that socket is still there, to the effect that nobody will actually notice that syslog crashed, because not a single message will be dropped. Every single message that is in the socket will be read by the syslog implementation. And this is really cool, because you can actually write robot software that can just crash. And the only thing you might lose is one transaction that it was actually processing while crashing. But otherwise, you don't lose anything at all. You can even use this for really amazing stuff, like upgrades. You say, OK, my syslog implementation, I got a new version. I can shut it down. I think I can start the up the new one. But because the socket listening is done by the init system, and it still always keeps that reference to that socket, you can do this. And you won't even lose a single log message. You can use it for a couple of other things, too. Like you can actually replace the implementations of what you do. And again, syslog is a good example for that. We, for example, have a little bit of a very tiny bridge that connects syslog to K-Message, K-Message being the current log buffer that you can see with D-Message. We always thought it's kind of sad that during early boot no proper logging is available. And in system D, that's different. Because in system D, we very, very early create that listening socket for devlog, then spawn this little, little bridge thing that just puts everything that comes in through devlog into the current log buffer. And eventually, when the real system goes up and the real syslog starts up, we can just start them and replace the implementation of syslog on the fly without losing a single message. And that is really, really great. And there are a couple of other things. With this kind of design, it is the kernel that schedules the execution order for us. Because let's, for example, a syslog example again. Let's say you create the socket, then you start syslog. At the same time, you start all kinds of clients of syslog. Then they will connect to that socket and they'll write the messages to it. But they will never actually block on it. Because a socket buffer of devlog is kind of large. So every time they write a message, they will just write the message. And the kernel will just put it into that socket buffer and will return immediately. So the clients do not have to wait ever for anything. They can just push data into it. And eventually, if the socket buffer really runs over, if you'll really log into a couple of megabytes of stuff, then they will have to wait until the other side caught up. But only in that case. And in the syslog case, it works really, really well. Because syslog is really strictly one way. You never expect to reply from syslog. You just push data into it and syslog takes it. But it will never actually respond to you. At the same time, then the client will write a message to the devas socket. And devas will take some time to reply. But it's only that one application that will wait for it. And at the same time, you can start everything else. And they can also access the socket and push data into it and stuff like that. But yeah, the kernel is the one that will order the execution for you. And that doesn't need to be a scheduler anymore in the inner system to make sure that the things are started at the right time and make the best use of this CPU. So yeah, this is socket-based activation. It's one of the greatest things that are in launch D. And so we thought, OK, this is so awesome. We want that in system D2. Because it simplifies everything, because you don't have to configure dependencies. It parallelizes things like nothing else. And it makes things more robust because it can change things and replace implementations and stuff like that. Is there a question? There's a question. There's a question. There, there, there, there, there. One thing is, how do you know which sockets have to be created in the first time? It's basically, if somebody wants a service to be started, like you install a Syslog implementation, then it will tell us not only that it's this Syslog binary that should be started eventually, but it will also tell us, OK, please create the devlog socket for us. So it's basically, at installation time, you drop in a service file and a socket file, that's how we call it in system D, that contains configuration parameter. Usually that's very, very short. You just say, listen on a datagram socket slash devlog for me, full stop. So that's really, really short. And you can actually maintain those sockets in the services and system D completely independently. You can start the Syslog socket very early and then very late actually start the service and you can even do stuff like this bridge I mentioned. It will actually terminate when idle. Like, if there's no log messages, if it didn't receive any log messages in the last 15 seconds or so, it will terminate itself. Because the unit system still listens on it, it's not a problem because the moment somebody actually writes something again to the thing, then the unit system will notice, oh my God, something wrote something to the socket, but there's nothing backing it. Oh, then let's just start it and then it's being started. Then we'll process the message that got queued in the socket and then eventually the bridge will terminate again. So you have this kind of on demand starting in this as well. All of this is not really a new idea in Unix. As mentioned, Apple came up with using this for parallelization. But actually this was known before in INITD. I already mentioned that earlier. INITD is one of those classic Unix socket that has been around for ages. Unix services that has been around for ages. It did very similar things, but it didn't do it for parallelization or for robustness reasons. It mostly did it to simplify implementation of diamonds and do on demand starting of diamonds. They had a little bit different models. They were mostly focusing on internet sockets, not so much on Unix sockets. We focused more on Unix sockets but also do internet sockets. And they had, although it was possible to start services that would then take the listening socket, it was mostly focused on spawning one instance of a diamond for each connection socket. But actually it supported both ways. And we support both ways too. It's just a little bit of a different focus. We want, if you start Apache on demand, we want that Apache get the real listening socket. And they back then wanted to just hand off one connection socket and have a couple of Apache instances. Which is not a recipe to make things fast. And we want things fast. But there was a question. Yeah. I'm assuming that demons need to be modified in some way to understand this or? That is a very good question. And I have a slide for that. So yeah, we need to patch damage for this. In some cases at least. It's actually very, very simple to patch damage for this. For a couple of reasons. The code of the diamond actually becomes much, much simpler. Because if you currently, if a diamond creates a socket that it wants to listen on, what it does, it calls a socket system call. It calls the bind system call. It calls the listen system call. And maybe calls a couple of sockets in between. If you use this kind of system deactivation, then the socket just gets passed to you via process of execution. You don't have to do anything. You just take the socket and it's there. So yeah, but you need to patch most of the diamonds. We already did most of the work for the stuff that is running on default on a Fedora system. Like we patched RSS log, we patched D bus and all these kind of things. So that they actually work like this. It's really, really simple interface. It's basically, you just get an environment variable which tells you, hey, you got a socket pass and then you just take it and use it. Also, quite a few diamonds already support a socket based activation due to INED history. For example, SSHD, we won't bother with actually patching that. Because SSHD-I can be used to start it per connection via INED. And we say, that's good enough. And Apple actually started the way too. We don't need any further socket based activation. We can just use this classic INED mode and we support it and it works fine. And there's a couple of other reasons where socket, this kind of patching becomes very simple. For example, because Apple supports this on MacOS, quite a few software that exists out there. It's already patched for these kind of the same mindset, basically. It doesn't use the same APIs as we do. Our APIs are much, much simpler. Our APIs really are just checks environment variable and just use a socket. While with the launch D APIs, you have to check in and say, yeah, give me those sockets and it's kind of complicated. But if a software already got patched for launch D, it's very, very simple to update it to support our mode as well. So yeah, you have to patch. But it's really, really simple. So regarding this socket based activation, have you given any thoughts to services that are associated with the user sessions? That is good questions. So yeah, later on, if you saw the original slide, it wasn't only saying system manager, but also session manager. So eventually, we do not only want to manage the system with it, because that is that the system boots up fast is one good thing. But that the session, after you logged in, boots up fast is almost as important, might actually be more important on most desktop machines because we start a shitload of stuff in the sessions nowadays. So yeah, we definitely want to also start GNOME like this. It's not on the to-do list right now, because first we need to get into distributions for the system stuff. But it's going to be the next thing that we look into. Actually, you can already run it like this. You can just run system D dash dash user, and it will work as something like a session manager. But you can't really use it yet to spawn up GNOME. In the long run, it's our secret hope, but we haven't really figured out all the details that we basically pull out the session management, take this away from the different desktop environments and just have one simple one that everybody can use so that you can, instead, that you have a GNOME session, a KDE session, and then something else. And people just can mix and match stuff. And there's only the Linux session, which would be system D. But yeah, all these things like paralyzing boot up, they are the same problem for starting up the system and starting up the session. I guess that answered your question. Okay, there's another question. Yeah, just a quick note. It's a good idea to try to replace GNOME session and KDE session and XFC4 session, for example. But you still have to keep in mind that system D will only work on Linux. So... I didn't follow? System D will still only work on Linux. So we still need the traditional session manager on the other operating systems. So yeah, in system D, we do care only about Linux. This offers us so much advantages that we... Yeah, I personally don't care about the other operating systems, basically. If people care, it's their problem. The thing is if we focus on Linux, then we have so many opportunities because we can use all those Linux-specific APIs. And Linux is simply the most advanced kernel there is. And it has all these awesome things, like Linux C groups, all these kinds of properties. And if we would want to make these portable to other Unixes, which basically, I don't know, was the API still in the, I don't know, 10 years ago, we could do this, but we couldn't use all these functionality. Also, it makes our code much, much simpler because we do not need to care or anything about abstraction. We do not need to abstract kernel interfaces because we could just develop it on Linux. So our code becomes much shorter. It becomes much easier to read and becomes much more powerful. And we actually use a shitload of Linux-specific interfaces. We use Eventfd, we use Timerfd, we use C groups, we use Slash, Procmount. Really, yeah. A lot of these things, you can't even do another operating systems. For example, just watching the mount table. What is mount and what's not. There is not really a nice API for this on any other operating system, but on Linux. So, well, I mean, I do respect, if people wanna spend time on other operating systems, they may, but I think the focal point of free software development is Linux. And I don't think I need to care. Oops, I need to care too much about keeping compatibility with 10 operating systems. Just one is good and it's going to create the best product. And I think most of the people who actually do the work in GNOME and do the work in the other operating systems do mostly care about Linux. So, yeah, basically my position on all of this is, yeah, my focus is Linux. I offer you something for Linux. If you care about something else, I won't make your life extra difficult. I will respect that. But don't ask me to care for FreeBSD or whatever. I won't. That's another question. What do you do if you have a dependency relationship that's not based on sockets? So, it's some other dependency. So, socket activation is not the only kind of activation we support. We also support Debas activation. So, on current Fedora systems, actually, by default, more Debas services are installed than actually services listening on sockets. So, everything I told you about socket activation, we also support for Debas activation. Meaning, if somebody wants to talk to a service that is not available on the bus right now, it will be spawned. This actually exists in Debas anyway. But in this new scheme, it will actually forward to systemD. SystemD will start the service and it works like this. This is much nicer than the traditional stuff, actually, because you can actually start a service, for example, Avahi. Avahi, I hope everybody knows that it's service discovery. You need it. If you have a network, you need it if an application requests for it via socket. Meaning, does an NSS request. And you need it if somebody requests it via Debas. So, these three triggers. And if nothing of this happens, you have really no reason to start Avahi, because if you don't have a network and no application wants to use it, why start it? So, in systemD, we have this scheme in there that you can have three triggers. Hardware, Debas, socket. And they will result in just one instance being started and that completely race-free and completely safe. And, yeah, does this kind of answer your question? Yeah, thanks. That was great. So, if you look at the main page, you'll see we have a couple of other triggers as well. Like, we have mount triggers and all these kind of things. So, you have quite a bit of flexibility. Yeah. There's another question from where? No, there wasn't. Everything doesn't explicitly do the thing. Oh, by the way. So, what Kai just, by the way, this is Kai Ziver. He is the other systemD guy here. So, if you have questions, you can not only ask me, but also him. But what he mentioned is that while in systemD, most of the dependencies completely go away. You don't have to configure them anymore. You still can if you want. And this is necessary, for example, for early bootup. Because at early bootup, you have dependencies like you want to first set the clock and then do a couple of things like writing log messages because you want to make sure that the right clock is used when writing those log messages. So, sometimes you do need explicitly configured dependencies. And systemD exposes a very elaborate system if you want to. But normally, for most of the services, most of the services like system log debuts, whatever, you actually do not have to configure a single dependency. It will all happen for you. So, the dependency configuration is mostly something for the people who actually build the operating system for you, like the MIA sub-stream or the Fedora maintainers and Demian maintainers and all these people, they will use explicitly configured dependencies. But the people who develop services, the administrators who want to write service files for existing services, they now actually have to deal with that. That's it. Yeah, I don't really understand, oh, this is supposed to be a drop-in replacement for system five-init if you need to patch the demons. And so, yeah, you don't want to think about operative auto-operating system, but you expect the demon people to patch specifically for systemD and Linux. So, to make this really clear, you do not actually have to patch anything. It's, you can if you want to. But systemD supports classic system five-init scripts just fine. It will read them as if it was native configuration. It's just different kinds to write configuration down. So, if you don't want to use a socket-based patched, I don't know, implementation of MySQL, you're welcome to. So far, nobody has patched it anyway, so you can't even use a socket-based activation. So, there is no need for you to patch. We welcome you to do this. We think it's a great improvement if you do, because you can get all the robustness. A lot of people agree with us that makes sense. For example, really not so mainstream software has even patched these days, like DoveCop, that IMAP server actually supports socket activation. But, yeah, you really don't have to. You can continue to use systemD in its scripts. You can continue to not use a socket activation. But, if you do, you gain all the advantages that we offer you, that you get rid of all the manual dependency configuration, that everything can be started up in parallel and all these kind of things. That answers the question, I hope. Okay, there's another question. Yeah. Somebody got a mic? Hello, yeah. I didn't understand the last answer you just gave there about optionally supporting system five-init script. Surely, for the dependency resolution process to work, you have to have a full set of dependencies. If you only have a few exposing their sockets and then a mish-mash of half-ass system five-init scripts that no one has been bothered to port, then the functionality that you're relying on to make systemD work just won't work very well. Are we pointless? It should really be all or nothing. No, it's not all or nothing, because actually, I mean, if you install 10 services on your machine, like an iMirror server and Apache, then iMirror and Apache do not actually have dependencies. So, but you're right. If one of the things that are more at the root of things is not socket-based activated, then you cannot parallelize the stuff that's started afterwards, that is true. But in real life, that actually doesn't happen that much because at least as far as we see it, all the core stuff we have patched. Syslog, debas, blah, blah, blah, blah, blah. But yeah, you're right. And in some cases, for example, for the debas activation becomes much easier because debas activation already existed. So manual configuration of dependencies is never necessary there. But you're right. If you have a long chain of system five init scripts that use classic dependencies between them, then yes, the system will make use of that. But so basically, my look at this is if you start converting, then start at the root. Don't start at the end. And then the problem doesn't exist. Okay, any further questions at this moment? The last one. So when you first actually blogged about this and did your first release, one of the things you talked about was aggressive starting of demons versus the sort of start them whenever they become required. And you'd put forth a few statements about how you thought that won't be faster but you needed further testing. I'm curious, I have not seen you post anything about any further testing you've done. And what has been the performance impact of that? So yeah, in system D, our focus is not only speed. It's also speed. But the central focus is to make things correct. That's what we wanna do. We wanna have clean code that does things correct. So if you look at system D and everything started parallel, then you have this problem that on a classic rotating disk, actually this is not necessarily improves the performance. The reason for that is that why traditionally if you started everything one after the other, then because all these blocks that these applications used were on disk one after the other, you would actually have linear reads through the disk and that is the best thing you can do in rotating media. If you however use system D and suddenly start everything in parallel, then basically the read requests to the hard disk come completely random because there's this service which needs to access there and the next one needs to access some completely different place. So yeah, this resulted in not an extreme improvement in speed if you use rotating media. So however, this is not unfixable. The reason is because it's not unfixable, we have these elevators, these XIO elevators and they still, a good elevator benefits if it gets a lot of requests that it can choose from. So far the classic Linux elevator wasn't very good at that but we now generating those workloads and elevators tend to be optimized for the workloads that they have. But the summary of it all is it isn't worse than the current stuff and it's much better on SSD and the future is SSD anyway. Because this of course a little bit of disappointing that this on rotating media because rotating media still, I guess most of the computers or the majority of computers probably still use rotating media, it's a little bit disappointing. So we looked a little bit into this and tried to find a couple of fixes. System D actually comes out of the box with a read ahead implementation. Read ahead implementation is something like that exists actually at least five or six versions of. Read ahead implementations basically, they look at one boot, deduced from that the sectors in which order they were accessed and the next time they had a very early boot to read all the sectors and in the right order that they are on disk, thus optimizing things and under the assumption that then they're already in memory when they're actually used and speed things up. We installed them by default to remedy this problem. It gets us about 10% depending on what kind of machine you have. If you don't start any services at all, like if you have very little to start, then of course the speed up of this stuff will be minimal. If you start a lot of stuff, then you will actually notice things. But yeah, but to be honest, I don't really want to really get too much into optimizing micro-optimizing the stuff like read ahead because read ahead is actually not a nice thing. Because what you do with read ahead is that you second guess the IO schedule of the kernel because you actually try to be smarter than the IO schedule of the kernel. So if we really want to do this, then you probably should upload the request that we will know will happen to the IO scheduler so that the IO scheduler, which has much more information about the actual seek times of the disk to make the decisions. But we talked to a couple of kernel guys, like, what's his name? Anyway, there's the IO scheduler guys. How about all of this? And to be honest, the interest in read ahead is not the biggest because they always say, well, if you want it fast, use SSD and all these problems go away anyway. So yeah, I hope this kind of answered your question. Well, I have a follow-up. I was probably speaking more directly in terms of D-Bus because obviously the way that system D works, you start everything in parallel up front that socket activated, but then D-Bus sort of trickles along. As dependencies arise, as things actually get called, then the process actually starts to handle the D-Bus request. So essentially, assuming, for instance, a typical desktop boot scenario, your desktop basically becomes available after the last daemon that had some request was started. So in that sense, you are parallelizing everything up front with socket activation, but with D-Bus, you're actually creating a long trail of dependencies. So has there been any thought about optimizing that? Well, I mean, we can start D-Bus at the same times as the process is also. I mean, there's a lot of stuff. The desktop can also peer up while still, for example, pulse audio starting, because if pulse audio is starting and you want to have the welcome sound, there's no need that this in any way actually delays the graphical stuff. But yeah, I mean, D-Bus is central. It's probably the central dependency of the desktop in this area. But I mean, yeah, we can paralyze it already. And yeah, of course, it's a bottleneck and there are a couple of people working with D-Bus optimization, but to be honest, D-Bus actually starts really, really fast these days. There's not, I don't feel too concerned about that. Are there any questions at this moment? I don't see anything, so let's go to the next slide. So yeah, one nice thing that we actually can do is paralyze file system jobs. So if you have a computer these days and it has a couple of hard disks, then you actually run FS check at boot. And then the entire boot waits until the FS check finishes and then you mount everything. As mentioned, system D actually supports auto mounting. And that enables us to actually paralyze the FS check with the actual startup of the system. Because what do we need to do? We need to, of course, do the FS check of the root file system, but we do not actually need to wait until slash home becomes available. Because what we can do here is we start FS check to file system checks a home directory, the home file system. And while that is active, we already install the auto mount point for slash home and continue booting. And then everything will work fine. And the moment some service or some user logging in actually accesses it because for example, Samba wants to share it. In that moment, Samba will do the file system access. This request via the automator will go to system D while that actually happens, Samba will wait for that. It will automatically freeze by the kernel. And eventually when system D then call up when the file system check finished and it replaced the auto mount point by the real mount, the execution of Samba will just go on. So all this kind of prioritization that we have with socket based activation or bus based activation, where we can start this lock and divas and all this stuff in parallel, we can extend to the file systems. We can run FS check, the quota check and everything else at the same time as other stuff that is started on the system is still in progress. So that's the question. And there's another one. Just a question about that. How does system D interact with early user space? And as an example, for instance, like crypto routes and things like that. So system D nowadays is not only this, this inner system that you can install and where you then can integrate your classic shell scripts with, you can't do that if you want. System D nowadays tries to standardize the entire boot process for you. We looked at all the boot processes of the different distributions and they all have these gigantic inner scripts that do the early boot stuff and we looked at them and noticed they all do the same thing and they all do it completely differently and in a gigantic shell script that is horrible mess usually. So what we did is we said we can do this better because shell scripting isn't necessarily nice. Shell scripting is necessarily slow. It's necessarily because it involves a shitload of forking of processes. We saw we can do this nicer. So what we did, we looked at this over a longer time and always picked as little pieces out of it and implemented that in C code. For example, most trivial thing, setting the hostname. We saw, well, if it's just about reading one configuration file and calling the set hostname system call, why do we need to fork a process? And then we looked into a couple of other things like this and then said, okay, let's find a nice advice where we can just do this in C. For the hostname stuff, for example, we said, okay, it's in system D itself. System D itself, when it's boot up, will now set the hostname for you and so you don't need to do that in the shell script anymore. So we covered actually everything now that is in the default Fedora boot. Everything, if you boot F-15, not a single shell, and didn't install any kind of magic stuff like NFS or for example, like that, you will not have executed a single shell while the system boots up because everything is now done properly in C. For the DM crypt stuff, we actually provided something. System D is nowadays extensible. So you can actually create, during boot up, system D units, system D, these units are basically what system D covers. A unit is a service, a unit is a socket, the unit is a mount point and stuff like that and you can have dependencies between them if you want to. So we have this plugin which looks at ETC Crypt tab and automatically generates on-the-fly units of it and system D then reads them and can integrate that into the usual flow. The effect of all of this is that the crypto loop stuff can be executed in parallel with FS checking the next file system already because instead of having this gigantic script where everything bit by bit is executed, we can actually parallelize that completely and because all this unit stuff in system D is perfectly parallelized, this is actually quite an improvement. So yeah, in system D, by default, if you install things, you actually get support for crypto stuff. It's optional dependency. Probably all the embedded people don't want to use it because they don't need crypto stuff but on the desktops we probably all want and so we support them. But your standby example, is it really a good idea to start up standby file system really isn't accessible and won't be for minutes or potentially hours if it's running FSK and waiting for it to finish? And that's a question for other services as well. Sometimes you really want it to be available if it will have a response time that's reasonable to the system as it is set up. I mean, do you take that into account? I'm not sure I understood the question. You gave the example of Samba starting it up and it will automatically block if it tries to access the file in the auto mounted file system because the file system is being checked. But that might take minutes or hours. So sometimes some administrators might not consider it a good idea to pretend that the service is available even though it won't actually be in practice. So my reply to that is in the traditional mode if the file system really takes half an hour or something then in the traditional system, your boot took half an hour to boot. So we allow you to already run the Samba at a much, much earlier point or something else. You can already access H login at the point where Samba is still waiting for the FS check to complete. But my reply to that is in system beam, all operations, really all of them, are actually time outed and you can actually configure that to the effect that if something really takes ages we will just go on booting for example or basically configure a robot what happens. So the idea in this case is if the file system check takes too long we will actually fail this request to Samba. Samba will get a clean error code but we'll just get this thing like yeah not a viable EIO or something like that and can then continue from that. But yeah, I don't think that actually really something changes there. If Samba previously had to wait completely for the FS check to wait for half an hour this is still what happens. Except that this delay to the moment where it actually accesses the file system. Well there is a difference in that Samba is actually appears on the network, it's visible so clients might try to use it. Well that there is some point in that. To be honest, I don't think that in the future all the file system checks will be that slow, I don't know. But yeah, it's a valid point that it will appear valid and accessible for a while but if you actually access it it will time out after a minute, that is true. Okay, my time's up. So thank you very much for your interest and if you have any questions then I won't bite and just ask me or Kai and yeah we'll be available for your questions all the time. Thank you.