 Okay, I guess it's one pass now, it's about time to get. I'm still in a tutoring, I'm going to talk today about SystemDs, the CoroS. The CoroS is some relatively newly coined terminology, it was actually coined in an LWN article about SystemD and we like to term very much so we adopted it and this talk should highlight a little bit what we actually understand as the CoroS, why we think SystemD is the CoroS or should be relevant for the CoroS and all the things around it. Yeah, I mentioned that yesterday already I like my talks interactive so if you have any kind of questions please interrupt me, I much rather prefer if we turn this talk into the direction you guys want rather than just what I like to talk about. So if you have questions just show up and answer them right away. Okay, so as mentioned the CoroS terminology for LWN, it's a bit of a vision of what SystemD should be now, when we started with SystemD we of course started with, we wanted it to be an inner system. An inner system is responsible for bringing up the system, that's why it's called inner system. But as we progressed with this vision we figured out that maybe it's not sufficient to just focus on the inner system alone and SystemD turned into something of basic building, a set of basic building blocks to build an OS from and for that we now have this nice term CoroS. So what the reason we shifted the focus a little bit from just being an inner system to being a CoroS. The reason is basically when we looked at solving the inner situation we figured out that well just being PID1 and bringing up the system is not enough because actually the inner system is not the only thing that is crucial for bringing up the system. Usually in traditional Linux in Unix there's a huge number of scripts which initialize certain parts of the system as well which are not part of the inner system traditionally but which are shipped along with it and are very much distribution specific. These can be various things for example like starting UDEF and doing device discovery and loading kernel modules. It could be something like registering new binary format that could be emptying slash temp, it could be all kinds of little things that each by its own are very simple and minimal but you all have to have them to have a proper operating system to be built from. Distributions had different scripts for that but they all very much did the same things like cleaned up slash temp and all the things that I just mentioned a couple of other things. Did it all with different code? Is it all different quality? So we looked at that and figured out okay we have cleaned up the inner system but it really would be interesting to clean up this basic set of inner scripts as well and replace complex shell scripts by a relatively simple and minimal AC program. So yeah we then started to move a couple of things into this to be proper meaning that system returned into just being one process into set of processes a set of very minimal processes that are responsible for very specific parts of system initialization and yeah bit by bit this vision extend a little bit like we move more and more stuff into it for example nowadays we have a replacement for console kit which manages sessions. The reason we move that in is very similar because we figured out okay service management is all about resource control and by about setting properties for services like process properties and the C group properties and all these kind of things and we figured out well services are not the only processes running on a system. You have all kinds of processes running for example also user processes I mean if you log into your GNOME session and that's not a system service that's user code basically and it is very interesting to set all the properties that you can set for service managers for the services you can also set for the user services you want that because I mean one interesting use case for example for C groups is that if you have Apache running SQL and Apache has 1000 CGI scripts and my SQL only has 20 worker processes you still want my SQL and Apache get roughly the same amount of CPU so for that you can use C groups but the same problem also applies to sessions if one user logs in and creates 1000 processes and another user logs in and only gets three they still should get in total each of them is the same amount of CPU and that's where C group comes in as well so you figure out okay so a lot of things that apply to the service problem also apply to the session problem and so we figured out okay maybe if we have that code anyway we should extend that and also include something consequence like the manager session tracks one user log in applies properties to them wraps them all nicely in C groups and these kind of things so yeah basically the vision grew a little bit by bit we added stuff here and there everywhere so yeah to summarize this vision of what the system in that case should be the core I should be is a minimal set of components to build an OS from of course by with the time the minimality increased bit by bit I guess that is I mean most computing programs end up growing growing growing that way so by minimal we actually don't really mean the absolute minimum you need to to to build an operating system from like if you want to do that then there are other solutions like busybox kind of things we don't want to have like go into that term but what we kind of try to focus on is providing everything that 90% of the use cases require and 90% of the use cases require also covering all the important bases like desktop stuff server stuff and other stuff as well the core OS having this on system D does not mean that system D should be one monolithic thing that covers everything but it means simply that we have a set of services that are tightly coupled that integrate nicely that behave similar that share code that reduce duplication but that are still individual processes so if you install system D these days it will cover a lot of ground but it's about I don't know 50 or so individual processes at boot we started at least I don't know 10 or 15 of those so yeah it's about it's not about making things monolithic not about pulling things all in the same process it's about simply streamlining the code a bit unifying things unifying things between distributions and unifying code paths where we all have the same stuff the core OS should be universal it should cover all the architectures and it should call all the users as mentioned servers containers and desktop all this kind of stuff we want to cover with system D we kind of want to focus on the same bandwidth as Linux itself focuses on and the same way as Linux works on mobile servers and embedded on desktop we want system D to cover the same ground as mentioned not everything like if you want to go really embedded then go for dvbox but everything that is like the more common stuff universal does not mean that we want to cover other kernels or anything like that we strictly want to focus on Linux with system D as the core OS we want to manage all the high level OS objects traditionally on Linux kernels and certainly concept to kernel new where basically files and directories and these kind of things and processes and a couple of other stuff like sockets we want to add a couple of or we added a couple of high level things that are now managed by system D of course services services we actually define them in terms of C groups mostly so that nowadays you have the identity of a service actually exposed as kernel objects by means of C groups and we did the same for user sessions for containers and apps where everything basically now exists in all layers as a stack on the kernel level of C groups and in user space by a user space maintained object so yeah that's our vision a little bit like basically finishing the high level object view on the system where basically postings left off and didn't go any further and it's basically just realizing the fact that I mean traditionally on Linux services we're just a single process each but nowadays that's not really how systems work because they haven't been in a long time because Apache is hardly one process it's a set of processes, one process, a couple of worker processes a couple of CDIs kind of things so yeah this vision of the core OS we try to basically complete this thing and have the same definitions that we have on all layers of our stacks of these high level objects system D as the core OS one of the key aspects we try to focus on is introspectability introspectability means whatever state the system is in currently you want to allow the administrator, the user, the developer be able to look into it more specifically than many cases most of the core components we have you can connect to primarily via debuts or other like individual components have different uses there but we basically give the administrator and developers complete access to the channel for system D mostly readable access that sometimes we can control it transparency means also that you can look into all the details specifically this can mean for example filing system D because we figured out a lot of things that new system booted up is very useful information and we generate this nowadays with system D build in so that when you boot up it will just measure everything it can tell you how much time the bias took, how much time the boot loader took, how much time the kernel took to initialize how much time the initial round was cooked to initialize and how much actual final user space it took so yeah we want to focus on making everything introspective so you always can figure out what's going on, how long did it take and what was involved in which order it was executed as mentioned a couple of times actually system D as a core OS is supposed to be something you build real world systems from so it's not the absolute minimum that boots it's more than that this of course is like if we decide if something belongs in the core OS and belongs to system D we of course always have this question like yeah where do we put the limit that's going to be more about that later on but yeah we knowingly put the limit not at the maximum absolute minimum which some people probably would do then we want to make sure that whatever you put into system D should be zero maintenance by that we mean that whatever happens in whatever condition you run this stuff it should not suck up more specifically like let me give you an example what we mean by that if we look at things like SysLock work now and SysLock traditionally if lock files were collected and went to disk rotation would come by time after a while rotate the files and free a little bit of disk space and delete those files basically and then rotation is finished and then after now it would come back into the same thing now this kind of design like from the first look at it kind of looks safe because after some time the disk always gets cleaned up and if the disk runs full then you just have to wait an hour and everything is done but if you actually look closely at it it's fundamentally flawed in a way because if somebody manages to generate a lot of lock messages in a very short time period he kind of can DOS a system and until the next time the lock rotation actually takes place the system will be basically clock nothing can write to disk anymore because the disk is full so in the system we context we added a little bit of logging functionality into it we really wanted to avoid these kind of vulnerability and what we did basically is that in the general context in the system we log in frame on contact everything we lock to disk we first check if we can actually do that if the disk is full and stuff like that and hence do the rotation in line synchronously to how we write stuff so yeah and then this thing is this kind of goes like this is kind of thinking of designing system that whatever happens we stay running and we don't knowingly enter a condition where it doesn't work anymore goes through the entire design I mean this ultimately is a lot of other things for example we put a strong focus on allowing the administrator and developers configure their services in a way that they automatically restarted if something failed this is something actually that was solely missing in system 5 in it because in system 5 in it when you start a service that basically all that happens and nobody watches if the service ever dies nobody is there nothing is there that could restart the service when the service dies and make sure it goes up again which is actually huge there's a huge hole in the entire model of Linux traditionally because I mean how can you do like a reliable service if something fucks up and you cannot automatically recover from that but this also means a couple of other things like in system 5 for example we have socket activation which allows a lot of interesting things because we can recover from damage dying without actually losing the connectivity because the socket part of the communication part of the damage always stays around and is maintained by the inner system and if the actual processing part dies then of course the specific transaction it's currently working on will fail and will be terminated but the actual connectivity is now a lot so yeah the entire process of design that we had in that was always to keep in mind whatever we do we want to have it in a way that it is reliable that it gives ISOS automatically reliable by the way so we want to have it in the design or at least give the administrator and developer the means to make it reliable by doing automatically starting this one the minimize test matrix is an absolutely key thing for us minimize test matrix by that we basically mean we want to remove a lot of variables from the base system the minimize test matrix basically means like if you look at the distribution like Debian where they have 10 implementations if you want to test everything then the test matrix becomes huge because you have to test every combination against every combination and the more combinations you have the larger the test matrix goes and effectively this means is like the more variables you have in the system the more unlikely is that you will actually find the bugs in every possible combination so we want to minimize the test matrix with stuff that we do in the system so basically we want to remove variables that make no sense anymore this has various effects like for example traditionally on Linux this is really simple program called Getty. Getty is responsible for bringing prompt on a terminal screen so that you can type in your username password and it will lock you it's an absolute trivial thing it will just really just ask for your username password then check TAM like the security the authentication library and if it's okay it will spawn a shell for you there's nothing like there's a few programs in our state that are trivial and more trivial than that now as it turned out over the history of Linux we got like seven implementations of that the core distributions ended up shipping at least three of them A Getty, Min Getty and something called Getty all these programs were completely trivial and could all do the same more or less functionality we looked at that and figured out this is bullshit all distributions pick a different one by default and all distributions ship them all we shouldn't do that and then there was all this irony that somebody created this Min Getty thing which was the minimal Getty which had a smaller binary size that actually took more memory than the other Getty that everybody else used anyway so we said we're not interested in testing all these combinations we don't want this kind of unnecessary variables in there and we just said yeah, it's just a myth we will default to only one of them to A Getty from your Pula Linux which is the best-maintained and one of the most featureful and that's how we ship our stuff and that's how everything will work by default but we will still make it possible that if people want to do something else they can plug it in but they have to do manual configuration for that yeah, this minimized test matrix goes through our entire design as well we try to look at the different components of our stack and really ask ourselves if we want the variability in there if we want to have this ability for people to replace a component unfairly in some cases it totally makes sense for example if people want to run Apache or Nginx or whatever kind of a piece of it it's totally fine I should always support it because like in the Getty case I don't think there's any point in making that overly configurable and suggesting this the effect of that is basically that if you run system D system because all the components we suggest you default configuration meaning you have to use that but yeah this default configuration because probably most people use that as the best test across the board and if it's best test then it's much likely to be actually robot another thing you wanted to really make sure about system D is to and then as used in the core OS is design the whole stuff in a way that it is useful for read only system images which is interesting for a lot of different use cases it's interesting for embedded case because yeah if you have the entire operating system in a way that it's entirely read only then you're less likely to run problems if you suddenly power off the machine but this also is very interesting for containers and things like that because if you are in a container containerized environment like cloud and stuff you usually have 5 front containers and they have very similar boot images if they don't have exactly the same images and if you can make them read only then you can be sure that you can basically run all the container from the same image and if one container changes the image that's just not possible because it's read only. So with system D we really tried hard to separate the read only parts from the operating system from the actual state parts of the operating system so that these kind of use cases work. Ultimately our goal with that is even it's probably not realistic to ever achieve that for normal systems but it's very realistic to achieve that for embedded container setup is that we actually want to be able to ship an empty slash EDC so that the definition of slash EDC is not anymore to ship defaults and these kind of things I mean it could still do that but it wouldn't need that so all the components in system D that we wrote basically they have default built in and if the EDC is empty it will still boot up with all these defaults and will just work and will not require any configuration but if you drop something in a slash EDC that is basically when the administrator changed the defaults and overrode what the default were. So the basic ideas it's a little bit inspired from what Android does in this era like if you have an Android phone they basically have three parts of the flash, three petitions one is for the system image, one is for actually user data like music files and photos and things like that and one is configuration and with this model of the allowing empty slash EDC and allowing with only slash user we kind of come close to that with model and as mentioned this is super interesting for more embedded kind of things and super interesting container kind of thing but probably not necessarily over all the back of really the depth of it. Another thing that we want to go for is state lessness. It took us a while to realize how awesome state lessness is. State lessness basically means that if you boot up a system it doesn't require to save any state to work. Let me explain this a little bit like I don't know most people probably use windows before if you have a windows operating system image and you boot it up on one machine it will write a ton lot of registry entries because it will just say okay I found this hardware let's say I'm crossing the disk about that and if you use the same OS image and boot it up on another system with slightly different hardware then suddenly there will be a ton lot of more registry things and everything just goes really bad. With everything we want to design for have designs so far with system D as the core OS we really made sure that the entire thing is state less meaning I can have the same image and I can boot it up in a container and I can boot it up on this bare metal machine on server and I can boot up on this bare metal desktop or something like that and it will work and it will not save anything to disk so that it's completely state less the same image can be used in any possible way and we actually really like this being able to boot the same image on bare metal on virtualizing solution on the container. It's actually really really interesting because with the system D image nowadays you can basically start your server with bare metal and then move it into container and at least all the system D components will automatically adapt to that and do the right thing. So yeah, statelessness we really funded this. Then here's another thing we tried to do with the core OS model we tried to basically copy a little bit how the BSD model works and the BSDs are, I mean I assume most of you know that is Unix implementation, the classic one, it predates Linux a bit and it's kind of lost traction, it's not popular at the next stage but their model of development is different from how Linux does that. In Linux like the kernel is developed by one group and the Lipsy is developed by another group and Utilinux by somebody else and system you by somebody else blah blah blah blah so you usually have, you pick a lot of different components from a lot of different people and then you build a system from that and you take different versions and you mix and match and you basically have this bazaar component that you put together in the way that you want it. The BSD model is different, the BSD model, everything is basically developed in a single repository more or less. So the Lipsy is updated at the same time with the kernel and the user space components are all in the same tree. We kind of wanted to not go all the way to the BSD model but at least move things a little bit closer. So basically effectively this means that we moved a lot of the basic building blocks of an operating system into the system tree, like Utilinux kind of thing for example and develop them in line so that the integration can be synchronized and we don't have to maintain all the different compatibility for all the different versions all the time. This helps us reduce the test metrics. This helps us to move a little bit faster because instead of updating 20 packages or 100 packages you can just update one. This does not mean that you can build the entire operating system just with one package but it will use the set of packages you need to build operating. So I mean traditionally if you want to build a Linux operating system you kind of need 60 packages or so. You need Lipsy, you need the kernel, you need Utilinux, you need agetty you get and these kind of things with by adopting this little bit going model that goes a little bit into the BSD direction in the system tree we hopefully kind of reduce that to just require kind of a set. The other thing is, yeah we try to innovate a little bit within the Unix design. There's actually, I mean there are a lot of people who are saying Unix, oh my god Unix is awesome and SystemD is totally not Unix and so SystemD sucks on this kind of thing. Actually we as the System Guys do believe that Unix is great inspiration. It's created something durable and we should always drive inspiration from that and we actually believe that in the system we often do that. More specifically this basically means yeah we try to, I mean in Insisting there are a lot of components that are kind of replacing specific components that people traditionally understood as the Unix. For example we have like the journal which kind of replaces this lock but not really replaces it. But this thing is and we believe that Unix is mostly about the concept that they used. More specifically in Unix everything was a file. In SystemD we said okay if everything is a file then we can go one step further and actually say that services are a file too. So in the SystemD world due to using C groups to group processes and C groups being exposed in the file system, we can actually say okay yeah we went one step further to be more Unix-ish. Because yeah in the file system you suddenly see old services and not just processes, that's the problem. But that's just one example that many other examples like for example in Unix traditionally you have this TTY logic. TTY is teletype, it's basically on traditional Unix system you have basically one central, you could have one central server and a couple of terminals connected to it and each terminal was basically just a keyboard plus a screen. And yeah this is how a machine worked in the 70s, 80s and 90s but not anymore really these days. But the concept is actually kind of cool because you have just a single machine that you have to administrate and it's cheaper and it's more energy efficient that you have to administrate but you have a couple of users on the same thing. With system we said okay it's a cool thing but we probably need to bring the concept to what is today required from a workstation because just the keyboard on the screen is not anymore what people expect from a workstation. Nowadays they expect the sound work but there's a mouse that is graphical that there might be in the webcam that might be something where you can plug in the USB key, your storage device will actually show up on your screen and not on your neighbor's screen. So yeah we looked at that and thought okay we can make that happen again, we can bring this bit of Unix history back, updated to how systems work these days and we did that by multi-seat support. Multi-seat support basically just means that due to creating these concepts that I mentioned earlier of sessions and seats in users, these high level operating system we just said okay yeah then let's right away from the beginning introduce multi-seat support. So basically it means that system is actually capable of tracking which device belongs to which seat and if you log in on a system system it will always keep in mind okay you log in on that seat and hence you have access to this set of hardware and if you log out and somebody else can log in then hardware is assigned to that user depending on which seat you log in. So yeah that's another thing where we kind of looked at back at what Unix actually was about and said okay that's cool let's bring that back, it kind of lost history but it was a good idea. So yeah and a couple of other things where you basically just said okay this is totally Unix. Again, I'm pretty sure a lot of people totally disagree without saying system is not Unix at all. Another thing that we try to follow with the system as the core OS is that we really should take what Linux offers and provide the user with it and empower the user with it. That basically means in Linux since time began basically people cared a lot about portability so they wanted not to limit themselves to one specific operating system or Linux but they were kind of focused on APIs such as POSIX and developing the APIs against POSIX and little else so that it would run on the BSDs and Solaris and all these kind of things at the same time. This is an interesting concept and it works quite well the higher up you go in the stack but we're systemally being like yeah basically the building block for the West which is way further down. We saw that it's probably not the best concept to follow. Since the Linux kernel actually offers so much more beyond POSIX that it's really useful to have. For example I mean starts with C groups like resource management. Running services these days is mostly like one of the primary facets of service management is the resource management. Regardless if you have an AMD device or a huge server, if you have an AMD device you probably want to run like 10 services on them and you have very little resources and you want to really distribute the resources you have in the way you want across several servers you have and hence resource management really matters. If you have a huge server you have a lot of resources but you probably run even more services on them because you have a lot of customers who sell a bunch of machines too for example. And if you have 500 machines there you probably want to use your machine to a very high degree because you made investment in the machine and buying it and hence the more customers you can sell it to at the same time the more money you'll make. So whatever side you look at resource management is absolutely key for running a modern computer. So resource management in classic post-exit is really minimal. They have resource limits they call their prep process but it makes them kind of useless because not even a web server like on a web server that doesn't even make sense anymore because a web server usually has a lot of workers right with CDR scripts and things like that and setting a prep process limit hence has very little effect because people can just park off and they get a new set of limits. So yeah we decided we looked at the Linux kernel and found so many awesome things in that that make real life like administrator lives and so much more useful that we say we will not limit our self to purpose. We will actually expose what the kernel has to offer awesome things awesome technologies and that covers everything like resource management as I just said a lot of security things that the kernel exposed hardware management all these kind of things. We really want to give the user the full power of the system and expose it as if it was normal and not add this artificial limitation and other things that we try to do with systemally as we quarrel as if we run into problem we always want to fix that problem instead of working around it. That basically meant that if something we need in the kernel then we will fix it in the kernel rather than work around. This is traditionally like probably done differently when people build in a system. I mean if you look into the initial scripts of various distributions you find tons and tons of sleep loops where they wait for something to happen and they sleep for one second and check if it happened now and then if it didn't they sleep again and all these kind of things that just grows hacks and we basically when we sat down and started with this and we said we'll never do that. We will really try hard to always fix the problem where it is and fix it properly instead of adding a work around it. Another design goal with systemally as we quarrel as is that we never want to be excused. That means while we will replace a lot of functionality that has been traditionally implemented into various unique services we never want to force people to actually use that. People can continue to use what they want. For example, more specifically this means you can run services based on time. You can say run this service every five minutes or so, or run this service on November 5th, 2013 at 6 a.m. in the morning. This of course has functionality that had been existing before in CRON and people are CRON and CRON is often because it's really simple to use. You just write one line and you can just find that. So when we added that systemally it's really useful to have that in system D because system D is really good at starting services. You can specify all the resource limits, all these other properties that are available for you. It can restart things, it can watch stuff, it can collect logs of the service which is much better than what CRON can do. But we acknowledge that people love CRON and people are across any kind of product from the page. So yeah, we never are excused. We don't have people to run whatever they want even if we provide something similar or equally powerful or somewhat related in the basic operating system. So this is like everything that I was talking about, just this fluff, divine thing that we kept in mind while turning system D from this inner system into this CoreOS kind of thing. The next part of the slide really focuses on the specific parts where we kind of took one of the existing components and replaced in system D itself. I hopefully can explain why we did this. So the obvious case of course system D replaces system 5 in it. Most distribution had something called Start Stop Demand which they, because actually spawning services from childs could suck, they wrote a little bit of a glue thing between called Stop Stop Demand which makes it starting to look a little bit nicer. It's all gone. In its scripts as mentioned, all the distributions had a different implementation of that. It's just a set of basic scripts that everybody used to set up to those modules, blah, blah, blah, blah. And we replaced that. These are kind of the obvious things because yeah, it's what system we started out. We kind of replaced INED. INED is this thing. Many modern machines don't even have that anymore, but around in Unix for longer you do know the thing. It's basically it allows you to run services only when somebody actually connects to them. Meaning, I don't know if you have a, if you run Git for example the Git Demand, there's no need to run it all the time. It's efficient if you actually run the Git Demand. Somebody connects to it and wants to check out your sources or something like that. So traditionally people used INED for that. It would just listen on a port or on a list of ports and IP addresses and then as soon as somebody connected would actually start to serve the actual Git Demand. This existed in Unix since the Saturdays basically and it was used as a way to kind of overcommit the functionality of the machine. So you have a very, very low resource machine and you can run a ton of services on them because the services are only started when they're needed. And systemally we have something called socket activation which makes use of this but it's a lot more powerful than the classic INED. But it's also compatible to INED. So whatever, if you have an INED service like Git Demand or whatever you have, it will actually work with this system as well. In a system you can do a lot of other things like you can have support for more particles than just internet sockets that are Unix sort of things. And you can say ton loads of parameters for the sockets like TOS fields and these kind of things. It is a lot more useful. Then we kind of reply to Cron on App. Cron being, I just talked about Cron before and App doing something very similar. Actually just two weeks ago something I actually implemented the calendar time support for for time archivation. So yeah if you have an app added system and don't actually need the functionality that actually real Cron provides to you nor the interface with Cron then this should absolutely be sufficient. Like if you want to start job space and time system you will do it out of the box. Another thing that we reply is Readahat. Readahat is something that's not so interesting on service but it is kind of relevant on desktops and an embedded hub or some embedded setup. It's basically something that optimizes food speeds. It basically, it starts during food and watches what part of the heart is actually accessed during food and then in the next food it uses that data collected and will load all this data into memory very quickly at food, like very early and then if the the food process actually needs that data already in memory. So it's a, it gives you like, depending on what kind of part we have it gives you like 20% of the food up. And we said, we looked at that and they were like probably like five different implementations of it and they all sucked in different ways and then really hard in different ways. Like for example some of them used the audit framework which is something for Securities to watch the files and system accesses which kind of broke audit if you enable this functionality. It's crazy. And other things used, yeah. It was all awful and we looked at that and said, okay, Readahat is actually something useful to have because even on modern hardware if you have SSDs and things like that it actually speeds up your food and there's a sufficient large number of people who actually will use this like amateur people and investor people. So we said, okay, we're moving implementation of that into system B. It's nicely integrated. It started at the right time in food very, very early on and collected data and will just work. Of course, this functionality is completely optional because on the server you don't need that. You can just compile time without like you remove it at compile time or remove it at runtime. All right, mention this. We replace content like session tracking, user-set tracking, MC tracking. This is all done by some component called log-indy and system B. It's also an optional component because if you have an embedded device where no user ever can log into because I don't know how would you log into your refrigerator, you can remove that part. But we said it's common enough which would be in system B so the people who need it can just use that and it's nicely integrated. We'll work and interface with all the rest of it. Then earlier this year we introduced something called the journal which can replace Syslog in a way. Talking about the journal probably could be another talk, or hours that we talk about that. So I'm not going to cover too much about that. But yeah, basically we looked at Syslog, figured out there are huge problems with the current Syslog because I mean it basically was designed in 1983 if something hadn't changed at all in its functionality. It doesn't do anything what people probably want these days. So we came up with a journal which can take the same input as Syslog including it's compatible to a certain degree with Syslog. But it provides you a lot of different functionality, much more powerful. So we added that to the journal because we inherently believe that collecting logs of services is absolutely essential about service management. And system D primarily is a lot of management. More specifically this means if we watch services dying and can we start them then this is only half of the equation because if you restart a service you need to give the administrator a way to figure out why it died so that he can do something about it for the next time. And this why question you can only answer with the logs. So we came to this conclusion that there's no way around it. Collecting logs and handling logs and indexing logs is absolutely essential for service management. It's just one asset of it. And keeping this entirely separate doesn't scale because I mean classics are not indexed and it's not trustable. So that's why we moved that into the player. Of course moving this and mentioned is never exclusive so people can continue to run the block. Here's a simple one we added watchdog support. And hardware watchdog is a really stupid thing but basically it's something that has the ability service and things like that. It's an embedded device. It's a small device. Even my laptop, even the snapbook that I'm using there has a hardware watchbox device these days. It's a very simple device that if the software doesn't ping it every, like it's configurable at the time but let's say every minute, then the hardware will be automatically reset and rebooted. So the idea is that if for some reason software hangs, the machine gets automatically rebooted and becomes usable again. The user space part or like the software part of watchdog support is absolutely trivial. All you need to do is wake up every now and then and send one or the other to the device. And we figured out, okay watchdog support in a way is absolutely essential to many, many use cases like GAMP service and embedded and all that kind of things. Not on the desktop. So we said, okay, executing one IOC is probably something we should be moving into system. Especially since we recursively actually need to support watchdog support services as well. Meaning that if for example a patchy stops responding, we probably should restart a patchy as well. Not just when a patchy is dying. And if you do that for the individual services so that system supervisor services does watchdog for them, then you probably need something that also watchdog system be and we can use the hardware for that. So yeah, we said, okay, we can just move this watchdog support into system. And it's trivial. It's like five lines of code. And yeah, we'll cover a lot of use cases. It's much nicer. Yeah, ACPID. ACPID is a program that whose only purpose is that when you press the power button on your desktop or on your server the machine shuts down. And if you, yeah, and if you press sleep button that's the only purpose of ACPID. And that's an absolute trivial thing to do. Absolutely trivial. If the key press comes in you shut down the machine. There's nothing about it. Traditionally though, people always have to install ACPID. Like a demo for this absolute trivial thing. Another thing is the inner system already controls like if somebody press control it reboots on its own. And that's already handled by the inner system anyway. So the question is, why do we actually need an individual demo to react to power presses if we already have one that's way more complicated for reboots inside PID 1? So when we said, okay, this is ridiculous. Everybody needs the handling of the power key. So we moved that one line down. So my time is over basically. And we moved up to a couple of other things like power management stuff. This is now the system D. Parts of auto fast. You let this move into system D handling of crypto devices and a couple of things and we innovated a couple of things and yeah, we have a couple of other things on the line. But this is basically the last thing I want to talk about. Where to draw the line? If people look at this then of course we moved a lot of things into system D and the question is, well, that's a lot of things in there. Isn't system D bloated? Because I mean previously we had so many different components that did different things and now we moved that all thing into one and then this one thing grew large and so it's bloated. I think that's a complete bullshit thing to say about system D because I'm pretty sure that system D actually works against bloat. Because it doesn't actually move everything into one single process but it just changes the distribution model how we do these things. Like instead of having 20 or so or 100 or so different packages it just moves a lot of them into one package and it moves a lot of duplication of code. Like if you look at, for example, a cron and look at INED and look at classic system 5-init and start-stop them and you realize that most of the stuff they do is actually just boring processes and all of them have different functionality like some of them can set alignment variables before starting something and some of them can do some other properties when starting them. But yeah with system we said okay system you should be really good at setting the environment up for services to run and we want to regardless how it's triggered and provide the same powerful functionality regarding resource controls, security and all these kind of things. So we said yeah let's unify all this, let's just teach system D to have a couple of different triggers without a service and remove all the duplicate code that spawns services with various amount of configuration. So yeah basically if somebody asked me if system D is loaded I would tell, hell no, it removes the application. It allows you to build your system for much fewer packages and yeah. So I don't know, of course different people have different ideas to draw the line. We still think that we haven't reached the line yet where we should stop moving things in the system D. But as we started out with just saying system should be an in-system and nothing else and now it's changed a little bit positions saying it's a CoroS I guess it's a little bit of a dynamic thing. But yeah anyway it's an interesting question to figure out where does to draw the line. But anyway this is all I have for now. Nobody asked the questions so far. Please do that now. Do we have a mic? How does it work? The first question I want to ask is about the adoption. Who is using this? Which distro is using this? I used this on Arch Linux for quite some time. I used system D on Arch Linux for quite some time. Which OS is using it now? Which OS will move on to this? So what is the future? So basically all distributions have it. All the big commercial distributions use it by default like meaning Fedora that uses it RAL7 will use it, Magai use it, Madriva uses it, Frugalway uses this Arch Linux by default. Debian again to have it in the distributions but don't use it by default. I mean they're kind of very conservative distributions and they're very slowly especially Debian that. So it would be surprising they already switched. Basically the big one outlier is Ubuntu. They're stuck to Upstart which is their own thing. And they maintain that and they don't have that and I do not expect that to change hands. And then there's of course all the weirdo distributions like Slackware and I'm never going to switch. But anyway basically all the big commercial distributions did and all the community distributions at least have it. You started developing this replacement for init, right? Then you made choices between things like in-getty, a-getty or between signs. If you like any one of those you adopted them or you just replaced like the thing which you did with ACPID. So this is applicable to the desktop environment you're going to choose one of them or you're going to do your own. I mean to clarify that we didn't even make the choice regarding ACPID you still can run ACPID and turn off this power button handling in system D. So we will not make things impossible. This is what I meant with this exclusive exclusivity thing that even though we provide this functionality in system D, we will always allow you to continue to run what you previously did. We will just have this nice suggestion by the way you can just use the built-in stuff that's much easier and it will just work out of the box. But you can still run any kind of, you can run min-getty if you wanted. We will not stop you from that. I mean there's even documentation on the internet somewhere where people figure it out how it works. And for the desktops of course it doesn't have no implication at all like D or GNOME or something like that. It's completely irrelevant. This is all low-level stuff. We do not make decisions for you. We basically just say, we make suggestions to you. We say if you use that one we have tested very well and we support it very well and if you have questions about it we can answer them right away. But so basically we don't remove the variables entirely. We just say, yeah this is the option we think is the best one. But we don't want to force anybody to do that. I listened to your talk and I couldn't help but wonder, most of this philosophies you talked about. Some of them are slightly conflicting. I was wondering about the future of the project. I mean this project needs really good BDFL or somewhere to measure when to draw the line or when not to draw the line. Can you discuss the future of this project? The question is always where to draw the line and then the approach to that is basically to decide on the individual case. I don't know what the future will bring and there are certain things that we believe is absolutely essential to the basic building blocks of the operating system where we say, okay obviously this belongs to the system. There's some case where it's obviously not, for example being UI or anything like that. It's definitely not on the card. And there's a lot of things that are in the middle and we will decide on the individual case. To keep this in mind insistently, even though it gained a lot of modules almost all of those modules are actually optional. There are very few that actually are required. The only thing that's required is basically PID1. You have to build that and you have to build UDAP but you don't even have to run UDAP and you have to build a journal. That's basically it. But all the other components like the replacements for the unit scripts most of them you can totally disable and that support you can just do that at configure time. So even though we have a lot of things in this build tree and you build if you do not specify anything with all of it it doesn't mean that you have to use that. We just say well this is really nice integrating our stuff and shares a lot of code so it moves a lot of complexity and a lot of duplication. But again if it's totally up to you to use that and in embedded cases for example I expect people to turn off a lot of it. So whatever we add it's very likely that things are going to stay optional. We will have to figure that out in individual cases but the thing is like the systems change a little bit how they used to be and how they are nowadays but basically everything that probably the vast majority of people will need to build an operating system might like and it is lower level, low level and that probably belongs to somebody. But again, individual cases. Yeah, hi. So looking at your vision of I'm here. I was looking at your vision of integrating some of the utilities that run in a distribution and my question is how do you see this effects say updates and other things that from the point of distribution. So today for example if Cron has a bug and getting out a new version of Cron takes is quite fast. But when you're moving into a monolithic app which does a lot of stuff. It's not monolithic. Okay, I mean even if it's modular so for example if you integrated Cron functionality into system D and a bug comes up in Cron does it require a new version of system D to be released for the maintainers of a distribution to pick up? I mean if there's a bug to fix then there's a bug to fix and it really depends on distribution how they package system D even. Like for example if they are concerned by that they could split up system D in a lot of individual packages and then they could individually update them. But honestly I mean looking at the bug history so far I mean we don't have that many bugs and currently at least in Fedora it's completely sufficient to do one update every two months or so at least the distribution is even more selling than that. So it really depends so far there's not been a problem at all with that and the bugs that were there usually were not the super critical ones. So I have a fatty positive view on all of this. So but yeah I mean if the thing is like system D like the system D RPM is not huge. It's a couple of megabytes but it doesn't really matter if you update all of system D. But does it cover all the services that you mentioned right now? I haven't actually looked at system D in Fedora yet. Does it handle Cron? I mean many other things that you're mentioning can be pulled into system D once that happens in the future. Then normally when there is a package for example we have seen it in KD something a small component of it has a bug and there's a lot of resistance to come out with a new bug fix for KD because it changes the bug fix as the versions have to be kept compatible and the modules are out there and there's a lot of resistance for coming up with a new version when a small component has a bug fix. I mean I'm not entirely sure I understand the question but you can update system D and it's not that big like if you have a bug then update system D is totally fine. But the thing is like I think actually that updating things like this is much easier because you get the test matrix down. You just apply the patch to system D build and you don't have to test this package against all the other packages. I mean most of the distributions allow you to mix and match not only different implementations of things but also different versions of things so you can run basically system 5 in it against any kind of Lipsy and these kind of things. But basically I don't think that's much of an issue because the package is not that large and it's completely up to the distribution how the package is so exactly nothing changes from the previous thing. None of that makes sense. I mean right now in Fedora system D is split up into four packages or something like that but there's probably plans to add a fifth or something like that and I think Debian splits it up and more but it's really up to the distributions how much they do. But I mean the other thing is like one should not assume that Cron has actually a lot of code. I mean just think about what Cron does if it actually executes something based on time that is not really rocket science. So they actually like when I added calendar support to system D calendar time support like calendar based events this was like I don't know 200 lines of code for parsing calendar specifications and that's kinda it. That is nothing. That is really nothing. So yeah there's a surface where where bugs can be gets smaller system D because you have so much less duplication because we now use just one code password and then have a little bit that's just time handling instead of having the code has for responding things in Cron and ATIN and so on and so on. Hope this can help. Yeah so Linux is traditionally been a loosely coupled subsystems which we alluded to as you know the Bazaar model. And you said that you know there are advantages and I see some of them in moving more of them in a single atomic unit but what are the disadvantages you see with that because especially when it fits in the distribution model or you know about the lifetime of those packages which are probably also being worked on in parallel by different groups. People see different like of course one of the disadvantages of moving everything together is compatibility because it does work differently than Cron did before. We try to deal with these issues by saying well if you need to compatibility with Cron then install Cron that's what I meant with this non-exclusivity that people can just continue to use what they want. But basically yeah it is about compatibility with the problem and then doing things new of course always comes with a learning curve. People know how things used to be and now they are slightly different. That is a problem absolutely. I don't think it is a big problem because we really try how to make system D as easy to use as possible. But yeah I don't know. There are other drawbacks of course. I think the net result though is that integrating these things has drastic advantages because it simplifies the code unifies the code and reduces the test matrix radically. How do you what is the biggest advantage you are saying to move to system D One day I moved to system D. I don't have Cron Syslog works before I mean Syslog is older than me. It works well Cron works well. So you are basically a big move right? To enable like Syslog it was just one line of configuration somewhere. To enable journal CTS it was like one huge command like system CTL or journal or something. How do you plan to move all the people to system D because it's anyway a very new thing. Almost all the commands change, configuration change and what is the biggest advantage you are saying to use this and throw away everything you know. I don't know. The thing is like I don't really try to move people to system D necessarily. Whenever we add something we say this is the offer you have but continue to use Cron if you want to. To convince people to actually use this stuff is basically just by providing very good documentation and providing a lot of functionality that didn't exist before. For example in the Cron case we will probably publicize a little bit more that we can replace Cron these days by pointing people to new features that we can do in system D that Cron can't do. And there's a lot of things to that. For example we want to add to Cron that you can specify things that you start even if the machine is off and then we use these wake up timers where the hardware actually boots up and then can have timers that can unsuspend and then start the machine if you want. And then for the cloud people we want to add some things that implicitly to every time event you can add a configurable jitter so that if you have a cloud like a cluster of machines it all start at the same time. Not all the Cron jobs are run at the very same time and clogs the network at the same time so that you avoid the DOS thing a little bit. So if we add this jitter which is relatively easy to add then this suddenly becomes interesting to cloud people. And there's a lot of things like that where for example we can integrate anacron and these kind of things and already actually the system D Cron replacement language is a lot more powerful than Cron itself is in very specific cases for example in system D you can express I want to start something on the first, second or third day of the month which is a Tuesday or something like that which you cannot actually express in Cron. And in system D you can express I want to start something on that very specific data and actually specify the year for it and you can't do that in Cron. So all these things are mostly smaller things but I think we can make this interesting for people by just saying yeah if you use that system D build and stuff you get something that is drastically more powerful than Cron used to be. And then there's lots of other things to that like system is actually more power efficient because the way traditionally yeah I think I should totally but yeah the thing how Cron traditionally used to work is it woke up every now and then figured out if there was anything to run, executed if there was if it didn't it would go to sleep so it would do this definitely every minute. Now in system D we never do that we actually wake up when we actually have to wake up with this program with timers now waking up every minute by itself is not expensive because like you wouldn't ever notice that the power graph but it suddenly starts to matter if you multiply your machines if you have containerized machines if you run 500 machines in one and every single container wakes up every minute or so then that basically means you wake up more or more than one second of the machine all the time and then the container set up and if you have this in the cluster the same thing happens. So basically yeah we are more power efficient we have a lot of things so we try to make this more accessible to people by saying we have more features and yeah it's a matter of telling people but I'm way over time I think and so thank you very much and I have a good lunch and yeah I hope this was interesting because a little bit of an idea what we trying to do with system D and what our vision behind it is just a small announcement we all haven't heard of NH7 already one of the best indie music festivals in India the festival is 15-16 in Bangalore and they are kind enough to give us 10% discount so please head to the the fostered install which is the right of the entrance and we are also giving our delegate kits at the swag area please don't put