 So, good morning. I think it's the morning. It's probably the first time I have questions before I even start the talk, but that's a good thing. It shows there is interest, I guess. So, yeah, I'm the co-maintainer of Geeks together with Ricardo, which you've probably seen this morning, or you should have seen maybe. I'm going to talk about GeeksSD, which is a free GNU Linux distribution, and about how we manage system services in GeeksSD. So, I'm pretty sure the Intotubes has already started making fun of me because I'm supposed to be the maintainer of a distribution and I'm just claiming that I recently discovered what a system service is. I'm pretty sure people are making fun of me. But anyway, so, yeah, first of all, let me maybe do a recap of what GeeksSD is. How many of you have a feeling, like how many of you have used it or read about it? Okay, 100%. That's pretty good start. Okay, so, well, as you know, GeeksSD is that free distribution where you can basically declare what your operating system is going to look like. So, that's different from most distributions we're used to, like, you know, Debian and all the traditional distributions because instead of, you know, logging in as root, installing packages and fiddling with files in Etsy, you go right from the start and you write this configuration file and that's it, right? That's how your operating system is going to be. So, previously John Darrington talked about the graphical installer for Geeks which is just brand new and until now, you know, you had to, when you were installing GeeksSD, you had to actually write this kind of configuration file or start from a template and actually instantiate it live from the USB installation image. So, what is it telling us? Well, here we see a number of useful pieces of information like we have, you know, the local time zone, user accounts, file systems and so on and so forth and we have this last bit here which is what I'm going to talk about which is system services, right? So, that's usually the most important part, I would say, maybe of the system configuration and it's also the most difficult part, the thing that is pretty hard to get right, right? So, we have sort of conflicting requirements here because for GeeksSD we want to make it as simple as possible for users to, you know, choose what system services are available, what options they use but at the same time we want to be able to handle pretty much any kind of system service. So, the thing is we want to be able to have essentially just one line per system service in the configuration that people actually write but it should be able to handle complex, you know, service compositions. So, I'm just showing briefly what users see and what users actually type in in their config. So, here we have two additional system services. So, we're saying I want to use, you know, the basic system services like, you know, login, console and so on and I want to add a couple of services to them. So, the first one is a DHCP client so I can get networking and the second one is the SSHD service, the SSHD man. And I'm just using default configuration values here but of course we want to be able to, you know, to specify options to that so let's say for example for OpenSSH I can just write this thing here and, you know, if you're not familiar with Scheme, that's okay. You can guess what it's going to do, right? We have an OpenSSH configuration object here that's being instantiated and we're saying I want to set this particular field X11 for writing to false and this has a field to, you know, that thing and that's how you customize services, okay. But sometimes also you want to be able to say, okay I want to use the desktop services which is to say that all the services that you expect from a desktop style installation which is, you know, a graphical server maybe GNOME, maybe that kind of thing, you know, networking and all that except that there's one thing I don't like so I want to remove it. How many people are familiar with Scheme in this room? Oh, yeah, that's 90%. Okay. Well, the thing is this is Scheme, right? And you have to know that remove is a procedure that, you know, you give it predicate here and it removes item from the list which is a second argument that's essentially the story. And so lambda just means this is a function, right? It takes one argument which is a service and if that service is a service of type NTP, well that's the one I want to filter out, okay. So here I'm just saying, okay I'm removing the NTP service from my machine because it's in this list of default desktop services but I don't want to have it, okay. So this is just standard Scheme, we're just manipulating values which happen to be services, okay. Sometimes you want to be able to do more complex modifications like again you want to be able to use those desktop services because it's pretty complete and so on but you want to change a few details. So let's say you want to change the way, you know, the mean getTTY login thing welcomes you like you want to change the message of the day. That's something you can do this way using the modify services thing and here we're changing the way you power it so the power management service, you know, handles various events and that's something we are doing here, right. And again, this is still a simple Scheme list of objects, service objects and so we can have variable holding that thing and so on. So if you're used to Puppet or Ansible, I know the person sitting next to me right before is presumably using Puppet or maybe hacking at it. Well, you know, this kind of declaration is not very different after all, right. It's just you specify all the details of your operating system and then you run Puppet or Ansible or something and you get your system which is going to instantiate roughly what you asked for. But there's a pretty big difference I would say between GeeksSD and Puppet which is that in GeeksSD this is functional in the sense that you give it a configuration and you get an operating system instance, right. And you can do that, you know, anytime on any machine you'll always get the exact same result even bitwise, modular, maybe a couple of packages that are not bit reproducible, right. So that's a very strong kind of guarantee. Whereas if you're using Puppet, well Puppet is essentially trying to modify the state of existing machines by running new commands and so on. And so if everything goes well, you'll end up with pretty much a state that you asked for but in some cases, you know, you might have problems. I don't know, maybe we can discuss it afterwards but yeah, it doesn't always work from what I heard. So that's the story. So what do we do with this configuration? Well, we have a bunch of commands so we have the GeekSystem command which is a central dispatch kind of command for GeeksSD and essentially we always provide one of those configuration files and then we can do a bunch of things with them. So first thing is we can build the system. So let me just show an example. Where is my shell? Oh, okay. Yeah, it's gonna get better afterwards. Yeah, so here I'm just running Geeksystem build some file and what I'm getting here, I'm just getting as a result one of those big directory names that you should be used to by now with a hash which is actually hidden in this case. I can show it just for the sake of it. So this is a big hash, right? You know, it doesn't seem very useful as such, right? But what's in there is that there's actually everything that defines the system instance. So if I jumped into that directory, I see a bunch of entries. So the first one is boot. It's actually a boot script. I probably don't need to go into the details here. There's each C which is the systems slash each C directory and then you have the initial run this to kernel and the number of other things, right? And again, you can run Geeksystem build with this particular configuration anywhere you always get the same result. I can also do things like Geeksystem VM and if I do Geeksystem VM, it's gonna do essentially the same thing as Geeksystem build but in addition, it's going to build a virtual machine image that runs the system I declared. And so what I get as a result here is a script that runs QEMU for that VM, right? Yeah, just to show I'm not cheating. Yeah, this is the system that's booting. Trust me. And again, if I want to instantiate it in the container, I can do the same thing but with Geeksystem container this time and I'm gonna get a script that spawns the container, right? So I have one declaration, operating system declaration and then I can instantiate it in different ways and of course I can instantiate it on the bare metal but I'm not gonna do it live because you know. But I could do Geeksystem reconfigure and I would essentially switch my system to that new declaration. So what happens at runtime? So we have this wonderful operating system declaration thing and at runtime when we boot GeeksSD what we get is of course Linux Libre so Linux without the proprietary blobs and then we have an initial RAM disk that initializes like mounts a root file system and it's actually written in guide scheme and then we have PID1. So PID1 in our case is the GNU Shepherd which is a very little known piece of software I guess. It's an in its system, right? So we're not using SystemD or RUnit or one of these we're using the Shepherd which is also written in guide so I show where it's interesting and finally we have applications which may or may not be written in guide. Yeah, some of them are still not written in guide, that happens. Okay, so before I've shown these things so what you as a user will enter in your operating system configuration and now I'm going to talk about the developer side of things, right? So how do you as a developer, as a GeeksSD developer define a service? So roughly the idea is that you specify a Shepherd service structure which has documentation requirements so you can say what other services it depends on and you can say how to start it, how to stop it, right? And here we have this hash till thing if you're familiar with some sort of list you could think of it like classic quote which means essentially introducing code here little code that will end up in the Shepherd itself so it's not being executed right now we're just producing code for execution at runtime, right? And so this is the code that the Shepherd PID1 is going to execute when we start my SQL so we're essentially using the API of the Shepherd here so we have this make4ksec constructor thing it's a bit of an obscure name but it's been there for ages so that's how it is which tells well you know to run my SQL you need to call this command with this argument and this needs to happen as this user that kind of thing so that's the basic idea but the good thing about Shepherd is that it's written in Scheme so we can actually inject Scheme in it so if you think about systemd, systemd is written in C and so when they want to add new features to systemd it actually has to get into that C thing so it's modular but still you have to get into that C thing it has to be within systemd whereas here we can just say well the Shepherd doesn't know how to mount file systems for instance yet it's a useful thing to do right so if you're familiar with systemd systemd has this localfs.target thing to basically say okay I'm mounting all the local file systems well we have something similar here we have a file system service and it needs to be able to mount file systems so we happen to have a module in Geeks that provides bindings to the C library and in turn that allows us to mount file systems so we just want to use this module here and so we say okay I'm going to import those modules for the Shepherd and then I can use my mount and new mount procedures provided by this module so essentially without touching the code of the Shepherd we've added new functionality to it right we've just imported the ability to mount and then mount file systems without actually changing it so if you look at the Git repo of the Shepherd you'll see it's quite inactive actually and one of the reasons for it is that well we can do things without actually modifying it so that's pretty cool now for another example so this is the typical service for a demon with the start method to launch the demon so B2B is an IRC gateway that I'm using on my laptop I don't know if you're using it it's pretty cool so B2B is software that needs to talk to the network obviously right but it shouldn't have to talk to have access to the file system like slash home and so on right so we've been discussing like for a month that we need to be able to showcase where it's so cool to have the Shepherd in scheme so the previous example I gave about mount and U-mount you know that's pretty cool but still kind of trivial right and so while on the train to foster them I thought okay let's make it happen so the thing here we want to be able to run B2B in a container so that at least we're reducing the attack surface well we happen to have a container module in Git itself which is used for other things and so why don't we just import it for that definition inside of the Shepherd so that we can run B2B inside a container well that's what we're doing here and it's a world premiere and so essentially what happens here is that we just added this slash container thing so we're using a different procedure to start the process but this procedure is not defined in the Shepherd itself it's defined in our own module here and it actually works as incredible as it may seem so this is my VM okay I'm afraid this is not super readable but anyway the thing is I can use the herd command to ask the Shepherd about my B2B service and it's telling me that it's running as PID 476 and so if I get into that container of PID 476 using Geeks container what's the story? Geeks container exec you know so I'm using Geeks container exec 476 also to change it to a white background of black writing I'm afraid this is QMU I'm afraid this is not possible okay but I'll just tell you what's happening on the screen essentially we have a shell that's running inside that container and if we look at slash proc well, slash proc slash 0-9 we see there's only one process inside that container so it'll be running as PID 1 inside that container that's a story okay so that's it for the world premiere so far I've been talking about services that are typically demons and that use the Shepherd to be spawn that's the main use case so I'm going to talk about the story of services in Geeks SD and how it went from sucking to being pretty cool so let's start with our first take on this so initially the focus when designing system services in Geeks SD was like well all we need is something like let's say system deunit files all we need is to be able to describe dependencies among services we want to be able to say okay first thing I do is to modify systems and then there are user processes and then there is syslogd and there's a bunch of stuff up there that you don't need to worry about but yeah that's the main idea we have a dependency graph this graph is actually generated by a command which is Geeks system Shepherd graph so you can give it a config file and produce the dependency graph for Shepherd so that was the initial take on this so essentially the model was well you know you have a daemon and you just write service start stop and you're done right well except that there are some cases where you know before starting the daemon you need to do some initialization right like you need to create volley mycquail something like that right so we added this activate field to our service structure to handle this case well it turns out that you know for some daemons you also need user accounts right so yeah like for Postgres you need the Postgres user account so we added these two fields to our service structure to handle this case and yeah turns out that for some services you also need PAM service declaration you know pluggable authentication module service declaration and well for some services you also need files at a precise location in slash etsy so I think you see what I'm getting at right this is not very modular or extensible right we ended up adding fields in that service structure and it didn't feel quite right although it did the job for a while and then it was like last year or maybe a bit more than a year ago Andy Wingo came to the mailing list and said ok this is the school but I want to be able to run GNOME ok fine and to run GNOME we actually need the whole free desktop set of services well I was like let's just add them right ok so what do we need well we need you dev and deburse to start with ok that's simple we have them and then we need things like Upower for power management and Udisk ok but deburse needs to know about Upower you know it needs its dot service declaration thing that allows it to know that Upower is a deburse service and same for Udisk so there is some sort of a connection between Upower and deburse but it's not like just you know a demon in the usual sense it's something else right and Udev also needs rules you know these dot rules files from Upower and Udisk or just Upower I don't remember anyway it needs info from those packages so it's like ok we can hack something we can manage it so at this point we are still fine right and then Andy kept explaining well turns out we also need Polkit, LoginD and well it would be nice if we had this thing called Colody which is for color management and Geoclue which is a deburse service for geolocation used by some applications and well I was like yeah ok well how does that work right you know there are these arrows these are not arrows like in dependency in a shepherd dependency graph it's something else right there are connections between all those services but it's not completely clear what these are right it's about you know passing files and pieces of information between services and so at that point we understood that we were essentially screwed right we had to do something about services in Geeks SD and to me yeah it really felt like this example in particular it felt like spaghetti right but at the same time we knew that we had something to do and that's how it all started so nowadays in Geeks SD we have composable services and we can handle this kind of situation and we actually have GNOME running and that kind of thing so the key insight here from this free desktop example is that services you could think of them as extending each other right so like color D you know color D provides a dot service file to D-Bus right so that you could think of it as extending the functionality of D-Bus in a way and you have Upower that provides Udev rules to Udev which is sort of like you know extending Udev and that was the key insight when we started redesigning services in Geeks SD so I'm going to make a short digression to talk about how NixOS does thing because I had been working on NixOS before and I obviously knew that they were doing better than we were doing right so let's see how it works in NixOS so this is how you would write a service definition in NixOS in the Nix language but again without being a Nix expert you could I guess you can understand what's happening here and it's so pretty concis because you're just defining values and those values have meaningful names right and the thing here about service extension is that NixOS seemed to get it right so we can see here that SSHD is adding new users it's extending what's in SSHD it's adding new services for SSHD all these things right so it's extending things in a way there's a difference still compared I mean there are a few things I was not completely happy about so the great thing is that it's super versatile right from your service definition you can change the whole system essentially right from PAM to SSHD to SSHD and this is the really cool thing that we want to have but the not so great thing is that in NixOS you're actually building a big key value object key value dictionary that contains the configuration of the whole system and so every service implementation received this config parameter as input and config actually represents the whole system configuration right and from there it actually produces additional key value pairs to add into that record, that dictionary and you might have been this thing also the MKIF so the thing is you can in a NixOS service you can look at the existing system configuration but then it's recursive right it's actually a fixed point because you're looking at the configuration that you are currently building and so NixOS is a lazy language but you have to pay attention to this thing because if you don't do that you might end up in endless loops because you know you're looking at the data structure that you're building at the same time that kind of thing and also my main grief is that it has good structure syntactically it has good structure of things it sees system D and so on but then that structure is purely syntactic right you cannot really tell what's going on how services are extending each other right you know that those arrows that I showed before we don't really have them here and so yeah this is what we want to have so we have this I would call it the system extension graph so we have color D again providing dot rules files E-log in no wait what is it geoclue providing dot service files and so on and so forth and we want to be able to model that graph really that's what we want to be able to do in NixOS D so we can actually see what's going on and be pretty clear about what service modifies what other parts of the system right you shouldn't be able to just change any part of the system configuration at some point you have to precisely say okay I need to change slash it see for instance but yeah so this is the basic graph form for free desktop but again if we look a bit further while we see this pattern in different places like D bus itself you could think that it's extending the shepherd by providing a service definition to start it and D bus is also extending slash it see by you know providing slash it see slash D bus one the directory so this is again what we want to be able to keep on the on the user side of things right the user should still be able to have that simple list of values that they manipulate and so we have some sort of a mismatch here because we want to have a model which is essentially a graph yet at the same time we want users to be able to manipulate a simple list of objects so how did we do that well the thing here was to introduce a notion of services which are first-class values representing services which may or may not be shepherd services and service types which represent you know classes of services so let's let's see an example okay so going back to my desktop service example this is the color D service type so this is how we define it in Gix's D we have we define a service type okay and it has a number of extensions so these are exactly those that I should be for in the graph right so it extends the so-called account service meaning that it's creating new user accounts it extends the activation service meaning that it provides code to run when you boot a system like you know creating varly, color D, something like that then it extends D bus to provide its dot service file and we're making a distinction between service types and services because you know so this is color D so typically you have only one instance of color D running on your system but in some cases like SSHD you may want to have several instances of them so likewise we have an SSHD service type well actually open SSH service type and you can have several instances of this particular type so in this way we are defining the graph that I showed before you know for all the services and we can pretty much achieve what I showed for an XOS which is that you can choose which parts of the system you want to extend and actually you know define how you extend it so for the account service extension we provide user account subject that's how we extend it and then for the color D activation we provide a snippet of code for D bus we provide a package object and then what do we have here and for pulk it we provide pulk it actions which are themselves contained in a package object so this is all you know we have disjoint types for all these things so we can know precisely what it is that we are manipulating right so it's not like this you know key value dictionary that I showed before you have to be very precise about the types that you're manipulating and things that you're providing as extensions right and then once we have that we can use that Geek's system extension graph command and we can visualize how the system services are composed which is I think pretty unique to Geeks SD and so even for a simple system like I showed before is essentially just the basic services plus SSHD we get to graph that with the you know kind of big but essentially we find the same thing things as before so at the top over there just above the screen we have OpenSSH and these pinkish arrows actually show what OpenSSH extends so we can see that it extends the activate service the account service and Shepherd and we have a bunch of other extensions going on here now if we look at that example that supposedly simple free desktop example well it's still spaghetti right but at least we can visualize those spaghetti right that's probably an improvement I guess but more seriously it allows you to reason about what's going on here kind of so if obviously you have to zoom in to get a feel of what's going on but essentially we see GNOME that's extending what is it Polkit and Polkit is doing a thing with Udev and whatever so we can right well the details don't really matter here just to show that here we have a well defined structure of services and service extensions and we can visualize it that that's the story so at each step on this kind of graph we are manipulating different types of objects so if we go to Polkit for instance Polkit is concerned with policy rules whereas Udev is concerned with Udev rules and the account service is concerned with user accounts right and at some point it has to be folded into something that's slow level that's close to what you have on disk and this is why at the bottom here just below the screen this time we have one node which is the system so you remember when I run gig system build I got one directory as a result well that's this node right here at the bottom so the system is a service that's itself extended by all these things so what that means is that we need to be able to fold values and to sort of compile them from this high level like user accounts Udev rules into something as low level as files on disk so how does that work well there is a fold services procedure that I don't know if I'm going to be able to present maybe yes maybe yes okay let's do it okay so Alex showed you the Gazer and the Ripple and all that so you're probably super familiar with it now so essentially I'm running Guile here and I'm saying okay I want to load my operating system configuration like this and I get an operating system object and from there I can do like I can query it like how many system services do I have in there well I have 27 of them fine and now how do I actually compile these to those low level files on disk essentially well I can use these fold services thing it's not in my history so I can say operating system services if I give no arguments then I'm getting as a result a single value which is the system service right and if I look at the parameters field of that value it's actually well the system itself right the derivation that be the system but I can also stop at intermediary intermediary levels so I can say okay this time I'm specifying a target system type let's say idc service type I'm getting this huge value which actually represents all the files that are going to end up in slash idc right so I can do some sort of a step-by-step compilation of this high level declaration to what's going to end up on disk any Haskell are in the room so this is a monoid I think so just to conclude I think the take away message is obviously that this you know we leverage a holistic approach to system services so more seriously the initial idea that essentially a system the unit file is what a system service is turned out to be flawed right system services are more than just you know actions to start demons stop demons stuff like that it's more than just this dependency graph and service extensions as I showed the capture all these aspects of service configuration and how they interact with each other and it makes complex configuration tractable so there is a geeks contributor maybe maybe somewhere here Christopher Bains who has been using it at their workplace to actually build very complex kind of things like web services in rubies that extend the engine service which in turn does crazy things that I don't really understand but yeah you know like complex configuration like though that you have on web services you can handle them in this way and you can still reason about them and find out and that's pretty cool I think and so you should come up with your own system services because then the the limit is imagination right I mean you can you can think of very high level services like you know the like I don't know GitLab service or you know something that provides a big service and in turns you can decompose all the details of what's going to happen and that's pretty cool so you're welcome to join us in having fun with Geeks SD and that's it any questions yeah so it's really interesting like the kind of services graphs that you were showing they basically execute this with a geeks command right so it's basically they're generated on the fly from a from a description system description right it seems that that is a really interesting way also for all that like complex software for this to try and like just to visualize their dependencies in a visual way yeah yeah thinking of my work environment of like providing web service that depends on on NGNAs and on my SQL databases so it would be really nice to see exactly all the things that are involved in the dependency browser right so yeah so just to repeat your comment is about visualizing graphs in general and how you combine things right I think it's pretty useful and actually I was really jealous of just yesterday in the her talk because I would have loved to have a D3.js kind of thing where you could you know show how you compile services down to the final value next time maybe are there questions do we still have time yeah yeah oh Sandra yeah so I'm actually curious about the products so what do you think we have a reasonably complicated configuration and you want to build that because it's faster, it's slower and also about for example the memory configuration yeah so yeah so the question is about memory and CPU usage when building a configuration and a complex configuration so I think I don't have any concrete figures and also I've never really toyed with configurations that are really that complex I mean a desktop configuration already has some complexity but maybe not as much as the web services I mentioned earlier so you've seen it in action it takes like I don't know maybe a couple of seconds at least where things are already in store I haven't done any memory usage measurement it's not not super efficient probably but still not maybe yeah I remember with large nix-west configurations you could have huge memory hags I think it's probably less bad but again I haven't done any measurements so I can't really tell still have three minutes yeah other questions or maybe a break okay thank you