 Okay, cool. So, hi, guys. I'm Alvaro Leyva. I'm a production engineer at Facebook and Instagram, and I'm here to talk to you about how about we use system D in high-level languages. System D is already written in C, so that's already a high-level language, but I'm talking like higher-level languages, like Python, Node, API, sorry, Node, JavaScript, PHP, all those languages that are not really, like, nuance and stuff. Okay, cool. So, basically, this talk will have, like, two portions. One is how we, as Facebook, did a library that talks to system D directly instead of shelling out, and then a few fun things that we do with it. The idea of this presentation is not to say, hey, guys, go and use our library is more on the sense, like, hey, like, this is all the things that we learn by creating this library. You should all go and create your own libraries in your own languages. And if you also want to do one in Python, that's the one that I'm going to present, go for it. Like, this is really fun and this is something that we can all learn from. So let me start with the initial journey on how we start with this. So, this started basically when we were at Instagram. We needed to migrate all of our fleet to Cento7 because that was part of what Facebook was doing. Davida has talked about this for two years now. He promised this that we, he promised in this year that we have already done this, and we were one of the last groups to do this. And the reason why it was hard for us is because we had a lot of maintenance scripts that rely on the underneath service manager. One of the things that we typically do is that we would execute stuff like ETC, ENT, Cassandra status, and then we parse the output, and then that's how we decided if Cassandra was running or not running, and all the things. So all the things that were not really, it's not that we're not possible, but that was not a good idea to do it with system D. And for good reasons. So, for instance, system D comes with system CTL status, which is really great for humans, but it's really hard to parse the output. And system D does provide a lot of toolings to do this from the command line. Even system CTL has a code status that will kind of ask you individual fields. You can also use bus CTL and all those stuff. But if you are in a high level language, again, I use this as an enonym for Python, to do this, you have to chill out. And at least for my personal experience, telling out, I consider it a little bit clowny on those things. So it would be really nice to have a really nice and native interface to talk to system D. Yeah. So it basically all starts with divas, right? So from an external developer, not a system developer, but from an external developer that wants to interact with system D, this is for me the best decision that could have ever gone into system D. Like, using divas as a way to interface with system D. This basically allows external services to communicate through the statement and get all the information that you need from it in a way that is not as clowny or hacky as like executing a command, getting a text out, parsing the text, and then if that text is supposed to be like a PIV, convert that into string, sorry, convert that into an integer that has its own problems. So it was really kind of a surprise to me when I started working with this that this is also how system D most internal tools work. And it's kind of because we all know that divas, the divas demon is not present at every stage of the, of the, of the machine. Like, it's not really present at the beginning and at the end, shut down and start. But for most general things, this is how you communicate, this is how even system D tool communicate. So, first of like, if you go and read system D code, which I'm sure that everybody here has done it, you will like see all this type of code. Like, system CTL, communicate to the manager, system D run, which is something that I really like to use and have used. Also communicate through divas to start the transient units. And even machine CTL, like instead of telling out the changing namespace or doing other things that you could do, just communicate to the divas demon running inside the machine. I'm pretty sure that I'm going to get corrected about this, but at least this is how I view it from the outside. So, divas is like really great. And it's something that, that we could really use. And in general, like, if you forget all about this and you need a really good IPC layer to communicate, like use divas. It's really reliable. I would expect people can tell you otherwise. So, since everything starts with divas, if you're going to interact with system D from a high-level language, you should literally just start with divas. So, let's see a little bit how divas things work. So, system D comes with a nice tool, that is bus CTL. Bus CTL, when you execute it without any comment, it will just show you all the services that are running on your system. And these are all things that you already know since system D1 or hostname, that is called the reverse domain, something. Sorry, I black out on that. Well, basically, these are all the services, right? If you go and explore one of the services, you will see objects. And objects, you can always look at this, like, name of instantiated attributes or instantiated objects on the system. For instance, for system D, these are all the objects that you can connect. And then you see, even though they have, like, weird names, they're all in target or service. So, if you don't know anything how this thing works, but you have only worked with system D, those doing, like, start creating units and stuff like that, you will start to see, like, there is a certain resemblance. And this looks like a really programmatic way to access this data. So, if you go and actually do an introspect into this, you will see that it has interfaces that have names that are really similar to what you would expect them to have. Like, if I'm inspecting the service, a service unit, you would expect that I would have R.free desktop, the system D.service. And then inside those, you have methods and properties. And I'm pretty sure that this is really known for everybody, but at least when I started this, this was all new. So, I think it's at least worth mentioning. Method is just a fancy way to say functions. It's something that you can give arguments to it. And properties, it's, again, it's another fancy word for just saying a property is something an attribute that your object has. And it can change through times. I always think of properties like functions that you call without arguments, even though that is not true, but that's a really mental model that I have. So, the thing that makes a divas object that you actually, that makes it really powerful are the three interfaces that are the beginning. The first interface is the one that we're going to start using and it was the one that made possible to actually create a library. It was the introspectable that allows you to, without knowing anything, except the address and the object, this will give you all the information that you have to reconstruct the object and actually interface with it. Peer and properties are things that are there and are really cool that we are not going to touch really much, but you can also use it. Okay, cool. So, introspect. Introspect is a method of that interface that if you call it, you will get an XML representation of the object. And once you have an XML representation of the object, anyone who has lived through the 90s knows that you can construct anything, right? Because so did it, so why can we do it with divas? So, in theory, that's all you need to do to work with it. So, how do you call it? I'm getting ahead of myself because I'm using a system D library. There's nothing in my talk so far that talks about system D. I'm just talking about divas, but I'm getting ahead. So, basically, you call that method and then you will get something like this, that is the XML. And you see that it gives me interfaces, a list of interfaces. Inside, you have properties that has the type that it will return. It has the type of, it has a real name that you will use as a unique identifier for the property. And then you have methods. And methods cannot take arguments or they can take arguments. And then you have arguments with the types and basically the direction. If they're in, it's arguments that you give. You also have, like, the return types that is eventually defined somewhere. So, some implementation of divas also returns the name of the argument that is kind of useful. I mean, I would prefer that it were there, but if it's not there, it's not a big deal. So, we now have a way to construct things. So, when we were going to start building this library, we had to find, like, underneath libraries that we wanted to use. Again, I'm a Python developer. If you haven't noticed by the 100 times that I say Python. But so, I wanted to use divas Python because it has Python in it and it also has divas. It makes sense. The problem is that that is abiding to live divas that is really old, that is basically written exactly, like, a month later after the implementation of a specification of divas was written. So, it's, it doesn't have all the experience that we have now that we, now this protocol is pretty mature. And also, the main author of this library says, a disclaimer that says divas Python might not be the best divas pending for you to use. So, if they also said don't use it, then thank you. I will move on. Okay. The next one was Python divas, which also is based on G divas. And it was really, it was a really good alternative that I wanted to use. But with that said, I spend a few times reading learners blog and I find SD bus as part of the system de-implementation. And this was actually the best thing that we wanted to use. So, we wanted to create a library to vint to system de, not to vint to divas. Divas was the protocol. And using this thing will basically give us the same way that system de itself vint to divas and do everything. And also, it gives you a little way of cheat in the sense that I could just go to system de and read the code and understand how to do it. And then I could implement it on my script in Python and say, hey, I create something, but no, I didn't. For your language, if you want to do it, there's this page over here. There's some free desktop that has all the bindings for many languages. You should go and, again, write this in Ruby. Have fun. Okay. So, the great thing is that, okay. So, we want to use SD bus. There's no Python library for SD bus. In fact, there's no Python library to vint to live system de. I mean, no, there is not. There's few libraries. None of them did whatever we want. So, as all developers do, when they can't find something, they just build them themselves. So, we at Facebook created system de. It sounds like system de, but also start with py as every library should. So, yeah, naming things is hard. This was easy. Great. Other options were pid uno, which basically is pid one, but in Spanish. So, yeah. We did it. Yeah. Okay. So, yeah. So, basically, we did it because we wanted to vint to SD bus, but we also want another word vint to system de. We could vint to other libraries that were present, like SD demon. And all the things that David talked about in his talk, that was really cool. So, how does this work? We basically Python this library, connects to divas, get the information, and you generate classes. Nothing far of the ordinary. I'm pretty sure that everybody can come out in five minutes with code version of what this does. But here's the fun thing that you learn while you are translating basically something that was written in C into Python. Is that there are seasons or Pythonisms that you need to translate from one language to another. First, in C, there is no propagation of exception. So, you see this pattern all the time when you read code, you make, you call to a method that returns exit status and there are side effects on the arguments. And then you need to check for the return status to get, to get a, you need to check for the exit status to realize if you need to fail or continue. And then if this method is returned by another method, then that method also needs to check and then there's this propagation that you have to explicitly do. In Python and other languages where you do have exception propagation, you all you need to do is like check the exit status of the first time that you call the command that actually has this behavior and then you raise. And when you raise the exception, that will bubble up everywhere else. So, you don't, if you want to stop when something fails, you don't do anything. You just let the application or the virtual VM does that for you. If you do want to stop, sorry, if you don't want to stop, then you just try to accept your errors. So, that's one cool thing that, that you get and then you're going to see that difference if you ever look to my code that I raise a lot instead of checking. And then I try to accept a lot. The other thing is that SystemD, at least from my point of view, it's really written in a way that it looks like it's supposed to be like object oriented, but it's not because it's not. So, you usually give like this struck at the beginning of almost old calls. So, this really lends to when you are translating this into a language that is object oriented, like heavily object oriented, you get into those things. So, you have a class STBUS that should be like then you delete the first name and then you just rename all the methods and then they call self. The idea of doing this is that if somebody is read, is used to write Python, this comes really natural. And it's something is used to write C and wants to write these things, but in Python, because somebody asked him to, it doesn't have to learn so much new things, it just needs to learn that it needs to replace STBUS underscore with STBUS and then a dot. That's not that big of a deal. So, now how does this look? It's basically you install everything with bib install.pistending. Then you call it like that and it looks very simple, like if you do org.free.stop system D1, you will install solid, sorry, call it from Python and then you call the unit and then you start the unit in this really pitonic way where you don't have any information about how it is and this will construct everything and then you can start and stop it. And this thing over here encodes the idea of the interface that is completely evaluated when you call for the first time unit. Now, this is, again, you can do it, like if you don't see, you can even do it and see there's nothing magic about this. The great thing is that now for a Python developer or these developers that are high level, that they have never touch C and they also don't want to touch C, you give them like a really nice interface to the stuff that they first didn't knew how to do or that they will chill out and then do a really hacky and let's face it clowny way to get the output. So these things are cool. Also, once you call this method that it's basically automatically calculates the thing that it needs to replace, put it on the bus and everything, you also get the real return value of the thing that in this case would be a job. So then you can verify like the job through Divas and everything right tight nicely and then you don't have to do any experimentation. I said that a really thing, I said that a really thing that SystemD was really good at making the decision of running Divas. The other thing that I really think SystemD does really good is giving you output or information about your service and I'm preaching to the choir probably, but when you look at the CTL status you get all this information and sometimes you actually want this to do in your code. So you do the same thing and then you ask the main PID and again the main PID you get it as an integer because this thing is typed. But then with SystemD you, oh fancy animations, okay so with SystemD you can do you can actually like do socket activation. I mean you don't do it with socket activation but you write information from socket activation when you just get the same headers that you would get by paying them with SystemD and you do socket from FD and then now a Python developer and I'm maybe I'm selling them short but that is used to like do socket, socket and then socket.list and then socket.asset or bind. Now in his own language without changing anything he can do the things. So that's the first part and now let's talk about fun things that we can do. And a little disclaimer everything that I'm going to show you you can already do it with like SystemD run and then BatchScript and everything. I'm just showing you how you can do things that may not be the best way to do them but again you can do it from a high level perspective. So the first thing that I really like when I start working with SystemD and the things I really wanted to have this idea that I could decorate a method like for instance this ping over here that it could be like even like a normal call and then I wanted to decorate and then just run this section of the code in a different secret or even in a different process. So I actually went on and did a few things. This will all be available on the SystemD page so you have repository. Right now they are not because I read write them only for this talk. So my idea was that I could decorate this method and then make this method run without any network access. The idea of this is that sometimes you have to run like non-secure code or code that users provide. So it would be a good idea to just make this thing run in a different secret. And again this is not the best way to do it but it's a at least interesting way of doing it. So yeah so this is how I solve it. I decorate that and then what that will make it will start a unit that it just be a really big no-op. It has remain after exit true. I really know that you don't need to do this but I just put it that as an example. So remain after exit true. So this will execute this. Will not do anything and then will stay there. And then you set the type one shot. You set the type delegate to true and then you set the C group settings that you actually want to preserve. And then you start your settings. Your method. To start your method I do a little trick that is the following. In a main process I start this as a child. I fork. The great thing about doing the fork is that I can still hold all the status or all the variables and all the information that I have from the main process. So I have like if I declare a global variable it's available to this method. But I start. I wait for this thing to actually start to have a P. And then I call this little great thing that is called attach process to unit that basically will allow me to make this process. Move them to this C group or to this other process. Yeah, I assume that I said something that makes sense. So then this is what happened in the parent. And then in the children I start the process. I wait for actually to change the unit and to be running on a different unit. And then I call the methods. What this will do is two things. The first thing is that I can call this. And then when I try to ping it will not allow me to ping. Even though my application does have this permission by me changing to a different C group it works. This is not the best way to do it. But it really works good for this. The other thing is that now since this work in the different process you can literally just do system CTL status, the other process and then you will see this running. And then this give you all the information that you want. So yeah, so this was one cool thing that we did. I think, okay never mind. I assume I still have time. So the other thing that I have is that just a raise of hand because how many people here are Python developers or coding Python? I love you all guys. Okay, so I'm not going to purchase the choir. In Python you have this idea of virtual environments, right? Which allows you to basically set similar but different environment that you have on your system with Python. So you can imagine that there's an analogous concept to the idea of portable services. But having portable Python environment. So instead of you just chipping like a whole system, like a whole distribution. Thank you. Like a whole distribution you just chip like your Python code. You just chip the Python executable and delete Python. And then through the magic of just setting up stuff you can make it run in a way that is completely isolated from the Python that you have. And this is one example of how we do this. But for instance, you can just start a system. That's awesome. And then you just mount all the Python that you bring from your application. You just mount it into this environment. So you can mount literally the binary and then you can mount whatever libraries you want to preserve. Then you do a few configurations and you run and then it's a good way to actually create virtual environments on the fly in the system D, in a system D context instead of doing virtual m. So this was actually really cool. With this set you can do the same thing. It doesn't have to be Python. One other thing that we have tried is that actually instead of doing this is bash. So we can actually make, when a developer for instance wants to try, like they have their own settings that looks really similar to production, but they actually want to try entertaining libraries with the ones that they are developing. We change this to bash and then the mounts just change the location where the library should go with the developer one and it works really well. The good thing is that this works in the context of the process and then when you exit the process it goes out. And again the mic here is that you create PTY something you've been, seen a STD in, STD out. This is the last example that I will give and I think with this are we running out of time and I think I really want to have questions at the end. So one typical thing that we usually do as infrastructure people is that we need to, for instance, we have to decommission a whole rack of machines. So for that if these are databases you usually need to train them, then take the data, put it somewhere else, and run it. And the first time that you do this you create a batch script. You SSH from a scheduler, you execute that batch script, but then that batch blocks. So you block the thing and then you cannot actually run more than a few because you don't have resources to run them. So the first thing that developers do, and I've been guilty of this, is I'm going to create a service that is going to be listening and then when it gets a call I'm going to sub process something created and then this is going to start working and then I'm going to monitoring what it's doing and when it's finished I'm going to call back the main process and every time that I get that is that that's a service manager. Why would you call that if you already are running in Linux with a service manager? So a good way to do that is that instead of doing that just write your same shell script that you have do a little bit do a little more interesting stuff but literally just run it as a transient unit. So like this could be like a normal unit and then you just start it but if you want to create this file on the fly and do all those stuff you could literally just create a service as a transient unit. You give all the attributes that you want you can put you can put limitation into your script change the user and the good thing is that you should always try to use notify when it's possible. The reason why well so this is not blocking and then you just return the name of the unit that you created and then you just monitor that unit. Now how do you monitor that unit? One thing that you could do and it's not really useful but then you could use like notify. So notify you say ready equals one if you like the status but what if you can put text into it and you can like you do like don't over have used this but like in your in your backup script you go through your backups and then every time that you make a change you change the status. If you do that then if you want if you monitor your service from normal tools you will always see the change that you did then you can actually get it from the text but that means that your scheduler what it can do it can always kind of go and connect to that particular unit and then retrieve the status text and this is useful when you create interfaces like like human interfaces like you allow the script that is doing the backup to communicate how it's doing the things that way you can read it that's mainly it I will not touch that one ask me later if you want to do that final thing is that bending to live system D fun you should all do it final language that doesn't have a bending do it yourself and that's it so thank you and questions I have one minute so at least one question or none go for it so I have two questions the first one is you showed before there is some auto generated code out of the D bus introspection stuff when is that happening like at build time so in my library you can do whatever you want you're free to write your library whatever you want it's here it's like when this with you create the object and this with in python world it's executed a method that is called enter and that method enter you get go to system go to the library sorry go to system go to divas call the introspection method get xml construct the object and then you return the object that you constructed the only the good thing is that these interfaces are really stable and you only need to do these ones we don't do it we do it every time that you create an object but you can only can only need to do that once okay and the second question was like if I'm not wrong there are a few function library function that are asynchronous already insistently oh yes I'll do you and that one is like his own event loop I think it's like how do you bridge with the python asynchronous stuff so I don't but I will so yes the that's the one so the first thing that I create this library I created with the specific intention that python 2 has to be supported which everybody in the python world hate me for and python 2 doesn't support I think IO like out of the box so now that we're moving into a python 3 world it's like I will do I think IO and I will make this library completely I think if you want to like it's not cool you're welcome cool gentlemen in the front that I don't that I have never seen now on that slide right there what is your place Oh so that is basically like the start and the stop method they this is like system D from yourself so then I can give you the better answers than that but they take arguments on how you want to start like how do you want the manager to actually start or stop your service and the most typical one is replace go ahead it's basically if you start a service and this contradicts with another operation that has been queued before what shall happen shall the new operation replace the older ones or shall the new one fail or something like that so I want I want before if you have a question before you ask I want to say like this actually was one of the coolest thing about working in this type of library is that you get to learn like one quote how these things actually work how this sausage is made because you get to like like you I call this method doesn't have arguments like what and then you say like oh there's all this option there's like three options I believe go ahead my question was regarding like if you got yeah on then this slide here like you're saying that the introspection happens at the time the the object is allocated right yeah so this works for any any diva service yes literally so how would that look like if I let's say talk to I don't know network manager oh yeah so so I don't have example let me see if I can get this one so basically this is how the might work you like I have sorry okay so this one it's like a custom object made understand that is the concept of unit but I have an object where where this inherit from that basically allows you to specify any any service and any object when you read that object and you see the xml I understand that I understand that everything that like if it is or.free.login.1 I understand that every every interface that start with that it's like an interface proper of the object so I exposed it like the thing as a proper attribute so then you have like your object that you created that you need so the user would be this one and then you will see it like it works like that and then the method works and the other one that has different names they are not exposed as separate attributes but you can always like get interface and then give their proper name and then they will have so what's the work then then let's say if I want to talk to network manager or any other divas interface what would I do I would first subclass this generic object thing fill like a handful in of things and then that's it right that the rest happens automatically yes exactly but in general like you don't even have to subclass it because the base class like it work on its own you just subclass it like to make like future calls not necessarily to actually give the the name but you will literally just if you want to talk for instance I do this a lot like with machines with machine one you give this one and then you give the object that would probably be either the machine manager or the machine or the machine itself and then like you say everything works like right off the box except for stuff like time date which are like really small and they don't really confirm this logic we can talk about that later but but for things like like machine CTL or logging which are like really big classes and they like work like exactly like system D1 perfect okay that's great I'm glad you like it I don't know if I have time for more questions I don't have time for more questions but