 Just for who doesn't know, this is Mr. James, and he's going to present MGMT config for containers. Thank you so much. Applause. So I'm just going to talk a little bit about containers, because it's such a hot topic. And a little bit about why we don't treat containers exactly like other resources. And by resources, I mean, like virtual machines or other things. We're going to talk about that. I'll show you some videos. Who am I? I'm a hacker. I talk really fast, because I have a lot of stuff. I'm going to show you some live demos. So if you really miss something, and you really desperately don't understand, raise your hand, and I'll try and answer quick questions. But I'm going to try and show you a lot of demos, and it's recorded, so you can watch it and slow at the end. I write a technical blog called The Technical Blog of James. Who's seen it? Just raise your hand. If you haven't seen it, just raise your hand, so I seem really popular. Thank you. Excellent. I work on, I used to be a physiologist. That's what I did by training in, and I don't really do much anymore. But if you have cardiology questions, I can try and help you out. I was big into DevOps. I used to do a lot of puppets. Who's done some puppet before? No, okay, so you know what I'm talking about a bit. A lot of infrastructure automation. I learned a lot of puppet, but I thought there were a lot of things wrong with it. And I realized very recently that I didn't really want to live in this Kubernetes monoculture. Do you want to live in this Kubernetes monoculture? Who said, yeah, get out of this room now. So this is my answer to whether I really want to live in that world, because it's just silly. It might be useful for some things. So this is my nope guy. Everything is nope. So instead, long story short, some time back I started, I sat down, I started working on a project called MGMT. It was really intended as a solution to the problems in puppet, but I think there's some overlap with what people are doing with Kubernetes and how they're doing it wrong. So I'm going to talk about that and hopefully convince you. So in this project, all open source, free software, whatever, there's two parts. There's the engine and the language. The engine actually does the work on the computer, so it sets up containers, VMs, files, packages, services, et cetera. Then there's the language, which is actually what we use to describe everything to the containers. So, oops, sorry about that. So there's three main parts to the engine. It can run in parallel. There's events, which I'll show you in a moment. And the whole thing works as a distributed system. So it's in contrast to something like Kubernetes for example, which is really a central orchestrator. The graphs that we run, we run graphs. That's our output. And these things are resources. So this might be install a package, start a service, and so on. And these black arrows are dependencies. So it's something you might really meet. Similar to what you might be familiar with, Puppet, and so on. Except we can run everything in parallel. And the events can be anything. They can be files, service. The resources can be anything. They can be files, services, packages, and spawn resources, kind of container thing. Docker containers, if you're into Docker. I'm personally not, but I'll show you a little demo of what I mean by event driven. So let me just actually... So... I have to actually turn off... Is this big enough? Can you see? No. No? Bigger? Yes, no? Don't be shy. Is it big enough? Bigger? Okay. So what I'm going to actually do is I'm going to actually just start up MGMT. I'm going to show you just... I'm going to start with a little example. Where I go... So I'm going to run this resource that basically says, please create one container, one file. And if we go into this directory here on the left, you can see... I can show you that it's created this file called Hello World. And if I remove the file, you can see it comes right back. So I remove it and it comes right back. And this is what I mean by event. So each resource actually is being watched by the engine. And whenever something goes wrong, it's so fast it even comes back before it gets to the end there. And this is not fake, you can see the engine running as this happens. So it sleeps and it wakes up. And it's so fast you can even run it in this watch command which just loops something over and over again really, really quickly. And you'll see that it just always sort of keeps you in the correct state. And I have actually an example with an N spawn resource if you'd like to see that. So N spawn zero. And this is actually kind of new so I have to see if this is going to work because there's a bug actually in this resource. But I wanted to show you it anyways. I think it's actually not working right now. I have one second to kill this. N spawn. So if you wanted to do an N spawn resource, you can do it just like this. Just basically declare the resource. And I have this demo. So I was working on a fancier demo with N spawn. But I'm not actually going to show you that right now. So I can actually just show you. I'm going to show you this one because I was really working to get this ready. Sorry, one second. What's my password? Who knows my root password? Anybody? So I used Fedoro on my laptop. I actually wanted to do a Debian demo. So this is what it is. So it's actually, you just need the name of the machine. See if this works. So these are just the declarative resources. And if I go back here and I run this one here, I think that's going to work. All right. So machine CTL, I think it's a list. And there you can see it's running. And the same sort of thing happens. Like if you, I forget if it's terminate stretch. Oops. If I kill the container and type my password you see it actually MGMT wakes up and it brings it right back. Does that make sense? Cool. So I like demo with a little bit. I just had to get it ready for you. So this is the whole point. So we have these primitives. They all have events. Now let's do some fancier stuff. And this works for everything. Virtual machines, files, packages, services. And I think this is actually kind of like monitoring. So typically we had set up infrastructure in a second step. But if we build this in this event driven nature into each resource, it's already partially done for you. So now we have to describe how you actually tell the computers what to do. So normally like in the Kubernetes land and Ansible land it's blasting these giant YAML files which to me makes no sense because it's not expressive. You can only play with the knobs that they've programmed for you. So I wanted to build a language, it's a DSL so a very small language that lets you actually build your own custom creative things. The language has to be really safe because you don't want to make an off-by-one error and blow away a data center. It has to be very powerful so it's a reactive language kind of like an FRP. If you know what FRPs are, that's okay. You'll understand in a minute. And here's the demo I'm going to show you. So what I do here, I can actually start off this demo. I'm going to start off this demo on the left and on the right here I'm just going to watch this file. Here we go. So what's happening? I'm running this code and here's what it basically looks like. I have this datetime function at the top and it adds the value from this output to a year which is some variable that has a whole bunch of integers multiplied together. Then there's this load value down here and it goes into this variable and then both of those variables go into this struct which gets put into this big printf string which gets set as the content of this file. So what's going on? But here's what's happening when it's running. Sorry. Here's what's happening when it's running. On the left you have MGMT running. On the right I'm just watching this file just so you can see what the output is. You can see it's actually changing about every second because MGMT is actually constantly running constantly deciding that the value should be different in the file and it's updating the file every second. So what's actually happening here is this datetime function is actually a stream of values. So it's the stream of number of seconds every second that goes by. So the data is always going up and it's putting all these values. It changes constantly. Putting all those values and it's re-evaluating just the parts of the code that it needs to which ultimately results in this file resource that has different contents. See that? And then that all gets put into this file and you can see that it actually changes. Does that make sense? And just to make it a little bit more fun I actually built a function, a custom function called VU Meter and what it's doing it's running. It's actually listening to my laptop microphone right now. And just to prove you this is real we're going to be very quiet and when I point to you I want you to make a whole lot of noise and see that it's working, okay? So nothing I could do and so this is the kind of thing we're taking real-time event sources and we're integrating them in this small, simple, safe language that lets you do very powerful things. So the kind of things you could do for example is you could take this and maybe you have this in your server room or in your little office and when there's lots of loud noises and people are screaming and fighting you automatically set all your systems read only so that's the kind of joke I make. So that's the sort of idea but you want to see some more stuff? Do you want to hear some more stuff? These people leaving are like nope I don't want to see some more stuff. Alright so we're going to do some fancy scheduling stuff. So in addition to just these silly examples we actually as I said work as a distributed system and we've built into the core of MGMT some functions and one of the special functions is called schedule and what it does is it actually takes all the machines that run it and produces a list of machines that are selected by a distributed scheduler. So what it's going to do it's going to do this and it's going to run in each machine so I'm going to just run it. This is actually a new demo so let's see if this works well. So I'm going to run it one at a time and oops schedule schedule dash docker so I decided to do this with a docker so I'm going to run the first one and over here and run a watch. I'm going to run this big fancy watch command and what it does is it's just going to sleep and it's going to tail some files and run docker ps and sleep and so on. So I'm just going to run that and you actually can see at the moment this is running. Did it work? The docker thing is quite new so let's see what's going on. Just kill this for a second. ps docker images are just starting up or is it unhappy? Oh I don't have internet. That's why. I need some internet. Sorry folks. If you're torrenting please turn off your torrents. It doesn't need a lot of internet because I pre-cached the images but docker is sneaky about this stuff. We have internet just kill this I'll start this up I need to CTL start Okay it's working. This is the first machine and what it did is it basically had it right to a text file if you're scheduled write it off. The first machine says I'm scheduled and it decided out of the available host currently there's only one this is the only one therefore I'm only a schedule on the host the first one. What you can actually see down here is one container was automatically created 24 seconds ago and instead of running four different VMs to do all this I'm just doing it all locally the machine is named D-hostname so H1, H2, H3 Let's run the second one and see what happens. I'm going to run the second MDMT they'll automatically cluster together and if we're lucky you can see now that it starts up and in a second now we see a second container running because MDMT decided I have two hosts available I asked for two containers to be running I'm going to put one on H1, one on H2 and you can see them running here and H1 down below. Do you see that? And each host is actually creating a text file just so it can visualize what's happening this is the text file from the first host that says I am scheduled and H1 and H2 are the two hosts in the schedule set. Does that make sense? Just a cheap little schedule. You want to start a third one? Come on, do you want to see a third one? All right, so the third one now I've only asked for two things to be scheduled so if you look at the example so watch what happens, nothing happens because there's only two and we've asked for two, a max of two and a TTL of ten so now if we were to actually shut down the second one we're going to shut down nicely watch what happens we shut down that. Oh, two went away and automatically the first one has said now we've got to reschedule to host three so H3 and then the third one finally came up we're going to have H3 and now you see we've got container H3 and container H1 running do you see that? H2 it's not going to flop because we don't want it to flop back and forth so it should get started up and actually it decided in this case to actually switch it so normally it's fairly static but you get the idea and so what we've basically done is with a very simple built-in function we've allowed, I built in like an hour, a custom little tiny play scheduler now this is just a silly schedule you could build something much more complicated that has a fancier scheduling algorithm like around Robin strategy and so on but that's the basic idea instead of building this huge monolith that has all these built-in components and built-in knobs I decided to build this engine in this language that allows you to put in whatever resource you want whether it's container, docker, nspawn, whatever and then other pieces that glue together to build what you want you might not have to write the code for this in the future someone just might write a scheduling package that does all this stuff for you and you just say what images you want to run but that's the basic idea any quick questions? wanna see some more stuff? you're like somewhat enthusiastic, it's very warm in this room it's like 300 Celsius so that's the scheduling thing you can kill this I'm just gonna actually kill it violently so I want it to go away so we can do another demo and you can, all this code is online so you can play with this at home so I said you don't need to have nspawn this could be a vert resource or anything else I think the Kubernetes project actually made a mistake by sort of hard coding the idea of containers as their primary thing but their opinion ended that way and I think that as a result because they were so specific about that one solution and that's the thing forever it's caused a lot of problems in the codebase very difficult to change the project for the future so what is this? this is a great example because this room is a million Celsius what is this? it's a photo I took in my parents house you can see it's in the international units of Celsius it's very warm I found a clear picture in some sort of arbitrary sauna units on the internet and these things have actually a very interesting property does anyone know what it is? hysteresis hysteresis you saw my sock before okay that's great you're just a smart gentleman so it's hysteresis and what hysteresis is it's also smart if you saw my talk but that's okay hysteresis is the property so let's say you get up to 20 you ask please set your temperature to 20 Celsius the temperature goes up and hits 20 then immediately clicks off and then heats up again it goes back and forth this would be a big disaster we don't want this to happen so hysteresis actually we have the most enthusiastic sign waivers like 10 minutes left I know don't worry thank you hire this man he needs a sign waving job so hysteresis stops that flopping back and forth so you can actually wait some amount of time to go threshold and I'm going to show you you want to see a demo of that come on do you want to see a demo who else is live demoing for you okay so I'm going to run this hysteresis example instead of containers I'm going to use virtual machines just because it's a little change and I'm going to run so I'm going to run this watch command over here it's going to print out the contents of this text file and run verse list and what you can see I'm actually it's mgmt on the left two virtual machines one and two and what I've actually done is it's also printing out the system load so the kernel generates a new system load value every five seconds and I've built in a threshold of 1.5 and the idea is that once we hit 1.5 we're asking mgmt to automatically shut down one of the VMs right there's too much load too much traffic so we're going to move that to some other machine somewhere and schedule it elsewhere and so we're going to do that but then when the load drops below 1.5 we don't want to reschedule that in this case for 10 seconds so let's try that does that make sense so I'm just going to artificially increase the load just so that imagine the VM is doing a whole bunch of work I just got a trick Linux into thinking it's busy oops so we're going to do that my poor little laptop is hot enough as it is it's going to be on fire after this so CPUs are going okay so you saw the first machine was there and the second machine disappeared you saw that we're just watching it and it's past 1.5 and we're going to watch it go down so it's still there and below 1.5 it's going to take 10 seconds so we're still not there 1.8 mgmt is still running it's just waiting watching the threshold reevaluating the code every time something changes 1.6 1.62 any second keep watching for it so it's below 1.5 nothing happened right we're waiting 10 seconds and then it should hopefully start it up again 1.3 so 5 seconds left 3 2 1 boom and it started right back up you like that? thank you so again if you wanted to build something like this with existing tools like Kubernetes you'd have to find some special knob that some programmer spent tons of time building and that's just absurd because there's always going to be some special infrastructure that's different that you want to do something custom your way without making it too complicated and again you don't have to necessarily build all this stuff you could actually have someone else who wrote a module that does the general things that you use so a bunch of companies use the same you know scheduling container module up to you but the point is it's possible and it's very easy you want to see some more stuff? how much time do we have left? let's see I gotta actually shut this down 10 minutes left awesome so I have some time I'm just gonna actually destroy these VMs I'll get some my poor laptop get some love so just some interesting properties about this stuff so the language in the engine these two parts there's actually a bit of a separation between the two so in theory the code actually is safe and it shouldn't crash but if there is some disaster that happens the engine can keep running even if the code actually hits an error it's actually a safe language so there's no nils you can't have like a nil pointer exception in these kinds of bugs it's an immutable language so if you did something like x equals 5 then x equals 6 it would be a compile time error and the whole idea is to eliminate classes of errors before you run the code in production so it's pretty cool you might remember that earlier example I showed where things were kind of out of order you're like what is this guy doing date times up here and the variables down there and the reason I did this is actually kind of as an example because the code actually doesn't need to be in a certain order the code is actually a graph it's just streams going from vertex to vertex of data so the code can actually be written in out of order now you shouldn't do this in practice you would be insane but just an example to show you that the engine the language engine doesn't actually care if it uses the code and it follows the data paths does that make sense? yeah it's a bit of a strange concept I talked to you about immutable variables I've shown you some hysteresis so there's different kinds of hysteresis that based on a time limit or distance things like that so all sorts of fun stuff hysteresis is actually very important because hysteresis is actually a real world thing that our brains are sort of programmed to understand for example if your toaster is not working you push the toaster thing in and the toaster doesn't work you might like you don't try right away again or you try once right away and then you wait you think why isn't the toaster working and then you try again so these delays and these backoffs and these logical things that humans do are actually possible because we think in time so this reactive language is a time based language for this reason it models real life and the idea is basically for error scenarios in real life instead of getting a page and having to wake up and fix something you just program in advance what the language should do for every second of the day in every situation possible and then you have built an automatic sysadmin hopefully good idea right? yeah some people are like yes so this is some stuff we've talked about this is the important module system I'm going to be talking about it at a different conference and I'll show you about it at the end and there's still a lot to do with this project it's still a relatively new project it's not big and fancy there's still some new stuff we need to do in the standard library there's a whole bunch of new functions that would be great to add new features and stuff are always something we need to do there's a bit of code we need to clean up that's a bit messy there's a few bugs and annoying things some legacy stuff that I didn't know better at the time and this is all about you how can you help? do you want to help? I'm going to leave do you want to help? yeah all right so what can you do? you can use this test it, patch it, share it, document it start it blog it, tweet it discuss it just hack on this stuff right this is like hacking stuff MGMT I worked at a mostly cool tech company and I left a little while back to sort of work on this full time because I'm trying to do the free software thing but the bad thing is I don't have really any money to do this I'm just living off my savings so if you want to help like send patches or money or both that's cool I have Patreon it's just an attempt to see if it's helpful and finding a hacker it's very sexy so don't be shy if you want to be sexy this is the easy way let's just recap there's no audio this is just a bad joke of Arthur Benjamin putting the cat back on his pen it's a bad joke I reuse all my jokes and I apologize so if you want to get involved we have an IRC channel MGMT config on free node we hang out I see at least one MGMT contributor in the room so join us we have a Twitter account and a mailing list which you can hang out and listen to there's a technical blog of James you all know about it now so you can check out RSS and read all the great blog posts I write I'm purple idea at IRC and Twitter and GitHub and so on so you can find me there more stuff today I'm actually somehow a main track talk just too bad so I gave a whole bunch of talks already which is just absurd that was sorry that was yesterday and then today I have one more talk in the config management and monitoring room which is basically just a five minute lightning talk so you probably won't see anything new but it never ends and these old talks you can go see the recordings of I showed some different demos and different stuff and talked faster so if you want to there's this really cool conference called config management camp which is happening tomorrow and the day after in Ghent giving two talks and Felix is also giving another talk who's a great contributor as well and on the sixth right after that conference in Ghent as part of it there's a hackathon so if you want to actually get your hands on MGMT with me and some other people writing code writing your own function writing your own resource doing stuff come check it out totally free totally cool if you like this talk please go find a Fostum organizer and send them little shirts and say hey I really like James's talk James Schubin purple idea and if everyone does that will be like a DDoS and it will be really funny and if you don't want to do that that's okay if you go to the schedule link on the Fostum website if you look there's a secret like link right here I didn't know about this for many years called submit feedback so you find my talk you click submit feedback say this was dope it was awesome it was fun if you give me some good advice cost you nothing it would really help deal? just do this like imagine 200 submissions of like awesome they're gonna eventually have me for a main track talk right alright so I have some free stickers there's a sticker cartel so they're incredibly expensive but if you promise to use them on your laptops come at the end out there in the corner I'll give you some away if you'd like and show them to your friends you're gonna have a cool laptop like me and yeah thank you very much before you leave I have probably a few more minutes left for questions maybe someone has questions yes questions this is an encourageable gentleman up here I'm gonna start putting stuff away so I'm out of the way yeah hi thanks for the talk here oh there you are hi like I'm a newbie so sorry if that's a completely absurd question that's been answered like many times before like in the first page of the fact or something but I imagine that the whole dealing with being responsive when you're dealing with stuff that can send you like interrupts and tell you oh I change it's relatively easy but what about like being reactive about stuff that doesn't announce it like newbies starting to write code and then it's suddenly getting MGMT getting 100% CPU just because it's like polling stuff that isn't reactive yeah so the way we work every event source whether the functions that actually look at data or the resources that actually look at changes they all are based on events so select calls I notify her files there's no polling so if there's something we cannot do and we must poll then the solution is to ask the kernel people hey we're missing something there's almost nothing there's a few things left that don't have things and in those cases we could poll but almost everything does have an event system or it's a bug so really good question any more questions got another question over here hi great talk you talked about monitoring does it only work for the system itself or can we use it for our for logging or stuff like that can we replace everything and just rely on the monitoring system I don't if you're leaving try and be a little quiet I don't 100% 100% understand what you're talking about but if you can explain to me better I think you're asking about can we log and do other things besides just automation the answer is yes so this is a general purpose engine for infrastructure automation stuff whether you use it to get sources from events that come from monitoring systems and integrate them into your decisions or not that's up to you for example you could write a Prometheus function from a Prometheus server that was sending data from something else and then respond to those things if you wanted to Patch is welcome for sure we have a space for one more question one more question gentlemen in the front here maybe and come get a sticker if you want yeah mine goes with the first one what's the overhead when you're looking what's the overhead if I'm looking for I don't know 1000 files 100 virtual machines so how painful it is it's incredibly fast so it's probably the fastest tool out there if you were to come to Puppet and all these other equivalents if you had some absurd requirements there actually is a feature called auto grouping which will take multiple resources and group them together so it's more efficient but just one second let me do that this guy really wants a sticker he's like hey so we actually can do some efficient things like that but if you did have something that was totally absurd the fastest are you're probably doing something that might not make sense so I if you find a use case that doesn't work for tell us but I think it's probably going to be good so it's really nice basically it's in Golang it's very fast and yeah thank you so much thank you