 How do I introduce this guy? The person who messed up my Spotify history with all his crazy rap songs We have two more talks left and the next one is by James and also known as Purple Idea And you will talk about next generation conflict management with MGMT Thank You Walter. Hi So I'm gonna go really fast, but I'm also gonna try and go a little bit slower on some other stuff So I'm gonna assume some knowledge of automation and config management or puppet and even MGMT to a certain extent because I've talked a Lot about MGMT, but I don't want to start giving the same beginner talk about MGMT So if you have missed something you'll have to go see a talk of mine I might sit down for a little bit. So if you can't see me, don't be afraid. I'm still here And have some questions at the end. So who am I? I'm a hacker I work on config management things. I write the technical blog of James Who's seen it just raise your hand who's seen my blog if you haven't just raised your hand anyway, so I seem really popular Everyone. Thank you Used to be physiologists, but now I'm doing hacker stuff. Just some background There's a lot of Libra office bugs, but behind the black square is is beaker So sorry about that So every day nowadays we're kind of everything's yamling it seems to be like Ansible and Kubernetes and all this yaml stuff to To describe things is at the end of the day The whole point of all this is to describe our infrastructure and ideally we'd like to describe it in some sort of way So that if we got all our code right and the word any bugs It would just sit there and do its thing and we could go like lie on the beach or chill in Canada or somewhere cool And that's what I want to build like I want to build autonomous systems And I want to figure out how we describe to the computers to do what we want over time Do we want everything in yaml? Like we just write this yaml and throw it at a tool Do we want to be yaml programmers? I don't think so So this is my answer to the yaml programmers I'm sorry if you've seen this before he pushes the note button and it's gone Anyways, so long story short. I sat down some years ago and started working on a project called MGMT Which has this logo there's two main parts the engine the language. I've talked about this a lot before And I'll show you just a quick demo I can actually even pull a demo that I was gonna do this morning. Should we do that now take a little bit longer I'll show you a You know waste two more seconds and show you a demo that I was gonna do this morning in my talk Which didn't work out because I forgot So just boot up a little machine the idea basically is that We have this engine and it can run things in parallel So the dag the graph of resources that gets work done Normally we run it one thing at a time We can run that in parallel and each resource is event driven So whether it's a file or a package or a service or a user or a virtual machine all those things can detect the state And just detect when that state changes So changing the the graph of resources can be done very efficiently and we never have to recheck that whole graph So every time you have a new puppet run or a new ansible run. It's rechecking every single thing We only ever have to check once And the whole thing works as a distributed system. Is that big enough? Can you see? Okay, the password is password, and I'll just show you this was the MGMT demo I was gonna do so if we just look at the code here It's just a small amount of code to create a user I'll even let me just SSH and it would be a little bit nicer at Damn 2020 don't wonder if Hava he's working. Oh, yeah cool, so we just go here and If we just run this Command run it's running MGMT. You can see it's adding a user And I'll just show you what the code is doing so run that sh. It's just literally running MGMT To create those users and it shuts down But the cool thing is we can actually run it in the normal mode Run continuously, so it's just gonna run that code. I just made this shell script. So I don't have to Run it if it's really important otherwise, you have to wait the end of the short talk so MGMT is running continuously but if I go over here oops and Our Delete user Sorry user delete MGMT one And I delete the user you can see MGMT wakes up and puts the user right back So ID user ID MGMT one there's it's Over the users call. Let's see. Oh, yeah, there they are so user down MGMT to And you see MGMT Goes right back and adds the user right back. So it's it's not doing work polling every time It's always just listening to see the state of the system whether it's a package or file or a user And it will guarantee that states always the case Okay, and this is just simple. This is just for a user resource But it's more fun if we do fancier things. Do you want to see some fancier stuff? Okay, let me just kill this Vm because my poor laptop does not have a lot of RAM first destroy stem 2020 What was the VM called? Oh my goodness. It's in boxes boxes. This is me hacking away How do you shut it down? Hmm. Oh my god when you close it it sleeps. Okay, that's why so anyways back to the talk The whole talk is me randomly hacking on my laptop. So be prepared. So how many resources do we have an MGMT now? Yes, call it a number a thousand thirty close. We have 27 so five more in the last year so we can do all these cool things can be managed in real-time even virtual machines EC2 instances and so on and now I want to talk a little bit more about the language So the language is this special DSL. It's not a full programming language What the pool me people are doing is cool But ultimately if you want to describe safe infrastructure I don't think a general purpose programming language is the right approach So that's why we have a DSL this domain specific language that allows us to model the time components and the coding components safely Here's a quote about that should be safe should be reactive as I showed you it wakes up and detects when something changes and so on So let's do some demos. I showed you the user demo I've showed this demo before bringing a show it once more just to get us all on the same page So I'm going to run this MGMT code here on the left I'm in my dev environment. So there's sometimes some bugs. So if there's something bad happens, we'll we'll fix it So MGMT has created this file and I'm just gonna on the right here I'm just gonna pull the file to see the contents and you can see that it's updating this file in real time And that's because the code Which is running continuously looks like this There's no syntax highlighting, but basically here's a slightly nicer version We have a date time function, which is a core function that's in MGMT You can have other ones that you write yourself and it takes this value. It's not really a single time point It's a stream of times right the current time a number of seconds the next time a second later And it's always updating and whenever this updates this expression updates whenever this expression updates It changes this struct which has a bunch of different values in it and all of this goes into a big template And that is the contents of this file down here at the bottom Right and that's what we're showing the other values that go into this big string for the file is the load and this view meter function Excuse me, and as you can see It's constantly changing so the load the time is going up the load is changing and if you look You can see one of the inputs is actually my laptop microphone And so the language is taking all these different real-time event sources Combining them in a safe way and doing something useful now This is a complete garbage example because it doesn't do something really useful part than show off at conferences but the really cool event example is something when you have the load and error scenarios and our say auto scaling up and down machines and Replacing machines on failure in real time before you even get an email that something happened. Does that make sense? It's a bit abstract, but we're gonna stop the abstract stuff. I have some more concrete examples Do you want to see some more demos? Yes Yes, or no, yes Everyone in the back. Do you want to see some more? Demos all right. I'm gonna show one silly example This is a new thing in MGMT if anyone's done puppet There's a fragment sort of pattern where you take a bunch of files and you glue them together to make a bigger file So I realized there was a better way to do this in MGMT So the file resource itself it has state right state exists or whatever But now instead of content for kind of like puppet We have this fragments pattern which can take a list of things in this case. It's a directory or file or both and You can see here's a file down here. That's just normal file and this is another file down here I'm just gonna run this fragment example right here on the left and You can see The actual code takes a bunch of different files and glues them together. So if we look at This file is it in Just watch It's this file whole Oops whole one. There we go So you can see this file is a combination of these three files and this bonus file, right? Basically this code here plus some more frags But the cool thing is if you go into that directory down here Fraggs you can see those files there and same thing if I delete f2 It comes right back right because that's an individually managed file That's always ensuring the right state and delete f2 and cat f2 it comes back But you could even put echo. Hey foster into A new file and see it picked it up and right away someone here said that's fast it is The real question is not why is this fast? Why are all the other tools so damn slow? Am I wrong like this is how it should be I Was getting super old like waiting for puppet runs and ansible runs to go for like, you know I'd go have a coffee and or whatever anyway, so this is the idea So we can build the fragments all with the core file resource and that's a new thing I think there's another demo. I wanted to show you with that That's most of it. There's not I mean basic thing, but just a thing that's done I think the right way so when you're building config files from different pieces just sort of native But I want to do something harder when it's something harder Okay, let's move on more real-world examples so reversible resources. This is a new thing so in my main talk today I was talking about some properties of config management one is item potents Which you all know one is convergence which means is the whole state converged as a whole and there's a third property Which almost never is it's almost never talked about which is called reversibility and what reversibility is so you take a resource It's item potent it gets you to a certain desired state if that state transition is what I call reversible That means there should be a reverse operation that takes you back to the original state Exactly okay now not not everything has this but everything ideally should have this To a certain extent so for example if I create a file and then I run the reverse operation. What should happen? It removes that file. Okay, so let me show you that first example, so I'm going to have an example temp MGMT hello and All I do is I say this is the file and I add this metaparameter This is built into MGMT. You don't have to do any of this You just say reverse true, please and here's what happens This is reverse Okay, so I'm going to run this file and What I've done I've actually I'll show you the code just so you can see what's happening I've decided to make it a little bit fancier So this code right here is just a very simple way to say every four seconds Switch the variable from true to false true to false so since MGMT can run in real time It's just going to flip a variable on and off on and off every four seconds so you can see it running and The thing that we actually do this variable is right here See that mod variable this one at the very bottom and so this is the equivalent of Anyone who's done puppet or something like that has written some code You ran the puppet code and then you changed it in git and you ran it again And in that new git version of the code that resource is just absent You didn't change the thing you just removed it from the code and what happens? That file is still there on your machine, right? So the whole point of reversible is if you have a resource which is declared and then at some later time That resource is no longer declared in your code MGMT will say aha this was asked to be removed This was set as reversible so it will do the reverse operation when that code disappears And what I'm effectively doing if mod equals true by saying true is false It's effectively taking that code out of MGMT You could obviously just change this code by deleting the code and running MGMT again, but this is more visual So we're running this code What was the file name temp MGMT hello, so it's running here and Temp MGMT Where's it it work, so I'll just remove the state time file and I'm just gonna Cat star and you can see there's the file Four seconds go by oh it disappears Four seconds come by and it goes true again, and when it goes to false you can see oh it's gonna disappear cool Is that cool you can applaud if you like I think that's cool, right? Okay, no no no wait wait wait wait wait wait let's make it more difficult so the second thing What about this one down here the second file, so this one? I'm saying I don't declare the contents. I'm just declaring the mode and When we run this Now I just have to just double-check it and Might take me a second to get this one right so there's the file and Okay, it disappeared so I just have to actually make a file first because it doesn't create one so echo Foo into hello right, so there's one that has to be there to start off Right, and so now when we run MGMT You can see it takes it away and Puts it back one second Does it start off as true or false? I'm gonna go Think that's it. I'm running the right code. Let me just check It might be a bug in this but it usually works Oh, sorry this one doesn't have the mode my mistake I went to do reverse three. Yeah, this one has the mode. That's why you got to pick the right demo So once more we run MGMT on the left Okay So there's no file initially. So we got to create one And you can see Watch the mode boom Okay, so it's just setting this the reason I had to create the file is because in my resource definition I did not specify state presence or straight state absent So if it's not it's just gonna leave that part on it's only managing the mode and it can only manage the mode if there's actually a file there And so that's basically what it's actually doing is when it changes the mode the first time just before it makes that change It actually stores the existing mode to know what to reverse it to so whatever that initial mode happened to be on your system Is there now? This is a bit of an abstract example, right? Because Changing a file mode is maybe not the kind of thing we want But the kind of place where this is gonna get super interesting is where in our code We're declaring the state of running containers and running virtual machines and all these other real-time systems and when they disappear We can turn them off and stuff like that so we can build fancy systems We could build a Kubernetes like scheduler in like hundred lines of MCL code And it would be super safe and easy to customize and No YAML involved How long do I have left till? 50 55. Oh my god the short talk. Yeah, I'm gonna show you another thing. So I Decided we have resources and our resource model has this event system built in which for programmers means that in the Implementation of each kind of resource. There's a main loop from like event-based programming main loop programming Maybe that rings a bell, but it basically means we can run some code inside that main loop and One of the things we can do is we can run a TFTP server in the resource It's just a simple demon could be a web server. It could be even a huge database server Although that might get a little huge, but just as a simple example We have a TFTP server resource which I've built which I'll show you in a second and the cool thing is so You have the resource itself when that first resource runs it sets up the server and starts listening It's in the MGMT binary. So it's just like a super powerful binary And then the other thing is say you want to actually have files that that resource serves I've defined down here a TFTP file. So there's no data on disk anywhere It's going directly into memory in the TFTP server and just for fun for traditionalists. I've made a normal folder Which is this root directory here So you can actually just have a folder of stuff on a disk and that will also be part of the TFTP server resource Now here's the magic part You might remember this from a very early talk. I gave many years ago called automatic grouping so this is a graph of resources that MGMT produces, right and MGMT does a really cool thing with this graph Which is it makes it more efficient and that if you see there's three package resources and with puppet and every other tool What's going to happen? It's going to see this graph. Those are the dependencies Those are the different resources and it's going to start up the package manager Install power top and then shut it down Install Kauze, shut it down, install SL which has a lot of overhead So MGMT can actually analyze this graph You don't have to do anything happens automatically and it will rearrange this particular graph to look like this and that will group Those three resources that can be run together and it will actually take that package resource and turn it into a like Yum install three packages in one transaction which makes it really fast and so the way this works for the TFTP resource is That different parts of the resource that all should be part of the same running code base in this case the server and all the different file Definitions will actually be grouped together automatically in the background and so they'll all function together as a single entity So magic stuff happens, but to the end user it looks simple and separate so different parts of your code different modules in The Apache package versus some other package can actually now contribute code to each others Functioning with automatic grouping that makes sense. This was always like a problem. We had in puppet There was no way for this module to play nicely with the firewall module or vice versa Exported resources got you for some of the stuff which we can do in a different way and other way as well But this is a core thing which was not in puppet, which I think is quite helpful. Do you want to see a demo of this? No I'm trying to put the energy. So All right, so let's run this TFTP. So I'm just running it here on the left and Forget how to actually I have a little It's not as exciting as you want TFTP So I have a go run. I just have a simple TFTP client in in the git repository So you can just run that yourself. It's on port 1069. So I don't need root and we can just ask it I'll just put some here after file zero And you can see on the left you see in the logs the resources like ah something happened I requested a file and you can see the output of the file right there Okay And you can do other stuff. There's another file on the file server. You can see that and so on and so forth cool This is actually practical. It's not useless to have a TFTP server because I have to do something real So I'm going to try this one demo over SSH just to show you Don't tell so this is some phone that I have running in Canada And over here, this is MGMT running and what it's doing. It's just an MGMT Cluster in Canada somewhere running a TFTP server and it has files for as you guess each MAC address for the device there's a MGMT template which builds the templates and Let's restart the phone. This is in someone's office. So let's hope they don't mind I Think factor your sets probably better I don't know if this is going to work because like I've been gone for a while and they might have moved the phone around but in theory this poor phone in some ladies office is going to reboot right now and after a while But yeah, I'll just show you MGMT is really running Oh, what was that about? Oh, that's my SSH forwarding after a while you should see This will take a little while because the phone is like from grand stream Which is this Chinese company that makes really bad software. So if anyone knows anyone grants you I found more bugs in these phones It's unbelievable So that's just the long-term demo It should eventually come back and it will provision itself and takes a little while and while that happens you'll see the logs and Like I'm really using this now in production for setting up crappy sip phones. That's cool, right Walter So I got a few minutes left 50 what is the time it's and okay, so I'm gonna so This the connection areas you're seeing is just my SSH tunnel because the web browser is trying to connect back And but in a while the phone should boot up while that happens I'm just going to sort of finish off the end of the talk and then we'll look at the logs because that's awesome This is a shorter talk. So what's left? So? I'd always like a few more functions the standard library We actually have a lot of the Golang standard library imported automatically into MGMT. So we have all the functions you could want There's some new features which are not there yet. There's a few bugs Unfortunately, there's a really yucky race in the engine and there's this function as values stuff Which I'm gonna probably talk about a bit tomorrow in the languages dev room, which is not finished But how can you help this is about you so use this test it patch it share it document it start it blog It tweet it discuss it hack it if you think it's cool hack on this stuff Are we at a hacker conference? Like I get surprisingly few patches from hacker conferences. So I'm kind of like worried And I basically I used to work at Red Hat and they stopped funding my work I decided to leave Red Hat and just sort of do this on my own dime So it's super sucks to pay for all this stuff and life's expensive So if you want to send me money on the Internet's funding a hacker is very sexy I'm trying to make this into something that will be viable on its own because I don't want to do proprietary software I like to do this because I believe in this project, but No, you got to eat so in the next six months or in 2020 I'm going to be trying to launch a corporate sponsorship thing So we're having an MGMT website that's launching and we're going to put your logo in the big front-end center For like some small amount of money per year So if you work for a company and you can convince them for like a few thousand euros per year or something like that small amount that would be great and Then we'll free we won't forget you when we're a big company and we like are feeling angry with small businesses I'll be like you are a founder sponsor So so yeah, please reach out to me send me a message to this contact forum saying hey I'd love to have my company be involved or I'd like to have a training session or like to help sponsor help out We'd love to work with you That cool. We will all do this ask your companies, please Let's just recap It's a recapping joke. It's not funny. I've used it before There's an IRC channel where I talk about what I'm working on and stuff like that. There's a Twitter bot And a mailing list There's the technical blog of James so everyone knows about this now so you can go to Google it has RSS even check it out I'm purple idea on Twitter and github and all this stuff. So if you want to ping me Which is fun. Oh, this is actually the wrong date. So not tomorrow, but Tomorrow there is one more technical talk about MGMT in the minimalistic languages dev room that I'm giving and then the day after Like in Ghent, there's this conference that Chris another one else organizing which there's more fun stuff and If you want if you come I will give you a sticker on your way out Now let's just quickly see actually you can see right here The TFTP server is in fact reading and the phone is provisioning. So It's starting up. So that actually worked You can thank Chris until shine if you like my talk he's here on the front. Thank you very much