 We're gonna go really fast today. Who's seen any of my talks before anyone? Oh, there's some people cool cool Good to go. All right. Hey, I'm James. Hi everyone. Thank you for coming. It's a full crowd Did you put the full sign? She'll let more people let more people in the foster news room in the sides Yeah, so should we let some more people in? quickly quickly let them in let them in so Who am I? I'm a hacker. I work on config management stuff I write a technical blog called the technical blog of James who seen my blog just raise your hand If you haven't just raised your hand, so I seem really popular Is that Richard Jones? Is he here? Is that Richard? Anyways, I work on I used to work in physiology or I study as a physiologist So if you have any cardiology questions, just let me know And big into DevOps stuff. I was working in DevOps and all the tools were just just this everything was on fire and terrible And you know people like we had all these servers, but now we have VMs. So let's manage some VMs But then like oh no now we just have more of these machines, and it's a big disaster and Now there's all these fancy pieces of software to deal with these things like I won't name any names And the real question is do we want to really use these giant and flexible monoliths to manage our VMs? Do we? Don't be shy. So this is the answer to that as far as I'm concerned. This is my nope guy Just like I don't know some people might enjoy those pieces of software for me. It was not what I was looking for And that's fine. You use the things you want to use But I wanted to see if I could do this in a better way. So long story short I reluctantly sat down and start writing some code. We have a project called MGM T got this lovely logo. Thank you And MGM T has two main parts. There's the engine and the language The engine is basically what actually does the work and the language I'm going to talk about a bit too. So in the engine just really quickly I'm going to go very fast. So if I'm going too fast Hopefully you can slow down the video at the end and watch everything. So the the engine actually has two parts So the engine has three main design points the graph of resources that it runs it runs in parallel They're event-driven and it works as a distributed system So basically for this sort of thing We have these DAGs these resource graphs and each blue block is work that we want to have done And the black arrows represent the dependency between each thing that we want to do and If you can see this red line here is something called a topological sort Or we just basically decide we're gonna do this and then that and so on But an MGM T instead of wasting a lot of time the way most other tools do can actually Parallelize this so everything on the left can actually be run at everything on the right And so even one a once this is done we can do these two resources and so on so that makes sense Pretty basic stuff and that's useful if we're working on really big Complex systems that have a lot of things going at the same time. We want them to be able to run in parallel And each box each of those blue boxes those resources are actually Event-driven so I'm gonna actually show you a quick example of this We have all sorts of resources including a vert resource So I'm just gonna sit down to do some demos. So don't worry. I'm still here. Don't be afraid I'm just gonna run this first demo here and Basically, it's very simple show you the code It's very very simple we basically just declared early. Oh, we can't see it there, but We declare this vert resource and we give it some properties. So one CPU memory that we want it to be running and so on and So let's just run this and see what happens. So we're gonna run this over here And if we run verse list can everyone see is that big enough on the right? So you can see that MGM T started up this VM right here It's running, but here's the cool thing. Everything is reactive. So if I a verse if I just destroy the VM You can see MGM T says hey you were supposed to have that running So it automatically starts it back up and you can see it's running again So you can destroy it comes right back up cool. You can even I think you can even undefine it Which is another VM thing oops and No, that one you can't because it's transient, but yeah, so you just do bad things to it and MGM T will fix a state make sense So that's just the basics. We're gonna want to do some fancier stuff Yeah, yeah, all right. So just one check. So now we can actually kill it and it's gonna go away So so this is actually what I think is config management, but this is really another technology as well Does anyone know what I'm getting at? I think this is monitoring right so traditionally system in's would like set up all this infrastructure Then you have to set up monitoring detect what's happening This way kind of merges the two in the same in the same Bunch of work. So hopefully you don't have to replicate too much work So now I'm gonna switch very quickly and talk about the language So this is sort of the design principle for the language you want to express Things that are very very complicated in very very safe ways. Hopefully so a safe language We want to be very powerful. So we use something called a reactive language And we want the language to also be very easy to reason about so our language is a DSL So it's a domain specific language, which means the language is useful for infrastructure automation But it's not useful for general purpose programming It's designed for this use case and because it's very specific to this use case It has some special magic powers that other languages would find it very hard to write. So here's a demo. I'm gonna show you Just to show you the example so we have this date time function which is in the language and We add the output of that to this a year variable and then over here There's like this a year variable is a bunch of Numbers multiple it together and then we put that into a struct with this load function Which then takes this value and goes in here and this is all super weird And then all of these we build this big string and then stick it into this file What is going on here? Does anyone know? Looks a little confusing. So let's run this code just to see what happens So I'm gonna run that code here on the left and again that whole thing just produced a file and I'm just going to actually just oops You run this watch command to show you the output of that file in real time So you can see what's happening and you can see that the output of the file is changing all the time Right and here's why these variables and these functions this function here this date time doesn't return the date time It returns a stream of the date time values. So every second. This actually changes It's the number of seconds since 1970 right so this changes every second Which means this variable changes every second which means this variable changes and so on the load valuable the load value comes from the kernel The kernel decides that the load is they recomputed every five seconds So that will change and all of this stuff gets output into this string And then the engine gets these resources that it builds and it happens in this particular graph that those resources are changing every second Which is the file contents and that's what we see so we can see this ticking away and that's changing in real time Does everyone follow that and just to make this a little bit more fun I have actually this VU function and what the VU function is actually doing is actually listening on my real laptop microphone Right now and sampling the sound and if you look it's printing out a little graph to express that so if we're very quiet See it goes up and if you want to test this I'm going to be really quiet When I point at you I want to hear you make a lot of noise. Are you ready? There you go, so again any value any sort of stream you can build all sorts of inputs that are changing over time It's really up to you to be creative and I kind of use this example a lot because it's a bit of a joke But you could have this in your server room building your real-time infrastructure if the volume is too loud for some number of seconds Maybe people are having a fight so you could automatically set all your file systems as read-only so that system ends Don't break things. Yeah random things like that whatever you want That's the idea. Does that make sense about the reactive you want to see more demos or you had enough? Yeah, okay. Let's gonna make this a little bit more complicated So magic CPUs so here's what I'm gonna do first thing to do is I have I showed you had that vert resource, right? So I have another one and I'm gonna start up MGMT on my local laptop And I just want to make sure I start the right one. So I'm gonna start up MGMT and What it's gonna do is I'm gonna run verse list So it started up MGMT Some VM and let's actually log into that VM Now normally when you have a VM startup, it's already been started up. That was fast. Normally my password is password Normally when you have a VM startup You might want it to automatically start MGMT and then run itself inside the VM as well But because I want you to see the logs of what's happening I'm actually going to run MGMT in a terminal so you can actually see what's happening instead of doing it automatically and the way I do that I just have the same thing I'm just gonna run MGMT. I just have a little script that runs it up So what I have here on the left I have MGMT running on my host Which declared the state of one VM to be running and then inside that VM on the right I have just a console of what's happening and here's the fun part. Here's what I'm gonna do I'm gonna tell you the trick and then I'm gonna show you happen. So the VM resource remember it's declarative So I say I want this number of CPUs that's much RAM and so on But if I tell MGMT that that value should be two instead of one for the number of CPUs MGMT will notice that the user requested a different value and in the case of virtual CPUs It will hot plug or hot unplug the CPU count in that running VM So in real time in seconds you can hot plug or hot unplug by just changing the number And to get that number I Have a little text file that MGMT is watching so just the way I had a sound microphone input in this case. I have a Little CPU count text file that just says a number and MGMT is taking the value the integer from that file as the input Now that's part one. So when it this value changes it hot plugs run hot unplugged CPU in the VM MGMT is also running and MGMT also can use the CPU count as a function So it has a function called CPU count and it reads how many CPUs there are and so I'm just gonna hide this over here and Open up a new window. I have a little display CPUs Shell script and it's just looping and it's doing two things one It's running LS CPU so you can see how many CPUs the kernel is reporting The other thing is it's editing a file to put in the number of CPUs This could be doing something fancier, but it's just for simple So I'm gonna ask you to hot plug a CPU or hot unplug it'll change the CPUs and on the right It should react very quickly and respond. Does that make sense? Let's hope that should we see if this works Okay, so I'm just gonna echo say one CPU count and boom it's done See that we change it to two Goes to two you go to four if we wanted so very quickly it responds and then instantly we see four CPUs Is that cool? You want to do it faster? Faster or you've had enough You guys are shy audience who's shy just raise your hand so we know where you are anybody All right So I actually can do this faster. So I made a little crappy shell script Over here and all it does is the shell script when I press plus or minus It just echoes to that file one two three or four just so you can see how fast it goes So I'll move this over here And just as I press one you can see basically in in in virtually a second it goes up Is that fast? Is that cool that virtualization can VMware do this? I don't remember if that's a feature or not And so what apart from cool demos, maybe that's too ego-y But like the reason this is useful is because imagine you have some big host Running a whole bunch of stuff and you just squish all those VMs together because most people have their crappy website About their cat that never gets visited until that one day when there's a lot of traffic So then the load goes up so you dynamically see that the load is going up So you add more virtual CPUs and inside the thing you schedule more services to start and deal with that load and do all sorts of stuff like that. Is that cool? That's the point. You have a quick question sir Can it be used for what? fencing I Don't know Yeah, yeah, wait wait wait. I have more demos, so I'll show you a thing answer. I got a lot of demos so quickly We can do everything actually we haven't implemented it for RAM because it's really hard if you will help I'll help you right at the RAM support. It's trickier. The CPUs work really great The RAM is like a bit sketchy and there's some bugs, but it's doable So do you want to see some more demos? I have some more cool So I'm just gonna shut this down. You actually have to kill MGMT before you shut it down Otherwise it will just start it up again, right? So you shut that down. So I showed you some magic D demo stuff This I'm gonna skip so really quickly what I'm not going to show you demo is this language this FRP language You can actually look at the historical values that a variable had so I know what the load is today But I also know what the load was five seconds ago or ten seconds ago or whatever So what is this lovely photo of anybody scream it out? It's a thermostat This is my parents thermostat in Canada. You can see it's in the correct international units of Celsius I found a prettier picture for some sort of weird us I don't know what the units are, but I think they have their house far too hot just for clarity And this thermostat has an interesting property. Does anyone know what it is? I? Found the hecklers anyone know what the property is The property is called hysteresis and what hysteresis is basically when you let's say you want to have your house heated to 20 Celsius the temperature is gonna go up and the moment it hits 20 Celsius It's gonna click off But then it drops down zero point one degrees and it clicks back on and then back off and back on and back off And if it was doing this rapid oscillation You would hear all this clicking and it potentially break your furnace and it's not good So what hysteresis lets you do is when you hit a threshold you can delay Some time or distance before you click back on and we're gonna show you how that's useful. I'll see a demo Come on. Do I want to hear do you want to see a demo? Okay, it's the end of the day. I'm tired. I'm sorry, so Where's that demo? What was the demo? So I have this demo, so I'm gonna run the demo on the left It's running though. I can show you the code if you want all the code It's all in git so you can try these demos at home on your own laptops and just to sort of Show what's going on? I'm gonna run this watch command on the right So it's basically just every half a second It's padding some text file and then running echo just to put a new line and then it's running verse Which is the VM list command with Libvert? Listing over and over again make sense so you can see what's running We got two VMs running one and two and MGMT is running here and what I've done you can see right here I'm printing out the system load in that text file and I'm also showing you a threshold of 1.5 so when the threshold gets above 1.5 I've told it to shut down one of the VMs and Then only when the threshold stays below 1.5 for 10 seconds will it start up that VM again? Does that make sense? And we'll talk about why this is useful after So I'm just gonna move this down here. So watch so I'm just gonna just artificially increase the load just to sort of Show you that something is happening And watch the value go up It's going up pretty fast so watch the second hits 1.5 There's actually a bug in this so I hope I didn't hit the bug There we go see it hit 1.7 and you see how MGMT to the second VM shut down you see that So watch so the load is still high Now watch what happens when it goes below 1.5 nothing's gonna happen for 10 seconds. Hopefully so it's at 1.6. It's dropping 1.6 1.5 0 0 so not quite or so so Five seconds, I don't know where the rounding error is Okay, so in five seconds four three two one there we go and it started right back up. Is that cool? You like that you can clap if you want So Obviously the implications are pretty obvious. You have a whole bunch of load on a whole bunch of machines You rescheduled them somewhere else and so on and and like Trying to buy the VMware plugin to do this would be like a million dollars and who knows how it works So the whole point of having this light flexible language is that you can build these dynamic systems in real time to do what you want and Say you're one of those scared people that doesn't want to touch code First of all you're crazy, but second all you don't have to someone else could build a module for your company That does the use cases of a bunch of things so instead of building this huge Kubernetes or open-stack monolith or whatever someone would just build a small MGMT module that's customized for that You know a whole bunch of use cases and if you want those use cases It's a month of work for some developer to build the thing for you and a different month for a different kind of product And so on so you fast dynamic Right. I think that's how software should be So I have more stuff to show you unless you're fed up you want to see more That was better. That was better This is pasta everybody. Okay, so hysteresis so some interesting nerd properties and the language and engine The language is something called FRP. I'm not a genius at FRPs So I'm kind of making it up as I go along, but the cool thing is the language and theory shouldn't crash But it actually can because like a runtime errors like a hardware failure or something like that, but technically We're trying to eliminate all sorts of errors at runtime But even if there was a runtime error in the language The language is actually pushing a stream of graphs to the engine that does the work So worst case scenario there the worst case scenario the engine would just be sitting there without a new graph for a while So not perfect Remember how the code was out of order the little strange But that's actually because the code is a graph like in the graph dag sense of the form So you could write code that's out of order and will still work properly You probably shouldn't do this because you would be insane, but just a weird sort of quirk about the language We had a lot of safety properties into this because we don't want you to blow away a data center Because you made a bug so variables are immutable So if you did x equals 5 and then x equals 7 this would be a compile time error, right? There's no nil variables in the language So you cannot set something that's like a nil pointer exception or anything bad like that I talked about hysteresis There's different kinds and you can use this to make all sorts of complex decisions Historicis is useful because it's a real-life thing that we like, you know think about like imagine like oh I'm going to open this door the door handle is hot. Huh? Okay. I'm not going to touch it again right away, right? Your brain has all these ideas Something's not working now you think about it. It's a real-life thing Reactive variables super useful Eventually we could have probably also model error scenarios as reactive variables I don't really want to talk about that because it's kind of niche But I'm talking at a really much more technical niche conference In two days, so there's going to be stuff about that if you're interested And if you're into graphs our import module system is actually a dag and it's this new system I just built to do all the module writing so you could actually build these huge standalone components. It worked out really elegantly I'm very proud of it and All upstream so if you want to take a look Excuse me So there's a lot of stuff still to do. How much time do I have Mr. Timekeeper? Six minutes, so I don't have time to show you another demo Unfortunately, but I'll just tell you a bit more about the tool So we still need stuff to do it's still new in the project It's not we're really on the borderline of being production ready You could probably start using this for a lot of simple use cases even just doing a small VM automation project Something that you couldn't do with your traditional tools Play with this we have an AC we have an Amazon EC2 resource if you want to manage VM in the cloud thing Wherever that is so same sort of stuff if someone deletes your EC2 instance It will bring it right back you can shuffle them around We have a scheduler that works in real time so you could actually move VMs from cloud to other cloud or from EC2 to You know some other guys cloud if you want to so think about all these things New features still a lot of stuff to do There's some bugs and things we need to there's some code quality stuff that I think it's probably fine But it probably needs a little bit of love and this is all about This is all about you how can you help you can use this test it patch it share with your friends document it Start on github if you're into that kind of thing you write a blog post about it Even if you don't code like this was cool you can tweet it if you have Twitter you can discuss it with people in the hallways Hey that purple idea guys pretty cool hack on it just hack on this stuff, right? It's code It's out there. I left a job at a reasonably cool tech company to sort of live off my savings and do this And I'm trying to keep it all free software, and that's the goal But if I don't have people sending patches or sending me money eventually I'll have to stop or make it proprietary And that sort of sucks. We don't have funding. I'm just like graph of my money going downwards So I have a patreon just for fun That's me And like finding a hacker. It's very sexy. So if you want to feel sexy send me some money. That's cool Let's just recap You can't hear the audio, but it's Arthur Benjamin putting the cap back on his pen. It's a really bad joke I reuse all my jokes, so I'm sorry about that IRC MGM T config on free node. We hang out talk about nerdy things. There's a Twitter account and a mailing list Hosted by my former employer There's a technical blog of James. You all know about it now, so you can check that out RSS in the house I'm purple idea on IRC and Twitter and so on And if you some crazy person accepted all of my Fostum talks except for the main track stuff So I'm giving a talk tomorrow These are some talks I gave earlier today because everyone just accepted the talks and it was kind of absurd So I'm gonna talk about the same sort of thing, but with container scheduling so I'm gonna do a scheduling demo so you can actually Schedule containers very easily. You don't have to have a whole container thingy And then there's a small lightning talk, which you might not find interesting Right after Fostum on the fourth I'm at config management camp where I'm gonna give a really hard talk and then a really really hard talk And also Felix Frank did some cool work in MGM2 who's gonna be presenting as well So that's free if you can come if you can get a spot like you should come And on the last day on the 6th and again 30 minutes away. I'm holding a hackathon So all day will be writing MGMT code playing with it diving in new users experienced users Everybody's gonna be there hacking away. It's free. So if you can come feel free to Feedback if you like this talk, please find in Fostum organizer in the yellow shirts and go up to them and be like hey Like James's talk purple idea was really cool And if everyone goes up to them, that'd be awesome if you can't go to the schedule page on the website There's a secret link if you click on the talk at the bottom submit feedback This is real go click on it and say hey his talk was awesome And then maybe they'll be like hey we should have him for a main track talk because then you get a free hotel And that's awesome. So I'm almost out of time. I have some very expensive stickers I will give you if you promise to use them and stick them on your laptops then you can have a cool laptop like mine So if you want a sticker, I'll be just over here outside, and I'll give you one And that's really that thank you very much So I have I have like a few minutes wait wait wait a minute if you want a question or two I will Find it. I think the famous if you haven't met him Richard Jones is in the audience He's a famous hacker that makes all the plumbing stuff like all the vert magic all these things aren't possible I didn't write all the things underneath MGMT. So guys like him do great work to make things possible The system deep people and so on so bother Richard because he's shy But he's a genius and he helped me on the FRP path. So yeah questions Any questions I can take yeah, go ahead So the question is which hypervisor supported so we actually wrap libvert so anything that libvert supports theoretically works I only test with like QMU stuff and like KVM like the core Linux stuff It's known to work with others, but I don't really know Technically it also works with LXC, but I've never really tried it and if you have a bug like submit it It's probably gonna be a libvert bug Yeah How do you measure it sorry? Yeah So the event driven there's two basically events stuff the resources each resource the question was about events basically So the resources themselves have events to know when their state changes Which is so that the engine can change the state or run the state whatever the second source of events is in the language So those functions like date time and so on those suck in events from something So anything you want to have events from needs somewhere to get it so in the case of liver to the liver demon actually is watching the state of each VM and That's where it gets its information from that answer your question The load the load comes from the kernel so the kernel has a thing No, look at the code. It's amazing. It just pops out a new value every five seconds I don't think it's pulling but pretty sure I'll check but yeah any value you want There's there's almost nowhere that we were actually pulling I have to go But I'll be right here if you have any more questions and come up and I'll give you a sticker. Thank you so much