 Cool, okay, my name is Dave Boucher, I work for SaltStack, and I'm really excited about this talk. It's a fun one for me, it's kind of a crazy stuff you would never do in production or in real life, but kind of give you an idea of, you know, a greater vision of possibilities of things you can do with Salt. So, who here has used Salt either in a testing or production environment? Okay, so about 80% of us probably, grumpy guy shaking his head. Yeah, he does. Okay, so I'm going to just talk a little about some of the very basics, some of the general use cases of Salt, the people use Salt for, and then I'm gonna talk about some of the other pieces of Salt that people often kind of miss when they think of config management, they don't realize there's so much more to Salt than just that, and I have a couple of demos that will be interesting, and so get to use your cell phone, if you have a cell phone sitting down here, mine doesn't unfortunately, but we'll be able to, everybody will get a chance to participate. So, beyond config management. All right, so Salt basics, there's generally you would use Salt with a running a Salt master that controls other servers that are running an agent called the Salt Minion. If you have a situation where you don't need a constant connection to your servers, you can use Salt SSH to have Salt work across SSH in a one time, or not have that constant connection all the time, but just go over to SSH when needed. So this is kind of what the typically would look like. So you have your Salt master, you have a couple of Minions here, connecting through zero and Q, and then also Salt master switching out to one server through SSH. You'll notice the arrows are pointing from the Minions up to the master, because when you're using zero and Q, Salt uses a Pub-Sub interface, and the Minions actually connect to the master. So the master needs to have two ports open for the Minions to connect to, and the Minions listen on that published port. Whereas when you're using SSH, Salt master does reach out to your Minions and does things. When you're using the zero and Q in this Pub-Sub interface, that allows Salt to scale into the tens of thousands of servers, because the master doesn't have to specifically manage every server that's connecting to. That's why it's done like that. You can also use the Salt Minion by itself, so with no master at all, so everything's just done locally. Sometimes I'll do that on my laptop. I don't want, sometimes I'll have a master and a Minion running on my laptop, but other times I'll just use the Minion by itself to configure my laptop. People do that in production as well. If for some reason they can't or don't need a Salt master, they'll just use the Salt Minion by itself and do all the configurations locally. They'll pull them down from like S3 or other locations. A Cyndic allows you to have hierarchies of masters. So you could have a top level master that can see everything in your entire infrastructure, but then maybe you have a lower level master in different data centers or maybe different business groups have their own master down below and that allows you to have kind of a high level view from this top master, but lower level control down below. It also helps with scaling as well. Minions can also talk to multiple masters. So you can have a multi master situation where they'll accept commands from masters. Some people use that for a kind of poor man's failover type thing for your master or also for different people needing control or visibility into your servers. So advanced Salt features. One of the things that really differentiates Salt from others in this space is that fast encrypted dynamic connection that your servers have between the master and the minion. So we have an event bus that runs locally on the minions and also runs from minions up to your master. So events, things that happen on your Salt infrastructure get passed up to the master for various purposes. Salt itself sends lots of events. So when a minion starts up, when a new minion tries to connect to the master, there'll be an event. There are a whole variety of things that would cause an event to be sent up to the master. You can also send your own events. So you can have your own application send an event to the Salt master. There's a whole variety of things you can do with that. One of those things is a beacon. So a beacon is a small service that'll run on your minion that will monitor something. So we have beacons that'll monitor your file system. They'll monitor your SSH logins, whole variety of things. And we'll be talking about a Twilio beacon that monitors a Twilio text message queue and then send an event on the event bus. Reactors and engines. So the reactor sits on your Salt master and it sits there and listens to that event bus and allows you to do things, react to those events. So maybe there's that one stupid application you have to manage that needs to be rebooted every time it hits a certain memory level. Maybe you could check for that and instead of having to manually go in and do that yourself and events get sent up to the master saying, hey, this dumb application needs to be restarted and then the reactor sees that event, restart your application, it moves on, logs it somewhere and you know, barely a hiccup. Just a simple example. So if the engines are similar to that, they're a Python interface. So you can create your own little programs that will listen to that event bus and do things in Python. So there's a little more flexibility there, your Python. Tonight we'll also use a Salt API. You can turn on a REST interface that allows you to reach out to Salt from any other application and do things with that. So if you have an existing CMDB and this is an application that manages things on your network, now Salt's entire, all the possibilities of Salt are now available to your application. So we have a Python interface as well as this REST API and really makes for some really interesting use cases there. So again, we'll demo that a little bit today as well. So again, it's just like an overview. They have a Minion here, beacons and engines run on the Minions, reactors, engines as well, and the API running your Salt Master. Okay, now we're gonna do live demos. So, yes. All right. I had to rebuild the server last night to make sure everything works smoothly. So if you see here, what's that? Yeah, exactly. All right, so you're probably all, I think after four years, I wouldn't remember that. Okay, so I have a master, I have one Minion that I'm controlling here, and we must be probably familiar with this whole slew of modules that you can run. So in all my Minions, I'm going to run disk.usage. We're gonna get that data back, almost instantly. Yeah, do we have, actually I could probably increase the, better? All right. This is how we work at Salt anyways. Over in the engineer section, all the lights are off like this, so. So you can see, you can also get that data back in different formats. So say I want that in JSON so I can ingest that with some other tool. There's the xm command output in JSON. You can get it in raw text, in the raw Python data structures that it comes back in. Everything in Salt is a data structure, so that allows you to do a whole variety of things, including managed output like this. You can also do command.run will allow you to run any arbitrary command on your systems here. And that was one of the things that got me really excited about Salt originally when I first found out about it was just, I felt like I had this power. I mean, finally I could reach out and know what's going on on the servers on my infrastructure. What's this space looking like? There's this one server that we're constantly checking. We have monitoring, but what's going on right now? You can run this commands, get that data back, and do something with that really, really fast. In fact, the remote execution was built first. When Salt was first written, it was just all it was, it was remote execution. And that kind of informs the rest of Salt, all the Salt states, all the event bus. Everything is based off that initial remote execution that Salt provides. This is a brand new server. So if we want to run a Salt state, just for those who haven't seen one before, let's do an Apache server here. Okay, so here we, this is a simple Salt state. Apache 2 is the ID declaration, which also will become the name of the package you want to use. We have a package state, and installed is the function that we're going to call. And you notice it's installed. So if the package is already there, it won't do anything. But if it's not, then it'll do the right thing. So let's install this thing. So we're on the state.sls Apache server. And the amount of time it's going to take here is mostly just apt checking to see if Apache is installed and then installing it here. So I know for a lot of you, this is like the very basics, but I just want to kind of give a quick overview of how this works. Yeah, it probably does. He mentioned it was probably doing an apt update as well. So that's a simple Salt state. Okay, there we go. 32 seconds. Installed all the dependencies we can see here. Again, this is all a data structure, so I could have gotten this back in JSON, but you can see here we got the new Apache 2. Installed all the dependencies and gave us information about that. Now at any time, I can run that again and it will do the same thing. It'll check to see if it's installed and it already has installed, so it's not going to do anything. I can also, here on the command line, this come here and we'll do the package.perge. And when you're on the command line, I'm using the actual execution module and this is actually going to do this. It's going to remove that package. So there we go, we're done. A friend of mine that runs a bunch of the websites for the major newspapers in Salt Lake was on a train heading home when Hartley was released a while back and he hopped onto the crappy Wi-Fi on the train, ran the little check using this to see which servers were vulnerable to it, got that list, got that list, ran the command to fix all of them and it was done in the whole process. Took six seconds to actually do. And so you just, seriously, that's one of the things literally that got my attention about Salt was the fast control that you had in the simplicity, yeah. Okay, so now I'm going to talk about beacon. So a beacon, like I mentioned before, is a little server so you can start on the minion that allows you to watch something. So I'm going to set up a beacon that will watch a specific file for any changes and when it sees a change in that file, it'll send that event up to the master saying, hey, this file changed. And I'm going to set up another one that will look in a directory and notify any changes there. So if I come in here, so here we have our beacon's definition. We use the I notify beacon. This does require the, I think it's Python, I notify Python module to be installed. So any event, anything that happens to this serve, slash serve slash blah dot txt file, Salt will send a message up to the master. For this slash serve slash testing directory, I'm just going to look for the close write event and send an event up to the master and we're going to recurse through any directories that are created. We're going to auto add any new files to it and then the interval says we're going to check every 10 seconds. Okay, so down here, I'm going to start a little listener. So you can see what's happening on the event bus. This is a great way to see what's going on when you're debugging. So let me go into slash serve directory. Here's that blah dot txt. So let's open this thing up and another line, close that. And then you can see right there, we've got a notification that, increase that size a little bit, that the file was modified, the change that happened, and the path to the file that happened. Going to that testing directory. So let's create a new file. You can see here in that slash serve testing directory, we had our close write, another write, you can also see there the swap file that Vim created was also modified there. So what could you do with this? Let's say you had some little sys admin that just won't, or some developer would not stop logging into production server, start editing files, getting in there and modifying things without going through your change management. You could watch your Etsy directory and anytime there was a change, an event was to be sent to the master and you could clobber it and say, nope. We do it. We had one more clients and they did the same thing with, yeah, you can send alert through pager duty, announce it to the world. You can also check for like SSH logins and things like that. So someone logs into a production web fraud end that you never do. You could just log them out and kick them out right off the server immediately if that gets, or you can just notify pager duty, notify their boss that someone was behaving. Do you have a question? Just spam half the company when somebody modified a file and they shouldn't be. Ha, ha, ha, ha, ha, ha, ha, ha, ha, ha, ha, ha, ha, ha, ha. Automated pink slip, I guess. I don't know. Krister. Yeah, so all this does is it sends on the vent bus. So then you can use an engine or a reactor to do whatever you wanna do. So it could be anything. It could be add something to your CMDB. Yeah, exactly. So Krister was talking about his security module for file integrity. This could be part of that. You know that file shouldn't be changing at four in the morning and you can get a notification. Then you could log that however, in whatever monitoring system you have, depending on the file, maybe do an alert if necessary. Just because you have visibility into that. There are a variety of modules. So if we come here to the beacons section. So we have a disk usage beacon. This one allows you to set thresholds for notifications to be sent. So you could say, hey, if you ever get above 85% on these specific file systems, send us a notification, that type of thing. That notify that we saw here, journal D versus system D. Same thing with load. You can do your criteria when you wanna send a message. People sometimes will use that for load balancing, that type of thing. Certain server gets maxed out. You know, you can get above so many, your load gets so high it allows me to speed up another VM to help manage the load. Services.ps for services that are running. There's a whole variety of things. So WTMP is for your logins, that type of thing. And we're gonna use a Twilio. Take the message today. Let's look at that code for these. You'll see that these are extremely simple. So if I come down to this Twilio thing here, you'll see that we have some documentation, we have some imports. We have a validate function. And then the only requirement you have is that you have a beacon function. You can see here we have our documentation. But this is the entirety of, so that's 30 lines of code including logging and nice clean formatting to check for these text messages as well as a check to see if they have images and things connected to it. So if you had a beacon, you had something else. Say you had an application that you wanted to monitor locally on your server, you very easily could create your own beacon to run your checks and send events as needed. Any questions on these beacons or anything? Sure. Yeah, so the question was, could you have a beacon that would watch a log file? And then if there's a specific error, notifies this log or something like that, you definitely could do that. Okay. So here we have a our Twilio account here. I think I exposed my account credentials in the previous went up in the file thing, so I'll change those before we, this gets on the internet, so. Yeah, I do. This is live streamed? Okay, well. We'll be changing that quickly. Again, send the interval. This WTMP again is for if I wanna check logins. So actually the first time I gave this talk I had a guy from the Apple in the audience. I didn't know he's from Apple and I forgot to take it off my server. It's like the next like day and a half I kept getting text messages that I had to think so if whenever somebody logged in it would send a text message to a person and he texts me back like, hey, could you take me off that server? So those are beacons. And again, you have reactors and engines that can do things with those events. So in the demo that I'm doing today, I'm not actually going to use a reactor or beacon. I'm going to have a little webpage that the SELT API is going to expose. And the JavaScript is actually going to bind to the event stream and listen for events from my Twilio account and update the webpage when it happens. So just a quick view of this. My friend and coworker, Seth House, actually wrote this in like four minutes for me. He's more of a JavaScript guru than I am. So here we're going to listen to the events on this event source. And then when we're going to update and create a new module or new elements on the webpage. So let's see what that looks like. Sorry, I'm losing track here. Okay, so here's our simple webpage. Also promoting SELT Conf in April. You got to come out, it's going to be awesome. Few of you have been to SELT Conf. And it's a good time, a lot of learning, and it's really fun conference. And is that you? No, it wasn't you, it was me going. Oh, that was Charles in the back. So if you text that number up there, you'll see the text will show up here. Somebody wants to send a... You can send a picture text as well. Please keep it G rated. This will be on the internet, but somebody wants to send a picture. Not on my zipper down, please. So again, what's happening here is my server is running a little beacon. And it's checking every five seconds that Twilio will count. And when it sees a text, it just sends an event on the event bus. And there's a multitude of things we could do there. We could have rackers happening, and you can do many things with each event that comes through. In this example here, we're just listening to that event stream. And on the Twilio ones, oh, that's a big one. Yeah. And we're just dumping these to the web page here. So obviously you wouldn't want to do this in production, but maybe you have a monitoring system. You have PagerDuty, you have a whole variety of things you might do to monitor things. And you can listen to these events and do things with that. Any questions about... Let's go so far, yeah. Good question. So all I did was added that config file. And then we started the menu service, and it'll process all the beacon definitions and do that. So... Yeah. Oh yeah. No, you easily could have that run. You could see that event, take that text of that event, and use command.run from the loop and do that. Yeah, so obviously all of the security precautions you would take, accepting input from external sources you would want to do here as well. Yeah. But this kind of thing you could do if you have a chat ops type scenario, people want to deploy from Slack or from IRC or something like that. You can do that with Salt. You could also set it up so they could text something and deploy from Salt. We had a customer that had this Windows application that would frequently lock up and we worked with them with their first line tech support. And it was a pretty simple thing to do, but their first line tech support people weren't technical enough to actually log in and restart this thing without screwing things up is really difficult. So they kept escalating it to someone who could do it correctly. But it was fairly frequent, like it had a lot of customers that would call in and do it. So we worked with them to add just this red button in their thing. And so the first level tech support could log in at the customer's account and say, hey, my thing, whatever's not working and hit that button. And that bottom one there, scary when I was a kid. Dave. Yeah, so they just hit this button and the higher level tech support people never had to bother with that again. They would just hit that and they gave them the ability to take care of the customers without having to manually go in and fix their problems there. So I was gonna text from here as well. I usually send it from my phone, but I get no bars in here at all. Screen on, there we go. All right, any other questions about this here? Is there anything you wanna, yeah, go ahead. Sure, should we get to the microphone? Hey, okay. First of all, the message queuing model that Salt uses seems to be quite different from any of the other configuration management systems that I've seen. And as I understand, it scales extremely well. What kind of volume can you actually handle in terms of agents that are connecting into that message queuing system for simultaneous processing on large fields of devices? A really good question. So it's one of those things that kind of depends, but we have people that are using up to, you know, 18,000, close to 20,000 minions connected to one master. You can, there are some difficulties when you get to that scale, sometimes just the flood of data that's flying back. You know, your NIC can't handle, your operating system can't handle that much data. But there's ways to humiliate that using Salt Syndix, we have lower level masters that handle all the cryptography and communication with the minions themselves that then wrap up all their results and send them back up to the high level master, that type of thing. But places like LinkedIn, I mean, they have, you know, every time I talk to them, it's like they're bigger, but like one point they had like 30,000 servers and I think it's like 100,000 servers now. Not necessarily all of them running off one Salt Master, but you know, a couple of different masters there. So, I mean, it's really common for people to run Salt on like one or two, three servers. And then we have a lot there in the, you know, five to 8,000 range very comfortably. Cool. I'm also very interested in network device configuration management. I don't know if there's anything you can speak to about Salt being used in that kind of a context. Sure. So we have some modules that we'll reach out to network devices and things. There's a somewhat new feature of Salt called a proxy minion that runs on a regular minion and allows you to take like a, say a network switch that you can't install the Salt minion on or something like that. Adobe, we're working with them on a, I hope I can mention this, but on a module, talk to some server hardware so they can configure the hardware and then bring up the servers and that type of thing. So this proxy minion allows you to, the proxy minion handles the communication with the device and then allows you to use all of Salt's modules and things like that as if it were a server and that type of thing. You have a small subset of, you know, it's things that make sense on that type of hardware. Yeah. Yeah, you're welcome. I can actually get that to work. Emoji we're actually breaking it because I had, I was trying to cast everything into a string and so it was actually breaking the, so I had to go in and hot patch that code. You're getting close to the line, people. Go ahead. When I run a command again, it's all my minions, which I don't have that many, like maybe 60 or something. The first time, if it hasn't been run in like a while, the first time a couple of the minions fail to respond, but then like if I hit it again, everything will be fine and happy for the next while as I'm using it, is that normal? Are they like sleeping minions? You have to kind of warm them up. Exactly. So what happens is by default, every 24 hours, the Salt Master will rotate its encryption key that uses for the communication on the published port and if the minions haven't communicated with the master, anyway, so the next time when that happens, the minions have to, when they connect to the published port, they can't read, they can't decrypt the load and so they'll request a key refresh. So if that happens like a midnight and your minions haven't talked to the master at eight o'clock in the morning when you log in, the first time you run a command, it can throw that off. So there is an option, so that's what happens. So you have to run like two commands to get them all working. There is an option to have the master do a test.ping after the key refresh so that it automatically updates all the keys like that. But that's what's happening there. Also, is there any kind of testing framework for Salt? So Salt tests itself internally because states inherently test themselves to an extent, but we don't have an official framework yet. We're kinda working on ideas on how to do that well. There is a plugin for Test Kitchen that's employee of Philip Packard wrote, so you can use Test Kitchen, which is a Chef tool to use Salt to test your states and build up a VM with all your states and then make sure that they're behaving appropriately. Some of the ideas we have with that is to be able to specify success parameters right next to the actual states themselves. So as you build up your states, you can specify, if I spent up a web server, when I access port 80, I should receive this type of text, that type of thing. But it's still kind of in the design stage at this point. And I'm not sure if that'll even be gotten to this year, so for the moment, that's all we have, so. So any other questions? Would you like to do? Okay, so you get immediate audience feedback, it's awesome. Yes, and Utah Dave is my IRC handle. I'm Utah underscore Dave on Twitter because there's a real estate agent that has Utah Dave. He actually yelled at me on Twitter once. Who's using my copyrighted nickname or whatever? I've been Utah Dave, yeah, that's for two. Actually don't have any on this server. So a reactor is a simple YAML file that sits on your master and you specify a tag that you want to listen to. And you can use Jinja to look at pieces of the data structure of the events that are coming through and do things with it. Now when you're creating this reactor loop needs to be really, really fast. So you do have access to Jinja, which means you have access to the salt modules and states and all that kind of thing. So technically you could run salt commands and various things from there, but you do not want to do that because you want the reactor matching on these tags really, really fast. Because then when it matches a tag, it will spin off another process, another worker process to actually run the commands you want. So here is a reactor that was listening for the Twilio thing. Or actually no, no, no, actually this is actually to send an SMS. So in the other demo that I was doing where every time somebody would log into the server it would send a message saying it's not so logged in. So what would happen? So this first thing here is the ID declaration. So it's just a name I made up for this. It could be anything you want. Local is, I'm gonna run an assault execution module, which is Twilio send SMS. Target is going to be Boucher, which was the name of my server. So that's the name of the ID of the server I had. I had to pass in some arguments. My Twilio count was just the arbitrary name I'd given the credentials that's from a Twilio count. And then this second thing here, the information from the event that was sent is inside this data structure that's inside of Jinja. So this double brace indicates we're inside of the Jinja environment. So I'm gonna get the host name. So it says host name logged into and then the ID of the server and then the timestamp. I'm wondering if I had that, that's wrong. Seems like it. Yeah, so yeah, I haven't used this in like a year. So I'm not sure if that should work, but essentially here I'm creating the text that we're gonna send in that, okay. Yeah, so I think this could also just be name. Depends on the data structure. So I would have to look at the data structure that gets sent when someone logs in. So if it was just names, for example, that's essentially the text that we're gonna send in the data, sorry, in the text we're sending. And then this is the two, so that's my cell phone number. And then this is the from, this is my Twilio number. So what that means is this is, this gets called whenever a specific tag is met. So if we come in here, so in your master config, I don't have one right now, so actually maybe I do. Yeah, so in your master config, you would have something like this. So your rector is the initial config section and then you have a tag you're gonna match on. So in this example, the tag is salt minion and then we have star, which is an actual wild card. So in reality, when a minion must start up, you would see like minion 01 there where the asterisk is and then start. So every time the master sees that, it will call first this start.sls and then this monitor.sls. So in our example here, we would have a rector and then we would look for the tag that came up for that login beacon. So you can also send your own events as well. So here's another example. When you get an event with a specific key, you would take that key, delete it, with this execution module. So really, you have the full power stall to do whatever you want. So you can run stage, you can do a whole variety of things. So people that use rectors to fix problems that have known solutions that happen regularly, the application developer just won't fix that thing. You can have salt fix that for you. You can send events, so if something happens, you can have it fixed that. They fire an order. So they start at the top and then move its way down. You can also do things with engines. I haven't actually had a chance to use engines yet, but they allow you to create a Python interface. It's like your own little program, so you can do whatever you want within this little Python program. See, there was, oh, it's gonna show something else. I forgot what it was. Okay, anyway, that's pretty much it. Yeah, well, there were a complementary than either or. So a beacon is the thing that's actually checking your file system. And it, yeah, in a generative event. So it's dumb in that it doesn't know what's gonna happen. You don't have to configure a beacon to do anything. It just check this thing, check it, check it, check it, and then send an event when the conditions are correct. And then you would use a reactor or an engine to listen for those events and then do something. It's that way you kind of get a separation of concerns because you may not want the beacon running on that server, it could be in a semi-hospital environment or something, or you may not want it to even know what's going on. So it would send an event and then the master would take care of doing something. Yeah, a master always runs the reactor. Actually, you know what? I think you can run reactors on minions now in standalone mode as well, but in the general case, the master runs the reactors, yeah. Well, that's pretty much the end of my talk. I'm going to stay and answer questions. Yes, so the question was, when would you use a reactor versus using an engine? I think you would do that when you need more program at control of everything that's going on. So when you're using a reactor, it's simple in that you just have the reactor, listens for a tag, and then does execute this list of SLS files, and then those SLS files are still essentially yaml with some ginger, and for more straightforward things, it's pretty easy to go. Sometimes it can be a little obtuse trying to figure out the data structure of the event that came up, and that can be a little tough. Whereas when you run an engine, you're basically, you're just in Python. So you kind of just, you get to decide all the flow of that, what's going on, and you get more program at control, whereas my experience has been, if you're doing a lot of ginger templating and things like that, I like to keep it at a minimum. If you're doing a whole bunch of it, you're gonna get into a big mess and give you a big headache. And so if you were doing complex things, I would look at using an engine, and you'd probably get into finding what complex means to you, but that's kind of where I would do that. Yeah, sure. Mm-hmm. Great question. This question was, he mentioned that there's a lot of configuration management tools out there, and what led to the creation of SALT and one of the main differences with SALT. So Tom Hatch was the founder of SALT. He worked for, first for a lot of large federal government agencies managing large, large infrastructures, and also worked for a music startup that also had some large infrastructures. And there were a couple things he just craved as the systems architect was, one, to know what's going on with the servers, you know? They would keep track of servers like in a CMDB or just in a spreadsheet, and it was always out of date, right? Everything should have this version of the software, but you go check in reality and things that happened, and people had updated things, some things auto-updated. He never knew what the current status was, and so he really wanted to be able to have that remote execution. He wanted fast control over his servers. That was kind of some of his main desires there, and he had used basically all the tools that were out there at the time, too. And so that's where SALT started, was the really fast remote execution, and that's one of the things that stands out, that SALT stands out with is it's really fast remote execution that's not just bolted on after the fact. It's the core part of what SALT is. SALT is also very pragmatic. So it's very rare. I've done a lot of, in the past, the first few years of SALT stack as a company, I did a lot of the professional services working with big infrastructures, JPMorgan Chase, LinkedIn, various places, and well I didn't work on LinkedIn's infrastructure, but worked with training on them and things, and it's rare that you can come in and just be like, everything's SALT. You always have people that are like, well this little team over here, I mean we're using Chef. We should be Chef to configure our servers, and we're not changing anything, we're not touching it. It's not gonna happen, and then there's other teams using another tool, and another team has their own homegrown, bash, Python, Perl behemoth that they've got going, and it's rare that you can just swap it all out. SALT makes it very easy to use it where it really makes sense for your infrastructure. It's very pluggable, so both from what SALT can do, if SALT doesn't do something you want it to do, you can create a new module and state for it very, very easily. If you have, especially for an execution module, it's very, very basic Python, people that never programmed before, people that have experienced in other languages, the Ruby developers, it's very straightforward to get involved in Python. But also, connecting to SALT, so we have a whole bunch of external authentication modules. Some people in this room here have written their own external authentication modules, pulling data, configuration data from their existing CMDBs. You know, they have their own CMDB that they had written, and it's not going anywhere. That's the source of truth for a lot of what they do. And just wrote a module, and now SALT pulls data out of their CMDB, and it just fits right in. So it's very easy to tie into everything that you already have existing on your infrastructure. We have puppet modules and chef modules that allow you to do chef solo runs, and puppet runs manage it. Wikipedia does that. They use SALT to orchestrate all their code releases and all their puppet runs, but they have something like, and the last time I talked to them they had something like 30,000 lines of puppet manifests, and they just didn't have the time or really desire to swap all that out. So they use SALT to manage that, to scale their puppet stuff out. We have Ansible, what do they call it, a list of servers? Playlists, no. Pardon? Not the playbooks, but they're a list of servers they have that they run commands against. I'm trying to blank, but you can use SALT. SALT can use those to run commands on servers. We're very pragmatic about integrating with everything that you want. So there's something like 15 or 16 different pluggable interfaces that SALT lets you touch and add onto if you need. One of our big customers in San Diego, they're spending like $5 million a year on their previous configuration management software that they were using, and they swapped that out for SALT. And the fact that they could get to the source code and they've basically totally revamped one of the modules. So everything else pretty much worked great for them. There's one module that just wasn't quite doing what they wanted. The fact that they could get on, do a pull request and make it work right and fix that bug was just fantastic for them. They just, they loved it. So they have the support of SALT stack at the company, but their expertise that they have on staff, syshabmonds and developers can go back into SALT itself. In fact, a lot of times when you're running code with SALT, you're running code that experts at LinkedIn that are managing 100,000 servers, some of the, a lot of the scalability that SALT has was based off of some of the larger users running into things at scale, because we don't have 100,000 bare metal servers sitting around a closet there at SALT stack at our headquarters to test against, right? So a lot of the best practices and lessons learned from some of the top people in the world get all the experience flows back into SALT. Like the reactors, so there's those people here in the room that was working with them and we were working on re-imaging, so that a tool that could re-image a server and just completely put a whole new OS and change it from like an application server to a database server or backwards or vice versa. So we're working on automating that. Then it occurred to us that like, well maybe what happens if something goes crazy and suddenly we want, it's thinking it wants to re-image 400 database servers. That could cause a major problem. So we edited an acuing system. I'm not sure if we actually got that fully implemented there, but edited an acuing system that would allow you to add things to a queue, so if there was like five that need to be re-imaged, okay that's normal, but if suddenly there was like 400, you could say stop and do that in an orderly way. And so, I don't know, I could go on and on, but there's really, I've done, professional services with some incredibly talented people and I've never used a tool that was so flexible. There's like, I've hardly ever run into a situation where Salt just could not do something. That's correct. So even in our enterprise software, Salt itself is Apache 2.0, completely open source. And for a couple of years, we actually provided packages specifically for our enterprise customers that we supported directly. And we're moving to a framework instead that they'll basically use the same packages, but we're building a tool called RAWS and a web GUI associated with that that sits on top of your masters that will allow you to have this really large scale to keep all that data and scale out to the tens and tens and tens of thousands of servers and view that at a pretty massive scale. That allows us to provide value for enterprise customers at these difficult, large scale situations, but also have everything else open source. So if you contribute code back to Salt, it is all Apache 2.0, so you're not gonna like contribute code back to Salt. And suddenly it's only in the enterprise version. I mean, it's all open source. It's available for use. That being said, we do now have support offerings for just open source. So if you have 500 servers and you would like to have support, but you're just running straight up on source, we actually support that now. So you can have a support contracts for that as well. So yeah, for a long time we didn't require you to do enterprise to get support, but we don't at this point. Yeah, so the question was the difference between writing a formula and having everything be like your top file or I guess your file roots. So Salt formula, if you go, we have a GitHub organization that we keep all of our Salt formulas in. And there's, I don't know, we're into the hundreds I think now at this point. So Salt formula is kind of a generalized, kind of a best practices way of doing something. So at this point they're pretty much generally, they're all community supported. We do provide some support to that, but our community really is rallied around and worked on those. And so basically you can just clone that Git repo or you can use Git FS on that Git repo. And so like most of them straight out of the box will give you an installation of, let's say, MySQL or something with some sane defaults. And then they're set up to allow you to pass in variables to configure that simply through the pillar system, that type of thing. So anytime you have like a generalized thing, Salt formulas are a good way to look at that. So say you have a MySQL expert on staff and she has everything just tuned just right. Well, other people in your organization can use her Salt formula to do the right thing without having to be an expert with MySQL, for example. And so that would be kind of where, whereas maybe somebody else has, a light H2PD state that they just use and so they would just stick that on their file roots on their master and no one else really cares about it. So they wouldn't necessarily make that like a public or at least public within your organization. The good thing about using the ones online as well that we have is that again you're using something that people have already gone through the pain of experience to get going. So obviously you need to test them to make sure they work according to your needs but sometimes it's a good way to get started on something complicated without having to do it all yourself. This is essentially what a Salt formula is. Yeah, there are three out right now. Actually no, there's four. They're all pretty good. So there's one, there's two by Joseph Hall who's one of the original four co-founders or original engineers of Salt. One's like kind of like a getting started Salt basics and then he just released another one that's more advanced use cases. I haven't read the second one yet but that just releases Joseph Hall. I'm not sure what the name of the book is. Yeah, there's another great one that was written by Colton Myers. And then a former engineer from LinkedIn also wrote one as well, drawing a blank on the name of that. But there's like three or four books out. And yeah, we have some Salt errors. We haven't done a lot of video tutorials yet. We do provide Salt training both in our office. We can do it in your office and we also do it remotely. So we have a remote video conferencing system that works really well for training as well. And it's actually a really great training. Our founder was actually a trainer for Red Hat for a long time and so is the training class. If you go to saltstat.com, the awesome thing on training. We have a guy in the back here that would love to talk to you about that as well. This guy in the green jacket. And I know I'm very biased but it's actually one of the better trainings that I've ever been a part of. I mean, you'll come out of it with a really good overview of everything salt can do as well as getting your hands dirty, doing all the configurations and even extending salt to an extent too. So you really come out with a great boost. In fact, when I was doing professional services, I would basically force them to do a training the first week we did stuff because it just helped for them to like know what we were doing because they had the training already. All right, I think we're out of time. I can sit around for a few minutes afterwards but thanks everybody for coming. I appreciate it.