 Yeah, so today, I'm going to be talking about how at Salt we moved from an agent-based solution to being able to use Salt without installing an agent. So yeah, my name is Megan Wilhite. I'm known as Shell on the interwebs. Bonus points to ever understand what Shell is. And I'm currently a software engineer at Saltstack, but I am working mostly on getting releases out right now. So that's me. So what is Salt? For anybody that doesn't know what Salt is, I think it's important to understand what Salt is before we go on and talking about Salt as a Sage, which is the agent-less solution. So Salt is a lot of things, but I would describe it as being able to do things such as config management, remote execution, automation, and orchestration. And it's very event-driven. So talking about how Salt is currently set up with the agent solution, it is a server agent communication model. So we refer to the server as the salt master, and we refer to the agent as the salt minion. And the way that you would set this up is you would install the master, and then you would install the minion. And there are a couple of network services that provide the transport layer within Salt. And we refer to them as the publisher and the request server. And these are using a zero on queues library. And the publisher essentially is sending out all the commands from the master to the minions. And the minions then determine from there whether they should actually run the command. And then we have the request server. And this is where we get a lot of the bi-directional communication between the master and the minion. So when talking about converting over to an agentless solution, obviously SSH is unidirectional. So we had to be careful in terms of how we architected that so that we could use the power of Salt but also use SSH. So I'm just going to show you a quick command within Salt that you can run. It's just kind of like our version of Hello World. It's just Salt, which is the name of the CLI tool that you use targeting whatever minion you want to target. And then test stop peeing is just what we refer to as an execution module. Execution modules are not meant to be idempotent. They're more meant to be run as one-off commands. And so, yeah, it would go through that job flow, as I mentioned before. The master would publish that command of the minion. The minion would then run it and then return it through the request server. So as you can see through the CLI, maybe you can't see that super well. But essentially all it does is it runs test stop peeing on the minion and then returns true if it's able to communicate with the minion. OK, another important thing when talking about how we architected Salt SSH is you need to understand the difference between Salt and Salt call. So Salt, as we show here, we're targeting all minions with the asterisks there and test stop peeing. That is the CLI tool you use on the master. And Salt call gives you the same power but on the minion. And specifically here, you can use either Salt call or Salt call dash dash local. Dash dash local will not try to communicate back to the master. It will keep all communications within the minion. And we use this within Salt SSH because that is how we create that unidirectional traffic so that it can all just run on the minion. So now that we understand a little bit about what Salt is, let's talk about how we're able to convert the Salt to being able to use SSH as well. So first of all, why? Why do we provide an avenue for people to run Salt? Obviously with SSH it is a bit slower, but there's a lot of positives to it as well. It gives our users a lot of variety. They can use both the agent and the agentless option. You don't have to choose one or the other. In fact, I know a lot of people have used Salt SSH to bootstrap their minions as well. So a lot of people use both of those. Obviously you don't have to install an agent. And it's pretty simple to set up and get running. So how do you get it up and running? There's a couple things. There's Python needs to be on the targeting system. Unless you're using the dash R argument. And that just runs a raw SSH command. So you don't get the power of Salt. But you need to also install Salt SSH. And then you need to set up a roster file. And this is an example of what a roster file is, just one of our default, what we refer to as a flat roster file. And as you can see, it's pretty simple. Web 1 is just an arbitrary name. Obviously you want to name it to something useful. Then the host is either the IP address or the DNS of the host name, user password to be able to properly authenticate to that host. And just in this example, we're using sudo, which just the user needs sudo privileges. Obviously there's more options that you can add within the roster. But that's just a simple example that you can get up and quickly get running. There are other rosters available. One great thing about Salt is it's very pluggable in a lot of aspects, including the roster. So you can actually write your own roster. But what's available now is it can work with Ansible. The cache, cache is the master cache. So if you've already got minions you're targeting with your master, you can use those that are in the cache already. Cloud, which is our Salt Cloud offering, interacts with AWS. If you create a VM in the cloud, you can then use Salt SSH directly to interact with that. ClusterShell, Flat is the default range scan. That's where you can just scan and define a network range. And also you can use your SSH config. But yeah, like I said, you can also create your own. OK, so let's show the same example that we showed with the Salt CLI tool. The only thing that's different is you now have to use the Salt SSH CLI command. We target what we want to target, whichever minion that is, and we run just the simple example, test.p. And as you would expect, we get the same thing. We get the response of true. So what is going on in the background? Obviously, you can see that we're getting the same result, but a lot of different things are happening to make this possible. So we're just using SSH as the transport layer. So what we're first going to do, and I've provided where in the code this is, that's just more for the curious minds. But we're going to set up what our target and our roster is. We're going to initialize that in our SSH class. And then we're going to generate what we refer to as the thin. So at this point, what it's going to be is we're going to determine what are all the dependencies that we're going to need in this thin to be able to run this on the minion. And it's also going to add the salt call binary to the thin. And then it's going to zip that up. And then we're going to generate a new process. And we just do that using the multiprocess library. And then in that new process, we're going to generate the file that does a lot of the logic within Salt SSH. And it's ssh underscore pi shim. And what we'll do is we'd have to take all the ops, like where's the directory, where's the salt call binary, all of those things that you need to run the command. And we're going to add that to that module. And then that module, when it's run, it's going to determine if it even needs to copy over that tar. So we use a checksum and a versioning. And if any of those have changed, then we would re-copy over the tar. But if we don't need to copy that, obviously the consecutive commands will be faster. So obviously that's a great part of Salt SSH and how we've tried to make it at least a bit faster, even though we're going over SSH. So yeah, once we've got the tar copied over, we're going to run that shim file. And that shim file actually runs the salt call command. So if we dive into the code a bit, we can kind of see what's going on here like we discussed in that flow chart. So we can see here, we're initializing the targets and the roster. And then there's a lot more things that are going on here, but this is just the key point. And then we're going to generate that thin. If we then look at this code, we can see that it's adding the salt call binary to that thin. It's adding what we refer to as talks, what that command does essentially is just go in and determine what are all of the library or modules that we need to include. And also, this extra mods a user can define additional modules that they want to add to that. We also add a version of the salt code, a Python version. And then we're going to use either gzip or zip. And we're going to tar that up. So then as I discussed, we spin up a new process. And this is just kind of showing what we do there. We're just using the multiprocessing library. And we're just queuing all of that up. And then a lot of the initiation of the shim starts here. It creates that command string. It's trying to determine what is going to be run in that shim command and adds all the arguments that we would need to it. And then it's going to run that shim command. And that shim command, for example, if the thin is not already copied over, it will determine that and return deploy. Or if the checksome is different or things like that. And then we see that deploy was returned. And we rerun that command. And then it would copy it over and run. The shim command just does a couple of things. It determines you can actually specify whether you need a TTY or not when using Salt SSH in the roster config. If it needs a TTY, it just runs our virtual terminal within the salt code. If you are using a TTY, sorry if not. And then if you are using a TTY, it needs to actually copy over that shim because you need to use that TTY. And it runs it from there. OK, so this is all the magic that's going on in that shim file. We're going to determine where the thin path would be. Then we're going to do a bunch of checks. There's more checks than I have shown here, but we do check the checksome. The version and all those things. And then, obviously, you can see if it's different, we're going to say we need a deployment. And then we're going to untar that new thin. And then as you can see here, we've got this salt call command. And that is the actual command that we're going to be running alongside that test stop ping that we passed to it. And then we just use the sub process module to run that. So all of that discussed is just talking and referring to salt's remote execution capabilities. Obviously, there's a lot more you can do within salt. So one of the really other popular aspects of salt is the state system. And this is where we get a lot of our item potent behavior. So we refer to these SLS files is where you would define everything that you would want your machine to have to be in that certain state. And this is just a simple example. This one's using YAML. There are multiple other renders that you can use within salt. But YAML is one of the defaults. And this is a pretty simple example. We're just managing a file and we want to state or we want to put it worked into that file. So as you can see, we run it with salt SSH. We run state.SLS, which is we're going to run that SLS file. And it adds that new file and shows that a new change has occurred. Now, if you were to run this again, as shown here, because it's item potent behavior, it doesn't try to change anything. And it just says it's in the correct state. So we have the execution modules, but we now have the state system. And the state system requires a little bit more because we have those SLS files. So we need to have some sort of deterministic behavior to figure out what files we also need to include and copy over to be able to run the state command. And what happens is it's the same as the execution module, except for when it comes to that command block. Command or function, we run this one. Run wfuncts. And it essentially stands for wrapper functions. We have a bunch of or a couple of wrapper modules within Salt SSH, and they're all displayed right there. And one of the things that makes it a bit tricky with using Salt SSH is because we've had to create these wrapper modules, and you'll see what we've done differently in the state.py example, it gets a little bit hard to manage both of those files. Because within Salt's code, we have a state.py in Salt SSH. We have a state.py. So sometimes it becomes a little bit difficult to manage both of those files and make sure if the feature is added to one that we have to add it to the other. So that is one problem that we've run into with this current setup. So this function will set up all the pillar and grains. And within Salt, pillar and grains is just a lot of data. Pillar being usually people use that for more of the sensitive information. And it's stored on the master. And grains being information about the minion stored on the minion. And any other ops that we need to be able to add to run these state files. Then we call all of those wrapper functions. And in this example, we're specifically going to call the state wrapper function. So as you can see that we're doing here is we're grabbing all the data that we're going to need for the call. We're initializing the function wrapper. And then we eventually we're just going to run it and return the JSON of the return. And this is how we're doing it in the state.py file. So in a normal SLS file, we do not have all these extra steps. Because what these extra steps are doing for Salt SSH is, as you can see, we're preparing the tar. So within the tar, we determined what SLS files need to be added. One of the things that is a bit tricky is when people use includes in their file, you actually have to specify those separately in a configuration. So that is one thing I think could be improved upon as well. But yeah, so that prep trans tar, that's going to do all that magic there. And then we're going to sum up that and get it a hash. And then we're going to do the same thing we did before initialize a single class, send over that tar, and then we'll run that command block again and run essentially that same logic. So now that we've shown you what's going on underneath, you'll probably not remember all of that code. So what are some of the key architectural decisions to summarize all of that? One thing that's important to know is Salt was not modified to add Salt SSH. Salt SSH was added on top of it. And that's where you can see things like the wrapper functions and things like that where we didn't want to edit Salt. We use the tar ball where we tar everything up on the master. We determined what needs to be included on that, and then we send it over and run it on the Minion. As shown before, we use that Salt call. So Salt call dash dash local, that's what provides us with the unidirectional communication. And we won't need to call out back to the master while the Minion is running. It's able to run everything it needs right there. It has all the information it needs. And then we use these wrapper functions where we're trying to provide the same capabilities that Salt has but add it to Salt SSH. So what were some of the challenges when doing this? I've mentioned a lot of them already. Obviously, we lose some of the bi-directional where with SSH we only have unidirectional, so that was a challenge. The wrapper functions, as I spoke before, is a bit difficult in terms of managing the code. And a lot of times when we see bugs within Salt SSH is because we don't have all the correct ops that we need, and ops is just a lot of the configs that we would need to run a Salt SSH command. So that is one of the challenges that I think we could do a bit better on. And kind of make sure we have everything we need to be able to run within Salt and Salt SSH so there is more parity between the two. OK, so what are our plans for the future? We actually are initially talking right now. We don't have any set plans, but we do want to rearchitect Salt SSH and fix some of these issues that we have presented. Some of the ideas that we've been thinking on is trying to remove that Python dependency. And how you do that, we have had some ideas of just including the minimal Python executable within the tarball. Another reason I think this will be beneficial is eventually Python 2 is going to go away. And obviously, users are still going to be using machines that have Python 2 on it. So if we remove the Python dependency and we include it in there, that won't be an issue. If we do go Python 3, we can use some of the capabilities of async.io. And hopefully, we can resolve some of the issues that we stated, such as the wrapper functions and so forth. Does anybody have questions? Yeah, I don't know how to turn the light on. Hi. By trial and error. Now that you can see me, does anybody have any questions? Yeah. Yeah, essentially, you don't have that agent. That is the benefit that I can see over Salt. It is slower because you don't, obviously, using SSH. But that is the benefit. You don't have to install the agent. Huh? Sorry. OK, go ahead. You can go again. I've been using Ansible a lot. OK. So I'm kind of curious, how do you, so have you done any testing on performance files if you compare this solution with Ansible, for example? I haven't personally done. Bo, have you done? I know you've done a lot of work with that. Have you done any performance testing between the two? That could be a question of my problem. OK. I'm not going to do it anymore. It's just our problem at the beginning. So like faster than Ansible, because it's all the time. No, no, France are basically, yeah, OK, France, yeah. But as you mentioned, do you copy the Salt binary? There's just the Salt call. And then, yeah, all the other dependencies you would need. OK, so if you have a system where no binary exists, what happens to it? The system doesn't have Salt on it. That's what it does. It copies over whatever salt binaries that it would need in all the modules within it, yeah. It's the same function as Ansible in that case. OK. I don't know. Ansible initiate the connection. After that, it copies some binaries. After that, it executes commands. OK, yeah. A little question here. Yeah. A colleague is asking about, what about Windows? Windows, we do have a Windows solution. It is enterprise only, though. WinRM? Yeah, yep, uses WinRM. Yep, oh yeah, I should probably repeat the question, too. It's been a long day, OK. Having Salt as H having the same functionality as Ansible, unless you can explain to me how I can talk to an, let's say, a dump device, which I can do with Ansible, which is much like using expect. Sorry. I quite understand that because the speakers are back there. Can you repeat the question? So I'm interested in functionality of, I'm currently also an Ansible user, but I'm interested in functionality in talking to, let's say, dump devices, which have nothing on it, where I need something like expect, which I can do with Ansible, but I don't know how to do it with Salt. With expect, is that what you're asking? How would you, sorry, I'm having a hard time understanding. Oh, like network switches, things like that? Oh yeah, so we have what's called Salt Proxy, and that's our solution for not even having to have Python. We actually have a solution with Salt as H, and it's Salt Proxy. I don't know what's going on underneath, but a lot of our network modules and all of that heavily use Salt Proxy, so you'd want to look at that. I don't use Salt, so maybe the question is not exactly appropriate, but I was thinking about where do you store the configuration about the machines? When you don't have a model like Minions and Servers, where do you store your assets, the information about the machine you will connect to? They are described on client. The information such as? Let me say if you use a password or a certificate which is the user you use to connect to the host, do you have different YAML or configuration files, one for each client which uses SS and Salt as H? I think what you're asking is where is the information stored about the Minion? Yeah, yeah. Because just to complete the question, because I was thinking about when you have a server, the information about the host you are managing with Salt is in a way centralized on the server. When you use this kind of approach, those disinformation are spread over the clients. OK, so I think what you're asking is and what you're getting at is we have a concept of what we call pillar in grains. And pillar is what you would specify on the master. And a lot of people use these for passwords. And yeah, it is just like a YAML file. You can use whatever render. The grains is initialized normally in the Saltmaster Salt Minion. It's initialized on the Minion. And it automatically determines all that information like operating system, host name, things like that. So it's automatically determining that. And that's what grains are. You also have the concept of the Saltmine. And the Saltmine will give you the ability to query all these Minions and cache the information on the master. This is the pattern where you have the Minions on your. Are you talking about like a saltmine vectoring? For example, for example. And then there is the roster. Yes, oh, OK. So where you're specifying what your hosts are and what. Yeah, it's just the roster is what you would use. Because I was thinking about the adoption of this kind of pattern. When you have, for example, your operation department, they are 10, 20 single clients. How do you ensure that the configuration they have is up to date and so on? Like the Ansible inventory? It doesn't work under the softmines file. Yeah. OK. Yeah, OK. And you need to distribute and sync this information. Distribute and see the roster file. I managed just a flat file. Or you can use a different sort of roster and query that information differently. But by default, it's just a flat file. Yeah, it could be, can it be remote to centralized or just a flat file? Yeah, it's just a flat file by default. But like I was showing you, the other rosters, like you mentioned, you can use the target Ansible. And you can do things like scan the network and things like that. Or you can create your own roster. If you want to query that information for something else, like a database or something like that, you can create your own module and do that as well. OK. Sorry, hopefully I answered that well enough. So it's where there's initialization about the host. Ansible does what? IP for fear, IP for sex, for sex. Yes. General information about the host, how is this distributed? So IPv4, IPv6, interface, information. Like how do you use, does it use IPv4 and IPv6? No, the host information about this variable, it's how they... How do you query facts about the host? Yeah, so it would be grains. So you could just query the minion and use that. And you can run grains.items and that would grab all the information about all the minions you'd need. Or like I was referring to, we have this idea of the salt mine and it can kind of do the same thing. You can clarify what information you want and it would cache it on the master. And that does work with salt as the sage as well. Yeah, yeah, yeah. So from what I understand, now you would have two different commands you want to invoke depending on what kind of setup you have or what kind of minion you're targeting. Is there any plans to kind of unify them so that you would define in your roster, I guess? I don't know enough about it, but if you define in roster, okay, use classic agent base management with this minion and with this other minion, use salt as sage. So I know in our last release, or it's coming up in this next release, one of our users added the functionality so that you can actually use the salt CLI and target both SSH and minions. And the same. Was that your question? Yeah, so that is actually added just recently. There is some sort of argument you'd have to use. I don't know it off the top of my head. Any other questions? I believe. Yeah, so I'm almost positive you can because I didn't mention this, but before it actually submits that command to the minion, it actually is initializing a connection with the master event bus. And then when it gets that return, it actually returns that to the master event bus. So you could react off of that event. Yeah, so. Thanks. Thank you very much.