 Good morning. Thank you for being here. This is my first talk, so please be gentle with me or not. Thanks. We're gonna talk about salt, but before we dive into it, I guess all of you are developers. Developers, show of hands. Okay. And system administrators. Do we have some? Oh, that's really nice. That's really nice, guys. Okay, you're both going to enjoy this. So, this is the boring part. I'm gonna skip this really quickly. My name is Juan Manas Santos, as the lady said. You can find me on Twitter. I'm godlike64 on GitHub 2, Gmail. If you use IRC, I'm godlike on FreeNode. I'm an information system engineer, and just recently I got my Red Hat certified architect degree. I do work at Red Hat as a team leader and senior technical support engineer. And this is completely true. I need beer to survive, so if you want to buy me a beer, I'll be really grateful. Okay, so the first question is this. Do you know what configuration management is? Anybody has any idea? Okay. So the first question would be why do we need configuration management? Do we need configuration management? I don't know. What is configuration management? Configuration management is the way that you have to ensure that the state of your systems are consistent and not changed. So this may not sound like pretty glamorous or anything, but you'll see why. Computers were like this a long time ago, right? Or something like that. And administering these computers was really easy because you only just had one, but now they're more like this. And the more time goes by, the more systems you have. So the first reason of why do we need configuration management is quantity. The more systems you have, the harder it gets to make them behave. And system administrators here know this for a fact. So anybody remember this? I really hope you don't. A longer time ago, computers were like this. You had information stored on pieces of cardboard paper, just zeroes on one, and it was really simple, actually, if you knew binary by heart. Let's go back to the future. Does anybody know what this number means? Yeah. This is how many lines of code roughly there are in Linux kernel 3.19. So we've come a long way, right? No more cardboard. So that's our second reason. We have quantity and now we have complexity. And a more complex system is also one that is hard to maintain. And the third and final reason, this is the most compelling truth of all. If you're not convinced by now, we want to tell you, I know this piece of wisdom for a fact. Since I'm a sesame, I'm going to share it with you today. Are you ready? Okay. Here we go. Trumbroll. We are lazy people. So that's the most important reason of all. I'm going to tell you briefly how it all began for me with configuration management. Some time ago, a friend came to me and he said, I need help. At the same time, he was offering me a small job, part-time. So he needed help with a non-configuration management system that if not the most known, maybe still is the most known today. You may figure it out. What followed was a month-long frustration, stress, and a deep desire to inflict hurt upon myself and others. So after hope rose over, I got to thinking, why was it so hard? I mean, it shouldn't be so hard. I mean, I know how systems work. And I realized that the tool we were using was not made in the language we all know and love, but it was actually made in an alien language, an alien hostile language. Yeah, you may love, but I honestly think that the features of a language trickled down to the tools made with that language. So it was a solution, right? I mean, what could we use that did not get in a way? There has to be something simple, right? Something that we can easily understand and start using just right away and get to work with it. Is there a tool like this? Of course there is. I wanted to meet Salt, not this Salt, this Salt, which is also known as Salt Stack. And let's get with the basics. Salt is written in Python, which is already a good thing, or as I wouldn't be here, but not only that, Salt also uses both YAML and Jinja. And do we have Django developers in the room? Django developers? Show of hands? Okay, this is going to sound really familiar to you then, because these two things are also used in Django. Well, moving on. The first step for you to understand how Salt works is to look at the terminology it uses. So Salt works under a master meaning architecture, and the names are upon intended. So as we said before, configuration management is all about keeping the state of your system consistent. So in Salt, everything is a file. And in these state files, you define what your system is going to look like. Then you have a high state, and I promise we'll get back to that later, but it sounds real cool, so I put it in the slide. We have grains and pillar. We're also going to cover that. We have matching, which is the way of targeting your hosts and more stuff. But anyway, first thing I said was we have a master meaning architecture. So the master is basically the one who rules them all. And the minions, well, they live to serve. And actually, I wish to put a small picture of the recent movie, but I think because of a thing called copyright, I cannot do so. So you're going to have to make do with this. Minions are identified by an ID. They may or may not belong to a node group, which we're going to look at that later when we're into matching. And they provide grains of information. So my environment that I'm using for this talk is based in Gentoo Linux. So some of the things you see may not apply to your everyday run of a meal. We have Gantrator is the master, and everybody else is the minion, because that's the way things are. So the first and most important part are states and high states. As I said before, the state is the state where in a system should be. And this is the core of configuration management. It's something you're going to see regardless of the system that you're using, where it'd be Puppet, Ansible, Jeff, Insult2, you have states, maybe with another name in other configuration management systems. And as I said before, everything in Salt is a file. So they reside in the file system. They're called SLS files or Stalt State files. And when you need to define your Salt files, your state files, you just put something like this in the configuration file for the master. It's really, really simple. You just point it to a path where all your Salt states are going to be, and you identify it with what is currently known as an environment. So you can tear it down between development, QA production, and stuff. So what we must look at first is the top file. This file is a special state file, which is the entry point for the environment. Meaning, here we will define which hosts will match that environment. So we're going to know, thanks to this file, if a developing host should be in the development environment. So, okay, moving on. And this is more or less the syntax of the top file. We have the environment. We have a match statement. We must tell it which match type we're going to use. I know it sounds a bit confusing, but we'll be right back here later. And then you specify the state trees. And this is how a Salt file looks like. So this is pure YAML, right? We have the base environment. This is just an identifier. I have another group defined, which I'm going to cover that later. And we said, okay, you have to match the node group and you're going to apply all these states. And this is just another identifier to match another node group and apply another state. So, moving on. You can really grow this into more complex things. Maybe it's not a subject for this talk, but we'll see. So states is where magic happens, as I said before. And this is the part where you get to be creative with your environment. You can write up anything you want. That makes sense. You have state trees. They're a combination of states. And the syntax is similar to the top.sls file. So this is something like this. You have an identifier for the state, a type, and the parameters. We're going to look at this right now. So this is a possible state that you may have. It's really, really simple. It just ensures that the Tmax package gets installed. And we also deploy a configuration file for Tmax. There will be no point in installing Tmax if we didn't have a configuration file previously defined. So this is simple and to the point that I think all of you are able to understand that solstice is like this. The syntax is really, really simple and that's the magic of it. But we can do more complex things. So instead of deploying a verbiting file, sometimes we need to deploy a template. For example, when you're dealing with Apache and virtual hosts. So you can do more complex things, as I was saying. Let's see. Here, I just said to Solstice that this file is a Jinxia template. And I am able to properly, you know, replace the values in it with data from Solstice. So in the next slide, we're going to look at it. And there you see. This one right here are grains, like I mentioned before, in Solstice. And they tell me for each minion the number of CPUs they have. So I don't need to write a static config file for each minion depending on the number of CPUs. I just write a template. And there we are. Okay. Moving on. This is another example of how a state could look like. This is a little more complex. But what I'm basically doing here is putting a small section in a file that is going to be banished by Solstice. And no matter what happens to that section in the file, Solstice will replace it if it has changed. So that's useful, for example, in an ETC host file, where you need to have a certain section of predefined hosts without changing the whole file. This lets you do that. And it even lets you properly mark up the section. You know, start and end. Well, I mentioned high state before. You got to like the name, right? High state is the collection of all states that must be applied to a particular system. For example, if we wish to apply all relevant states to all relevant systems in our fictitious whole three tier environment, we would just do this. And this will apply all states that apply to any system that is currently registered to the salt master. So the way is more something like this. The high state will download the top SLS file. They will attempt to match against that file, see where they go. And then if a match is found, it will download all the modules or salt states, compile them and execute them. So by the end of it, we will have a happy system. So assuming we only have the previous TMAX file we defined, if we call a high state, we're going to see something like this. Salt is going to list the states that it applied. And it's going to say whether or not it could do it. And when they're started and the duration and okay. So the next important part for you to understand are matching and not groups. And matching is the way the salt master has to refer to its minions. They can be matched by ID, which is basically the host name. They can be matched by no group, which we're going to cover in a bit. Subnet, IP address, grains, or you can even do a compound match, which is a combination of any of the previous types of match you have. So we can match a single host. We can match a list of them. But this matching actually doesn't make any sense unless it's just a one time thing. If you're matching more than one host, we should really use node groups. Node groups are defined inside a special directory that comes with salt, which is ETC salt master T. It has a similar syntax to what we're used to. And now if we define the previous one was like this. So if we define a node group like that one, now we can match by the node group instead of putting all the host names or IDs actually. This is just a special salt command that you used to see if your servers are up, which more often than not, they're obviously not up. That's how we roll. And we can have as many node groups as we like. This last one here is a node group match using grains, which I know it's coming. Be patient. We're going to cover that in a bit. And if there's a word to define node groups, I think it will be flexibility. So as I said before, we can match whatever we want with them. Whole domains, single host, grain base matching, and we can even include node groups into another node group definition. So there's actually no excuse. You can apply this no matter what. Okay, grains and pillars. Grains are bits and pieces of information about the minions, about the hardware they're running in, the operating system, the Python version, and stuff like that. They are generated and loaded on Minion startup. So they are kind of static information, but still. And then you can list them and you can get their values. And with salt, there are already 50, 50 plus grains that are shipped by default. But here's the beautiful part, is that you can write your own. So writing your own grain is basically writing Python code. You just have to Python and return a dict. They go in a special directory which you are seeing there, underscore grains inside the master tree of your states. And well this is something I just did because when I started using salt, I needed to know the number of cores a system had. And right then and there using arithmetic expressions in the matching clause would not cut it. It wasn't pretty well developed. It is coming along though. So asking the number of cores was more than one wasn't feasible or more than anything. So I just made a small Python script that is a grain and it simply returns through if the system has more than one CPU. So after you place the grain in the directory, it gets seen to the minions and you can query it. You can ask for your custom grain and it will tell you whether or not your system in this case is a multi-core system. So that's it for grains for now. We also have pillar. And pillar is also data but it's not data that gets generated on its own. It's one defined by the administrator. It's torn the master and it is sent securely to the minions when needed. So again, since everything in salt is a file, pillar is also storing the file system. And you have a similar definition to tell salt where your pillar files will be. And I get it might be a little confusing the difference between grains and pillars. I know I got confused many, many times while using salt. So during my time with salt I came up with this little rule. I did it all by myself. Grains are data from minions. They're generated on the minions. They're retrieved by the master and they tell us things about the minions. And pillar is data that it gets sent to the minions. It's arbitrary data. We're going to have a couple of examples right now. But that's the important definition. Okay, so we're going to query CER pillar for our faithful Antioch hosts and we have nothing. Right, because we're supposed to be generating pillar data. So let's write up some pillar. I know this is getting boring, but please bear with me for a second. Yeah, pillar has a top file too. So this is an example of what a pillar will look like. It's just names and variables, right? So we're basically replacing our previous definition of assault state where we had the package name and the place where the file was stored. And after we do so we can query the pillar and we can see that it's already defined on the minion. So what does this allow us to do? With pillar data you can do fancy stuff like trimming down new states. So our team state from previously, it could look fairly clean now. You have all the relevant information in one place and you can just template your state in this way. But not also it looks fairly clean and we're avoiding information repetition because now the important parts are already in one place and should we need to change it, we just go to that place, which is the pillar. Okay, docs. This is the most important part. Actually the documentation of the assault project is really, really good. It's basically what I've used to make this presentation, apart from my small knowledge of assault. The URL is that one you have over there. Okay, I really encourage you to give it a try. It's Python. It's really easy on the eyes and you should be able to learn it pretty quickly should you need to. Okay, thank you. And now we have time for some questions. Are there any questions? Is this on? I have a two-parter. So the first part is I kind of came from the puppet world too. I wanted to do kind of a node classifier kind of thing, which is sort of a concept from puppet. So I started off pushing stuff to the grains like that this is the DNS master, for example. So I have now I can in my templates say to all of the things matching grains DNS master do these things, but then I realized there's a race condition where someone who could change the grains file on the endpoint between high state runs could promote something to be a role that it shouldn't be between runs. Because first, anyway, so then I said okay now I do it with pillar. Pillar makes sense, but it looks like the what you call the host groups that you just talked about is actually the best way to do that. The other groups. Yeah, and actually you cannot use pillar for node groups matching. You need to use grains. I am currently using pillar as my kind of most classifier and it works, but I think I'm doing it wrong. So thank you. I always like to find out when I'm doing it wrong. The second question is I guess that was a comment, mansplaining it to you. When a node's configuration state changes, as far as I can tell, the only place that gets logged is on the menu and I haven't been able to figure out how to have because the salt master, when you run a high state, you get back this report on standard out that tells you what changed, but I haven't seen any way in the salt master to make that get logged on the master. You crank up the debugging of the master for the logs to see if it changes and well, the bugging should really show you. It's going to be a lot of output, but if the bugging doesn't show you, you can run the master in foreground. I know this is not feasible, but yeah, first step would be to crank up debugging in the log file. So something foreman-like would be nice for salt one. Actually, foreman just recently included in the latest release support for salt integration. Yeah, I know foreman two and that's the thing that I really, really wanted to see in salt. Still, foreman is written in an alien language, which I don't like, but anyway, so it is coming. It is coming along. One thing I failed to mention is that salt is in heavy development, but things are getting way better by the minute. The documentation, as you say, is good, but sometimes it's hard to find the proper idiom to do with saying it's evolving. Yeah, yeah, yeah, I understand. That's my master minion, Gray and Spillers. Hi, thank you for the talk. It was very nice to see salt for the first time for me. I've got a few more beginner questions. The communication between master and minions. Is that encrypted? Yeah, it is. And you said that the grains come from the minion and pillar data is being sent to it. Yeah. This data on the minion, is that encrypted as well? Am I, or let me rephrase that, if I have to send, I don't know, AWS keys or something else in some way in my configuration, how would I do that? Okay, well, from what I understand, the pillar data gets sent to the minions on a state or high state run, and it gets interpreted there. So I actually don't know if it gets even stored on the minion. So it just gets sent. We could totally look that up afterwards. It's interesting what you mentioned, but from what I understand doesn't get stored. Hi. In my company we've been using BCFG2 for a few months now, and just after moving to BCFG2, I looked into salt, of course, because it was a good alternative. It came to me that, well, BCFG2 has a centralized configuration, a bit like salt, maybe a bit simpler to implement. Because salt, well, it's very, verbose, in fact, I think. Now, my question is, what happens if on a minion, the service that is running fails? How do you know that the host is not responsive? Do you have somehow on the master some monitoring of the aliveness of the minions, or do you have to do this manually? Okay. Whenever you need to interact with the minion, and the minion is not responding, the master has a timeout, and it will tell you that it couldn't contact the minion. But from my experience, there's no active connection. I mean, of course, the minions are listening to 0MQQ, that's where the orders from the master get sent, but other than that, it's not actively listening. So, follow-up question, you said that the connections were encrypted before that, but 0MQ does not have a secure protocol, I mean, as far as I know. Okay. I'm not that much into 0MQ, sorry. But from what you get from the documentation of salt is that it is a secure connection. So, actually, I don't know how they did it. Was language the only reason why you decided to choose salt stack over puppet, or was there any other reason? Salt stack over puppet, yeah. The language was the main reason, and the other reason was that I hated puppets so much. Any specific reason why you hated puppet? Sorry? Any specific reason why you hated puppet? Why do we need puppet? Why do you hate that puppet? I never really understood it. I mean, I don't get along with Ruby syntax, to be honest. It's just curious. It's okay. So, yeah, I've also been using puppet and Ansible. The way I see the pilot thing, it seems very similar to me to hiring puppet. So, is there any way in pilot to encrypt data, because normally that kind of centralized configuration is kept in a Git repo? Do you know if Pilar has any kind of that functionality to encrypt that? You can use, if you mean, can you place your pillar data somewhere else, other than a normal file system? Yes, you can do that, for example, Git. No, I mean, the content that you put there, I mean, do you know if Pilar has any kind of encryption, like if I wanted to store secrets in Pilar, like passwords or, you know, SSH keys, whatever, is there any kind of functionality with Pilar that you can encrypt that? So, you can have that in a Git repo that is public? I understand. Okay. Not by itself, but there are just files. So, they have to be accessible when Salt is running and needs to retrieve Pilar data. Other than that, Salt does not have an encryption mechanism for the files that you have stored, but you can encrypt a file system and, you know, if it ever gets high check or anything. But ultimately, if they do gain access to the server, I mean, you're pretty much in the oven. I seriously didn't think I was going to get this much questions. So, they're really good, they're really bad. Just to the point about encrypting Pilar data, I think it's important to think about what the threat model is. If I had sensitive data encrypted, I still wouldn't want to put it on public GitHub because maybe somebody's going to be trying to brute force it. Maybe there's some vulnerability in the encryption mechanism I'm using. Maybe one of my sysadmins gets fired but the stuff is still out there and he has something in his head. And so, let's say you do have some magic way to encrypt the Pilar file. You still have to decrypt it on the master, which means that the decryption key lives in memory, which means if your threat model involves an attack or pawning your master server, then he has access to retrieve that clear text from memory. So, just don't get too paranoid. Think about your threat model. Yeah, thank you. Hi. So, it looks like Salt is awfully similar to Puppet. All the concepts are basically the same. So, I wonder what's the difference, like, feature-wise? And second part of the question, what are the differences community-wise? Like, if there's anything like Forge for Salt or you can download not working module that you'll have to fix, but it's still in start. Feature-wise, I didn't have the chance to dig that much into Puppet. I believe it has a bit more features. It's also older. But the basics for configuration management are there for Salt. I really want to check out the integration with Foreman. Hey, turn it on. I really want to check out the integration. Are you done? Okay. Check out Foreman and see where it leads because most of the Puppet magic actually comes from Foreman. And community-wise, unfortunately, not many people use Salt. So, this is also kind of like the aim of my talk to get you people using Salt. So, we have a bigger community. How old is it? Sorry? How old is it? Because when I looked at three years ago, I didn't find anything written in Python, so I had to use Puppet. Yeah, it's around that time, more or less, but it's quite new. Okay, thanks. You're welcome. Any more questions? Hi. I wanted to know how do you manage dependencies with Salt? For example, if you can make layers of states to implement dependencies and how do you manage the removal of files, for example? Okay. If you have to change a configuration, okay, you're going to add something new and until then, it is usually fine, but if you have to purge stuff. Okay. Well, due to the length of time we have, and because I had to cover from the basics right up to what you saw, I wasn't able to cover it, but you can use includes in your state files to indicate dependencies between each other. So, you can include states and make it depend on one another. I believe that answers your first question. About the second one, I never actually had to remove files. What I did do was remove pieces of text from the files. Which was in a state over there. You can use, for example, regex. I believe that answers your question. You can split it by files and you can require on the other definitions. For example, you're installing a database and you need a certain service. Of course, you have to have the service defined in another state file, but you can require. Yeah, the important thing to understand is that since we're in a declarative world, order inside the state file doesn't matter. So, you have to define your requirements inside the states, because if you trust things to be executed sequentially, maybe they're going to be out of order and something's going to fail. I have one more question. Actually, it's two questions, but related. What happens if the grains raise an exception, unhandled exception? What happens if a grain raises an exception? If when you look at something, it just doesn't work and something breaks, because somehow the local configuration on the minion is not compatible and things just blows up. Using a custom grain that you wrote, for example. Something like that. Well, the grain is not going to get indexed on the minion. You simply won't have that value. So, the question is, now the grain is interpreted when the master, when you issue the request, or is it cached when you create it and it's run regularly and the result is cached somewhere on the minion? When you create the grains, the grains are synced when the minion starts, but you can also force it to get synced. So, it's stored, but when is the code actually executed? On any of those two situations. But it doesn't get executed normally during the minion's lifespan. It just gets executed on startup when the minion starts to communicate with the master and gets the grains from the master, then it gets executed locally. And should you choose to sync the grains manually, they get executed at that point. Now, the second question which is related is, how do you manage logging? Let's say, I want that when my master command is run on the minions, the result, if there is an error or something like this is sent to, let's say, some external service like Sentry, is there an API to configure this? Yes, there is an API. It's kind of more advanced and I really do hope to be able to come back next year and do part two of the presentation. More questions? And if you do have more questions, you can look me up in the conference, I'll be here. Feel free to come to me and we'll talk and maybe I can show you my environment live. Thank you.