 Alright, let's get moving. I hope you guys have had an awesome DEF CON so far. Let's dive right in. So this is Bad Salt Adversarial DevOps. I'm going to start with a quick introduction, some background about myself. My name is Casey Erdman. You may know me by my handle ingame on Twitter or other aspects of the interwebs. I'm a giant offensive security nerd. That's why I'm here in the Red Team Village. I am the co-founder of DC 706, which is a Columbus Georgia DEF CON group. So if you're ever in the area, feel free to roll through. These are some of my professional certifications if you care about that type of thing. In my day-to-day though, I am a manager of an information security department. We are a very small team so I touch pretty much everything. We're in the kind of financial space so lots of offensive security stuff, pentesting that type of thing, incident response, good old PCI compliance, stuff like that. But I think the most interesting thing you should all know about me is that I'm a programmer gone rogue. Most of my background is in software development. And I just kind of got into CTFs and started hacking and I decided no more software development. Let's do the offensive security stuff. That's a lot more fun. So that's where a lot of my passion is and that's just a little bit about me. This is probably the most important slide today. If you don't take a picture of anything else, I'm going to give everybody a second to take a picture of this. All of the code, the slides for this talk, any of the PCAPs or stuff like that are at this GitHub repo and we'll be there. So give you guys a minute. Perfect. So let's move along. Here's our agenda for today. First we're going to hit some of the high points, introduce anybody to the concepts of configuration management, how we landed on Salt Stack, why this technology is even the focal point of this talk today. Then we're going to talk about some attacks against that platform. And then the meat of the talk is actually going to be hacking with that platform. We're talking about adversarial DevOps. We're talking about using these new fancy tools that are in our industry as an offensive person. How can I take advantage of these? And that's really where we're going to dive in and look at a couple of those things. And then of course we're going to also give our blue teamers the chance to stop anything that they see like this and we'll look over those investigation techniques at the end. All right. So what is Salt Stack? So to talk about Salt Stack, first we need to talk about configuration management. So configuration management isn't really new. We've got configurations we've been managing for a long time in the industry. But there's kind of new technologies that help us facilitate this idea. And the goal here is to establish some baseline and some integrity across our systems to make sure they look and act how they should and don't have any variance here. Some common new tools for configuration management you may have heard of are things like Puppet, Chef, Anseful and Salt Stack. So we're going to take a high level look at some of these tools. All of these tools can essentially accomplish the same goal. So I'm not saying anyone tools better than the other. I'm not sponsored by any of these companies. I just want to kind of give a narrative of how I landed on what I landed on. So starting with Puppet, Puppet is kind of one of the industry leading tools in this space here. They've been around for a while. It integrates really well with other products because this baseline purpose is the concept of configuration management. So when these tools first start coming out, you may have heard things like maybe Puppet and Chef. And it's this bundle of tools where Puppet would be the foundation and help you do the configuration management where something like Chef or Anseful would go then execute on those configurations. So that's kind of where things kind of get started. It's got a little bit of maturity, good foundation. But there's also other tools out there that have matured a lot more. And so Puppet and Chef can kind of just be replaced by Chef nowadays. Chef has come a long way, but at first it was more focused on infrastructure as code and not really configuration management. However, inherently when you're talking about infrastructure as code, you're defining how your machines look in some kind of code syntax or configuration. So naturally Chef kind of became its own configuration management utility. And the focus here is of course that continuous automation. You can continue upgrading, changing your configurations in your systems and also manage those configurations. Chef also provides a couple of neat other tools like Inspect or Habitat. I don't know a whole lot about these tools. I know Inspect is supposed to be some kind of cool like code audit tool of some sort. But the point is it's matured a lot more now and there's a lot of feature sets. It could be a good option when you're looking to explore the concept of configuration management for your infrastructure. Now Ansible is a little bit different from these other tools. So Ansible is more about the infrastructure automation, not so much the configuration management. Well, again, you can do pretty much everything with all of these tools. Ansible I like to think of as like mass SSHing or something like that. You're able to go and execute on hundreds and hundreds of systems at one time to apply whatever state or configuration that you're trying to apply. And so I like to equate maybe Puppet and Ansible as something a little bit closer to what SaltStack is. But again, it's less about configuration management and just more about the execution of the automation of whatever your configurations are. Alright, so we got all these great tools. Why did it land on SaltStack? So from a professional standpoint, when myself and the other members of like the DevOps team are kind of coordinating and looking at different ways to do configuration management on our infrastructure, we found that SaltStack scales a little bit easier. And the reason is because it's agent-based. So a lot of these other tools may require you to have access or credentials or go open up some firewall rules or network communications. There seems to be a lot more setup. Things are changing very rapidly with a lot of these tools. So what's true today, of course, may not be true later, but at the time of selection for a tool, SaltStack really scaled well. And of course, it does everything all the other tools do, and it just does it all in one. You don't need an Ansible and a Puppet. You could just use SaltStack and get all the same capabilities out. It doesn't make it better. It doesn't make it worse. You just have those options available to you. And SaltStack's primary purpose is for configuration management. That's kind of what it's built around. It's built around the concept of desired state. So desired state is this idea that your machine is going to match your configuration all the time no matter what. And if at any point it's not matching that desired state, it's going to correct itself and make sure it maintains that integrity. And that's kind of what SaltStack's built around. And because of kind of the agent-based nature, it integrates with kind of some older or more standardized systems a little bit easier. We're able to deploy this in anywhere from like cloud environments like AWS or Google Cloud or traditional data centers or co-locations. It's really easy. You just kind of set up a master and look at the architecture in a couple of slides. You set up a salt master, you deploy minions, you set the egress out of where you want. And you're good to go start configuring hundreds and hundreds of systems of SaltStack. So that's kind of how we landed on this from the standpoint. And I kind of took it in random with it for the majority of the talk. But just how we got there. This is SaltStack architecture. I'm not going to dive into this and I'll expect anybody to actually read it. It's really complex though. That's kind of why I wanted to show this slide. So the kind of key points here that we're going to look at is this idea of the master and then these little beacons or minions as they're actually called. So there's a lot of functionality in SaltStack. Please read more into the documentation. It's really cool. But this is the simplified version that's a lot easier to understand. So salt minions and salt masters. That's kind of what we're primarily focused on here. So the salt master is going to be listening on two ports, 4505 and 4506. 4505 is the publisher port. This is where the master is going to issue a command say, hey, go run. Who am I? Or whatever. And then 4506 is where the minion is going to pull up and publish its results. So the minion is going to go up to port 4505, grab that command, execute it and then return up to 4506 with whatever its output is. Now this request server also handles file management. So one of the cool things we'll do that we'll kind of look at a little bit later is manage files. So if you want to copy other configurations for other components or install applications from a custom repo, whatever have you, you can copy those. And that port is where the salt minion will go fetch or send commands, either one. So that request server. Now, of course, we have this execution of commands and the sending of response. Sounds kind of like a C2 server. So keep that in the back of your brain as we go forward. All right. Kind of wrap up the section configuration management. Why is it cool? What are we using this for? Well, you can enforce hardening standards with it. You can force those nasty compliance standards with it. Create uniformity across all your environments. And then ultimately the coolest thing that works both on the offensive side and the defensive side is making sure stuff is installed that should be and making sure that stuff that isn't installed shouldn't be. So you're really able to enforce that and help lock that down with this concept of desired state where the system will go and see what it should look like. If it doesn't match, it'll go try to correct itself. All right. So hopefully everybody's got a good overview of the concepts of configuration management, desired state, what all these tools are for. Let's hack SaltStack, right? So quick history lesson. SaltStack was released back in March 2011. This was a very stable or very not stable, very early version. The first stable release came out a couple months later in November 2011. And we're now on version 2019 to which was like in February. And this is true as of the time of typing of the slide. There may have been a new release like this week. I don't know. But the point here that I'm trying to illustrate is this is back in 2011. We're in what, 2019 now? About eight years. That doesn't sound like long ago because 2011, but it was eight years ago. Old things have bugs. SaltStack is now kind of an old thing. It's kind of weird to think about. But to illustrate the old things have bugs. Here's a list of kind of recent CDs for SaltStack. Just to kind of go over and glance real quick. This list is longer than here. This is just a snap of part of the NBD. But we got some code execution. We got some directory traversal, some information type of issues. The goal of this research wasn't any kind of vulnerability research specifically. So I encourage you to go dive down these your own. Just to show that SaltStack has had some problems. And what I did from there is kind of see if there's any proof of concepts out there. See if, based on what is known, what type of attack factor should I be challenging myself to go look for in a current version of SaltStack? And the first thing I kind of stumbled upon was this Evil Minions project. Now, this isn't really a proof of concept of exploiting any kind of vulnerability. It's actually just a load generator. It was made for purposes of going and testing how many Minions or processes a particular Minion service could handle. So it just spawned a whole bunch of Minions and test the load. So nothing really crazy in terms of exploiting vulnerabilities, but might be cool for some, like, a denial of service attack or whatever have you. That was just the first thing I stumbled on with some simple Googling and looking at search terms. I dug a little bit deeper, though, and I found Chicken Salt. So Chicken Salt is this neat little POC for abusing the communication between Minions and Masters when they do this handshake for the public key authentication. So this allows for a man-in-the-mill attack, which ultimately allows you to take over Minions. Now, this is important because while you may think, oh, well, if I can get access to the Master, well, then I can control everything on the network, right? Well, what if you could just stand up your own Master? You don't need access to that Master. And then you get everybody to talk to you, and now you can control all of the Minions and controls. That's what this kind of attack allows you to do. And what happens here is you'll sniff the handshake upon initial communication between a Minion and a Master. You would be able to grab the public key and the verification token for that particular communication. Then you command in the middle, serve up that public key and token you require, and bam, you can pawn all the Minions. Couldn't really find a CVE for this one, though. It didn't match any of the things that I had already found when doing initial research. It looks like it was probably patched in version 0.17.1. It might be related to that CVE that's on the slide there that has to do with Salt not validating SHH key request, which also resulted in a man-in-the-middle attack vector. It's just really hard to tell. All I know is it doesn't work now because I tried. However, when I tried, I found a new attack vector. So I kind of want to talk about modern man-in-the-middle attacks against Salt Stack. Now, this isn't any kind of crazy O-day. We're not doing anything super sophisticated. This is a really easy attack, and it's really not a high risk. It's actually just by design, how Salt works. And what I'm trying to communicate here is just awareness for good configuration standards of your Salt environment, because it's really easy to mitigate this type of issue, but by default, it is not mitigated. And that's just by design. It's not a vulnerability. It's not any kind of issue, but it is a potential risk that you need to be aware of, especially if you have Salt in your environments. So the issue is caused by the need to distribute keys. Once you install an agent on a box, it's got to go talk to the master, and it's got to do this kind of key exception thing. So we'll look at that on the next slide. So here's how the key exchange works. So you have a Salt Minion and your Salt Master, as we've seen. The Salt Minion is going to communicate initially the Salt Master and say, hey, here's my public key. Do you want to accept it? If you choose to accept it on the Salt Master side, it'll add that Minion into your queue, and now you can execute commands and facilitate communications with the Minion. So the problem here is that the Salt Minions will trust a master on initial communication no matter what. This is any master at any point in time. The solution to this problem, which is built into Salt already, is to give Salt Minions a fingerprint for the good known master. So everything's good, right? We give them the fingerprint and we're good to go. Well, how do you give the Minion the fingerprint? Fingerprints have to be issued on initial deployment. If you don't issue them on initial deployment, they won't be set. Therefore, making you potentially vulnerable to this type of attack factor. Now, it's really easy to fix this issue, as we'll see, but here's the kind of capabilities and control that you have. So let's hop into this demo real quick. So here we've set up Better Cap with a man in the middle listening on our two Salt Ports for our fake master. And then here is our Salt Minion. So the Salt Minion is going to just show you what our configurations are. That's pointing to this 128.13 master. There's no master fingerprint set by default. This is Fresh and Salt. And it started the service. So now it should call home. This is going to show this is the legitimate master. It's going to be .13. And we'll list our keys here. And there's no unaccepted keys, even though there should be. We should be getting a call back from that Minion, right? Well, if you go back over to our Evil Master and see we got some communications from our interception from men in the middle and run this. We're going to show that we're not the same IP. And now we're going to see what unaccepted keys are in our Bandmasters queue. Oh, it looks like we've got one. So we're going to accept that key. And now we've successfully man in the middle of that Minion and we're ready to execute commands on this host. So that's going to have the attack plays out. Real simple, not anything complex to it, but it is very much an attack vector and is set, it is caused by this default configuration. The good news here is this only works on initial communication. Like I said, it's fairly low risk. What happens is on that initial communication, once a master accepts the Minion communication, it's going to cache that fingerprint that we talked about. And it's temporary, to some extent, I say temporary, it is stored on the file system and is there, but it is not in the configuration, which means if at any point that cache gets cleared out for any reason by whether manual deletion or some debugging or reinstallation of a Minion, whatever have you, it could potentially go away. However, if you have Minions already in your environment, they've already communicated with your legitimate master, what will actually happen if I try to do that man in the middle attack is there's a conditional set. So the people who make SaltStack are already aware of this particular issue and this nice helpful log message that says, hey, you might be getting man in the middle attack, well dump into the SaltMinion logs. So it kind of fixes itself if you've already had that initial communication, but if you haven't, then you're open to this potential vulnerability. Now detecting this type of attack is fairly simple. In the first image we have here, a successful attack basically looks like nothing. It's completely normal, you don't see anything going wrong. A failed attack, however, has that log message that we looked at on the previous slide. Now there are two ways to prevent this type of attack. Of course, the best way is to deploy your SaltMinions with the fingerprint and that's part of the configuration. Real simple, there's no reason not to do it, doesn't break anything, doesn't cost anything extra, you literally just spit out on your legit master whatever the fingerprint is, go pop it in your Minion Configure whenever you're setting up Minions and you will never be vulnerable to this type of attack. Real easy. Option two is you could automatically set all of your Minions so it'll always be accepted on master. This will work and kind of mitigate after that initial communication, but of course if an attack is already on your network man in the middle and all the things, option two doesn't really work and kind of keep intercepting stuff. With that said, this is a pretty tight attack vector. It's lower risk. The conditions to kind of pull this off, you would first need access to the target environment. That's easy, we can get into any environment, right? Next up would be that we need to be able to pull off man in the middle attacks in the environment. This could be a little bit harder if you have a fancier network segmentation or other kind of rules to catch you going on, but still easier than most other types of attacks. The catch is that you would definitely need to catch your target in the middle of some kind of minion deployment. So hopefully you hacked a DevOps engineers calendar and you know when they're doing the next salt deployment or something like that. That's really neat. Good luck, you could try to pull it off. I don't recommend doing this if you're on like some kind of red team or a pentest engagement. It's probably not going to be your best way in, but if all the cards align, you could potentially get in with this method. And there's no really notion of them patching this because like I said, it's not really a vulnerability. It's not us managing our configurations properly. If you go out and you establish your fingerprint, you're never going to have a problem with this. So I just kind of wanted to raise awareness there because this is not the default. There's no fingerprints that it can't be because it's not going to know what your custom master is once you spun it up. Be sure you go set that fingerprint so you avoid this type of issue. All right, now that we've looked at attacking salt a few different ways, let's get into the meat of what I want to bring to the table today. And that is hacking with salt stack. To start off, I want to talk about some C2s. So hopefully most of us are familiar with a couple of command and control frameworks such as Merlin or PowerShell Empire or the Metasploit framework and Salt Stack. Yeah, so if you remember that little architecture slide from Saltmaster and Minions, we had that little C2 type communication. We can execute commands from the master and the Minions will post their responses. So I kind of wanted to just pick out two C2s that I felt related to Salt Stack. And I felt Empire and Merlin were probably the best candidates, but I'm a fan boy of Merlin, so pick Merlin instead. And the reason being is because the communications work very similar. Now while Merlin and PowerShell and Empire both operate on like a single port communication, the idea of the master service posting commands, your agent coming and connecting, establishing that command, and then pulling back up to send the response is kind of the same type of thing. So it's just this pulling agent idea that makes them similar. So let's compare these two. First thing at Merlin, is it a C2? Yeah, that's what it's built for. It does command and control. Does it pretty well. The traffic for Merlin is disguised to look like HTTPS traffic. So it looks like normal web calls coming out of your environment, helpful for getting around some firewall rules or maybe just doing some light evasion so you're not instantly being detected on a network. It's got some really cool post-exploitation modules built in now so you can run things like Mimicats or whatever have you. And it's got some AV evasion. This isn't because it's built to evade AV. This is just because GoLing's kind of new and it's really hard for most modern AV to detect GoLing. That will definitely change as we progress forward in the future and GoLing becomes even more popular than it is now. But for the time being, I've found that on most of my engagements I can drop a Merlin agent, don't have any problems with any AV. So just throw that out there if you haven't used it on engagement. Pretty neat tool. All right, let's compare that to SaltStack. Is it a C2? Yeah. Hey, execute commands, get control, do your thing. Kind of covered that a little bit. What about the communications? Well, everything is encrypted. You have a complete encrypted channel. But it's not quite HTTPS. It's pretty much clear, or not clear, but it's straightforward TCP communications to the master. You can do some things to kind of change up ports that you're listening on and change your configuration to your master to kind of be more stealthy and we'll see that a little bit later. But overall, it's going to be pretty clear that the communication is abnormal. It's not looking like any kind of standard thing in your environment. It's going to be pretty exposed in that regard. But it is encrypted and it is a secure channel. Post-exploitation modules? Well, I mean they're not baked in, but we call this desired state. And this is kind of where the cool stuff gets in that we'll look at in a couple of slides. So post-exploitation modules for SaltStack is going to be this idea that we are going to take the concept of configuration management and build our own configurations that are bad. They do things that an attacker wants to do rather than what maybe an infrastructure engineer wants to do. AV evasion? Yeah, these are trusted agents from professional software. Yeah, they're not going to be picked up by anything. Real easy guaranteed to go through. So we get some leverage there from being able to cash in on this trusted software. All right, so in terms of using this platform as an adversary, like I mentioned, the desired state configuration concept is going to be kind of the key here. This is what makes it really cool, because we've got command and control for days. What makes us any different? Why is this a need option to explore? Well, I call this like post-exploitation modules in advanced mode. It's kind of cheesy. But it's really easy to automate your persistence. And basically what this means is you can pre-build these states to say on all these types of systems, Windows, Linux, Mac, whatever have you, do this. So if you have compromise, you know, I don't know, 10, 200 systems, and they're all different operating systems types, you can have conditional set in your states and say, okay, I want to do the same type of persistence across all of these different platforms, have them categorized out beforehand, hit state apply, it's just going to run through and draw your persistence for you, and you're good to go. And so that's really, really freaking powerful. And I think it's really cool. You're able to enforce that persistence. So of course, desired state, you don't want your system to deviate from how it looks. So this is your server now. Think of how you as an adversary want your server to look at all times that you've compromised. You want to make sure you're maintaining the access and being on the box, and as long as your agent doesn't get removed, well, you can keep enforcing that additional persistence. And as long as that additional persistence is getting enforced, even if your agent does get removed, you can play your cat and mouse game, and you can get back in with your other methods. So that's kind of the real power here, is that being able to correct the server from how it looks to how you want it to be. And you're able to do that from the attacker's perspective. So it's really just a super cool post-exploitation, T2. It works a little different than things that are built for post-exploitation, because it wasn't made for that. But we can make it for that. So kind of want to introduce bad salt. So that's kind of where this comes into play. Salt sex is really hard to learn. It can take some time to set up. You actually get pretty snappy at it once you've done it a couple of times. But when I was doing all this research, I got tired of setting up new boxes and reinstalling agents and going through the whole rigmarole. So automated everything. So that's the bad salt project. It's not really anything fancy right now. Don't expect much. It's literally just a couple automation scripts to help you get started and get things set up. But if this is all brand new to you, this is a good way to get started, especially if you're interested in this so here it eventually would be to pre-build out some of these post-exploitation states and also kind of make some sexier stages to easily deploy on the potential targets because right now it's just a bunch of like batch and batch scripts. Really truly nothing fancy, but it did help automate a lot of things. So let's look at using salt as like a C2 from an adversarial's perspective. So let's see this in action. So I want to demonstrate a couple things. The first one's going to be, of course, a trusted minion. So we've got a fresh smell or bite. It's all live protection, salt set up, ready to go. We're going to go. See it's not even detecting anything wrong with the repo. There's nothing malicious going on here. Obviously you'll see some pretty janky stuff going on, pretty clear things that you can see as IOCs. And we'll cover that later once we talk about detecting this type of stuff. So purely just for automation and proof of concept purposes. This will take a minute to set up, but what's going on here is it's pulling down the main repo.salt.com, the installer binary for a salt agent. And then it's running an install command. And then the last thing it'll do is pull down my custom bad configuration. And all I've done really at this point is change 4505 and 4506 to 80 and 443 to look a little bit more stealthy and maybe get past some egress filters. So this is going to take a minute to set up. Once it does though, we'll switch over to our salt master and apply one of those bad states that we've set up as an attacker. Any day now? This is what Windows is my least favorite, though that jab out there. All right, cool. So we've copied our config over. And now we're going to go over to our master. We're going to accept the key. And then we're going to see that there is no backdoor user install. So we're going to run on that user command. This is like an IE test box. This is the default user on that IE test box for Vagrant. And now we're going to run our persistent state. And this is going to be very simple. We'll look at the code for this on the next slide. But it's going to install this backdoor user pretty quickly. And now we have a backdoor user installed on the box called Backdoor has administrative privileges and RDP privilege. It's pretty quick, pretty effective. What does that actually look like? We'll start actually with a Linux state. So this is going to look similar. We're actually going to look at the Windows state as well. And then we're going to go back to our normal configurations. They're real simple, real easy to adapt. And basically just think of how systems are configured and what you want to do on those systems. And you declare it in YAML. And it's pretty straightforward and easy to read. So here I've named this configuration add backdoor Nix user. So when I call it later, I'll say, hey, for Linux machines or Unix machines, go do this thing. And then what it's going to do is this is going to check if a user is present. So it's a user dot present. It's very declarative. In this way, if it's not, go do it. Rather than saying, go do this to this machine. It's going to check first and validate. So if it's not present, it's going to add a user with name backdoor, password's going to be door back, it'll hash it, give us a home directory of slash home slash red, and then add us to like wheel, suitors, root, et cetera. And now we have a root account ready to go. In a much more powerful vein, you can also do these file managed states. So here we've got the sample one for how you might go deploy a interpreter payload on a particular target. So here we just have our assault repo set up. Assault colon slash slash is actually a local directory on your master system. It'll be in like slash serve slash salt. And you'll kind of set a directory structure here. I've got one called payloads nix. And then I put any of my Linux payloads that I might want to do manage file stuff with there. In this case, it's interpreter payload. I'm going to tell it to go, hey, install this into slash temp. Give it executable read write permissions, et cetera. And then go run it. And then this BG is go run it in the background. So it'll go execute it, run in the background, and then I should get a interpreter session that pops over once that happens. Now, with just these two ideas, you can go crazy. So from just the managed file, being able to run all your custom binaries from one place, you could get SaltStack in there and say, all right, I want to do some crazy stuff with my custom tool sets that I already have. So you can apply it like that and solve it across as many systems as you have into your SaltStack instance and go from there. So whether it's a interpreter, Merlin, or some crazy custom tool kit you have, that's about how you do it. Now, for Windows, it's about the same. This is just to show kind of the difference between the language that you're going to be using in the Amelfall, but this is the exact state we just ran in that demo. So we're checking if the user is present. If it's not, we add a user backdoor, and add it to the administrator's and RDP user's groups. Really simple, really effective. That's all it was. That's all we did. The managed file payload or configuration state is the exact same as the Linux one. Only we don't have to set any executable permissions because Windows is going to know it's an ATT, and we just copy it to where we want in this case, Windows Temp, and run it in the background as well. So these are really, really, really simple and really, really, really effective. And I hope it kind of gets your brain turning into new ideas of things you could build really easily with these types of desired state configurations. And what's really great about these is, like I said, that concept of desired state. You are missing this from the system you just installed. Your user's not there anymore. Any of the persistence you just set up is gone. As long as it's still calling home to your Saltmaster, you can even set it up as a job. It'll go correct itself on its own, but if you're there monitoring it or babysitting it, it's as simple as a state that apply, and it'll go fix itself. You can do it manually, whatever have you, but you can always correct to fix things as they should be. All right, so let's talk about detecting hackers that are abusing Saltstack. No red team or off-security team is worth their salt without helping our blue teams. We're not doing any good. We're not helping anyone. This is probably the most important piece. First and foremost, simple log analysis. Go back to basics. Check the salt menu and master logs, especially in the context of like a man-in-the-mill attack, kind of harkening back to that screenshot from last where we pull out, hey, this might be a man-in-the-mill attack of some sort. Those logs will just tell them themselves. It's kind of your golden ticket for seeing all of the minion and master communications. It'll tell you who, what, when, where, what time, etc. that minions are talking to a given master, even if it is the wrong master. And so you kind of use that in your investigation to see what's going on. All right, so that's kind of basics. Another thing we can do is serve the network traffic patterns. While everything is encrypted, it's actually kind of easy to spot. And of course, PCAP cert didn't happen. In that repo, there's all the actual PCAPs from the researcher did. You can go look at them yourself if you don't feel like taking your own packet captures. I'm not going to dive into this. I don't expect anybody to read it. But one thing I want to point out here is that we've remapped the 4505 to 80 and 4505 to 6443, kind of like I mentioned a couple slides ago. This is actually what's going on, though, in that PCAP. So the first thing we're doing is accepting a key from a potential minion that we've actually made in the middle in this case. So we've successfully pulled off middle attack. We've got our key accepted. We're going to run a test.ping, which just says that we can talk to that minion. Then we're going to run a couple commands like QMI, ID, LS, et cetera. And that's just to kind of show what's going on here. But what's actually happening in terms of communication pattern is what's important from a blue teamer's perspective to detect. So when a minion calls home, in this case on port 80, it's going to check in on port 80 first, and the master is not going to publish anything, because they're not associated yet. And then it's going to go up and say, hey, on port 443, do you want to trust me? If you accept the master's in, now it'll send you its public key, and you're good to go to keep your communications running. Here, we can execute commands. So maybe on the salt master, we'll run a who am I. And then the master will publish that on port 80 in this case to say, hey, go run who am I. When the minion checks in on port 80, it'll go fetch that, run it, and then come back up to 443 and say, hey, I was rude. And kind of the pattern here that I want everybody to notice is that you'll have a connection to port 80 or port 4505, followed by one or more connections to port 443 to the exact same host. That's a pretty clear pattern that we can see carved out there. And it's even easier if they are using the default ports for 4505 and 4506, because it might give away that it's salt sack right away. But that's a pattern that you could use. So obtain them with those log analysis and kind of your traffic, your packet captures. You can kind of correlate these two events and say, hey, maybe this is some kind of attacker abusing salt sack as a command and control service. Another way to kind of detect this type of activity is the fact that salt master and minion installations are very, very noisy. Now, obviously, an attacker could change all of the configurations and move things to different directories or whatever. It's an open source project, so there's a lot of power they have. But by default, it is extremely noisy. So if any of these directories exist on your system, whether it's bar log salt, at sea salt, serve salt, if you're some sort of master on your system for whatever reason possible, or sea cold and back to salt for windows, if any of those things you exist, pretty clear sign that somebody might have installed salt sack in some shape or form on that box. Another thing to do is check your repo files for your network communications to repo.saltstack.com, because if they're pulling from the default or not compiling their own binaries, it's going to be a clear communication just like with that batch file. So if somebody just wants to go be a script kitty and pull down that GitHub project and go use it on a potential target, it's going to be real easy to pinpoint that that's happening. Like I said, it is an open source project. There's always ways to modify and change this. Those are things that I'm going to be looking into more in the future, but most part, it's pretty clear when somebody installs these logs and these existence of these directories is a clear indication that that's happened. A crucial thing, though, to take a look at is if you are legitimately using salt stack in your environment, an attacker could still install a duplicate minion on a particular target they've compromised. So you won't be able to identify multiple minions in that case, because you should only really ever have one minion running on a box. It's supported to have more than one, which is why this is possible, but in a legitimate configuration, that's not something that if you are, here's how you might be able to detect a bad one. So when an attacker goes to install a second minion on a box, they're going to either A, bring their own minion binary, or B, use the one that you already have installed, and then just point it at their bad configurations. So what you're going to want to look for in these cases are the potential usage of the same log file. So if two minions are installed and they haven't reconfigured their minion to point to a different log file, you're going to see really weird activity going on in that legitimate log from like two different minions going on the box. So you can check that out. Another thing would be like your system B logs, you could look for activity that's going on on one minion that might be similar to activity going on a minion, but one's not named like a minion, it's just a behavior type thing. So you'll notice that two services are behaving the same, one is a legitimate minion, one's like weird XYZ name in or something. So that's stuff you may want to look for and then of course kind of a dead giveaway that I think is going to be your network traffic. It's got to communicate back home, right? So not talking to your master, it's probably not your minion, and so you can go trace that back to whatever your process is. So that's pretty important if you are using SaltSack in your environment because all the other ways are a little bit easier to detect. These are the things you're going to look for if you already have minions installed. So that's kind of all the things I wanted to say. Okay, go check out SaltSack if you're looking for a potential tool for your configuration management or any of the tools really, I'd just favor this. It's great for operations, great for blue teams and seemed pretty fun to use as a red team tool potentially as well. A couple of appreciation shoutouts, first and foremost, Seth, the only reason I even chased down this research, he was one of our DevOps team at the time, really educated me a lot on the platform and I kind of took this home and ran with it from there. Always being real supportive. Daniel Robertson and Dan Borges, both of these two have been great influencers and mentors whether they realize it or not. And then Camilla, also a great supporter of my life. Couple of references, like I said, these slides will be available on that repo. I want to point out Seth's brain again. That guy's really smart. And with that, we have questions. Can you repeat that? I'm sorry, I didn't hear you. Yeah, so the question was is it better to, or not is it better, was the second option for mitigating the issue to automatically accept minions to the master? And yes, that was the second recommendation. I just wanted to... Yeah, and that's absolutely true, which is why is okay, but definitely not the best idea. The best solution is just to install the fingerprint where it should be in your configuration. That's absolutely true. After an intense engagement, we want to get everything back to normal. Can we use desired states to... Yeah, so you'd have to know and coordinate with your red teams or your pentest teams to know exactly what they installed and anything like that as well. But if you potentially wanted to use that as a way to clean up after known good engagements rather than having to go back and main up, that is absolutely an option. So you could build your states around what you know that you've done and work with those offensive security people to execute on those states. So that's definitely an option to help make cleanup quicker. Yeah, so I don't know exactly where I'll dive down in terms of that, but I do definitely want to explore new tools or existing tools to see if there's anything better than doing things with SaltStack as we've done it, because it's definitely possible. Like I said, any of these all do about the same thing, so you can accomplish anything that we've shown in SaltStack today with another one of those tools like PuppetChef and Ansible. It's the idea of just using these tools from the perspective of the adversary. I'll be around if you want to ask me any other questions, but thank you all for coming.