 Hey, absent village. Welcome back and get ready for another talk. Coming up next, we've got can't touch this, detecting lateral movement in zero-touch environments. Philip Marlowe is going to be presenting this. Philip is a cybersecurity and DevOps engineer. He helps organizations understand how to adopt DevOps practices to increase their security rather than sacrifice it in the name of speed. Philip holds several security, cloud and agile certifications, and it's currently pursuing a master's degree in information security engineering at Sands Technology Institute. Please give it up for Philip Marlowe. Hello, everyone, and welcome to can't touch this, detecting lateral movement in zero-touch environments. In this talk, we're going to talk some about DevOps and how to detect attackers trying to move into your production environments. Before we get started, a few disclaimers and acknowledgments. First off, I work for the MITRE Corporation, but nothing I'm saying today reflects their opinion. Also, I'm a student at the Sands Technology Institute working on my master's degree. Huge shout out to my faculty research advisor, but I'm not speaking for her or Sands today either. And lastly, big thank you to my wife, Madeline. Once again, not speaking for her in this presentation today. So who am I? I'm Philip, and professionally, I really enjoy the intersection of security and DevOps. Yes, you can call it DevSecOps if you want. I don't really care either way. I wrote my first vulnerable code in elementary school. I was writing a blogging platform for my family and right in there because I didn't know any better, put a SQL injection vulnerability. I was a big surprise for them when they found out we've been hacked because of that. I first began learning to exploit applications in middle school when I received as a president the book, Hacking the Art of Exploitation. And I've been involved in application security one way or another ever since then. Today's my first time as a DevCon speaker, and I'm super excited to be here. And one of the things I really enjoy is being able to learn through hacking. And you'll see a flavor of that throughout this presentation as we take both the attacker and the defender point of view. So if you're not already on board, why should I care about DevOps? Well, for one, you probably don't have any choice. If you're running any kind of application, that's the way it is now. You probably just need to learn about it. If your red team standing up command and control infrastructure helps to have DevOps, your blue team trying to implement monitoring and detection helps to have DevOps. Application security DevOps is the way of life now. And provides all these security benefits on top of reliability benefits and maintainability benefits for the other parts of the business. So let's take a look at a very simplified environment. And let's take a look at it from the attacker's point of view. We want to get to something on the application server. Maybe we want to steal data off of it, or maybe we just want to use the hefty server power for crypto mining. Either way, we've got three basic options. Option one is to just go straight in the front door and attack the application itself in order to get access to the server. This is pretty simple. Maybe there's a firewall or IDS you have to evade, but you're not making a lot of hops around to finally get to where you want to be. But it's also not 2002 anymore. The situation has gotten a lot better in how we protect our application servers. So while this once might have been the best way in, it's not really anymore. Report after report from places like FireEye and Verizon have shown a much more popular way into networks these days is by social engineering users in one way or another. In this case, we're going to take a look at phishing a operator or developer on their employee workstation to give us our initial access. Then in method two, we're going to move laterally through the bastion host and finally to the app server. We'll talk in a couple of slides about why this works so well. But before we get there, let's talk a little bit about option three. With DevOps implemented these days, another way into that protected app server is through the DevOps pipeline itself. But there are a lot more hops here and a lot more ways that can go wrong and a lot more opportunities to detect this attempt. It goes through source control repos, test servers and configuration servers before we ever get to our intended target. So how does traditional application deployment work? Well, developer give ops a deployment package. And if we're lucky, some install instructions, and then ops is going to go and log into the application server manually install that software. Pretty simple. And then an update comes along and you have to go and manually log in and update. And then the security patch comes along and you have to go log in and update. And this manual logging in and updating is going to be a real problem for detecting lateral movement. So what does traditional lateral movement look like? Because they're logging in to do configuration and updates all the time, ops has highly privileged credentials. This might take the form of SSH keys or API tokens. And oftentimes, they're stored in plain text on their employee workstations. If you've already fished that user, those are really easy and really juicy targets to steal those credentials and move further into the environment. All right, so that sounds like it's a problem. What is this zero touch concept? Google defined zero touch networking and zero touch production. And they've given several presentations on what that means to them and how they've implemented it in their environment. Overall, it's about how we can abstract away managing the production environment so that we're not doing those manual logins, change things, configure and update anymore. And while Google is one of the only companies to use the zero touch nomenclature, a lot of companies do this. Often, it's just a result of being very mature in their DevOps practices that everything is handled by automation rather than human users. So zero touch deployment looks very different. Every change is made by automation and pre validated. This means no humans logging in. And that's key for our talk today. You'll also notice on Google's presentation that there is an exception. And that's when something is made by an audited break glass mechanism. We'll talk a little bit more about that later as well. So what does this zero touch deployment look like in practice? Well, rather than having a user log into the app server via a bastion host, since that's no longer allowed, every change has to go through that DevOps deployment pipeline. Looking at how attackers are going to move laterally, however, let's compare and contrast between the traditional and the zero touch deployments. So I'm about to play an animation and keep an eye out for the malicious connection between the bastion host and the app server. It's going to be a slightly different colored arrow. Did you catch it? Have you seen it yet? It's kind of tricky to find. And this is when it stands out in some way. If we're looking at encrypted connections like SSH or HTTPS, it may not stand out in any way, unless you're doing some kind of break and inspect in the middle. Now let's compare it to zero touch. In the zero touch deployment, you're going to see a bunch of arrows again showing normal benign traffic. And you'll also see one arrow that'll look a little different being our attacker trying to move laterally. See if it's easier to spot this time. I'll bet that was a lot easier and as soon as it came up, you noticed it. This is the whole concept that we want to use writing this detection. So how do we actually go about detecting this lateral movement? There's a few things we need to do. First, we need to define all of the protected servers. In our simplified environment, that's just the app server. But it might also include DevOps automation servers, it might include web servers, database servers, all of which might be behind load balancers. Whatever humans should not be logging into, that goes in this list. We also need to define all of the human access points into the environment. Again, in our simplified environment, that's just the bastion host. But depending on your network architecture, you might need to include other sets of load balance machines, virtual desktops, employee workstations, anything of that nature. That goes in the second list. And then all we do to detect is look for any connections between those two sets of servers. Then alert, investigate, enjoy detecting this lateral movement. Let's take a quick look at what this looks like in practice. So for the demo, we're going to look at a couple of scenarios. First up, let me tell you what you're seeing. In this environment, I've got a bastion host that's used for manually accessing it when needed, an application server, and a configuration server running puppet. So just to make sure that there's nothing up my sleeves, what we're going to do for this demo is run some puppet configuration in the background. This will be traffic both within the environment to the puppet server and external to the environment, SSH connections and HTTPS connections out to the wider internet. So let's get that started. And then while this runs, jump over to the bastion host. And as you can see, we've already compromised the bastion host. And we see some SSH credentials that we found. And in fact, some SSH credentials for the application server that has the goodies we want to get to. So we'll run our evil script file. And in real life, this might be installing a crypto miner or any number of things in this case, just modifying the configuration files that puppet is managing to show that we've actually gotten in and had the ability to do evil if we want it to. So now we're going to jump over to the network monitor. This is a a plain instance of security onion with all of the usual defaults and alerts installed. And what we're looking at right now are the Zeke notices. And me turn on the refresh, but you'll see a whole lot of nothing. Zero, no results found. No results found. If we switch over to squirt and refresh it, same thing. No result, no result. And this is exactly what we expect from a default instance of a network monitor. It's not going to alert on connections like this. Let me enable the rules we've been talking about, and then we'll come back and we'll try this again. And now that I've enabled the zero touch rules, let's try it again and see what's changed. As you can see, the lab server is still emulating real traffic by reaching out to the puppet configuration server and to the external internet. Let's go over to the bastion and run our evil script again. Same thing, utilizing those compromised SSH credentials to reach in and modify the app server. Now we'll jump over to the network monitor. And as you can see, we're starting to get alerts. Zeke has noticed a couple of connections from the bastion host to the app server. And if we switch over, you'll see the same thing in squirt as well. Notice how they work slightly differently. The squirt gives us many more alerts because it's looking individually at the packets traversing the network versus Zeke gives us total number of connections. But either way, we're seeing the alerts that we're violating this zero touch policy, and that it's a good indicator of lateral movement. So you want to start doing this in your environment as well. Well, first off, if you're not zero touch yet, you should move that direction. Implement zero touch. It's got a lot of benefits besides this one. When you're ready to implement this detection, use your platform of choice, whatever you're already using is going to be best. There's no need to go get specialized software or hardware in order to do this. Use your existing ideas or network monitor. It's very simple detection to include. But then also pay attention to how you tailor it for your specific environment. In the example, it was looking for all TCP connections because I wanted to catch the broadest range of lateral movement. But that might not make sense for your environment. So maybe you need to tailor it down to only look for SSH connections or, you know, whatever makes sense for your environment specifically. And then the detections I showed, for all very simple, they just looked for that connection and alerted. But that can be just the beginning of a bigger process. Imagine with Zeek, instead of raising a notice, if we correlated that with other suspicious traffic that we'd seen, maybe some data exfiltration traffic, and then we could automatically provide analysts with here's the map of the attacker's path through the network. Starting to add some of those higher end capabilities really makes this powerful. And finally, I wanted to leave you with some lessons learned from this project. And there are two that really jumped out at me. The first is know your network. And this is not just know your networking. It is know your specific network. Justin Henderson and Mick Douglas, who teach the tactical seam class for Sands, really nailed this. And I learned this from them. So huge shout out to them. Knowing the specifics of your network lets you do really cool things in monitoring it. Things that aren't possible in generic networks, such as the detection we talked about today. And then secondly, don't be afraid to look for stupid simple things. All we were looking at today was the presence of a connection. Tying together these two lessons learned is really this whole project is looking for something simple like the presence of a connection and knowing your network well enough to be able to say, that's unusual. That's a problem. Thank you so much for attending the presentation today. Shout out again to the Sands team for their help and support with the research project. If you've got any questions about this project, AppSec, DevOps, the Sands masters program or anything related, the best way to get in contact with me is via Twitter, my handles on the screen now. If you're viewing the live presentation, I look forward to chatting with you during the Q&A immediately afterwards in Discord. Again, I'm Philip Marlowe. Thanks for coming.