 Tom here from Lauren's Systems and I'll be joined by Amanda Berlin, lead incident detection engineer and Blue Mera. Being heard is going to discuss how she utilizes Sysmon for threat hunting, testing detections by looking at real world examples and detecting malicious behavior in the wild. She's a longtime cybersecurity professional and she's going to share with us how she looks at the world, essentially through the eyes of a threat detection engineer. So we're going to talk about anomaly detection and then we'll get into some of those more fine details and the tools that we're going to be talking about. You'll find links to down below. And for those of you wondering if you can do this yourself, absolutely. I have a video showing how we can use the Sysmon modular and even export all the logs out of your window system or view them within windows itself. You'll find that video link down below if you want to try these yourself. And these are the same rules and threat detection that she uses. So they're open source, they're available. This is one of the things that Amanda participates a lot in the community of sharing all this knowledge and threat intelligence knowledge because that's how we get better. And that's why I'm doing this video. So you can kind of look through the eyes of someone actually doing this. Well, daily looking at millions of logs and understanding threat detection. So let's get started. So how are you doing today, Amanda? I'm doing great. It's a, you know, nice blustery cold weather in northern Ohio. Same here in Detroit. We're pretty close together. So I get it here and then pretty soon you try to get a little snow as well. Oh yeah. Which sounds like a great day to talk about how to pull out logs from windows using Sysmon. You did this talk and I was like, wow, this was solid. And of course I already, I kind of know you from talks you've done before. And then you landed over there at Bloomerra some time ago, which is a company we use. Definitely a fun, fun tool there and pulling windows logs is your specialty. Yeah, it's one of my favorite things. Windows by default, just doesn't give you good tooling for this. Sysmon is the glue to bring this together so we can actually get some useful information out of windows. And that's what a man is going to be presenting today. So I'm ready when you are to get started. Awesome. All right. So let's just dive in. This is one of my favorite ways to start out a presentation. And I'm sure a lot of people watching have either stories to share of this or something that you've faced firsthand and the stories around that. So let's call this company Paths Grocery Store. My team is involved after a PS exec command runs over their VPN. This triggers an alert to fire and they have servers start to be ransomware, which is definitely way later in the process than you would want to know about an attack, right? And pretty much too late in this specific case. So there's no Sysmon installed and you'll find out why that's a big deal throughout this presentation. And for some reason, their domain controllers stop sending windows altogether. They aren't sending any of their endpoint agent logs, but that wouldn't have mattered because the domain controllers didn't have the endpoint agents installed anyways. The account that was used to send that PS exec command ended up being a shared admin user with their Pulse secure VPN admin account. So same user, same password. That account had zero MFA set up and full domain admin access across the environment. I think the people don't realize that the first thing you notice is ransomware, but that's by far not the first thing that happened. That is, that is the end. That's the end of, that's the end of you having a good day. Right, right. Yeah, if, if something getting ransomed is the first indication that you have, it's not good. You definitely have had them on there for a while before that. So other than the lucky fact that one of their endpoints ended up receiving that PS exec command was sending us windows logs, we would have been completely blind to all of that activity happening in that environment. And it's always so nice if anybody's had to deal with this before. Threat actors, a lot of times offer to give discounts for speedy turnaround, which is they actually have from what I've heard amazing customer service. Yeah, they build a reputation on it. It's kind of they do, which blows your mind because they're doing terrible things, right? I do believe Pat's grocery store had had to end up paying the ransom because their backups were also not working for a significant amount of time. And that could probably be a total talk all in its own. And they needed that data on the servers. Otherwise they were just going to have to shut down all of their stores, right, and build from scratch. Exactly. So they needed that server data that was ransom. And sadly, it's just one of those cases that, you know, you can't. A lot of times we tell people, you know, you should do this thing. This would be a great idea security wise. And sometimes that just doesn't it doesn't work until something like this happens. And a lot of times this is also where you end up getting security budget, sadly. So as defenders, above and beyond all of the other roles that we play, you know, there's strategic thinking, process creation, research, writing, all of that kind of stuff. One of the main goals we have, no matter what vertical we're in, is defending against threats. That's, you know, a lot of times it's in our title. So there's entire conferences, companies, frameworks, podcasts, whatever you name it built around those three words. And so prior to 2014 or so, endpoint AV products were just like detections of MD5 hashes. So just plain virus virus signatures for the most part, we're doing all of the work as the amount of all of the things on the internet just started to boom off control, right? And that paled in comparison to what we see today. So now we rely on what AI, machine learning, artificial intelligence boxes or maybe just waiting for that time that you see a splash page about the server being ransomed. And as enterprise networks grow and hopefully mature, the majority of what we see is largely misconfigured or underconfigured networks and endpoint security products that just cost upwards of hundreds of thousands of dollars a year. Right? Like if you look at the budgets that security companies or security teams have to spend on these things, it just gets more and more every year. And there's this repeated behavior of just trying to throw money at a problem and hope a new Blinky Box is going to fix something. Right? I was looking for some magic solution. Right. Right. Like, I don't know. Our auditors say that if we get this WAF, everything's going to be fine. Right? So you have these Blinky Boxes that are compliance checkboxes that without the time and care and effort aren't going to do a whole lot. Anyways, yeah, I see a lot of the Blinky Checkbox Compliance stuff. And that's not where the real security comes from. We have to know what's going on in these boxes. Right. Right. And visibility is one of the anytime I get interviewed or right or whatever, one of the common questions is, you know, what do you suggest people do or learn about or whatever to prevent, you know, next year cyber attacks? Visibility, visibility asset management, top two, hands down all the time. Know what you have and know what it's doing. Yeah, exactly, exactly. So that being said, how can we make sure you're getting the biggest bang for your buck, especially when that buck is free? So Sysmon is free here to save the day. This is the Microsoft definition up there, just in case anyone wants a refresher. This was also released in 2014 as a part of the Sysinternals suite of products. And ever since then, I'm sure security admins, server admins, practitioners have been perplexed as to why it's not just installed on every endpoint by default. Yeah, Microsoft acquired Sysinternals back then. They did. Yeah. And it's just one of those, this is something you need to load on your Windows systems that was great. Those tools were so popular for years, it was the missing component. Of Microsoft for a long time. Yeah, for all admins, right? Like I remember all the time needing to use Sysmon and Process Monitor and all of those things for troubleshooting, right? Like it was just not there and you had to install it as an extra thing. So we'll cover some of the use cases around why we want this and then we'll dive into some threat hunting stuff too. Yes. So here's some fancy stats. So enterprise orgs have a lot on their plate on the cybersecurity front. There's fast adoption and move to cloud, which I'm actually just writing about now, like the move from Exchange on prem to 365 Exchange. And then the last two years, you know, just the largest amount of move from a corporate world being from home from work to home. Right. So you have this huge boom in work from home now where prior to COVID, a lot of us were in offices and that attack surface has significantly expanded, right? Yeah, all over the place. You don't have everybody in the office from nine to five behind a firewall. This has created some really interesting challenges because now they're scattered. All my users are you can't even hire someone if you don't put a hybrid in the job title somewhere. So at least part of their time is not going to be behind your firewall but behind maybe some consumer router with access to things. So monitoring what's going on in those endpoints is as critical as it's ever been. Oh, yeah. And, you know, as a former CIS admin, that always just like scared us to death, right? Because I worked at a hospital. We didn't have anybody that worked from home, but we did have like physicians that would take laptops home and stuff like that. And they were always, always like just full of spyware and just all of this stuff, right? So going from that to like, no, everybody works from home is a huge shift. So going based on our average implementation at Blumera, so we would, we were seeing like a single digit percentage of orgs that come with us, come to us with CISmon already installed. So we had so few that now it's just included in our onboarding process, right? Like you want to get Blumera licensed like not our free one because that's just cloud stuff. But if you want to do end point stuff, yeah, you get CISmon. Because it's so helpful for detections and for incident response and it being free. You know, why not? Yeah, absolutely. Just load it. Oh, yeah. And I'm a huge proponent of using the tools that are at your disposal. So whether they're open source, free, whatever, you kind of I've always felt that the the responsibility to whatever organization that I work with to look up those kind of viable options prior to just asking for like capital expenditure, right? So why not try? Like if it doesn't do what you want it to do, then you're not really out a whole lot of money, right? Just a little bit of time is setting it up, right? So if you can reduce like time to detection, time to remediation with a price tag of zero, why not? So I'm not one to read slides, so I'm not going to read this one. I'll let everybody watching since you can pause it if you really want to. But, you know, I I've always wanted the tattoo live, laugh, log. Live, laugh, log. I love that. I've not made that jump yet. But if I were to get another one, I think this might be a little bit too wordy. But, you know, this could be a possible possibility, too. So at any point when you have an incense or if you had one or whatever and you have Sysmon installed, configured and logging, you'll be able to find your breach scope, right? I can guarantee you that time to detection, time to response, time, all those buzzwords are going to be key security metrics that people end up using in coming years. And I know a lot of people are already using it already. And it's one of the best things that you can pay attention to for program maturity. And then I'm sure a lot of people have also heard like the assume you've been breached. Yeah, just because, you know, in the beginning, getting breached was a huge thing, right? When it happened to Target or stocks plummeted, when it happened to Equifax, stocks plummeted like they did. Terrible for a couple of years. And then, you know, it all recovered. But now it's just another one of the news. Right. The stocks don't even dip anymore. You know, people are less likely to get fired anymore when it's just so prevalent to get breached that people are like, oh, just assume it. And then if you have been, what should you be doing to find evidence of that? Right. Well, and that's, as you know, is me and you both work in the infosec community with instant response teams. And they're always complaining as they should be about. We couldn't figure out what happened because they didn't have any logging tools. We know we know the results. We don't know how we don't know the backstory. We don't know what led up to it. There's an absolute lack of information. We just see the boom, the results, the disaster that we're dealing with today. But that's why getting these tools and solve the beginning is so important for that forensics information. Because if you don't know how they got there, you don't know exactly how to prevent it. Exactly. So that being said, we can go on some Sysmon use cases. First, we're going to cover some of the specific benefits you can get from Sysmon. So if you're beginning a hunt, more than likely you are inserting yourself into an event related to a Windows host that's in the process of happening. Or maybe like we do. A lot of times it's a retroactive step to find that activity that's been missed. In either of those situations, Sysmon logs give you by default an amazing amount of logs compared to anything plain windows can provide you, even one fully configured. And honestly, way better than a handful of endpoint solutions. So, yeah. So here you see a difference on the left hand side in the difference of Windows IDs and then on the right compared to Sysmon. So there's 10 total types of data that fail to appear if you don't have Sysmon. And then there's others that are possible that are extremely difficult to configure to even get a shred of information out of it. If anybody has ever tried to log DNS from a domain controller without using Sysmon, you have to like turn on debug logging. And half the time it doesn't even tell you what device requested. You know, may the DNS request or it's just horrible. And Sysmon just does it for you. It says, hey, this host reached out to this website. It's fantastic. So for threat hunting, what can we do with this kind of information? One thing that we do every now and then is using... And I think a lot of us take for granted that we do this. It's kind of just part of how our brain works sometimes. And it's using standard deviation to weed out baseline activity. So this is the normal like curve of standard deviation. And in this example graph, like 90% of the results live in one and a half standard deviation. And the rest you can call outliers. That can be really helpful in threat detection because looking at those outliers regularly gives you a good look into some of the less common activities that happen in the environment. Granted, if you take your standard deviation and you already are already are having an incident and it's been around for a while, it's just going to fall into normal activity, right? So you have to have this for a while and collect a lot of data to be able to see what is normal in your environment. Here, I just took... I know this is like a crazy graph, but I take an example of some Sysmon logs across all of our customers. And so the y-axis on the left is total destinations. So this is just like IP addresses that things have reached out to, starting at 50, going up to the upwards of 500 to 5 million range. And the x-axis shows standard deviation score. So this, we're taking all process names on devices across all of our customer data set, which is why those numbers are so large, and looking for destination IP addresses that processes are accessing, right? So some of them make complete sense. And then we calculate the standard deviations. So we're going to zoom into this, which you'll get that joke in a second. That makes sense, right? Processes zoom, it's reaching out to a ton of IP addresses. Teams, Java, Chrome, like this makes sense. These are applications that reach out to stuff all the time. Edge, go to meeting, that's Cisco, WebEx. Ring central, everyone's phone system. And then we see Notepad, which should Notepad ever reach out to IP addresses on anything, no. Good news is that was just our lab. Because if we start to see things in that standard deviation, or outside of the majority of that standard deviation, that are like this, like that's something that you're gonna wanna look into. And this becomes different with every customer, right? Like this is interesting data set because it's across to everybody. But it's nice to see that it actually captured our Notepad going out to 2 million IP addresses. Another main detection creation strategy we use is adversary annihilation. So there's a bunch of tools out there like Red Canary has stuff. There's atomic red team. There's a whole bunch of different tools out there that you can use. This one is also free. This is called Vector, it's B-E-C-T-R dot I-O. And you can import different frameworks into this. So I'll give you some examples of a couple of things that we started to do. And this is like a heat map of MITRE. So you can import all the things in MITRE and you can test against it whether the endpoint agent you use blocks or detects it or your SIM or your IPS IDS. Like there's so much that you can do to track all of the different levels of MITRE, right? Because there's different stages. We used to have the tech, the cyber intrusion kill chain, right? And now we have MITRE, which I think is a little bit more extensive. But there's all of these different TTPs that people use and it's good to test these. Now, can you test all of these all the time? Probably not without having several staff dedicated to that all the time. It's really hard to test all of those. But just think about, you know, like spot testing this stuff. So the amount of things that can go wrong in any given day with any kind of technology is, you know, overwhelming. But you can pick, all right, today, I wanna check to make sure I can detect on if somebody installed a new remote access tool, right? Just things that you, I always, you know, back in the beginning when I started doing this, it's something that I've kind of always used as an example. Think of the things that keep you up at night. Like, oh my gosh, I woke up in a cold sweat because I just realized that these users don't have 2FA enabled. I don't know if anybody else does that, but there's things out there, right? Well, I think it's important too, one of the things that we don't use TeamViewer within our organization. So when we see a TeamViewer install, it absolutely flags, it opens a ticket, it creates error. We don't even have to test it because, well, clients tested for us. Yes. And you're just crashing head, wait a minute. How'd they get, they're supposed to have admin privilege. So there's the first problem. How'd they get TeamViewer installed? Exactly. But yeah, going through your organization in validating, not just saying, okay, we monitor for installs of new files. We monitor for installs of remote tools. Being able to actually test and validate that finding is really important. It's a heavy lift to ask an IT team, but it's kind of an important one to make sure that those flags are still being raised. Yeah. And you can pick like 10 a quarter, right? Like you could probably do 10 tests easy in a day, especially with some of these tools. So like an example is this is importing, this is if inside vector, this is an example of some stuff that you can track, right? So this is the red team section of this one mitre technique, right? So this is an example of T1482 and it's people doing domain trust discovery. So you can run this and it shows right in there when you import it. This is actually importing the atomic red team stuff that the phase of discovery tells you what it's about and like trick bot malware uses it, but it also tells you like, these are the commands to run. So you can just go to a Windows endpoint and if you can run this without being detected, that's probably bad. Yeah, and that's the important part because if you have, we'll say someone in your accounting office, suddenly that accounting computer is running this, you're like, that seems unusual. That is not what they run in accounting. Yes, yeah. As that backs to that whole anomaly detection, this is how you have to look at things holistically to say, all right, here's the whole picture, here's this anomaly happening, QuickBooks reaches out to too many IPs and et cetera, but all of a sudden that same computer is now running this and they're looking at what is in the trust list. So this is the instantly starts an investigation flag and getting that data out. Yeah, exactly. And that discovery phase is way before the ransomware. Yep. Those are all of the things that attackers, like we just saw one not too long ago where the attacker was using who am I and T, like doing user commands via the command line, just doing NS look up via the command line. There's a lot of like discovery things that most attackers either do or just have scripted to give them more information. And Sysmon helps with that too. Yep. And this is the blue team version of that same thing. Invector and this is like where we'll store. Yeah, we detected it. And here's the logic that we used in this specific affection, right? And you can track those, you know, however you want. And this is just, you know, our tagging system and internally how we track all the, all the MITRE things, right? All right, so now that we covered kind of how you can organize that stuff, we can dive into some specific use cases. This is use case one. There's a ton of different ways that process memory can be extracted from a Windows endpoint. You can run things like meme caps locally. You can gain access to hashes, a multitude of different discrete ways. I can think of like five off top of my head. Here you can see they use com services which is a com plus service DLL that was introduced in Windows XP, yay, that you can use to extract local hashes. There's a handful of detections and a couple of these we'll cover a little bit that are called finishing moves, meaning the attacker has all your keys to the kingdom. The security teams lives have gotten much harder than they ever wanted to be. Or you're, you know, calling mandiant or some other IR firm to come help you. This isn't necessarily one of those game ending moves, but it does mean that someone has local access to a machine and the files in their possession that they can crack to get credentials to your devices, right? Especially if you're sharing usernames and passwords across devices, you know, it could potentially be the entire environment. So we consider this a priority one threat, meaning that you have to immediately act on it or you should immediately act on it. I do have stuff blurred out in this presentation because they're actually from customer events. When it's not blurred out, it's me doing it in the lab. Understand. Yeah. So here we see the exact commands and the timestamp on the device in question. So this is com services. This is the actual command that was run, right? Com services is doing a mini dump, which it lists the process ID and then where it wants to store that memory dump and then the keyword full. You see here, let's see here, there's other related findings and that kind of just goes along with, you know, there's a lot of stuff that happens along with this kind of stuff. Yeah, I'm gonna mention too, on that page, the law goes from December 7th of 2021 and these are all local tools that are found on there. I finally was happy to hear in 2023, Lullbin come part of the common vernacular cybersecurity, but yeah, I mean, cool that we got a name for it. I like Lullbin as a name, but it's not that new folks, but Thread Actors have been using the tools that are available to them for a long time. It's just been slipping under a lot of radars. So how'd they do that? Well, these tools are all in there. I mean, they're using built-in facilities of Windows to attack us essentially. This is why it doesn't flag in malware scanners. This is why monitoring what's going on from a behavioral standpoint is so critical to cybersecurity. And difficult too. I remember when PowerShell by default started coming on Windows OSes and they're like, oh, well, we have to uninstall this. Like we can't have PowerShell on every device. Like how are we, how are we ever gonna manage security if everyone has access to PowerShell? And then it didn't matter if you uninstall it or not. Like you could just reinstall it super easy and then there was all that stuff. So yeah, like lullabins and what is it? Lullabins and lullabaws, whatever they changed it, or these are all ones they have now, living off the land, right? Which means there's so much already in Windows that can be used, right? So this is, and I have a couple examples of that, yeah. Yeah, it's definitely, I left because I think in 2019 that we had our last time we had a Detroit screen size, my friend, David, presented, showing how all the live-off-the-land techniques how I'm gonna, it was the most fun these types of, how I'm gonna hone your system with everything you have. Right, right, which is easy, right? There's entire attack frameworks built around PowerShell. Yep. So I apologize in advance for all of the screenshots of Event Viewer. There's a lot of them, but there won't be too much of a quiz. So here's when we see the incredible differences in native Windows logging on the left compared to a Sysmon install. So even if you, Oh yeah. Even if you can't make out all the specific ones, you can see just like a share volume of information. Event one in Sysmon is the same kind of as 4688 in just plain Windows. And it just pales in comparison, right? And the reason that 4688 even shows up is because we have to configure it in group policy to output those results, right? You have to do command line logging and process creation to get everything that you need where you just get it in Event ID one. Here's an example of the specific configurations that you can do for Sysmon. So I'll cover it a little bit later, but one of the things that you can do is there's like the Swift on security Sysmon configuration. And then there's also Sysmon modular, which is the one that we usually use. And the Sysmon modular actually gives you MITRE techniques at the top of the rule names that you have in your configuration. And then we can correlate that to a couple different things. The second one down is like we were just talking about before the living off the land techniques. And that is a screenshot from the actual Sysmon config. And then the next one down is from atomic red team tests, yeah, yeah. So it kind of breaks them all down and it's kind of cool that you can see all of the techniques kind of match each other throughout those tools. So in summary, this is what kind of the detection looks like. You can put it in most things that you have for a SIM. So we're looking for Windows Event ID one. And then we're just looking for command to the parent command line to be that com services with a mini dump. Granted, maybe you had an admin doing that for some reason, but you're probably gonna wanna know about it anyways, right? So nice and short and simple. And then we can move on to use case two. So this is an example, I talked about a little bit before, the finishing move, something you wanna know definitely as soon as possible. Another thing that just comes with Windows is NTDS util. If anybody's ever managed 80 databases, that's the utility that's built into Microsoft for doing that via the command line. And it's been used for years, right? You can use NTDS.dit and that's the database of all of your active directory user information. And you can just dump all of that information also. And as a threat actor, you can begin exfiltrating that and cracking all the passwords and looking and seeing what the directory forest looks like. I can tell you, I have not had to do this, but it was really interesting. The first time I gave this talk, the person, if anybody's ever heard of the, there's a Microsoft guide on network eviction process. The first time I gave this talk, I'm like, ah, by a show of hands who has, you know, had to follow this or had to start a forest over from scratch because I don't know if it is still, but that at one time are the two options you have if somebody breaks into your active directory, you have to start over from scratch and manually do everything again, or you have to follow this network eviction process. And the person that wrote it was actually in the audience. Like, oh, all right, well, that's great. And he actually told me, instead of doing network eviction anymore, he's like, I just tell people to move to the cloud. Like, all right. It's time to just throw away your on-prem. Yeah, just no more DCs. That's it. I'm like, oh, that's real interesting. It's not an easy place to unwind when people bury things and hide them in there. Yeah, there's a lot there. I mean, there's some tooling around it that exists. I believe Bloodhounds and all those tools that you can do some auditing with, but it's not easy. Even with the tooling, it's a project. It's its own project to itself after the incident. Yeah, I would not want to do that. Just, I'm glad I kind of misadmin stuff sometimes, but not always. There's a day when you're happy you're not doing that. Exactly. So here in our matched evidence, we see the full command, right? So ACI NTDS sets that NTDS as the active instance. IFM means install from media for like non-read-only stuff. And then the backups created and the two cues are just quitting the previous two commands. And it gives you a nice download of active directory information to go crack. Yeah, and with humans tendency to reuse all their passwords and the absolute amazing availability of rainbow tables, they actually don't spend as much time cracking. They're doing more of a matching scenario. Right, right. It's much faster than people think. Yeah, sadly. And then here are the differences. If you pay attention to the Windows logging versus Sysmon logging. On the left-hand side, it's just a huge list of 47.99, which is just enumerating security groups in active directory, which that's how active directory works. You do enumerate security groups all the time. A 4688 was generated and we saw that in the first use case. And then here on the right-hand side, you can see huge wall of tiny text with the amount of information you can gather when that command is run using Sysmon. So the first one is that event ID one again. It's comparable to the LSAS dump we went over in the use case one. You can see NTDS Utils, the original file name, the command that was run from the parent image of PowerShell. It even provides extended information like that a newly created process when the command was run and then the full command line. And we also see, is it in this one? Yeah, so it gives a terminal session ID. So it shows that I was connected via terminal services when I ran this as well as I use PowerShell ICE instead of just running plain PowerShell. And then here's the first place where we see event ID 10 which is a process being accessed. It's one of those 10 events that isn't included with Windows no matter what you configure. And then you see that the source image of PowerShell is using NTDS Util. And then event ID 13 is a registry value being set and that populates when that process actually executes successfully. And you can essentially, without the system, there's not the entire chain of the event. Right, yep, yeah. You get all three chain together for that instead of just enumeration and a process. So again, we see the detection screen with what you can look at as far as the detection goes. NTDS Util can be used again, legitimately across the environment but it's extremely rare. So you can still look for event ID one and then the process name of NTDS Util. And this is just one of our detection versions that has the command or parent command that has those two quick commands in it also. Because a lot of times that's pretty much all the script has. All right, then we have use case three. This is another priority one threat, another living on the land which is comm spec modifying the registry. So on the bottom there, you can see comm specs and environment variable that just opens the command line. So if you type in echo and then comm spec, all it does is open up cmd.exe. You can't really see what's going on here from the description of the command itself because it's base 64 encoded. But using that comm spec environment variable along with hidden encoded power shell commands is extremely sketchy. I've not seen this false positive yet of this type of behavior. And I say yet because there's an amazing amount of really terrible software used in production environments that like to mimic threat actor activity. Yeah, I have worked with some industrial engineering people like why are they base 64? Yes, yes. They thought they were hiding as a secret and apparently they didn't know about CyberShift. Right, right. And it's weird things too. Like I've dug into power shell encoding before and it's just been like, oh, we're updating the firmware on this scuzzy drive. I'm like, what in the world? Why would you base 64 that? I don't understand. Yeah, any time there's base 64, it's at least raise suspicion on it. Right, right. And then kind of a sidebar on that. When I was looking into some data for this, for examples for this talk, I was weeding through some of these power shell commands and notice something really weird. And normally when you see a bunch of words with upper and lower case, you just assume it's like obfuscation. Yeah. So I took that base 64, decoded it and found that it was Matt Graber's reflection method. Right, which is a tech testing tool that bypasses AMSI. So right away I was starting to worry. I'm like, oh my gosh, somebody's using Matt's like stuff in this network and they're attacking it, whatever. And it turns out there's actually a really popular tool that people use as an endpoint agent that for some reason, I don't know if they still do this, but they were using these locally on all devices as I'm assuming as some kind of, let's test our own detections. But I'm not sure. It's a legitimate endpoint security solution and it just made me freak out because I thought they were actually being attacked, which they were not, so that's good. So back to the use case is that com spec. And here we can see it's being used to run PowerShell and install on the service. Normally you see a random service name, but this talk was originally created for RSA. So I had to make the service name RSAcon. And then here we see the full command. And I've already, let's see, this is on, shoot, what event ID is this? I missed it. Let's go back. Oh, I think this is event ID 10. Okay, here we go. So this is, yeah, the sneaky way of calling command line and then you see no profile, the window's hidden. It's an encoded PowerShell command. All flags that make you raise an eyebrow at it. So you can use your window hidden encoded. Okay, there's something going on here, no profile. Yes, which are common ways to bypass PowerShell execution policy, which turns out wasn't even created as a security measure. Execution policies were created as a, let's hope these admins don't shoot themselves in the foot by running these PowerShell commands. It's kind of like having to type pseudo when you do something on Linux. So we're gonna look into event ID eight because we already looked into those other two that were there. Most true positive process injection attempts you can find in both of these Sysmon event IDs. This is create remote thread detected and another one that you don't have the ability to see it all with plain windows logs. And it lets you know here that PowerShell has injected code into DLL host. And you can see it here in the source process as PowerShell and then the target is DLL host. And then the next is 10. And this reports when a process opens another process and that's usually followed by information queries or reading and writing that address space of that target process. And during this attack in the lab there is a generous amount of both event IDs because the command was run and it attempted to inject into all of the processes. So this is just lab data. And it kind of shows that that encoded content isn't exactly the same as our actual customer event that we had in the use case. So if we dive into that, we take that encoded command at the beginning of this use case which was base 64 and I decoded that. And then I was really confused because if the top parts where I grabbed it from PowerShell on the left, the bottom is where it got decoded to but then it looks even more decoded and I tried to like double decode it, but it didn't work. Turns out that it was a base 64 GZIP file. And then after looking that on the left hand side, I'm like, I don't know what this code is. So I started to look into that decoded GZIP and it was cobalt strike. And this is one of the methods that cobalt strike uses to avoid detection. And another funny story, when you include this kind of thing in a presentation, it's best to take a screenshot of it as opposed to just copying the code because it turns out that a lot of places, it was very hard to email and share this presentation and no one could open it because all the endpoint agents were freaking out thinking that I was trying to use cobalt strike hidden in a keynote file. That's great. So you had to make sure it's an image. Yes, yeah. So definitely learning moment for me there is to always just take a screenshot. And then here's a summary. Really just look for ComSpec, right? Super easy. No one ever does that. Nothing good is happening if ComSpec is running. No, it's so weird. It's just this environment variable that you'll rarely see and is worth detecting on as a threat. So now that everybody's a little bit more familiar with event IDs and Sysmon, we're gonna tie it all together and a customer exchange compromise. I won't replay that lab because that's gonna be another million screenshots of event viewer. But this is an example of proxy logon which came out in 2021, proxylogon.com explains all of it if you really wanna know. But it's basically just a remote code execution that as long as your exchange server is open on port 443, people could attack you. So it generated tens of thousands of breaches of exchange online. So many incidents. Yeah, people's on-prem exchanges. Yeah, so not too long ago after, I mean, I guess this would have been a year ago now, we had a customer come to us and asked for help around a notification they received. There's two MSI files that are installed on the exchange server. Other commands were being run that didn't think should be running. And there were actually 11 rapid-fire findings. And so you'll notice that's the next several slides as they all happen in rapid succession. First, you see something along the same lines as that process injection we talked about in the other use case. So we basically have PowerShell being run, injecting into DLL host and acting alone with nothing else. It could be something not super terrible because we've seen legitimate software do this before, but then you get something like this, which is a discovery command, again, on its own, not terrible, but this is net group DOM domain admins, which is a command to find all domain admins in the current domain. Helpful for admins, right? Also helpful for attackers. Yeah, if you only have a couple admins and you're all looking at each other going, who's looking up the rest of us? Right, you start to be like, all right, well, that wasn't me, uh-oh, that's not good. And then here, you have a little bit more suspect activity. Granted, this is something I've done before, right? You need to transfer files and you just decide, okay, I need to open up SMB and transfer from one to another. Not great, especially if you're doing it to a domain controller from your exchange server. Very bad, right? That's definitely somebody just copying stuff back and forth. Even the effect that they could do that means that something is not set up properly. But yeah, this is when they started to really freak out. Talked a little bit already about PowerShell encoded commands, but this is something that happens in malicious PowerShell all the time, is that net web client, so you can, you know, a couple lines of code in PowerShell call and execute strings, like it'll download that string and execute it, right? Super easy, it's seen all the time. And then finally, you see this IIS web service process spawning another child's process. So in this case, it was PowerShell and associated with that full exchange compromise. This did turn into like a full IR response investigation, but without the use of Sysmon, who knows how long it would have been to see that compromise, right? It could have been way after this. You wouldn't notice it happening, right? Oh yeah, you would have just had a ransom exchange server at that point, right? And then with that last detection of IIS, so if you have on-prem exchange, definitely be logging your IIS logs and your process logs on your IIS server. That W3WP-EXE is the EXE name of IIS, which the, what is it, Outlook Web Access uses, and it should never respond PowerShell or command line. Never, there are no circumstances for each, that is a good thing. No, there's, unless you are being, maybe being pen tested, but above and beyond that, that is a terrible thing to ever have. Even then you let them know we found you. Yeah, yeah, for sure. So, you know, rip off your blindfold. There's a lot of different things you can do. You know, you can have a gap analysis and figure out where your quality of logs are now versus where you want them to be. You have critical devices that are not logging the right stuff. You have to plan for a lot of these attacks, which can be daunting because there's attacks for everything, right? But one of the first things that you can do is improve the processes of, you know, include installings this month, right? It could be a change in configuration. You can align these with company objectives, like, almost everything in security can be tied to something in your business, right? Whether it's reduced downtime, money savings, whatever you want to tie it to, you can usually align this kind of activity with bolstering like your security posture, okay? Something that we created, and it's out there for anybody to use, it's called Posham. One of the most difficult pieces we had in the beginning of Starting Boomera was installing all this stuff at scale, right? Because you can use, we've used NX Log, right? Which has its own configuration file and Sysmon, which has its own configuration file. But if you're not paying for them, you have to have like the right configuration or it doesn't start or you have to have the right logs being generated just like silently fails. So we were tired of having to do that and figure out, all right, well, this server has IIS, this one doesn't. This one's a 2016 server, this one's a 2012, whatever. This actually just does it automatically, which is super nice. It figures out what channels you actually have and just shoves it into config file. So it makes it much, much easier. You can, you don't have to have Blomera to use this. You could just plug in whatever you want, but yeah, it was much easier to just do that than manually. And then pick a configuration. A lot of times you saw the Enrichments that Sysmon Modular gave us with all those MITRE TTPs that it lists and it gives the ability to turn on and off certain levels of logging, especially Sysmon isn't made for all devices, right? One of the things we ran into is if we had a couple of customers that had really old servers with spinning disk drives, like their DCs were just some 10-year-old box on the floor, the amount of IOPS that Sysmon has to use to write all of those DC logs that are coming in kills the drive. It makes it very slow. So if you have SSDs, you're probably fine, but just know that Sysmon Modular is out there. So for the slower devices that we have, you have the ability to not overwhelm them if you really, really want to. And I always include this too. Always test your SIM. Whether you're doing it yourself, whether you have a third party doing it, having an active relationship with your MSP, MSSP, SIM vendor or whoever, there needs to be some kind of regular verification that you're detecting stuff, right? Whether you have a pen test or whatever, there's just so many things that can go wrong to make detections break, right? Service stops, a firewall rule gets implemented, something, right? There's a lot of different things that you can do to test. Like there's easy password spray stuff out there. There's just real easy commands that you can run a command line to test it. Yeah. One of my favorites, and I did this when I was doing a Blumerid demo was you could just go to who am I and you'll notice it'll flag that because this is not something commonly run. The testing doesn't have to be that hard. My favorite way to test is get some accounting firms and have them as clients. There's a secret class. They give all accountants that say, click on everything that comes through your email. I'm convinced of this. I don't know what the class is called, but there's something about accounting firms. Great, because, you know. They're quite vicious as I have named some of them. I'm like, this person will click on anything. Oh, that's terrible. Yeah, accounting for, I don't know what it is about that particular industry. I thought sales was bad, but that's very interesting. Yeah. We need some industry stats on what's- Yeah, I know, right? There'll be a panel discussion. What's the worst industry for clicking on data? Yeah, we have to ask no before. I'm sure they have stats on that. Yeah. And then last but not least, if you want to do your stuff internally, this is a great combination of tools to track what you're detecting and how you're detecting those things. And that's it. Yeah, Vector, Aramik, that's not, I'm sure there's links to everything. So it's easy for you to be able to click on everything in the description. Yeah, so this is me. Thank you so much for having me on. I love doing this stuff. And I love preaching the great word about Sismon. Well, it's fun. This is such a deep discussion on, we started with some of the theory and then some of the practical as you go through it and understanding the big picture, if you will. It's not just the commands that are run. This is like the whole top to bottom of why you need these things, why we need some of the alerting and how we're looking at it as security researchers. I think it's just, it's critical. Cause I get that a lot with the, as I'm sure you do with the younger people starting in, newer in their career. How did you see that or what made you think about it? And that's where the theory and anomaly detection really comes into play. Oh yeah, yeah. I definitely suggest reading up on if anybody ever has walkthroughs on instant response, right? So like either they've seen an attack in their lab or they've seen an attack at a customer and it's been like anonymized or whatever. They're always really interesting to read like the really technical, well, at least to me, the really technical versions of those because you can actually see, oh my gosh, like they went from this process to this process and that's how they did that. And the better articles, I think anyways, tell you what to detect on in those areas, right? I spend time reading different reports, you know, it's good, good reading. There's some great sites for that. And one of them was interesting, I posted last year, I can probably find a link to leave that down there. They had actually gotten a treasure trove of a weird different report because it came from the threat actor. They were actually monitoring a threat actor. So they got to walk through what their day to day was. Of how they were doing it. It was really cool. I put it on LinkedIn a few places. I can't remember if it was Recorded Future, I think it was Recorded Future who actually published it. It was a great read because it's very, very detailed. And it just, they had logs of everything this person had done, how they made their notes on how they attacked, what was successful, what wasn't. But this is how we update our threat modeling for what you should protect against. Cause I'll see people kind of go off and really worry about this or that when they look at their threat models. I'm like, just read different reports. You'll understand it because you only have finite number of resources. People get excited about some of the physical layer stuff because don't get me wrong. Go into a hacker conference and you watch someone like Jay Street give a talk. You're scared, you're blown away, you're amused. But honestly, he is probably not in most people's threat model of the, it's just from our day to day with all of our businesses. It's not physical. It's always phishing emails and some of the silly stuff. Hey, lock down. Use security cards. Don't let people tailgate. I'll agree with all those things. But if you only have so much resources putting a better door access system and probably they're getting into phishing, that's the most likely. Once you solved all the other problems then work on the other stuff. Yes, yep. Yeah, for sure. Well, thank you for this. This was a lot of fun. I love diving into these things and all that fun stuff. Links will be down below and thanks. Awesome, thank you.