 And welcome back everybody to DEF CON 28 safe mode blue team village. We're going to start our next set of speakers. Got a great topic coming up today. The title of this talk is building blue spawn an open source active defense and EDR software. Jake Smith and Jack McDowell be presenting. As always, please direct all questions to the text talks track one channel. Or you can connect the discord server as well and send messages to them directly. I'll have that in there in one second. And on that note, over to you, Jake and Jack. All right. Thank you. So how's everyone doing today? Just a few quick notes before we get into the presentation. We definitely want to take a moment to say thank you to all the volunteers and organizers of the blue team village. Putting this event on in a normal year is a lot of work and I can only imagine what they've done to make this happen. So a huge thank you to them. So today, Jack and I are going to be talking about an open source project we've been developing for about a year now. It's called blue spawn and you'll notice the emphasis on active defense here. So we'll get into more about that later. But first, a little bit of background about the two of us. We went to the University of Virginia. Both of us got our start in infosec doing really all kinds of cyber competitions, CTFs and whatnot. And that's been a fantastic way to kind of get into security. And these competitions, we'll talk about more are some of the reasons why we created blue spawn and all. So an important caveat to before we get into the presentation when we say we this afternoon. We're referring to all six of us on the slide. It turns out that developing a large open source project with already like 20,000 lines of C plus plus is a huge, huge undertaking. So it's really a team effort that we've been doing to create this. So, and also before we get into talking about blue spawn specifically, we're going to do a brief overview. Some of the available tools out there and kind of where they shine, where they have some short calls and more. And really, this is just all the tools that we're going to mention do a fantastic job, which is part of the reason why we want to talk about them as well. So first we'll start with your endpoint detection and response or EDR tools. That's your crowd strikes, your carbon black, your defender ATP, Sentinel one and all. They obviously used to like 10, 15 years ago, AVs were pretty simple. They look for bad hashes and whatnot. And now kind of that's they've increased and they've gotten really, really good. So these commercial tools and they're a bit expensive in some ways, but they really raised the bar for finding the low hanging fruit, especially in some even some of the more advanced malware. They're obviously not all created equal, but one of the things we noticed when we started developing blue spawn is that these defensive security solutions are pretty opaque overall there. There's obviously a lot of offensive tools out there, but it's really hard to get your hands on and kind of understand how these modern AV and EDR work. Which leads us into a bit about some of the free tools that are out there. There's not a whole lot of great ones, but definitely system internals is probably one of the best if you haven't used the Sysmon. If you haven't used much just beyond like Sysmon to improve your logging. Highly recommend taking a look at like auto runs and process Explorer and Procmon just they do a great job of finding malware on the system. They're more geared though toward the system analysis side so they don't really they're not going to say like hey this is malware they might show like a different color if this is like encoded or something but they're not going to point out and like say this is might or attack technique 1547 right for example. Next is process hacker process hacker is another really great tool. There's not much automation or security focus, but it's really good at just analyzing it's like task manager on steroids really. And you can even look at like a specific region in memory. So like if you know that one process was injected, for example, then you could go and analyze each bit of the memory yourself. And then on the like purely defensive side, we've definitely found PESive to be one of the best free and open source tools out there by far. The idea is pretty simple beyond the tool. It's basically look at all the processes running on the system, and then it can find hooks process Halloween process injection shell code, all that kind of good stuff. The only downside if it's really a downside is that it's limited to scanning processes so it's not going to find like other malware say like that's hiding in the registry on a system. And finally, then this kind of available tool section is like the collections of Yara rules, Sigma rules, system on campaigns out there. Even just a few years ago, there just wasn't really a whole lot out there, especially on the defensive side in way of signatures. And but these projects are doing a really good job of starting to share threatened tell a bit more widely and can help you find threats just on a particular system. The only thing I guess to caveat is like with hardening the scripts for example, a lot of the stuff you'll find out there is very, I guess more script like and not fully fleshed out. So that's one thing to be aware of like I'll set conflicting values or miss critical settings and whatnot. But so that I mean that's, that's kind of the reasons why we got into building blue spawn which we'll talk about here more shortly. One of the key things we want to emphasize this afternoon is that is the ideas and principles on which we built this tool, you're probably not going to go into work Monday and deploy blue spawn in production. And I guess you could but that's maybe not something you want to do quite yet, because the project's not quite that far along. But that said, you can use the information in today's talk to to conduct your own security research and begin kind of improving your own defenses to and looking at how these kinds of solutions work. With all things, all great ideas I guess started in Slack and whatnot. So this was I went into our archives. This was last year like we started with this really simple idea like what if we could like scan one particular thing like look at just registry run keys like as a way to detect threats and whatnot on a system and like an open source phone we didn't find a whole lot out there. Let us to do creating blue spawn. And as I said we were students or we are students so we compete in a lot of these different cyber defense competitions and whatnot. Honestly, frankly, I tell some people that I just do go to school for competitions and all. But basically these cyber defense competitions are basically you get a network and you're asked to secure it against active attackers who unfortunately for the students are industry red team so you can imagine how that kind of thing goes. And the red team pretty much always wins. So in these competitions though we can't really we can't use commercial tools and we can't bring in scripts. So we have to rely on open source tools that's kind of another reason why we focus so heavily on the open source aspect. And so we built blue spawn on one really particular focus is that we want to be able to stop threats as quickly as possible so think that you should be able to run this tool and find as many if not the majority or more of all the malware that's running on your system as quickly as possible. And so that is where the active defense idea of active defenses come comes in. And then the next is coverage so when doing these cyber defense competitions it's it's time limited your stress, your stress and whatnot so you don't have the luxury of say being able to look at everything very closely and you can't pull up process explore and look at every individual process to hunt for process injection. So our idea is that if you can get 6070 80% of the way there. I'm finding the most common threats. And you know what your tool is going to be looking for you can spend more time on this other areas as well. And finally open source software so this is a call to everyone out there. There's a lot of stuff on the offensive side, but there's just not really as much on the defensive side so definitely building more open source blue team software I think is really important and this is a great way for people to learn about how these kinds of things work and how you might detect say one particular technique in the attack matrix. So as time went on from that simple idea that we posted a while ago in slack. We started moving blue spawn towards what you might term as a colony DR type solution today. So that's where it's doing continuously monitoring. And then asking questions like what if we could just deploy blue spawn across the whole network, like you might deploy like Falcon. This is kind of the ideas that we're heading towards as we spend more and more time developing this, and then also proactive So it's not enough that you can detect malware and respond to it right we also one of the core focuses is the idea of being able to prevent these malicious activities in the first place. So we've talked a lot about active defense so far, but what we really mean by it and how does blue spawn approach the idea of finding anomalous or malicious behavior. That's kind of what I'll get into in this section. The typical antivirus or EDR products are meant to kind of run in the background behind everything only raising alerts and acting when there's something definitely bad going on for their use cases. This is a very good thing. People don't want alerts about everything that may or may not be malicious, but when you've been compromised or even when you think you've been compromised, that might not be enough. This is where our concept of active defense comes into play. Rather than sit in the background only really making noise, blue spawn is meant to be a tool that's actively used. The goal is to find absolutely everything that could be malicious, not just the things that knows for sure are malicious. In such an approach there are drawbacks in catching all the things we're sure to catch some things that aren't actually bad, but for an active defense situation, that's a trade off we want to make. How does blue spawn go about approaching this? Originally, this worked by just performing a scan of the system at a point in time to identify all the threats that were currently on there. And this hunting involved the typical registry scans, file scans, or process scans that an AV or EDR product would use. But again, blue spawn tried to be more in depth, taking more time to look at things with more focus on finding anything that's not normal. And I think part of what makes blue spawn different is we also factor event logs into our hunts. So correlating what Windows events have occurred with what is currently on system to identify any sort of malicious behavior there. And we're trying to basically use that to bridge the gap between incident response and what EDR products do. And as we move towards continuous monitoring, we extended these hunts to be able to run as needed. And I'll talk more about how we did that shortly. So as I mentioned on the previous slide, blue spawn is built pretty heavily around the idea of what we call hunts. For each mitre attack technique we support, which is basically a way attackers do things if you're not familiar, we created a hunt, which basically is a search for any sort of evidence of malicious activity using technique. We'll get into more detail about how specifically these hunts work in a bit. But for now, just know that they're really the starting point for everything blue spawn does. Modern Turing, as I mentioned a while back, is really just an extension of the hunts we've already written. Each hunt creates triggers for when it should be rerun and when those triggers occur, it gets run again. Now mitre recently released sub techniques, which break down each technique attackers may use into subcategories, and blue spawn has just moved to handle that by making each hunt responsible for all of the sub techniques that fall under it. So these hunts are really great for finding the starting points in malicious behavior. We can find registry keys that run files, files around startup libraries that aren't meant to be included at all, all sorts of fun stuff. But malware tends to have a number of different components that it uses, or even if it doesn't, it exists in multiple different forms in multiple different places. Maybe there's a registry key that points to a file, and then that file is loaded into a process. And on that process is changing other register keys and those keys point to files. And those files, you get the idea. To address this blue spawn searches each pieces of evidence and finds or to see if they can catch anything related to it. The result from this is a network of evidence of malicious activity. And the next section I'll go into a bit more detail about how this actually gets built. Now that I've given a higher level overview of our methodology, I'm going to take a deeper dive into the different things blue spawn can do with a focus on how it does them. And this section is going to get a little bit technical, but I'll finish it with a more broader review if you don't catch me anything. So, before I get into any of the features I want to preface with a disclaimer blue spawn is still an alpha development. So far we've been mostly focusing on infrastructure that is building out the core of the features I'm getting into. The other thing I'm presenting is currently implemented, but it's not entirely without its bugs. And the basic, the biggest example of what I'm talking about is our scanning system. I'm going to say things like this is where blue spawn scans x, but the actual scans performed are new and not necessarily heavily tested. At this point, I'd say a lot of the features described here are better talked about as a framework that fully fresh fleshed out product. One of this is to say blue spawn isn't effective. We spent a long time building our hunts and expanding our coverage, and it's proven to be pretty good as its job as Jake will showcase in his case studies section. And now that's out of the way onto the features blue spawn has five major components, each of which will discuss in detail in this section. There's mitigate which applies security settings hunt, which runs the funds I was talking about earlier scan with determines how bad the detections are and find associations monitor, which hunts over time and react, which handles responses to what hunt mode and scan mode identified. So mitigate mode sits somewhat apart from the rest of blue spawn without really any integrations to the other mode. So I'll just talk about that first to get out of the way. Mitigations applied by blue spawn are all mapped to either a security technical implementation guide or stick released by DSA, or the minor attack medications framework. When run mitigate mode blue spawn can either audit the current state of the system which just says this is what it's currently set up as, or it can go ahead and apply the medications it finds aren't properly configured with user confirmation of course. Now as of now our mitigation framework is very quite simple, but we think it has a lot of potential as blue spawn continues to grow. We see it eventually being configurable through some sort of configuration files and mitigations can be more uniformly and quickly applied across a network. We also see it mitigating services and software installed on a system rather than just the base operating system. And finally our biggest envision change is for mitigation mode to serve as really an aided performing system audit. So this would perform some of the same tasks as enumeration tools such as seatbelt or jaws with a focus on finding and fixing things actually being vulnerable to being exploited. Now before I get into the weeds of how everything works I'm going to throw some definitions at you for how I refer to the ideas and concepts blue spawn uses. So I'm going to be using the word detection a lot and it's just that detection. It could be something good it could be something bad it's just something blue spawn identified is taking a closer look at. It could be like a file or registry key a process or something else entirely. Next there's an association which is also just what it sounds like. It's a connection between two different detections and the blue spawn, we represent associations as having a weight representing how closely they are associated. The next thing is certainty and this is the metric we use for determining the likelihood that a detection is actually bad. There are three components to it, the two of them get combined so in practice there are really only two. The first is what I call intrinsic certainty. This is the certainty that comes from the detection itself. This would be what we're talking about if a service is called mini cats, or if file match to your rule from interpreter or something like that. The next is contextual certainty. This comes from the context surrounding this surrounding the detection. In a lot of ways this is very much like the intrinsic certainty. And in fact blue spawn doesn't actually differentiate between the two are really combined. And what I mean by context is things like a file in system 32 that's not signed by Microsoft. It's probably bad just because they usually are, or any value was under an app in it DLL probably bad just because they aren't really used legitimately anymore. The last type of certainty I'm going to talk about is associative certainty, and this is the certainty that comes from the associations of detection. And we found it loaded into a process. This if the process is behaving maliciously, and the file appears malicious, the prop the file should be treated with a higher certainty than if all we had was a malicious looking file. Now that I've got those definitions out of the way I can move on to how blue spawn actually works. The core of all detection management and really the core of blue spawn itself is our detection register. This tracks everything that blue spawns found. Whenever a hunter scan or whatever finds anything that might be bad, it sends it to the detection register. The register will record the detection and send back a reference to it. Now this reference may or may not be the section that was recorded and I'll explain why next slide when I explain how the registration process actually works. The detection register has three storage mechanisms. So whenever detection gets registered, it gets queued for a scan and we track all Q detection one set and all scan detections and another. That way we can just make sure we're not redoing the same work. We also keep a record of all detections regardless of whether or not they've been scanned. So now for how registration actually works. The first thing the register actually does when a new detection is being added is it checks the detection already exists. If it does, they get merged together. And generally the only thing that changes when this happens is their contextual certainties and their contacts get merged. If this merge causes their overall certainty to exceed a certain threshold, it then gets logged and its reaction to get triggered. If the detection being registered was merged with the pre-existing one, the reference returned is actually to the pre-existing one. That way it's just keeping track of the detection that we care about. Otherwise it's to the new detection. After all of this is handled, if the detection register needs to queue scans for the detection, it will. This constitutes calculating the intrinsic certainty, identifying associated detections, and then if the certainty is high enough after that, logging it and handling reactions. The associated detections then get registered and the scans for those get queued and so on. So here I've got a diagram for how the detection register lays things out. It looks kind of complicated, but I'll try to explain it so it's maybe not quite as bad as it looks. So the register itself has three main storage mechanisms. There's the queue of things that haven't been scanned yet, things that have been scanned, and then everything overall. And you can see that all of these storage mechanisms really just store references to detections that are stored elsewhere. I've also broken down one detection, in this case ID2, to show what kind of information the detection actually holds. So it first has this ID, which is just an ID unique identifier to refer to it. It has a context, which is the information surrounding how it was made and how it was found and when it happened and all that. Instead be what time the detection occurred, what hunt found it, and when the first evidence of the detection was found. There's also the data, and this stores information about what actually did it find. So in this case, I've just decided, for the purpose of an example, this detection is going to refer to a run key. And it's value name. Its name persists and it's pointing to this file called bad.exe. What I'm going to have is the scan information which stores information about how it was scanned. So that has the certainty which combines the intrinsic and the contextual certainties, and then the associate of certainty. The last part of this is the associations which identifies which detections are associated with this one. In this case detection ID one which I'm using to refer to as bad.exe is associated with it. There's also our mediator and I'll get to that a bit later. So in the methodology section I mentioned how hunts represent the starting point for pretty much all the threatening that blue spawn does, each one covers one minor attack technique. Each hunt, we then break down into sub techniques, which can be run independently of each other within each sub technique. We then break it down then into sub sections. And one sub section generally checks just one specific attack surface, whether it's a folder, a file, an event ID or a red issue key or something else. The idea is to keep it as simple and small as possible. This design allows for much more targeted hunts if we know what we're specifically we're looking for. And I'll get into why that matters when I talk about how monitoring works. One of the things we check get a quick scan, which just performs in those elementary scans is a very quick test to determine whether or not something deserves more detailed scans. If it does, we created a detection for it and registered with the detection register and kick off the whole scanning now just finished talking about then the scans I discussed get run on it and make a determination final related things so on. Here's an example of what a hunt looks like. In this case, this is a T 1547 the hunt itself really just runs all sub techniques underneath it, you can see here the and then here's the sub technique number four, which I picked out because it was pretty simple. The first part of it just compares the wind log on values for the shell and user net against known goods. To match, we create a detection with a moderate contextual certainty because anything that doesn't match chances are it's bad, even though the file itself might not be bad. The next subsection refers to the wind log on notify values. Any of these that we find get detection created for it just because they aren't used very commonly. And we create that with a weak contextual certainty because again, if you find one of the chances are it's not great. Now you'll notice during these I specify cursory. And I initialize a subsection that comes from what when you run blue spawn you specify how closely it should be looking for things. There's cursory, which is a very quick check. There's intensive which is very slow but careful check and there's normal which is somewhere in between by specifying cursory here, we indicate that blue spawn should run these subsections whenever the intensity is greater or equal to the cursory. So when we originally had our first working version of blue spawn, it was just a few points we'd written and not too much else. Since then everything we've done really has been taking what we had and proving it when it came time to start monitoring systems in real time rather than just forming a single scan. We realized we could reuse the hunts we had already written to do this. Each hunt we created already defines a number of events that indicate something pertinent to the hunt has changed. For example, we might make an event watching a folder for new files or old files being edited. When the event gets triggered, hunts that care about that folder get rerun. Of course, there are some obvious inefficiencies in this, which is a big reason why we divided hunts into subsections. Now, when hunts create events they specify which subsection should get run for each event. Now I'll admit monitoring is unfortunately one of the areas that hasn't received as much attention as others. It's certainly functional and it's effective at watching for things that hunts can catch, but so far we don't have that much that acts based off of behavior alone that we observe while monitoring. In the coming months this is going to be one of our big focuses, adding things like event tracing for Windows or API monitoring should position blue spawn to be able to catch much more. So here's an example of how monitoring sets up hunts of all of its monitoring triggers. So if you noticed my example hunt two slides back, each of these sub-takes gets a scope passed into it. Well, under the hood what's happening is the subsection macros I have actually check the scope against that subsection ID to determine whether or not that subsection should be run. So whenever I create an event here I specify which subsections are pertinent to it. And yeah, so here we have registry events file and file events, but we also have event log events to watch for Windows events being created. So the last component I'm going to talk about here is the reaction framework. As the name would imply this is responsible for taking care of detections that blue spawn has determined are bad. Since blue spawn is designed around finding as much as possible and is bound to have some false positives. Most things here require some sort of level of user confirmation before taking action. Now, obviously when I delete a file it's no longer an active threat, but there's still a reason for blue spawn to track it. To handle this sort of situation, each detection has a flag where we mark it as stale or not. Whenever reaction mitigates the detection you mark it as stale so that way other reactions don't go and try to mitigate it their way. Currently we have five reactions available that there is somewhat of a less defined six and I'll get into that. Blue spawn can delete files, quarantine files, delete registry entries and that can be values or keys to spend processes or carve out memory. And I'll talk about that in the next slide. The less defined reaction I just mentioned is for special cases. Remember back to the win log on hunt where it was comparing some values against known goods. It wouldn't be very good to just delete these values that they were malicious. They need to be properly restored. And to handle things like that when the detection is created it can be given a customer mediator. This remediator gets precedents over all their reactions if it's present. Now, as we continue to develop blue spawn, we of course intend to continue adding reactions and improving the ones we do have. Future reactions might include things like the register and service or blocking certain commands. So I promise that I get into war when I'm meant by memory carving. So here we are. The idea behind memory carving is that sometimes libraries or shellcode get shoved into processes and these processes still serve a valid purpose and need to remain active. In cases like these where a restart isn't a good solution, we cannot put the weight and mitigate the effect of whatever malicious library or shellcode is looking inside a process. It's not perfect and it can cause problems but it's better than nothing. This reaction works in five steps. First we suspend the process so that nothing we're doing causes problems with the way the process works. First what we do is we enumerate the threads in the process searching for any thread whose stack overlaps the memory section in question. And that basically is an indicator that the thread is acting based off of what that malicious memory section wants it to do. Or when we find this, we kill any thread that does that. Since malware generally doesn't like to share a thread or something benign, it shouldn't kill anything important to the process. Of course, if you've got things like APC injection going on, there's not too much we can do about that right now. After that we scan all memory pointers to the memory segment in question. If or when we find them, we check the address in question. If it's data, we patch it with zero so it can't read anything important there. If it's code, we patch it with return instruction. So if it ever tries to call there or jump there, it gets neatly bounced right back. Then we go through and patch the entry point and all functions that are explored with returned instructions. So anything trying to run any function inside the memory gets messed up. This can cause problems if we end up patching any function that requires the call to clean up the stack. But with that extensive work to figure out how much the stack needs to be cleaned up, there's not too much we can do. Once we're done with all that, we resume the process and let things continue. We have tested this with a couple different pieces of malware, including Cobot Strike Beacons and Materpter, and it seems to work without causing too many problems. Unless processes are storing function pointers in an abnormal manner, it should be very difficult to get anything in this memory segment working again after what we do occurs. Now, we are working on a few features to improve the stability of this. This includes things like forcibly walking threads back instead of just killing them, or finding out how much the stack needs to be cleaned up when patching returns, or properly unloading libraries as seen in Process Hacker rather than just breaking them. Now, Blue Smong would have been nearly impossible to build from the ground up without any libraries or integration. As has been a major theme in our presentation, the MITRE attack framework has been invaluable in providing guidance around what sort of things we should go for. YARA and YARA rules written by the community have provided immense value in being able to scan files and make determinations. The stigs around which we built our mitigations make it very clear what we should prioritize and give us a framework to convey our coverage. Finally, PE SIP is a great open source tool we use to scan any process for any sort of fileless execution or any form of execution that's not properly mapped to a file. In summary, blue spawns abilities can be broken into three categories, prevention, detection, and response. Under prevention, we have Mitigate mode, which integrates stigs and miners attack framework. Under detection, we've got monitor mode, which watches for changes to re-run hunts, hunt mode, which is built around the MITRE attack framework, and then checks these threats. Scan mode then analyzes what was found in hunt mode to find associations and make determinations. And finally, under response, we have our reactions, which aim to mitigate or remove the threat and then logging to report what was found. All right, so that was a lot, but we wanted to really give you a deep insight into kind of how some of these AV and EDR type programs and obviously blue spawn in particular work. And so now we're going to move into talking about just like seeing the tool in action, right? So we're going to walk through a case study with some kind of specific point in time screenshots of blue spawn catching malware. And then also we're going to have a video of kind of the tool in action. And I mentioned cyber defense competitions have been a really important learning tool at the beginning. So first, I want to emphasize that again. The largest is probably CCDC and it's held in about 10 regions across the U.S. with qualifiers, regionals and nationals, and about 230 universities compete each year actually. So that's one way if you're looking to volunteer to help in the community, highly recommend looking into a competition like that or others. So these competitions are basically what's known as an inherit and defend competition. The student teams work in a group of eight and they're basically work as a fictional company. So like they have a whole set of networks they have to defend and typically that's on-prem. There's some cloud aspects and that's usually done in like ESXi or AWS. So here you can see an example of a network map from actually last year. So you have some on-prem workstations. There's like on-prem cloud if you will and like ESXi. And then there's even like an AWS component and you'll see the stuff in there ranges from Windows Server 2008. So like really old stuff to Windows Server 2019 and every flavor of Linux distro really. So they kind of emulate a modern corporate environment the best they can. And there's of course stuff like exchange and web apps and databases and all. And so basically there's two goals in these competitions. You have to keep your services online and then just like you would in a real company and then you have to stop the red team from really just totally destroying everything. And really frankly that ends up in situations like this. So on the left hand side you see this is from the national competition this year. And it's where a picture of the red team's cobalt strike instance where you can see they just have some dozens of machines against the support team. And they also do things like defacing the websites and RMRFing boxes or overwriting MBRs when they get too much access. But really what that means is that in all reality is that these are a great testing ground for defensive tools, right? So the red team spends all year writing malware for these competitions and are probably watching right now. So I read team. But so their tool set is everything from like metasploiting cobalt strike what's pretty commonly used to really custom malware that's just written for the purposes of CCDC. So with that kind of backstop in mind we're going to look at some examples screenshots of blue spawn. So here we just have the system. There's two cobalt strike beacons running. One is just an HTTP beacon and then the other is an SMB beacon living in Explorer.exe. And so this is how blue spawn goes about finding it. It's looking at when Jack talked about process injection earlier and the card memory reaction. You can see blue spawns finding finding the malware in both Explorer.exe and then also in the PowerShell that's running. And then it starts prompting the user like, hey, we found this malicious activity and all the user did was just run blue spawn. And then it's offering to kill the threads and once it kills it, then we reach a situation like this where those beacons basically just stop calling back in. So that's an example of how blue spawn is responding to the threat. And here's another classic threat. It's not used obviously as much now but the sticky keys backdoor is where the registry key is configured with command prompt as a debugger. So blue spawn is able to look for that and that's T1546 sub-technique 8 if you're following along on the Mitre attack matrix, right? So it knows to look for that and then it goes and finds it and it lets you remove that option. Next, this was probably one of the coolest examples we saw at a cyber defense competition that blue spawn caught earlier this year. Windows has these things called authentication packages and notification packages and these exist in like the LSA sub-system, which is used for authentication and all. So blue spawn is able to map and it's like, hey, we found a registry key and it knows it understands that in the hunt that this hunt refers to things on disk. So then it goes and correlates that with the file associated with it. So we have two detections, one for the file and one for the registry key that contains the badness. Next here is the good old run key. So this is, there's a malicious BBS script that's configured to run in the registry. And again, blue spawn is able to correlate the two and be able to connect that to generate detections and find both pieces of malware. Now here's a web shell. This is a great, I like to use this for showing how the context works. So blue spawn uses a combination of regexes and YAR rules and it's scanning all the stuff on the system in the web accessible directories in say like I ask for PHP files. And it's able to say like, oh, this was caught by this YAR rule. This piece of this file probably contains a web shell. And the next here is a pretty simple example with like backdooring services. So as hackers like to do this, of course, to get their malware to persist. And they can do this in a lot of ways. And so this is just a very simple example. The blue spawn will scan all the services on the system to find which might contain a piece of malware or what services suspicious. So we want to get some feedback on red team, of course, when we made this tool and this was actually probably one of the first years they've ever had to contend with an open source tool being built by one of the teams. No one's really done that before just because it's a lot of work and if other teams are using it then right maybe you're not getting quite as much value out out of it. But I think it worked out really well. And as you'd expect, our red teams spend a lot of time analyzing the tool since it's open source. So the attackers know exactly what it can detect on, which is it's a bit hard to when you think about it, you're like, I don't know if I want the red team seeing it. But what we found is that it really pushed the attackers to move to new techniques so they couldn't just rely on the same things over and over again because they knew we were going to catch that. So it forces them to keep moving and shifting their tactics, which means they might be less detectable, but it also really increases the barrier to make it harder for them. And honestly, one of the coolest parts is that the red team unveiled something called red spawn this year. So that was a tool that was apparently supposed to like attack and defeat blue spawn. So that feels like a validation to us for sure that blue spawn is effectively killing stuff that they used to rely on. So from the blue team perspective, what we found in our experiences and other teams and all is that blue spawn works kind of like an EDR product that you'd expect to. So you can identify it'll identify and respond to malware on the system. One of the cool things we found is that it's like a force multiplier, especially in these active defense situations. So imagine if your tools deployed out across the multiple systems, right, you don't have to manually look through all like the auto run yourself. It'll just flag which ones are most likely to be suspicious. And that way you can spend more of your efforts hunting where it counts to find the malware. And so that's really important. So we've prepared a little demonstration of what blue spawn looks like an action and how it can be actually used to hunt and then destroy malware. So I prepared a Windows Server 2012 R2 VM here and I filled up with a bunch of different random pieces of malware and persistence. So you can see here that I've got two active meterpreter sessions on it. And I'm going to use blue spawn to go hunt and destroy them among our things. So first I'll show you blue spawn something and you just see you can see all the options it has and that's a pretty simple interface. Because I want to go find bad things I'm going to run it in hunt mode. I'm going to specify a normal aggressiveness just because I don't need to go too in depth to find anything. When I find stuff I want to react by deleting files so easily delete file reaction want to remove values so remove value reaction and car memory. And then I went to log to a file so I can view it later and then the console so I can see it now. Press center and now it runs. That's pretty much all there is to get it to run. So here it found a process injection into SVC host PID 632 and I'll open it up in a process hacker. So you can see here that it actually tells me the memory address where it found the injection. So I can actually go take a look at that in process hacker. And you'll see here that it's RWX memory section which generally is not a great thing. So I'll go ahead and carve that out. And here I found a couple more so I'll go ahead and carve those out too. And here it found some process injection into when startup the EXE. And so I'll go ahead and carve those out. And now here it found the sticky keys back door that Jake was talking about earlier that image file execution options debugger. And so I'll open that up in auto runs. So you can see that it's also found by auto runs. And it's under the image hijacks tabs. Sometimes everything doesn't show it. You can see it there and there. We'll go ahead and remove that. And here it found the run key that was running the process we found some injection in earlier. The windows startup actions which definitely is not legitimate. So go ahead and remove that run key. Here it found a net as a helper DLL, which is a sneakier business technique. You can see it in reg edit and blue spawn actually protects the files during scans. So this actually won't be able to show the signatures on it. But I assure you it's not good. So we'll go ahead and remove that value there. And now I found the PHP web shell and we can go ahead and remove. And here is the malicious service. So go ahead and remove that. Now, for some reason, auto runs can't find it, but I'll show you in this command prompt that absolutely is there. So I'll go ahead and remove that was gone. And now I'll go ahead and remove the net as each helper DLL file. And here's the wind start that you exceeded it didn't like. And here's the most service. And that's it. All you have to do. And here's how to run showing you that stuff that found has been out cleared out. So next slide. So here's what happened to the interpreter sessions that I had open. They got killed my car about the memory. And now onto the future work. Blue spawn still has a long way to go. A lot more features to be added. I'm going to talk here about some of the ideas and plans to come up with for future development on blue spawn. So most of our goals right now fall under three categories. Improving coverage, improving configurability and improving integrity. So under coverage, we plan to continue adding new kinds of mitigations. Arguably, our biggest goal here is to expand our data sources. So this would constitute things like API monitoring through a DLL would and all the processes or kernel driver network traffic monitoring user auditing. That's right thing. We've also placed a large focus on improving our scanning capabilities. Admittedly, they are somewhat black right now using only the very basic metrics. One of our goals with blue spawn from the start has been to kind of bridge the gap between incident response and active defense, taking past incidents into account for current threat hunting. Currently blue spawn has taken some smaller steps towards integration between the two, but we intend to work on building that out further and making it really a much bigger feature. As we develop blue spawn, we built most of our features to be very configurable from API perspective. But as of now, blue spawn doesn't really do a good job of exposing those APIs. In the command line interface, we have barely taps into the surface of that. We'd really like to expose more of these configurations to allow things like much more targeted hunts, customizable scanning and mitigation presets. We'd also like to add a way for signatures and definitions to be configurable, so that if users have their own signature-based device to integrate, they'd be able to. As for integrity, right now blue spawn has very little defenses against malware or malicious users actually tampering with it. To that end, we intend to work on hardening blue spawn itself, making it much more difficult for a malicious process from modifying it or causing problems with its execution. And this would most likely be through a kernel driver. Blue spawn also assumes it has all the rights that an admin normally has. But if malware is trying to protect itself, it might restrict access to certain resources that blue spawn was assuming it would just get with no problem. This also would likely be addressed to the same kernel driver. Finally, and this is more of a development feature, we intend to set up detailed atomic red team tests, which are basically tests that test against one specific malware technique to ensure that each build has proper coverage for everything it should. Now, everything I've presented so far has been focused on our Windows client, but our plans go a lot beyond that. Just this past week actually, we finished porting a version of blue spawn over to Linux, where we are now able to write hunts and mitigations. The Linux client is a decent bit behind the Windows one in terms of capabilities and coverage, but we plan to have an initial release at some point soon. We've also begun developing a server used to deploy a blue spawn across a network and then aggregate the logs and manage the clients at scale. This server is still in its early stages, but we plan to ramp up development in the next few months. Finally, the last major component to blue spawn is the cloud. Now, this is a long way off and I'll admit we haven't even started thinking about developing it yet. But the idea is that the clients or servers could send off malware to the cloud to perform better, more detailed scans, likely doing some form of sandboxing or heuristic analysis. This could also serve as a threat intelligence repository, sending out IOCs to clients and servers. So, we've talked a lot. Hopefully, we've given you a really good overview of kind of at a high level, like why we built blue spawn, a bit really into the weeds of how it works and how these types of solutions really work in the real world and then kind of showing you what it's like in action, right? So, we're going to conclude. First, remember that blue spawn is very much still in the alpha stage. So, it can detect a lot of the most popular techniques, but there are certainly some things that it's going to miss and it's a bit rough around some of the edges. Since all the code and detections are open source though, we highly recommend you taking it out, just going to the GitHub page and you can kind of find and look through, see exactly how it works and because it's really important to us to shed light on kind of how these black box EDR solutions that you might have deployed out in your network might actually work. And we also want to extend a huge thank you again to the community projects that are listed here. Really building something like this is not possible without these frameworks and libraries. So, definitely check those two out as well. So, most importantly, if you've gained anything from this talk, we'd like you to walk away with these three things. First, using Mitre Attack to be able to detect threats and think about your defenses is a great springboard and really think about it as a springboard. So, like we found when we were developing blue spawn that Mitre Attack might not have the most in-depth information on a particular technique, right? But you can use that and go as a rabbit hole. So, if you know attackers are going to abuse it, you can go look into that technique more to figure out how attackers might use it. So, that's a really good place to start. Second, knowing your coverage. So, we've used this more times than I can count, right? In our cyber defense competitions and all. And it translates really good to the real world, right? So, if you start thinking about, like, if you're xing techniques off of the attack matrix, right? You can actually get pretty far because if you think about it, the attacker isn't going to be able to use those techniques as well against you once you've got a little x over them. So, that's a great way to think about defense. And then finally, really blue spawn is kind of like a massive experiment just to figure out and understand how modern-day defensive software works. So, just like you might tear apart like exploits and malware, we recommend doing the same thing for your defensive security solutions, right? And thinking about learning about how they work, and then you can use that to kind of build and improve your own defenses. So, just a few housekeeping items, I guess before we wrap up. Thank you all for listening today. We hope you've had as much fun as we have in creating this and talking this afternoon. We also like to thank the conference volunteers, right? This would not have been possible without all their hard work in setting up the streaming and just making it all flow. Finally, we link the slides on the README. We'll have the DEF CON release, if you will, go live here in a bit after this talk-in so you can download the tool as we've been talking about it and kind of see a bit more what it's like. You can look at the slides. We also have a Discord for the project. So, if you have any questions, and I think they're going to post a link to the Discord and all, hop in there and you can go chat with us afterwards and we'll let you know. We'll be happy to answer any questions you might have. So, thanks again, guys, for listening. We hope you really enjoyed it. Thank you. Well, thanks, Jake and Jack. That was a great talk. I look forward to seeing what BlueSpawn continues to develop into. I appreciate you coming out to DEF CON and presenting to us. As a reminder, as they've got a pair on the slide, the Discord channel takes you over to their server. It is also up in the general, under the general tab of our Discord channel, as well as a link in the actual text talk channel as well. Other than that, have a great day, guys.