 Both of these folks have a pretty extensive background, so I'm just going to cover some of it, introduce them, but before I do that, quick show of hands, how many people are doing threat hunting in memory, looking for hostile actors in memory, right? So it's a difficult problem. And so I'm really, really excited personally to hear this talk. This is an area that needs a whole lot of work to be done in it, and it's, you know, folks like Carlos and Marcos that are, you know, on the edge of providing this kind of content for us. So Marcos is a software architect at McAfee. He's leading the development of exploit prevention technology components, which are part of the company, McAfee's next generation flagship product called End Point Security. Marco also led the organization of the first ever B-Sites conference in Cordova, Argentina. Carlos is an experienced InfoSec analyst who drives strategic initiatives and provides thought leadership and insight regarding the ever-changing global landscape at Claro, America, Movil Offices in South America. He organized the one hack para los chicos loco security conference and is now working on organizing the first ever B-Sites conference in Cordova, Argentina. So great, great to have these folks come in and share their research with us. Give them a round of applause. Thank you all for being here and thank you for attending this call, this talk, and thank you Blue Team Villas for having us here. This talk is going to be about this project that we started like a year ago called Mem Hunter. For us, it was really a way to learn on new and past code injection techniques and process manipulation in general, and also it was a way for us to explore and have to cope up with a problem that we were having on the field, which was around doing this threat hunting process in memory at scale. So we will talk about that during the slide and we will talk about the problem that we are trying to solve. So this is the agenda for today. So probably we will skip the first slide and then the introduction part. Then we will jump into the why, the how, and the what part of this issue. So on the why, we will dive into why hunting in memory is important, at least from our perspective. I will leave some hints on that. Then on the how part of things, we will discuss on the how someone can go and hunt for this stuff in memory by looking at specific properties of the processes. And then on the what part of the agenda, we will discuss the tools. On this tool that we have created called Mem Hunter, we will discuss on the characteristics of the tool. And we will also talk about the current state of the tool and then the future. And we will wrap up everything with a demo, with two demos actually. Okay, about that. So, okay, did you want to? Hello, thank you for coming. I'm Carlos, he's Marcos. He is an InfoSeglover. He works in McAfee. He's a software architect. Sorry for my English. You're doing it great. Okay, we are beside Cordova Organizators. Okay, and I work in ARSEC. It's a startup of InfoSecurity, okay? Okay, yeah, thank you. Thank you, Carlos. So why hunting in memory? So, a couple of things here. So why hunting in memory these days, I mean hunting in memory is an important because of the shift that we haven't seen on the threat landscape. So the threat landscape has evolved over the last few years. Mainly, the attackers have mainly started to adopt new techniques, to adopt new things to evade the current detections. So they shifted their attacker trade craft from a purely failed base approach to a file-less base approach. So one friend or one aspect of this evolution is what we call the different vectors that they are using for that. So one of these vectors is what we call the process manipulation. So as part of the process manipulation, there are several things that attackers do in order to evade the current defenses. One of those manipulations are what is called the code injection, right? There is another type of process manipulation called defensivation. So we will touch on that during the presentation. What I wanted to mention here is that mainly because of this trend is what people work on the blue side of the house up to come up with the new ways to detect this stuff. And another thing that which is actually quite important is the fact that in the past these techniques, especially the code injection techniques, they used to be something that was only available for these big sponsor APT groups, mostly national state groups. And they know how to take advantage of this just to avoid being detected and being spotted on their intrusions. But nowadays that is not actually the case. Nowadays it's code injection techniques especially are a commodity. I mean, anyone can go to GitHub, for example, and grab the latest and greatest code injection and include it on their code. So accessing to these techniques are available to anyone right now. So that's why we need to start thinking and including ways to take this stuff on memory. So going back to the file list state that I just mentioned. So there are, I really like this diagram that was taken from a sans white paper. So the idea here is that from the diagram you can see where code injection came into play on the file list attack scenario. So you see that code injection is used as part of, I mean it's mostly used, these series of techniques are mostly used as stages on a stages attack. So, for example, in this scenario where we got a remote code execution attack where someone is executing or getting execution on our end point. An example of this is the WannaCry example, for example, where we have the SMB vulnerable driver and then by exploiting that vulnerability someone can get a code to execute the end point. And then they will most likely use, they will mostly trigger the second stage of that attack by injecting their malicious payload into the memory, the user's memory of a running process, a system process, most likely. So in the case of WannaCry, for example, they use a technique called QU user APC which allows them to execute code, to inject code into the context of ELSAs from a kernel space. So in that case, code injection was used as a second stage of an attack. And the most common attacks that we are seeing nowadays, which are mostly driven by download, we see that code injection is used as a second stage of the attack. So the attacker gain code execution through a Word document, for example, just by having the user to open a document with a macro. Then the macro will exercise some Win32 APIs and then the injection will happen and the attacker will be able to run code in the context of a different process. So that's where code injection is actually a thing nowadays. So yeah, so we know what these techniques are. We know how to catch them, actually. But why is that difficult to hunt for this stuff at scale? So the problem comes on the tools and frameworks that we have to hunt for this stuff. So currently, I mean, if you want to hunt for this stuff, you only have two ways from my point of view. You either go and buy an expensive EDR product that supports these techniques. There are several EDR products out there that either does a member inspection, which is the same thing that I'm doing with this tool, or a memory scanner. And they do, you know, a pattern matching on the memory of the different processes to look for IOCs and see if there is an injection going on. So you either have to go and do that, or you get the volatility and you get a full memory dump of the machine that you want to analyze and take that full memory dump into the volatility framework and run the different scanners that volatility have in order to detect for this stuff on that dump. The problem with that approach is that it doesn't scale. I mean, if you had to do that, you have to go to the machine that you think might contain some of something living in memory. You have to dump, you have to get a full memory dump of that machine. You have to graph that memory dump into a forensic environment and then analyze the stuff in there. The problem with that is that you cannot do that scale. You cannot do that every day on your organization, on any machine in your organization. So mainly because of that, we created this tool called Mem Hunter. So what Mem Hunter will do, so I'm just giving a brief spoiler because we are seeing that in detail in a moment. But what Mem Hunter will do is that it will perform what is called a memory inspection on every process on the endpoint and it will apply the same type of heuristics and an extra couple of more that detects for this stuff in memory and it will report that back to you. So through that, you can run this periodically either as a service or on demand and get the results of your entire organization at scale. So OK, so we just talked about the why. So now we will talk a bit about how we can hand for this stuff in memory. So on the how part, we will mention a bit. Sorry, we will do a quick overview on what the process manipulation is, what the properties of a process are affected by a process manipulation and a code injection techniques. And then we will also describe, we will also give you a few details on how a process is created on the Windows environment. So talking about the process manipulation. So process manipulation for me is an umbrella term that can be used for any type of technique used to compromise a legitimate process and have it to execute the malicious code. So you can do that either by performing a code injection through memory allocation or memory writing. You can do that by getting and forcing the process to execute something. Or you can do that through a mechanism that are available in the Windows environment to force a DLL to be loaded during the process start. So going into the Win32 process internals, the thing that I wanted you to take from here, I mean, we can spend hours talking about this. But the most important things for this talk is the fact that the Windows kernel is the one in charge of allocating the, I mean, when a process is started, the Windows kernel is the one in charge of creating what is called a E-process object that will take care of the different areas that that process container will have. Among those areas, there will be memories areas designated for the different threads that will run within that process. So every thread, which are, by the way, the ones that will be running code, I mean, the process itself is not running in code, it's just a thread, the ones that are running the code. So there will be a call stack designated for every thread. There will be also an area within the process internal called PEV, which is a process environment block that will contain information on the common line used for the process, some information about the properties of the image file. Yeah, and a couple of other things that we'll mention in a second. Then there will be areas that will hold information regarding the dynamic libraries that the process will load. Within those dynamic libraries, I wanted to mention that every library loaded within the Windows process will have, will follow what is called a PE format, which is a format described by Microsoft to load stuff in memory. So that is going to be important for one of the things that will be discussed in a second. So just to wrap up a couple of things that I just mentioned. So the process properties, the thread properties and the memory properties are the things that define the normal or standard behavior of a process. So, okay, so why am I mentioning this and why I put in so emphasis on this? It's because of the fact that we will use that as a way to detect anomalies in either the process properties, the thread properties, or the memory properties to, and we will use that to hunt for stuff in memory. So I'm giving three examples here as a bit of a spoiler. We will discuss this in a bit more detail in a second, but the idea is that, for example, as I mentioned, every piece of code that you run on a process have to run in a thread. So you can, you will expect, I mean, in a normal scenario that that thread will have to run from code that is actually backed by a file on disk. So usually what is happening is that you have your process, you load your process and you create threads. Those threads have to run from code that they will be, that will be part of the main module, the main executable, or they will be part of a DLL within the, that was loaded by the process, right? So there might be scenarios where the process is not running, sorry, that the thread is not running from a code that is backed on disk. And those scenarios are mostly, not 100% of the cases, but are mostly caused by these side effects and the artifacts left by these code injection techniques. So whenever we found a thread whose start address is running from memory that is not backed by disk, we might say, okay, you know, this heuristic, because we call that heuristic, the heuristic might be telling us that there might be something suspicious on that thread that might require further attention. And that's what the tool will actually report. So the tool will basically flag those processes with anomalies for thread hunters to go and look in detail. Another potentially, another anomaly that we might end up using is the memory permissions for the base address of the threads. So the threads are supposed to be from memory regions that are marked as read execute, but there are code injections that left those memory regions marked as a read write execute. So that will cause one of our heuristics to fire, basically. Then in terms of memory properties, we can also look for, as I mentioned, memory regions with read write execute permissions. We can also look for all allocations. So whenever you allocate a piece of, allocate and commit a piece of virtual memory within the context of a process, there will be two metadata fields within that memory regions that will indicate the state of those, sorry, the protection, the state of that memory region, I mean, is that memory region is read write execute region or something else? So there are cases that techniques initially allocates the memory region with read write execute, and then called a Win32 API called virtual protect to move those memory regions back to read execute so it can look legit. But there is a flag within those memory regions to make that data that will still contain the initial allocation value. So by looking into the mismatch between those values is a way to determine if there have been a potential code injection in place. Then in terms of memory content, there are a couple of things that we can do here. I mean, one thing is just go and do, you know, the regular dam IOC match, which is effective but is very costly in terms of performance. So there are EDR vendors that, you know, have modules that are specialized on doing this. They are mostly running drivers that scan the memory of the process and looking for specific patterns and flag, you know, a threat event whenever something malicious is found. In our case, we were not able to do that. Again, mainly because of performance reason, my monitor is just a user space application. So we cannot afford, you know, just going and scanning the memory of the entire memory base of a process. So mainly because of that, what we are doing is something called the virtual query walk. So what basically we're doing is that we are walking the different memory regions of every process. And then in some cases, we are looking for, you know, just a few first bytes within those memory regions, looking for, you know, signs of a P file within that region, like looking for the MC header within those first few bytes. Or we look for, you know, associated strings left by attacker thread graph. But again, this is optimized just to look for a specific stuff and not for entire content of the memory region. Okay, so having said that, we will jump into two examples of both code injection and defensivation approaches. These examples are actually not quite new. They are being in the field for quite some time now. The first one is reflective DLL injection. So reflective DLL injection is a technique that got popularized a few, a couple of years back. It was created and released by a guy from Germany called Stephen Feuer. What this technique was to, I mean, the purpose of this technique was to allow mostly the people interested on this kind of approach to allow code in the context of a remote process just by using a DLL. So the idea is that you get the reflective DLL framework, you create the DLL, you include part of that framework within your code, compile your DLL, and then the result of that compilation is something that you can embed into your attacking process, or something that you can read from the file system if that is what you wanted to do. But the idea is that once you get that blow of data into the context of the attacking process, you will write that app into the victim process. Actually, sorry, you will allocate first, you know, read, write, execute a big chunk of memory in the context of a remote process, then you will write that down into the context of a remote process, and then you will spawn a thread, a new thread that will run out of that floating read, write, execute code. One of the biggest or the most well-known consumers of these techniques was Metesploy with this Meta interpreter payload. So initially, they used these to move the payload around processes using this technique to go in a pure file-less way. Nowadays, I think that they are using it as a kind of a fallback. I think that they start with a different technique, the Q-User APC one, and then use this as a fallback in case that the previous one fails. But yeah, but the important thing here is that the anomalies that these techniques will introduce within the context of the process is that after the techniques execute, you will run a new thread running from a floating code, right? So there will be a new thread that was not there before, and the base address of that thread will be a code memory region, mark as read, write, execute, which is not backed by a model on disk. Then if you look into the call stack of that thread, there will be stock frames that will, again, will be not backed by a symbol on one of the models on disk. And that basically means that, I mean, for this particular technique, that basically means that that thread was created out of shell code, that in the case of these techniques was used to bootstrap the malicious payload that you are using. So basically, the techniques simplifies that bootstrapping for you, so you are putting your regular DLL code, and the bootstrapping shell code will take care of mapping sections, resolving import, fixing up relocations, and a couple of extra things. But the idea is that after the execution happens, these three anomalies can be seen, I mean, we will be able to see these three anomalies in the process. Okay, so now jumping into a defense evasion example, the process Halloween one. So the idea here is that you want to evade, you know, the available endpoint protection stack, so you want to evade the AVs and the EDR, most likely on the endpoint. You want to, your malicious process to look as legit for these solutions, and yeah, and you want to use a legit process as a container for your malicious code. And this technique works by again, using legal Win32 APIs. So the idea is that you create a process in suspended state. Then you get the, I mean, when a process is created in suspended state, that process will not be assigned with any, you know, time, in the scaling time in the kernel, it will be just, you know, freeze, with just one single thread that will be also freeze, so nothing will happen. Then once you, the process is created in suspended state, you can get the execution context of that thread from the execution context and by also walking, you know, the memory of that process. You can, with those two inputs, you can determine which is the main module of that thread. And then you can call this API called NT and map view of section, which will end up unmapping, you know, the main module from the process. And then there are a couple of things that you can do at that point in time. There are more evasive ways of writing your malicious payload into that process. In this case, and just for simplification, I just mentioned that one of the possibilities that the attacker can do, can follow, is that they can call virtual.doc, again allocating a big chunk of read write execute memory. They can write, you know, the payload into that memory, and then they can redirect the execution context of the thread that is posed to this new memory region, and then they can resume the thread. So that will force the malicious, the host the process to run your code, but at the same time as no other thing has been modified, and the process just, we just force the victim process to run a different module. The process for the operating system and for EDR solutions, the process will look like, will look as legit. So the PV, yeah, most of the stuff on the process will not be touched. Yeah, so we will see that in a moment too. So in terms of anomalies, and in terms of ways to take this stuff, so most likely the best heuristic to take something like this will be to check for the P header fields of the main module on memory versus the P header fields of the counterpart on disk. So by checking that, you can detect if there is a mismatch, which should never happen, and if there is a mismatch, we can report that as something suspicious. So yeah. Okay, so we just talked about those two examples. We will just talk about the process properties and how we can detect stuff. We will now jump into the what part of the talk, which is around how to perform this at scale as part of our threat hunting process. And this is where we are introducing Mem Hunter. So this is a tool that we, I probably mentioned this already, but anyways, this is a tool that we started some time ago. And again, it was a way for us to explore on these techniques and ways to explore these capabilities. So Mem Hunter. Mem Hunter is a tool that deployed itself as a standalone binary. It will, once the tool is running, it will perform a series of inspection heuristics, which are the anomalies thing that I was just mentioning. So, and then whenever there is a finding, the tool will report that to the user. So the idea is that you have all these, this is the architecture of the tool, the idea is that you have all these different heuristics in the abstracted in this concept called Hunter Modules. So every Hunter is specialized of one specific heuristics. And then along with the Hunter, there is a part of a company in the tool in charge of doing the process context enumeration. There is a part of the tool in charge of doing the orchestration of the Hunters. And there is a part of the tool in charge of forgetting the forensic data. And something that I didn't quite mention is the fact that the tool can run either as a standalone service, which is mainly a way for the tool to get the Hunters to be driven by events. So we are using ETW to determine when the Hunters should be triggered. So for example, we are listening for kernel providers from ETW kernel providers to determine if there is a process got created, if a thread got created, if an image was load. So those kinds of things will draw, you know, some Hunters to run on the context of a process. So that's the real-time eventing way. And then you also have the point in time of standalone execution analysis, which is a thing that we are going to demo. So the idea here is that you get the tool, you get the binary of the tool, you run it with admin privilege and the tool will perform the process enumeration and it will perform, it will apply the heuristics over every process on the endpoint. Okay, so currently the tool support nine Hunter heuristics. So we will see that in the next slide. Along with the tool, I have also created as a way to exercise the different Hunter heuristics. I created this tool, separate tool called MInjector, which is basically a test tool, which have 15 code injection techniques. I'm probably going to include the new injection framework to add a couple of extra techniques. So there was a great talk yesterday on code injection in general, which they released a framework, which I can't remember the name, to be honest, sorry. But yeah, but anyways, that framework is great. I will start to use it. And then the tool will also generate some even logs. And it also supports some baseline, some exclusion, because as I mentioned, the tool will not give you data on which malware, which attacker tradecraft is running on the endpoint. It will just give you a hint that there is something suspicious on the endpoint. So you have to go and find yourself, you know what is going on. And that also means that the tool might end up reporting false positives. So for the false positive scenario, you have to have a way to exclude some processes. The processes that most likely are going to cause false positives are the process that use these techniques for legit purposes. Example of that is the .NET processes, the managed processes, which compile things on the fly. And the JIT compilers, there are also products from security vendors that use code injection as a way to hook into stuff, to monitor behavior and so on. So there might be some false positives. You have a exclusion way to baseline your environment. Okay, talking about the hunters. We have nine hunters, so we are running out of time for demo, so I will go really quick on this one. So the first one is a suspicious threat. It will look for the base star address and it will check for read write execution flag. Suspicious call stack, will look for symbols that are not backed. Suspicious memory regions, it will check for the first few bytes to look for signs of P models that was loaded or that currently being loaded and used. Suspicious base address is something that indicates that the main model is running from private commit and marked as read write execute memory regions. Suspicious pair is the thing that I just mentioned. Suspicious models, models associated with read write execute memory regions. Suspicious hollow models is a cross check between the P header field from the memory versus the counterpart on disk. Register persistent is just to look for stuff like this functionality from Windows to force a DLL to load into a remote process. And the last one, suspicious shellcode, is mainly to look for the well-known products from 86 and 864 shellcodes. So there are a good amount of shellcodes that start with the same set of opcodes. And this is something that I'm developing right now. So this is some evasions that have started to appear. I think it was last year, between last year and the year before the last one. So ways to evade the IDR solutions like a parent PID spoofing and command line spoofing. So the way to do that is to, again, to cross check information from the OS with the current things on the PAB. Okay, this is the eventing information and this will be later available for reference. The set of information, forensic information that the tool reports. And after that, we will jump into the demo. Okay, so Carlos here will help me with the demo. Thank you Carlos. Okay, so the idea, the idea, yeah, the idea is that we will look first at the, you know, the expected behavior of a process. So Carlos will open a notepad. Are you able to see? Yeah, so he will open notepad with a process hacker, which is a really convenient tool to look for, you know, the details and the property of a process. And within notepad, he will check for the running threads of the processes of that process. It will check for the call stack of those threads and he will also check for the memory regions currently located just to allow you to see that there are no memory regions with read write execute permissions. And after that, we will exercise the reflective DLL injection, the technique that we just mentioned. And we will come back again to the tool, to the process hacker tool. We will look for the anomalies and then we will run Mem Hunter to see the thing that reports. Okay, just a second guys. Okay, we'll switch computers. Okay, just a second. So we will use a Win10 RS4 machine for this testing. Okay, just one more second. There you go. Okay, so we have our environment over here. We will, as I mentioned, we will run notepad first. Then we will go and process hacker, we will check for the properties of notepad. We see that there are a bunch of threads running because of Windows. But the important thing is that there is this thread running out of the main executable image from notepad. If we open that thread, we can see the call stack. And the call stack, you see that all the symbols in the call stack are mapped to a file on disk, right? And then if we go and check the memory, we will see that there are, I mean, no memory region with read, write, execute. There are only read, either read, read, execute, or read, write. And there is also memory regions with copy and write, which is the thing that Microsoft used for his copy and write capabilities. Okay, so let's go ahead and execute an attack against notepad. So it's notepad is process six zero to zero. So we will, we will run this tool. So this is an M-injector tool, supports 15 techniques. I'm going to add a couple of more after DEF CON. So if we run the technique number five, which is reflective DLL injection, and then we ask the tool to run a payload that is part of the M-injector project called reflective payload. And then we target the PID of notepad, which is six zero, six zero to zero. And we run this, we'll get that this popup coming from notepad that indicates there is now code that was injected and this running out of notepad. And then if we go back to the project, so the process, sorry, and check on the threads area, we will see now that there is a new thread RTL use that I started, that supposed to start, that showing to start from RTL user thread start. And if we check on the call stack of that thread, we will see that there is now two symbols that are not backed by anything on this process hacker just showing the memory addresses of those symbols. And then on the memory area, we can check that there will be a read write, execute allocation that will contain the malicious payload, which is this private commit allocation. If we open the location, we will see the P header signs in here. Okay, so now if we run mem hunter, so let's go ahead and run mem hunter. So mem hunter, this is the output that you get from mem hunter, so we will apply a specific heuristic to mem hunter. The one that we're interested in is the suspicious thread. You don't have to apply any heuristic when you run this. In this case, we're just doing it for the sake of time. And then we will ask mem hunter to be verbose on his output. So mem hunter will run a bit, it will again enumerate the process, it will perform the virtual query walk on all the processes. And after that, it will report that there is something suspicious on nopad.dxc on this specific thread ID. And these are the details of that memory region. Okay, so that's demo number one. I don't know if that's the right word, I don't know. Okay, demo number two, again, is the defensivation example that we just mentioned. Demo number two, we are launching, let's close this nopad instance and we will use also nopad as a target, as a hosting process for our attack. So we are running technique number seven, which is process Halloween. We are pointing a M injector to use another payload, oh, sorry, another payload, which is in this case, is MSX payload, there you go. And then the target is going to be nopad again. Yeah, there you go. So the technique just executed, so we just hollow out the main memory of nopad. So if we check on nopad, the process looks pretty legit, looks as a sign process for Microsoft. It contains some threads, the thread contains a potentially valid coca-cola stack. I mean, there are no issues from a simple site. So we'll run mem hunter, but this time, we will use a different heuristic, a heuristic number four, that will perform the cross check between the P header field from the things on memory versus something on disk, on all the processes running on the endpoint, and there will be a mismatch on nopad.exe because of process Halloween. So mem hunter will flag that out. Okay, so that's it. Thank you very much for being here, for attending our call. There are some stickers about the tool on the tables. Yeah, thank you very much.