 Hey everyone, hopefully everybody's having fun at all the parties and the DEF CON general. So I'm going to give a talk today about doing memory forensics on cold drives. Has anyone here ever done that before? Okay, I see a couple of you, so don't give it away. Awesome, so let's get started. So a little bit about myself. My name is Lior, I do blue team research for DeMisto. Playbooks, you know, orchestration, all of that stuff. Before that I used to do some reverse engineering, vulnerability research, malware research, and I recently moved from Tel Aviv to the SF Bay area. So, you know, still very cool guys, lots of technical like-minded people, cool environment, different DC group number. Yeah, so I'm going to speak a little bit about memory forensics in general and hopefully convince you guys that although you have all these different tools that are either commercial open source to do forensics on the drives you get during cases, it's still worthwhile to pull out the memory forensics tools and run those on your drives. So what is your typical memory forensics process? First of all, you need to get the bit copy of memory from whatever system. That can end up being a lot of data, especially in servers where you got like 64 gig RAM or above. So I even did it once on like a 256 gig server and that was quite an experience. And then you need to somehow collect the file over to your analysis environment and that's very, you know, bandwidth consuming. That's why there's recently a trend of doing live forensics remotely directly on the endpoint and running your analysis tools on the endpoint you're investigating and that has its own limitations but in certain cases where you need to do a quick triage, it might be cost prohibitive to copy the whole thing over. In our case, we already have all the data on the drive and it's dead box essentially. So there's no issues around that, right? And then we need to somehow interpret those bits. So maybe you're some kind of a super human who can parse memory images into your head or with a hex editor or with like just a shirking analyzer or something but most of us need some kind of tool to parse those structures out and figure out the state of the machine. And then we need to interpret the results and for that you need to understand the OS, you're investigating all the different objects and there's lots of funny edge cases with like exited processes or processes that didn't exit cleanly, things like that. So there's experience that builds up over time. So why even do memory forensics? Like what's the actual motivation behind taking apart memory? Obviously there's a lot of undocumented structures. There's a lot of changes that get made to the OS. So it's very time and effort consuming to maintain all those tools and do that. So what are the motivations behind it? And in general, it's about creating, first of all, attacker costs and it's also about attacker awareness. So most attackers know that they can be detected on disk if they leave some kind of evidence that can make it to an investigation. Fewer attackers know that they can be detected in memory, purely in memory even if they don't hit the disk at all. And also it's harder to evade in memory. You need to take a lot more mechanisms into account. So by doing that, they're actually forcing the attackers to create more evasions, to spend more time building their tools, which is always good to do. There's also certain evasion methods such as unlinking processes that's been around forever. If you do a scan of memory and you look for unlinked processes, you can find that. Same for network sockets. You can also find objects that are no longer really there at the time of the capture. So for example, a socket that was created way back, even though it's not active, the object will still be in memory. So you can get that out and that can give you, for example, the start time for a certain malware. If the first thing it does is call back home, you can pull out that timestamp and then you get a pivot point for your investigation. Also, some malware will be packed on disk but it's much easier to dump it from memory. So certain mitigations that malware authors do to kind of evade scans will not be effective in memory. And obviously, certain hooks in API calls, et cetera, if you dump the entire structures from the OS, you can detect those, right? So the leading tool for parsing memory images is volatility. There's also Recall, which is an early fork of volatility and that's managed by the Google team. And there's advantages to both. It's always better to use more than one tool. It's like a general, as a general forensic practice, it's better to validate your findings. Volatility has a lot of very useful plugins that you can use to detect certain kinds of activity. And I'm going to go over a couple of them for now. So sometimes it's really as simple as using what you already know. There is no need to go hours deep into an investigation just to later find out that it's actually something that's already well-known. So you can see a process list here and can anyone spot what's wrong in here, by the way? Right, dusk-sk, a process there that's a known WannaCry process name and that's really all it takes to say, oh, okay, this might be WannaCry, better pull it out, pull the process out, analyze it, make sure whether it's really WannaCry or not, but you already found your first hit right there. So do the simple stuff first before you go into the deeper stuff, right? Another example is using the PScan plugin that will get you all the process objects from memory. This is another state, this is actually a memory snapshot of Stuxnet and can anyone spot what's wrong here? Have any of you analyzed Stuxnet before, kind of seen it? So there's more than one else's process and in a normal Windows machine that's not supposed to happen. So just by doing these simple checks, you can already detect a certain malware that's just trying to essentially hide in plain sight, fooling analysts who are just reading through these lists and some of these checks have already been automated and there's volatility plugins to look for these things. So when it comes to looking for just repetitions, counting instances, use automation, don't waste your time, there's also when you do this all day, you can miss stuff. So whatever you know, just automated, volatility has Python, has JSON outputs, so you can actually use or you can include it in your Python scripts and you can just automate all these checks. So network connections, right? In this case actually hit the name of the VMM file. So you're looking at network connections for the PID 1484, so that's a process that's associated with those connections and we get the ports and the IP addresses, right? So that obvious, there's indicators online and documented malicious IP addresses that certain exploit kits, threat actors are using and these rotate fairly often, so these indicators don't last long, but my point here is use what's known. So that one on the top right is actually a known IP address for Cridex, so it was actually a black hole exploit kit that was serving Cridex. So again, you don't need to be a top expert to do these simple steps. Use what's known and then go into the most more complicated steps, right? And when we look up the process that's actually responsible for these connections, so let's say you don't get any hits on the IP addresses, try to figure out does it make sense for this process to be communicating over these ports or to be communicating at all? Maybe it's not supposed to create network connections. In this case it's explorer.exe, so it's definitely worth looking into, okay? Another thing you can do in memory is look for mutexes, so those are just used by malware to essentially flag the system, the fact that a certain piece of malware has already infected the system. What the malware authors do not want is to hit the same machine again and have two instances of the same malware competing against each other, breaking each other, right? So what they will do is they plant these mutexes which are legitimate Windows functionality, but once we know the name that the mutex uses and they have to stick to certain names, certain strains of the same malware, different strains might have different mutexes, but if you just have a list of those and there's plenty of people curating those online, if you see that, then very clearly it's a piece of malware, right? So the Avira mutex there, it's known for certain zoo strains, so you do the mutant scan, you run that plugin, you run it past your list of known bad mutexes, done, right? Super simple, no reason not to use it, right? Sock scan, we mentioned unlinked sockets before. It can show you sockets that are no longer there, it can show you sockets that are hidden or unlinked and the timestamps are super useful if you're doing forensic investigations on timelines and you need a pivot point to try and figure out, okay, where did this actually start? How long has this been there? This is a very useful artifact to have. So, okay, at this point you might say great, but I do not have the memory capture, so how do I do all this cool stuff that you just mentioned? If all I got in my case is a hard drive, right? All I got is a cold drive. How do I get this out? I was actually considering putting, I looked for cold disk online and I got the cover for the CD of the frozen soundtrack, so I was like, hmm, my lawsuit, not so much, okay, don't forget it, I'll just use the ICM drive, I like it anyway. So, if you've looked on your C drive, you probably have seen these guys, so you can actually use these to do all this cool stuff I just showed, even though you do not have the memory image, so even though your evidence recovery team or whoever gave you the machine, unplugged it, and kind of destroyed all that evidence, you still have some memory forensics artifacts that you can explore and why not use that, right? So obviously, the hibernation file will be appropriate to the last time the machine hibernated, that might be a while ago, but still in many cases, especially when the malware has been there for like eight months, that's still useful, okay? So the first thing to look at is actually the write and modify times for these files that can give you sort of a clue. You cannot access these while the machine is alive because it's locked by the OS, so you will have to either mount it from a second OS, like boot from Kali or SIFT, workstation, whatever it is, or if you're getting the drive cold and obvious write, you just mount it. So the first thing you wanna do, again, start with the simple stuff, strings, okay? Any kind of IOCs or strings that you know you're looking for, you can scan for those, right? Any kind of indicators, URLs, IP addresses, domains, start with those. If you're looking for a specific type of activity, like mining or black market activity or something related to certain companies, other companies, competitors, in cases of industrial espionage, you might wanna scan for those. That'll give you a good clue, right? And you haven't yet parsed anything, you do not need any expertise. So, and obviously, you can have a wipe list to filter out strings that just come up all the time, right? And there are a couple caveats to that. The hibernation file is compressed on disk, so you will need to convert it first. And you do need some context for all this data. In the case of the page file, it's essentially a bunch of pages that were moved to disk, right? There's no real context there. You're missing some tables from memory. If you have the hibernation file and a matching page file, you may be able to reconcile those. But the page file, essentially, you just need to iterate over all these pages and there are some tools to do that. I'm gonna show you right now. So, the next thing you can do is file carving, right? So, other than just simple strings, and you should use both ASCII Unicode, if you're dealing with non-English text, you might wanna make sure to test your tools, make sure that you're actually getting those other languages as well, so you don't miss important evidence. And then for file carving, we're essentially just looking for the headers for known file types, and we can find images, documents, encrypted volume sometimes. So, there's a lot to be found and there's known tools to do that. They have the databases of magic numbers who just look for the different file types and you'd be amazed what you can get from these tools when you just scan and get out all kinds of documents and stuff people thought they deleted, but not so much. So, the page file, where is it? Usually, it's at the C drive, pagefile.sys, just like I showed earlier, but really, windows can have up to 16 different page files and they can be anywhere. So, do not assume that it's on the C drive. Go, hit the registry, look for this key, then you can get exactly where the page files are, where, how many there are, et cetera. So, what's in a page file? I actually already said that it's a bunch of memory pages that were swapped to disk. So, there's no particular order, there's no real context tying those together, but there's a tool called pagebrute and I have the GitHub link on here. Essentially, that can run the YARA scan on each page separately. So, you can use your entire YARA arsenal to scan against that entire page file, look for hits. You can find fragments of malware, you can find IOCs, et cetera. The swap file, that's actually a less common one. It's newer, it's on Windows 8 and 10 and it's actually used by the newer apps that are on the Windows UI that are Metro style apps. It's a separate swap space for those. So, if you're looking into any kind of activity used by those apps, you kind of have it separately in a dedicated file. So, that really helps you weed out all the other stuff you don't need. In those cases, it's really worth looking at. In general, if you're not sure what you're looking for and it's not very focused, I would probably have a look at that, see maybe there's some useful data in there. So, crash dumps. You know, when you run some kind of software and it crashes and it creates those nice dump files, that's actually data from memory from that process. So, it really depends which process crashed. So, if it's a browser, you might have some data from the cache, you might have some information on pages that were open at the time. So, it can be very useful, especially if it's a miner that crashes the browser, you might have that miner still in memory, right? And if it's a password manager, I actually haven't done this research yet, but that could be a next step for any of you guys. If you want to try running a password manager, crashing it, looking at the memory dump, see if there's anything useful in there, because if that's the case, then it's definitely worrisome. Definitely want to make sure those are not in there, right? So, now we're going to move to talk about hibernation. So, the hibernation mechanism in Windows, everybody knows when you close kind of your laptop, you can either suspend it, kind of keep it in low memory, lower power mode, or you can entirely take the memory, move it to disk, save it as a hibernation file, and then it does not consume any power, right? That's essentially turned off. So, Windows will actually mark the page file, if it becomes corrupt, it will change the magic number at the beginning. Yeah, so when you get a page hibernation file, you can take a look at that magic number and see the current state of that hibernation file. When you get the hibernation file, you want to convert it to a format that's usable by volatility, and then you can actually use all those cool plugins that I showed you earlier on that, on that resulting raw image file, right? So, the image copy plugin for volatility, you run that once, you get a raw memory image out of that, and then you can use that from now on. You can also operate directly on the hibernation file, but it will need to do decompression every time, that's kind of a waste, so just do that once, work on the raw memory file from there on. There's also a tool from Kame, from Matt Swish, Hybrid2Ben, so he's one of the first people to do research on this whole hibernation file forensics and analysis, and he wrote some very useful tools around that, so you might want to check out his website as well, if you're interested. So, some caveats. In certain cases, the hibernation file will be zeroed by Windows, especially in later versions. Volatility has some brute forcing kind of handlers to work around that. If you want to know more about that part, I recommend The Art of Memory Forensics, it's a very cool book, very comprehensive if you want to get more into that, and also getting the profile, so volatility does need to know the OS it's analyzing, so it needs to know the profile of the OS. So, if you have an offline drive, and you do not have enough information around the case, you might need to go into the registry and pull that out, so you know which profile to use, and then you can go from there. Another caveat about the hibernation file is that before the Windows system hibernates, it will actually close all the network connections. So, you can still scan for resident sockets in memory of connections that were closed, but you won't see any connections still open. So, just keep that in mind when you're analyzing hibernation files, and that's why it's also important to know when you get a raw image of memory, you should ask where did this come from? Is it from like a hibernation file? Is it directly from a box? You might be able to figure that out yourself, but it can just save you time, so it depends on the context. And obviously, if malware is hooking that functionality, it can actually realize that oh no, the system's going into hibernation, I might want to shut down and clean everything up before that happens, right? So, in those cases, you might not see malware inside of the hibernation file. So, that's not easy to do, but it's technically possible because the malware controls the OS, it can see that as well. So, that's important to keep in mind. So, the last thing I wanted to mention is the volume shadow copies. So, have any of you ever analyzed the piece of malware and seen VSS admin in there as a string? So, in recent years, that's becoming increasingly popular because of ransomware, trying to clear those volume shadow copies so that you can recover your files. But in this case, I'm actually mentioning another useful part of volume shadow copies that's useful for defenders. The volume shadow copies will include previous versions of those files. So, even if some crash dumps were deleted, if it's a server, volume shadow copies will be turned on by default. So, you will have multiple versions of the hibernation file to work with. You can actually diff those and you can find changes, you can find processes that have been opened the whole time, right? You can do a timeline of that, right? So, you can take all of those different hibernation files, do a diff on your findings, throw all of those into plazo, log to timeline. You can have a huge amount of data to work with. So, just keep in mind those volume shadow copies. You can find very useful stuff on there. Always remember to look for those. And when you do that offline, SIFT Workstation has a very useful tool to do that. You can just mount any of those volume shadow copies, get the data out, and you're pretty much done. Okay, so, I actually ran that a lot faster, through that a lot faster than I thought. Are there any questions, any follow-ups? You wanna share, have you done any of this before yourself? I see a question back there. Sorry? Yes, it does. Yes, awesome. Yeah, go ahead. Absolutely, yeah, you can definitely do that. You can turn on your power cfg slash h, turn it on, and then you do a shutdown. And yeah, absolutely, so that's a very good idea. If you have access to the machine to run commands on it, you can do that, drop the hibernation file, and get it later, just grab the disk and go, right? Thanks for that. Okay, great. So, thanks everyone for coming, and have a great rest of your nap time. Thank you.