 Hello, I am Didier Stevens and I would like you to make you welcome to my network device forensics presentation, so let's start with the slides. So in this presentation here, I'm going to cover network device forensics, so network devices like switches and routers, how we can do forensics with these devices. So I'm not talking about network forensics where you capture traffic on a network, on a device, on a host, and then you try to analyze that capture file for forensic artifacts. So that's not what I'm going to do here, what I'm going to present here, the thing that I want to present here is forensics specific for network devices like switches and routers. Now this presentation comes with an article. I've written an article for the ISSA journal about network device forensics, and this article was the feature article of the December 2012 issue of the ISSA journal. So that means that that article is public to everyone, so you can download it and read it. I will give you the links later. Now imagine that you are in a situation, you are at your company, and that there will be a former employee that comes to the premises and connects one of his laptops, a rogue laptop, for you it's a rogue laptop, he connects that laptop to your network. Will you detect this? And if you don't detect it, but if you get warned about this later on, for example after a couple of days, you get some warning that tells you that this happened, for example somebody told you, will you be able to gather enough forensic evidence to see what happens to be confident to build up a good timeline of what happened? Another example I want to cover here is say that you have an edge router. So that's a router that sits between your network and the internet. And one of your competitors trojanized that edge router so that they can take over that router and inspect the traffic and have access to your network. Again here, will you be able to detect this or not? If you don't detect it, but later on you get warned about the situation, will you be able to analyze, perform a forensic analysis of this situation? So here in the talk I'm going to focus on forensic evidence that we gather with network devices and second part forensic artifacts that we find in network devices. So first of all forensic evidence gathered by network devices so your network traffic flows through your network devices so they leave pieces of information on those machines and if you can recover that information, you can have forensic artifacts to help you with your case. One of the very important aspects here is logging. If you have a professional network device, it will be capable of logging. So check if logging is enabled, if it isn't do that, enable logging. And also check if you have levels of logging and maybe you need to change the amount of logging that you get more logging information than you get by default. It's also very important not to keep those logs on the network devices themselves but to have them in a centralized place. This way it becomes much harder for criminals to re-erase their traces by removing their entries from the logs. You can do that for example with a syslog where every network device sends all its events to a syslog server. Let me show an example of information that we can use with a switch port state. So our first example, the former employee who comes in with a rogue laptop, he connects that to an RG45 cable, to a network cable. Will you be able to detect that? Well, the switch port state is one way to do this. On a Cisco switch, if a network device is connected to the switch port, so the cable is plugged into the laptop, when that happens, you will have a couple of events telling you, for example, that the state of the logical and the physical port changed from down to up and state has changed to up and it identifies which interface that is. I also want to get you on board by enabling onboard security features. So a lot of professional network devices comes with a lot of security features. And some of those security features can help you with your forensic examination. And they don't need to be installed in a blocking mode, they can just be there in the monitoring mode, but the fact that they are monitoring will help you much more with recovering artifacts. So enable onboard security features. One example here, again with Cisco iOS is DHCP snooping. So DHCP snooping does the following, on switch ports, the switch will listen in on DHCP traffic that comes in over that switch port. So that's a say, DHCP requests that come from the clients connected to that port and DHCP replies coming from the DHCP server that is also connected to the switch. So in that case here, in that configuration, you will identify one port as a trusted port and that means that this is the port where your corporate DHCP server is connected and from which it is serving client answers, so client replies. The other switches, the other switch ports are considered untrusted and that is there, every client can connect. So if you do that configuration, you enable DHCP snooping and you define your trusted port, then iOS will build up for you a list of all devices that they have connected to that switch. So you will have a MAC address, an IP address, VLAN eventually, and an interface number. This is an example here. I issued a command show IP DHCP snoop bindings and I get this input. I have the MAC address, IP address, the VLAN and the interface. So in that case of that former employee, if you would have DHCP snooping enabled on a switch to which he connected, then later on you can connect to that switch and look at all the connections that have been made and maybe identify a MAC address from an OUI, so the three first bytes, the manufacturer, the OUI, that is not one of the manufacturers you use in your company. Say that here it says all the time HP because you use HP and then, for example, you see a Dell. Then there's a strong indication that that is the MAC address of the rogue laptop. You will find its IP address and very important the interface to which it was connected. And then you can use other physical surveillance methods to find out who that rogue employee was and where he was acting. For example, you can have access control logs, physical access control logs with badges to see who came into that room with that network connection, maybe also CCTV images. And of course, don't forget to ask around people working in that room or working in neighbor rooms to see if they saw something unusual. So this was about forensic evidence that we can find with the help of network devices because they log in formation, they log in because we require them to do and we ask them to do, but also in the case of the HCP snooping, they do that logging, they build that table so that they can be able to do their work, like, for example, as a first instance, starting to implement a system like NAC, but then there is also the case where you will find real forensic artifacts on the network devices themselves. And the thing I'm talking about, for example, is a compromised configuration. So for example, you have a rotor with a routing table that is the configuration and the routing table is part of the configuration and that routing table has been compromised. It has been changed so that some of the information is sent off to another route. And this is something you will find in your configuration. But for this to work, you need to have a system that allows you to verify that a configuration has been compromised so that you can detect the changes. So the best thing to have is a system of release management and version control for your network device, configurations and firmware. Because if you have no ID, if you have no baseline, then it will be very hard to detect changes to a configuration. While if you have release management, you will have documentation, but when were configurations released and if you have a version control, you can also see how these versions changed in between versions. Another thing that you can have as an artifact on a network device is a script or could also be a compromised script. So it can be a rogue script, a script that you didn't put there, but that the attacker put in there. Or it can be a compromised script, as you say, one of your scripts that was modified to make the router behave differently. And that is also something for which you require release management and version control for force of scripts here, and that you have to do that of scripts, not of your configuration files but of your scripts, so that you can easily detect after the facts what changes were made to the script. Compromised OS. So that is the network operating system, so iOS in our six core examples that has been compromised. So that means that it has been patched, changes have been made to the operating system so that it will behave differently. This can be done in two major ways. Attackers can change the OS image, so that is the file, before it is loaded and executed into the router, or they can change the operating system directly in RAM. So they don't touch the operating system on the file system, but they patch a running operating system directly in RAM. This can, for example, be done if you find exploit, if you find a certain exploit for iOS that gives you full right privileges to the RAM, then you can modify the running operating system so that it behaves differently. If you are dealing with a compromised operating system in memory, you will have to perform a memory analysis. That's the only way to go about, to detect these things. Luckily Cisco iOS has a feature to dump the memory of routers, it's administrative command, and when the memory is written, it is sent to a TFTP server which captures the files of the memory dump. It's also important to understand that such a memory dump does not really have a negative impact on the operation of the router itself on the performance unless that router was working at its maximum of capacity. Because the thing is that while the memory dump is ongoing, the router will still continue to work. And that, of course, is also a negative impact on your memory dump because that means that you are taking a memory dump and when you do that, it takes a couple of minutes. But you are actually taking a kind of rolling memory dump because when you are making a dump of the first part of the memory, there will be changes going on in the last part. And then when you end up dumping the last part, then there will be changes in the first part and those are already lost because you've already written that. So it's not a flash snapshot, an instant snapshot, it's a rolling memory dump, it takes time and while it is performing the dump, the data is written to disk but the router is still functional and so it is also changing the memory while it is being dumped. Now, until recently, you only had one option to analyze a memory dump and that was to send it to Cisco's technical assistant sensor, the TAC, the tag. So they would ask you to write a memory dump and send that to them and then they would analyze that and you would get answers to your problems. Several years ago, Felix Lindner released a tool called CIR and that tool was designed to analyze memory dumps from Cisco devices. So the thing that happened is that this tool was a closed source tool, it was written in .NET, it was a closed source tool and you could buy it from Felix company, Recuritlap, or you could upload your memory dumps to a website where this tool was running and then it would send you the analysis but of course that implies that you share your memory dump with somebody else outside of your organization and that is not always something you want to do and that is one of the reasons why I started to develop a similar tool but an open source tool which I call NAFT, NAFT, the network appliance forensic toolkit, it is written in pure Python and it can handle for the moment Cisco iOS and also generic network devices. Now the interesting thing to know is that when I was working on NAFT and tweeting about it, I got in a conversation with Felix and in the end he decided to release his CIR tool as an open source tool. So nowadays situation has changed, you have two open source tools, CIR and NAFT to perform memory dump analysis of Cisco iOS devices, CIR is in .NET C sharp if I am not mistaken and NAFT is in pure Python. Okay so now let's go into the demo of my NAFT tool, you can download it from here, I have the page for my NAFT toolkit on my blog so let's take the terminal with the command line and here I have my NAFT, so the network appliance forensic toolkit, my NAFT Python files, the NAFT dash here, NAFT minus, those are actually commands while NAFT underscore all those here NAFT underscore those are supporting modules so you don't use them directly, once you use directly here NAFT, GFE, ICD, II and IIE, so we will see what these do. I also have a couple of memory dumps in this folder here and a couple of iOS images in this folder here. So let's start with NAFT for the generic, so NAFT GFE is generic frame extraction, so this tool NAFT GFE works on the memory dump or even files of any network appliance, it will look for IPv4 frames in that file, it will look for headers that resemble an IPv4 header and then it will calculate a checksum and if the checksum matches with the checksum that is found in memory then we have found a valid IP packet and we will put it in a pcap file. Let's have a look here, the command, I'm going to store the packets that I find in the memory dump in result.pcap and inside the dumps I'm going for the core dump of my R, my router 877, the IOMEM dump, okay and as you can see here we identify 213 frames, 1 packet and there are 194 frames inside the pcap file. The reason why this is lower is that there is duplication, when we do this we only write the unique packets to the pcap file, if two packets are identical we will only write one of them. The final here, do my computer, NAFT demo, okay so here the resulting pcap file, if I open this with Wireshark you can see here that we have all those packets extracted from the memory dump, okay. Now NAFT GFE can actually extract packets from any file, it doesn't have to be a memory dump, it could also be for example a memory dump of an XP Windows machine. Let's do that, result XP.pcap and in the dumps I want the XP laptop here. So now in this dump, this is actually a dump that you can find online. Search for memory forensics, Windows XP memory dumps and you will find some of them and then you can use that as input to my NAFT tool here. So this here is a rather large dump, if I'm not mistaken 512 megabytes so it takes some time to search through that whole file. So it is done searching for IPv4, then search for RP packets and that was quick and now we have a full result. With the total of 2054 packets inside the pcap file, so that's not bad at all. Let us open that file, okay and now look for example let's filter for DNS, yeah and we see DNS to USGS Gov, that's a geological survey I don't know if I'm not mistaken, Benz Bargain.net, CNET Shopper ESPNGo.com, okay and HTTP, yeah we can see different HTTP requests here. Now here an XML file, another RSS XML, yeah, okay let's quit and then if we look inside the dump, I have this one here 500 and here I have also one from OSX which is 2 gigabytes so we can try this one, see what happens, so result OS6.pcap dumps Mac Mountain Lion. So now again the program is searching for IPv4 packets in that file, it is so looking for headers, so bytes like 45, 46, 47 and that can be the start of an IPv4 packet. Then it looks for the header, the checksum in the header and it calculates the checksum and compares it with the search checksum, if those two are equal, we assume that we have found a valid IPv4 packet in memory and we extract it. And after searching for IPv4 packets, we will be looking for R packets, those are also easy to identify. I will also work on IPv6 but that will become a bit more difficult because there is no checksum there, okay and here we have our results, well this is much less than 184, okay. So that is a demo of Naft GFE for generic frame extraction, it can extract frames and put it inside a pick up file from virtually any file, any memory dump. It doesn't know about the layout of the memory dump so it looks everywhere. Well if we are working with Cisco iOS memory dumps then we know more about the structure and for that we have another Naft command and that is iOS ICD. This program here, see iOS core dumps, ICD, this program here has a lot of options. You in fact provide it with a command and then with extra options, for example ask for the regions, for the processes, for the heap, for the frames, CW strings, check text, check text that is an interesting command that I will show you, events and history. So let's start simple with regions, ICD, I want to have the regions of the dump of my 877 router, okay. And this is the information, so the memory that is found in that router has split up over these addresses, start and end addresses and the size for text, data, BSS and heap. You can have exactly the same result when you do the corresponding iOS command on the router itself and then it will also show you the regions. But here we extract the regions from the memory core dump. Another very interesting command for Cisco iOS memory dumps is CW strings because, well let me just run it first, we'll make it more clear to you, that's it. So every Cisco iOS image and every Cisco iOS memory dump contains CW underscore strings and these strings contain information about the running software, like for example here CW version, this is the version that was running inside that memory dump, family C870, the type of image here, advanced security services, okay. And this is also something you can do just with a simple strings command, if you would do strings on dump R87 and then grep, I lost my window here again, and then do a grep for I believe it that, but what I wanted to say is do a grep for CW underscore and you will have a similar result. But here it's a bit more efficient because the command knows in which portions of memory to look for those strings, okay. And now let's have a look at the heap, the heap of that dump, okay. And this is all the heap functions that you can find that were running on that machine when the core dump was made. Heaps are pieces of memory and heap manager allocates pieces of memory for the different courses to work with. They have purposes and names, but this here doesn't show up in this simple dump. You can do the same dump again on a live Cisco machine with the CLI and then you will see the heaps and the names and their use. Now let's just issue the same command, but with another option minus R and then we resolve the names. You see that here now, instead of numbers, we have tried to replace the names of the processes like here SSH memory. So this piece of the heap, this memory chunk is used as SSH memory. Here another one for the HTTP process, IRP input. So you can find a lot of information here, but you are not limited just to view the location and size of the different regions, pieces in the heap. We can also filter them. So let's go back to the same command and now I will filter for TTY data. So now I will only see the pieces of the heap that have been assigned to a TTY data process and there they are. See that's much less, but in here, in those pieces of memory, you will find the data that we typed into the TTY that we saw. Now to view that data, we issue again the same command, but with another option S so that we can see the strings inside. And now here I did perform a dump of the strings that I found in the TTY data memory. This one is a very interesting one. This is the password that I used to log on to the machine. So the password is in clear text in the memory, which shouldn't be a surprise with a monolithic operating system like iOS. Here this is part of the command that I issued to write the core, write core and the name of the TFTP server. We can also look for data and grip the strings. So NAFTICD, I'm going to look, so in the heap command, I'm going to look for pieces of the memory manager that have been allocated to the init process. I will dump the strings that I find in those pieces of memory, in those blocks of memory, and then I will grab, minus j is to grab, four string cmd colon space, and that in the core dump. Okay. And you can see now here that you can see all the different commands that were executed by the consoles on that router and were still in memory. Now let's leave the heap and have a look at processes, iOS core dump processes, and these are all the processes running at the time that we took the snapshot. Other processes, IPsec processes, HTTP process here, SSL VPN process, all kinds of processes with the number, location and flags that indicate what they are up to. This is similar to the show process command you can also issue on a live Cisco router with the command line interface. The history is also very interesting enough, ICD history. Then I get a list of all the commands that were recently executed on that machine. Now this command here come from the config file. When router, Cisco iOS router launches, it executes the configuration file, so it is as if somebody was typing the configuration file to the console. So that's why it appears here. And then here later you can see that I do a dump, that I show a region, that I configure the terminal and go into line console. Those are all commands that I issued. So we have history processes. Now let's take a look at events, that's also very important. The events that we find in the dump, and here you can see all the events in the formatting as you are custom to them for Cisco. So you should be also having a syslog server where you find these events, but this is a way to extract the events that are still in memory. Now remember that you can have a compromised operating system. The operating system is compromised in RAM, so it has been modified in RAM, or it is the image that has been compromised. Now how can you figure it out if actually the RAM has been modified, so the operating system has been tampered with in RAM, but not on the flash disk? For that I also have a command, and that is the check text command. So I have an ICD, I issue the command check text. And this will compare the code that is in an image file with the code that can be found in memory. Code here is typically called text, so that's why I call this command check text. And I do this on a dump of this router, the 2610XM router, and I also provide it with the image for that router. So in images I will start with this one, this bin image, and I run the command. So now it compares the image of the disk and the image as it is written in memory to see if it finds any differences. And indeed there is a difference here, so that is not normal. If you have it provided the correct image, then this should be the same. Here you have the information of the image that is on the disk and the image that is found in the memory dump. This information comes from this CW string, the system description, and they are equivalent. They look the same and they contain the same information. There is some small formatting difference, but essentially they are equivalent. But if you look at the text from the image file and the text found in a memory dump, then there is a difference. And here we list where the difference starts and it's a small difference, it's only 48 bytes. While if you would do this with another image, with actually the original image, canary image, which was run, where they have an equivalent and text region and section are identical. So here there has been no tampering with it. And that is actually what I did here for this demo. So I have an iOS image that works with the standard canary values and canary values are values that are used in the blocks in the memory manager, in the heap, to detect when they are overwritten, but because they shouldn't be overwritten. And I have modified a version of iOS so that it uses different canary values, just as a proof of concept. And that is what I'm using here in this check text command. If I check the memory dump with the correct image file, the one which was booted, then you get an identical result and that there are no differences. But when I use a core dump of the Cisco router with a slightly modified image, only modified in total 48 bytes, then this will be detected and that is what was reported. So that is important to understand that if you have doubts on what is running, then you can make a difference with this here. We check the text tool to check if what you have in memory corresponds with what you have on disk. Now on caveat, of course, it is possible that the right dump memory, so right core, that command that creates the memory, that dump, that command has also been tampered with and that you cannot rely on it. And in that case, you have to break out of iOS, go to Homo and from there on do a memory dump. But that is much more tedious because you will have to do a memory dump as an ex-dump and then you will have to take that ex-dump and pass that ex-dump again into binary values. So that is not so easy. So that is what I can do with iOS core dumps and I have one other command that I want to show here, IiPy, that is for iOS images. So with this command you can analyze iOS images, let's go into images, this one bin, okay. And then you get a lot of information about that iOS image, the version, the family, the features and so on. The entry point, the number of sections, the embedded MD5 hash, which is here. These images are compressed. So the size decompressed, the compressed size and the uncompressed size, the original file name here. What I also provided as an option is that you can extract the image from that file, the ELF image, in a form that is suitable for the file to be analyzed with Ida Pro. So when you launch this command, okay. So you get again the same info. And now, look, this file here has been written to disk and it has been slightly modified. So from now on you can open that file in Ida Pro and have it disassembled and see what iOS does with all these functions in there, okay. So those are the three major functions here of my NAF tool. It is something that you will use to extract forensic evidence from memory dumps when you have to do it on your own, when you can for example not contact Cisco TAC to help you but when you have to do it on your own. And of course, like I said, it is important to have release management and version control to have a baseline so that you know what is supposed to be on the system and compare that to what is actually running on the system and from there on perform an analysis that tries to explain what new features and what new secrets are hiding in that tampered iOS image bead on flash disk or in RAM, okay.