 Okay, good afternoon everybody. Uh, next up is Chris's trunk with what's the difference for the FRR INS for ICS. So, give it over to Chris. Thanks for coming. Can everybody hear me? Okay, great. So, hopefully you've been able to go see the ICS Village and that's what we're here representing. Uh, if you get a chance to, oh thanks, nice. Um, if you get a chance to go over there and play around, actually try to plug in and see things. Uh, because a lot of people don't get a chance to do that and a lot of the equipment is cost prohibitive for most researchers. So, go get a chance to see that. So, my name is Chris Isstrunk. I'm a professional engineer. Um, I can talk more about myself later. Um, this is a lot of information to cram into a talk. But I'm going to cover digital forensics and incident response like an overview. How many of you have ever done digital forensics before or incident response? Okay. How many have done it for control systems? Yeah, very few people. When I gave this talk at Black Hat last year, it was like three people raised their hand that had done for embedded systems. So, we're going to talk about DFIR for embedded systems and ICS, embedded devices, what to collect, what to analyze and do some. Since I'm an electrical engineer, I work for a power company. Um, what I want to do is cover some examples for substation RTUs and how to do digital forensics on those. And it covers a couple of operating systems there. So, what we want to do for incident response, we have Find Evil, you know, assess the situation, define objectives, collect evidence. This is from the book, uh, the incident response book. And this is the same, uh, framework that we want to do for any incident response. So, if it's for IT, uh, or anything, it's almost akin to doing root cause analysis for an engineering. And so, we, we do an overview where we're collecting files, operating system, volatile and nonvolatile things, network traffic, application data. Uh, we do an examination in digital forensics to do an analysis and reporting. And so, this is from the, you know, NIST standards and the incident response and computer forensics book. Those are some really great, uh, resources. Um, if you get a chance, go get those books or get that standard as a free standard, NIST SP 800-86. So, there's a lot of, uh, traditional on DFI are tools that are mature and red line and volatility. Anybody ever use those before? Um, I mean, uh, I think the, the, it's great that, uh, the SANS ICS team has done some challenges where they actually wanted you to use red line and volatility for an ICS type, uh, challenge. That's great. And there's websites like a virus total malware and a couple of others. We have cheat sheets and books. But what do we have for, you know, control systems? We don't have any of those kind of tools. So, what is the difference for ICS? Hopefully that's the pun of the day. So, for assess the situation, you know, when, where, how has ICS affected? Define objectives, return the control system to normal quickly and safely, collect evidence, you know. ICS devices have real-time operating systems. I won't read the rest, but here's the difference. You, you, you have a similar situation between IT, DFIR and control system, DFIR for assess the situation. But for define the objectives, its physical processes is not just network. It's not just a data center. It's, you have to deal with, you know, uh, oil moving in a pipeline or, you know, power moving through power lines or water through pipes, uh, manufacturing facilities where you have a chain of, you know, things happening. Those are the things that are going to be important when you do DFIR. In collecting evidence, there's a difference there. Uh, must be collected manually most of the time because we don't have any tools. Um, and a lot of times you'll have to use the tools that are in the engineering workstation like a software from a vendor or something like that. Uh, when we do analysis, same thing. There's no ICS specific DFIR tools. You can't use a red line or a volatility. You can if it's a Windows, uh, based HMI. You can do that, but that's not the focus of this talk. We want to focus on the embedded systems and the custom operating systems. Uh, communicates the same, develop a remediation plan, how when to regain control of the control system, uh, ICS devices have constraints. So you can't just go reimage a PLC like you can a laptop. Uh, sometimes you have to load firmware and the configuration and make sure you have, you know, a lot more things. So it's not just willingly go, just like we do in IT. Reimages the laptop, no big deal. So with control systems, uh, you have to really think about how we do the remediation plan, kick the bad guys out or, or if they're still attacking, uh, and find out what to do and do it safely. So let's talk about ICS anomalies. I know some of you in here know about anomalies on control systems. Uh, but we want to say an anomaly of some kind has occurred, increased network activity, strange behavior, some kind of a failure. Now we need to investigate it. Is it known bad? Is it unknown bad? Do we escalate this to as a security incident? Do we have to call in, uh, our security teams or notify EISAC or any, you know, ICS or whoever? And who do we call? Do we call engineers, admins, PR safety vendors, legal? So here's an example, um, as of an incident might be an anomaly. So someone made a mistake in the, uh, an example configuration. So instead of 2400 bod, they put 2600 bod. And so how many times if, if you were an engineer, you've been called at five o'clock on Friday because someone missed a configuration. And that's happened to me a lot. So then you have, you know, maybe squirrels that cause an incident. You may have some kind of unknown thing doing something weird that you've never seen before. So don't reboot it. Don't reboot it. A lot of times, unless it's a safety issue, um, or there's some other issue that you can do to fix it. But normally we want to preserve the, the forensics. And if you reserve, if we reboot the system, you lose the memory and a lot of other things. So have you tried turning off it again? Don't do that. ICS collection tools. So we don't have any ICS specific DFR tools for, especially for embedded devices, but we do have other ways to collect the data manually. We can use Teraterm or some terminal program to connect, you know, Telnet or serial or whatever. Uh, serial cable, right? Connect to the console. You can use the software like from GE or Switzer or whoever, Siemens to collect information about the PLCs, the configurations, all that stuff. So you have to use the tools that are designed to go with that system. So what, what can we collect? Uh, two types, uh, physical data and digital data. So whenever you have a response that you need to do something, you need to figure out exactly what device it is and where it is and what rack it's in or what panel or what platform, whatever it is. Uh, any identifying information, manufacturer serial number, part number name, because most control systems, you try to have a good asset inventory, but we all know that there may not be an up to date as a asset inventory. And you got to make sure that you collect that right there. And then what kind of connections does it have? What are the LEDs on the front and the back? They have diagnostic LEDs or maybe green or red or, you know, the RT's down. Well, what are the lights? Are they green? Well, yeah. Well, it's, it's good. There's something else that's wrong. Um, power consumption. Is it using more power than normal? If you're monitoring that, uh, evidence of tampering. So, uh, temperature running hot. That's an interesting one. Mark Fabro found a, uh, a HMI that was running hot by using a flier camera and it was hotter than the other HMI and they found out it had a Trojan on it and it was using more, uh, than normal, uh, CPU. So it made it run hotter. So that's how they detected it with, with a flier camera. Uh, digital data. You want to be able to get the running configuration, whatever user accounts that's in there, last known good configuration. So if it's in a repository or if you have a CD or a thumb drive, however they store it, you know, whatever, if it's on the engineering workstation, try to find the last known good configuration. What is the running firmware approved firmware? Uh, because we've seen, um, in the real world, uh, someone put in non approved firmware, CPU usage, percentage, memory usage, RAM storage, all that stuff, what running processes, just like you do with windows, you know, net stat and, and in all these things you can find out what running processes, what running ports are active, um, logs. If you have any, uh, you might be able to do, you might not have Syslog, but you can log in and look at the console logs. You may have to, you know, use terminal server with, uh, a serial port. And then if you have an advanced system, I may be running VX works, you can do a memory dump and I'll show you how we did that. So what to analyze, find evil or ways for evil to do evil. So we have a, uh, a time base. So things, you know, things you want to do fast and first and things that take more analysis. So, you know, first responders, uh, what do the user and event logs say? Um, sometimes, uh, an RT may only have 255 spots in its buffer before it rolls over. So you've got to go grab that quick. Uh, what, what is the configuration? Does it match the firmware? Is it the firmware approved? Um, same thing. A lot of these things that you've got to go send a technician out there to collect, uh, or an engineer or whoever is the configuration and logic correct for the process. So an engineer can look at the configuration and say, oh, that someone made a mistake. So that is not as fast as grabbing the configuration, but you've got to do a little analysis there. Art of communications, normal. So you're gonna have to look at a serial packet capture or an ethernet packet capture and that takes a little bit more time. And what takes even more slower? Uh, I mean, uh, is things where we're having to do getting a vendor, a digital forensics specialist and embedded systems analyst. So in analyze embedded operating system files, capture data, arrest capture data in transit. And you know, if in the case of like, indistroyer, crash override or any like Stuxnet, you know, this takes time. It takes a lot of time to analyze it and reproduce it, volatile memory and to look for code injection or potential root kits. So keep these things in mind when you have a DFR process for control systems. So let's do DFR and two substation RTs. One on the left is a D20 from GE. The one on the right is a Switzer, uh, ARTAC. And if you are working in the electrical sector, you've seen these. So time to read the manual. So that's the Apollo manual and you have to have good manuals. So let's look at the GED20 MX manual. Anybody here ever seen a G, a D20 before? Okay. I didn't bring it, but this is just an example. If you have different PLCs, you don't have to know this stuff, but just remember, go look in the manuals that you have when you get back home, because you may have to get back to work. When you, when you look at the manuals, you may have all the same information. So it's just going to show you. So, you know, we found out the specs. It has an embedded PowerQuick Pro. It tells you how much memory it has. What is running VxWorks. And then, and then, you know, I've got the tools there. I've got the software SG Config. I can use terminal. I can use WinSCP. And I can, uh, look at the, the manual that came with it. And then in chapter 11 in there, it actually shows how to troubleshoot serial communications, firmware version, mismatches, shell commands, and logs. So in this manual, it actually shows you how to do some digital forensics. So here's some examples that it has, uh, some shells, uh, that's belongs, uh, to G. And then there's the VxWorks shells. So let's look at some of the error logs. Whenever, uh, an RTU technician first logs in, they want to look at the error logs. So you see, you know, some diagnostic information. If there's any errors in the configuration, they're going to show up right here. Um, I don't know if you can see my laser pointer, but you can see the top part, it tells some information about, uh, the ethernet ports and radius is not configured. So there's some errors, but if there's other errors, they would show up in that screen. Now the user log, uh, is really nice. You can export it to a CSV file and it also shows every command. So if there's, you know, what they type and all that stuff. So that's very, very important for digital forensics. So the power of the three shells, there's a D20 shell and two VxWorks shells. And a lot of these shells, uh, you know, sometimes GE, the factory may want to use these commands when, with a technician on site. So just be careful when you're in these things. Other, uh, uh, like Ovation and other GE devices, they use VxWorks. So some of these same, uh, things apply. So anybody remember this movie? He doesn't know how to use the three C shells. So you gotta know how to, gotta know how to use the shells. So the main shell shows you all these different commands. So you can just type in help and it tells you in, in the D20. And then what you can do is look in the user manual. And if you type IFINF, it'll tell you what the ethernet port is. And if you look in the manual, it tells you what, what that command does. So read your manuals. And this is great for an attacker, too, because if you see, you can also do things like restart, inject memory. You can dump the memory. You can do a lot of things from this. So it's very juicy for an attacker's perspective. So let's do data collection. We can get the running configuration. It's a very common task. Last known good configuration. Like I said before, you can type in IMGE to get the running firmware. You can query the processes. You can query the performance and get the CPU usage. Get all these things that we would do in Windows forensics. You just have to type these in manually. You can do a serial analyzer. Remember what we had before, Wireshark? Have to look in serial analyzer. So we can look in the RTU and say, it can tell us, this is an example of DMP3 string. So to and from, and we get this and send it off to, for troubleshooting, if we're in the field, if there was a communication problem. Now, this is really great. VxWorks allows you to dump the memory. So you can do a system information dump and it tells you the start of where the memory is in the addresses. So you need to know that information when you get ready to dump the information. You type in D and you can see I put a D, a bunch of zeros and then FF at the end. So I just grabbed the first 255 bytes of the memory. But in the D20, you can only do it over serial. They turned off doing it over SSH. So there's a gig of RAM and it takes a while to dump all the memory. But we're going to show you some more commands in VxWorks. So the, because like the, the Marjorover uses VxWorks. A lot of these, it's the most popular embedded operating system outside of it, like Android. So VxWorks is everywhere and you can use the same commands. You can, you can in the D20, they run VxWorks, but anything that runs VxWorks, if they have it enabled in the firmware, you can run a P cap straight from the command line and which is really great for forensics. Okay. Now we're going to put on our evil hat for a minute and we're going to do a live memory code injection and memory dump on the D20. So we injected EM to, to do a memory edit and then we put in dead beef and you can kind of see it there. And then we can do D and dump the memory and we see dead beef when we dumped it. So there's no live demo here. We just took screenshots. So we show that if you are an attacker and you have console access to this RTU, you can inject memory just like you can do with Windows or Linux or whatever. Okay. Now we need tools to enable us to perform DFR for ICS and embedded devices. So we at FRI, we got, I got with my good friend on the Flare team, they reverse engineers and I said, can you write us a tool that will dump memory out of VxWorks and then do a differential analysis on that. So we've written it and I'm going to show you the links to it and show you some things about it. So we wrote VxWorks DFR tool. It has a bunch of, it's all in Python and it's up on our GitHub. So there's the address ICS mem collect on the FRI GitHub. What it can do is it will read and write to memory of the device. It can cache the live memory locally and it can compare the system image. So from GE, you get a firmware set when you download it. And then what we do is we compare the memory that we dump against with the firmware image and then it does that comparison. It can work over serial TCP IP and it supports caching. So if you're say on a D20, you have to do it over serial and it takes a long time. So we had some problems dumping it and it takes like six days. Because as fast as you can go, it's 115, 200 baud. If you could do it over SSH, it'd be a lot faster to dump the memory. So we allowed to resume if connectivity is lost and it works on anything that looks like a Python file object. So cache files, memory dumps, sparse memory maps and other special objects. So here's an example. Josh Triplett explains this a little bit better than I can. If you go back and watch the Black Hat presentation from last year, you can hear his explanation. But here's a, it says Python validate image, disk image, Vx works, and it looks at the cache and you can see a section of the memory says IP firewall start. So in Vx works, that is where it starts the firewall. So what we did is we injected something to make the firewall not start. So we turned the firewall off in memory. And so here's some more about the, I'll show you, we're going to show you the, the, some of the outputs from the tool. But it uses clea loads everything. It works with the architecture of that device, which is really cool. I don't know how all that works. But it, you, it basically pulls from these two things, clea loads everything in capstone. Basically, it assembles and disassembles all the things that go into the firmware for that instruction set for that processor. Now the plans for the future, we want to do better documentation, testing with other embedded devices using Vx works like a ovation controller or a GE digital relay, expand the tool to work on other devices and other embedded operating systems and refine the scripts in the easy to use modules. Maybe that you might can have a volatility plug in of some kind and allow for feedback. So if you are interested in using this, go to that fire I get hub and you can download it and you'll have to compile it and do a lot of things. But if you have any questions about it, just send us an email and we'll help. Okay, let's go to the sweatshirt or tech. So this is, we wanted to cover a Linux embedded device. And this one doesn't give us any of the shells like the, the Vx works does. They protected the, the command line, but it runs on a power PC, has a giga RAM and embedded Linux. And, you know, it has the instruction manual that you can get and the software for it. And you can use a web browser and terminal for SSH. So let's do some collection here. We have to look at the digital data so we can get the running configuration, user accounts, running firmware. All of this stuff is on the screen when you log into it on the web version. And then physical data, there's a password jumper inside the relay in case you need to reset the device. So you can log in. And I've highlighted over, there's a button at the top if you highlight over it says if the password jumper is in or not. So it's just something to keep track of. If attacker was physically there, they could put that password jumper in and then get into your system. But then it's a physical security problem. So here's an example of the screens that we did from whenever we did an attack with DMP3. So here's an example of the screens that we did from whenever we did an attack with DMP3. So here's an example of the screens that we did from whenever we did an attack with DMP3. So back in 2003, 2013, Adam Crane and I were doing DMP3 fuzzing. And we had a Switzer 3530. It was vulnerable to DMP3. It basically didn't validate the incoming DMP3 messages. So this is a screenshot, just a little snippet of the web page. And when we send a DMP3 packet unsolicited, it made the CPU go to 100%, which is that's not normal. And then you can see the memory usage was very large. And then over time, the RAM crept up to 100%. And then what happened was is hardware watchdog rebooted. So you can see the middle screen there. This application has failed continue anyway. When it rebooted, the configuration was gone. So you can see that the configuration on the right side is scratched through and it's got the failed project error message. And you can see that the memory went back to nothing. And then the storage usage was zero kilobytes. So it was really strange that this message caused this thing to happen. But we could monitor that. If we were monitoring these things, we could have some really good digital forensics and instant response capabilities. So data collection, you can get several different reports and logs and especially syslog. So a good friend of mine, Nathan, he what he did was is he took syslog, but in the swatter device, you can make anything going to syslog. So he did all kinds of cool things to custom make syslog so if someone plugged in the ethernet cable, he could write that to syslog. If you unplug it, it writes it to syslog. If a port is active and then he tied in a door switch to the cabinet, so he tied that in to syslog. So all these really cool things that you can do with this device. It talks about all these different reports on the devices that's connected to it and you can access it, Microsoft Access, so you can run some database type things. There's no Linux shell, so that's sad, but there's pros and cons of that. That means that an attacker can't write memory to it like they could with a D20 or any VX works device. We can also capture packets just like with a D20 could and you can, it ties right into Warshark, so it's really great and this is an example message on the right and it can do, they actually wrote serial Warshark dissectors for self-ass messaging in telegear 89, 79. Those are two serial protocols that they wrote Warshark parsers for, which is really great because we can tunnel those over TCP IP now to make things work as we upgrade our systems to newer things. So we're getting close to the end of the talk for further reading about digital forensics and VX works and other embedded systems. HD more had a blog post about VX works seven years ago and they wrote a metasploit module for VX works remote memory dump so it does the same thing that our tool does, but it's you can look it up, WD BRPC memory dump is the module name. David Odell wrote a blog post about QNICS and that's a really good blog post to check out and then ICS-RT has recommended practices for ICS forensics. If you need to take a picture of that, we'll go to the next one. Travis Goodspeed, hopefully you guys know Travis. He wrote some very good information about MSP 430, so he did some digital forensics on that software family and the hardware family there with that specific chip. Ralph Lagner did a really great paper, especially in forensics data about Stuxnet, so if you read his paper to kill a centrifuge it's really great. And then if you read sans duck 5 about the Ukrainian power grid attack included parts of writing firmware of the embedded serial devices so just more things that's embedded. That's what we've got to get to. Does anybody have any questions? Yeah, so I've presented this at Black Hat so you can download the signs from Black Hat. If you want these with the ICS Village logo that's fine, but the video from Black Hat is up as well. You can see Josh's explanation is a lot better about the IP firewall start and how he injected that and doing the Python memory differential analysis which is really great. Hopefully this will inspire people to write more tools for embedded systems and we need a community around that so please try to install some of these things and if you have that ability and maybe we can work on making more modules for this type of tool. Any other questions? Thank you so much for coming and come see the ICS Village. We appreciate it.