 So I'm Jimmy Wiley. I'm a principal malware analyst at Drago's and primarily my job is to search for and analyze malware that targets critical infrastructure companies and equipment. So this could be lame shit like ransomware or it could be the dangerous stuff like Trisys or Crash Override and now Pipe Dream, which is what I'm going to talk to you about today. And mostly the sort of problems and the results from testing this toolkit in our lab. So first we're going to start with a little bit of background, talk about what Pipe Dream's capabilities are, talk about the hardware that we acquired, the lab setup, and then the fun stuff which is the actual results from testing in our lab and then talk a little bit about real-world usage of this toolkit. Real quick before I go forward, I just want to acknowledge all the people that helped me with this, be like disingenuous for me to represent all of this as my own work. There was like a whole team behind me helping with the analysis and the writing and just fielding my stupid questions about certain critical infrastructure things. And one more caveat, you can read that yourselves, but really the short summary here is I'm under an NDA and I can only say certain things about the malware. So in particular, I can't tell you where it came from. I can't show you screenshots of the malware, but I can do it, show you what happens when we run it. The important thing to take away from this, what I'm allowed to say is that we found the malware before it was used. So in the ICS community, that does provide us with sort of a unique opportunity to start building defenses for this kind of stuff. So what is pipe dream? It's the it's a seventh ICS impacting malware after the S-word and Havix and Black Energy, Crash Override, Trisys, Indistroyer 2, right. That would distinguish his pipe dream from all these other sample sets. It's the first cross industry scalable flexible framework, like attack framework for ICS equipment. So there's five components in total. We're only going to focus on the three on the left, which are Evil Scholar, Bad Omen, and Mousehole. All three of these are remotely interactive command line utilities for manipulating equipment. Evil Scholar is a Codesys version 3 attack tool with an initial targeting of Schneider electric controllers. Bad Omen is a similar utility command line, but it targets the Omron controllers, and you have Mousehole, which is a tool for interacting with OPC UA servers. So when we talk about our lab setup and some of the equipment, these are all the impacted devices of sort of the initial set, what we know right now. But the thing to take away from this slide is that column on the left. Codesys, Modbus, and OPC UA are everywhere. If you go to Codesys's website, there's like hundreds of vendors that use Codesys, and they have multiple types of equipment that they sell. Tons of equipment from all the vendors speak OPC UA, and Modbus is another extremely common protocol. So the breadth of impact of this malware set is more than like we've seen in the past 10 years. There we go. So our lab, what we did was we set up an NX1 P2 Omron controller with a couple servo drives and a servo motor, and then there's a requisite stuff that we need to actually connect it to the network, like EtherCAT couplers, and we have power supplies and all that kind of stuff. The Schneider Electric devices, we had a TM241 and a TM251 with two different versions of the firmware. We have an OPC UA server set up with our mock pipeline process, such a Kepler server with a bunch of Rockwell devices that are controlling a pipe and a pump, and then just some other hardware that we wanted to play with. So basically we just buy these devices off eBay or we ask a friend, Tom Van Norman over at ICS Village, like, ship me a 251 and a pinch because I couldn't find one with the right firmware. It turned out it still wasn't the right firmware. Ended up not being a problem. We tried to downgrade and it bricked the device. We could consider asking the vendor, but something to keep in mind with the industry is, you know, in the regular IT world there have been thousands and thousands and thousands of malware samples, and these kinds of things are a little bit more common, but in the ICS space, aside from the deluge of ransomware, we have seven cases of ICS malware and there's all kinds of other things that you have to worry about when talking to the vendor. They may not have the processes in place for even dealing with this. The NDAs can take some time. Personally as an analyst, I have reporting deadlines and things. I need to get things to our customers. So generally refer to just purchase the stuff where we can, stay independent, and then approach the vendors with the information once we have it. So when we go about analyzing this kind of thing, we had five pieces of malware. It was a big organizational task. We're basically focusing on a lot of the PLC malware first, statically analyzing it. So most of these toolkits are written in Python, a little bit of C++. Pretty easy to read. So we do that static analysis while we're acquiring the hardware. We're going to release initial details and mitigation advice, and we do just a shit ton of runtime collection, collecting PCAPs of everything, check which techniques work which don't, and then we're going to release updated details, mitigation advice, and then later on, all the follow on work that isn't as important for like defending the networks. And that's going to be things like actually stripping the firmware off the devices. We can't do that now while we're runtime testing it because we could break them. And like I said, sometimes they're not easy to find. So we're like basically in step five at this point. And that's been since, I think, April is when we started doing all of this. So the kinds of questions that we're asking ourselves before we get started with the lab testing, the main one, it's like, what's current? What do we have right now? Does it work? Right? Because we don't know yet. Right? And then there's the Codesys, the Codesys issue in Evil Scholar. Does that actually work? Is that something specific to Schneider Electric? Or does it work on any Codesys version three device? Bad Omen has a module for manipulating servo drives and motors. We wanted to see if that worked on only the specific motor that was targeted in the malware or if it worked on other drives in that series. And then more generally, and this is the most important thing, can these tools manipulate PLC logic? Meaning, can we actually manipulate this process specifically to create some kind of destructive effect? Then we think about future capabilities. Can we automate an attack with these tools? What do we think these tools are going to look like in six months? We know we have them before they were actually used. So sort of the implication there is that they're probably being actively developed. So these are going to change by the time they actually get deployed. And we want to hypothesize what's going to look like so that we know what to look out for in the future. And then there's the last category victimology which is squarely in the vein of threat intel. Can we actually narrow down a target? So let's get to the fun stuff. So the first thing we started with was Mousehole. Mousehole is a remote console for interacting with OPC UA servers. So you can sort of think about like OPC as like a translation mechanism. You speak to the OPC UA server, and then it translates those messages to whatever language the other devices speak in order to manipulate them. It's kind of the best way you can just think about it as sort of like an intermediary. So what this tool can do is it can scan for OPC servers in the network. It can enumerate the OPC UA structure and change specific attributes that then change attributes on those devices. So this can be things like opening and closing a valve, right? Or maybe perhaps changing the speed of something or rotation direction. I mean, it can be all kinds of things. It just depends on what the device is. In general, this is sort of a better implementation of the crash overrides attack. Crash override used the older OPC DA attack methodology. So OPC DA is the older OPC protocol that was based on DCOM. The OPC foundation was like that stupid. And they decided to make it cross platform and that turned into OPC UA. The structure of it is a little bit different, but it's basically what happened. So with Mousehole, it just worked. It's based on an open source library that was already well vetted, I think. So it was very easy to get it to work and manipulate the process. We had no problems on our pipeline or lab. So after we verified it worked, I spoke with the analyst that was working on this and we decided let's automate an attack to produce a more real world attack scenario. The hypothesis being that you would deploy these remotely interactive utilities to sort of recon the OPC server, figure out which parts you're going to manipulate, pull the important like sort of reconnaissance tool back, and then just like wrap up a very small version of it and just throw that onto the box to actually disrupt the process. So I want to show you a video here of our lab. And this is us actually testing the malware so you can see how exciting my job is. What's it going to play? There we go. So you'll see we have a water tank. There's a pump. There's a little valve. And on the bottom right is our HMI screen. This is us actually running the malware and creating an unsafe state in our pipeline. See that? It's super exciting, right? So it turns out, so it turns out if you ask your management if you can blow up a pipe in a water tank in your office they tell you they respond with a bunch of words like safety and liability and expenses. So what we're going to do is actually, the reason nothing actually went wrong there is that lab is specifically set up so that if we did like overdo it, it wouldn't break. So the pump that's attached to that lab is like a little baby pool pump and it can never like over pressurize that pipe in a way that would actually cause a problem. And that's good because that pipeline process sits right behind our CFO. So probably be a bad idea if we screwed it up. All right, so before we can actually talk about what an unsafe state is, I want to show you what a safe shutdown looks like. So we're talking about a pipeline. So we have a few important things here. There's a pump and how fast is pumping water. There's the valve position whether or not it's on or off. And then there's the actual PSI of the pipe. And in particular, the safe value for the PSI of that pipe. So there's a max PSI that this pipe is allowed to reach and any higher is considered unsafe. And when that unsafe value is exceeded, right, the process is going to take over and say, okay, we're going to turn off the pump and we're going to get everything back down so that you can restart that process and keep going. So this is what it looks like if we try to exceed that PSI, oh man, why is it, oh there we go. If you try to exceed the max PSI, we've slowed it down a bit because otherwise it goes really fast. So you'll see there we have the flow rate increasing. The max is about two gallons per minute, right? And the next thing you're going to see is that solenoid box, that's the green box at the bottom, is going to turn red and switch to closed. And then the pressure starts to increase, right? We're at 14 here. And you'll see that it's probably already been exceeded at 15. The flow rate is already decreasing. So it's going to go ahead and turn off the pump. So you're going to see that go down dramatically. That's at 80 to the negative, whatever stuff. And then the flow rate decreases and the pump switches off. And then what the operator is going to get is a reset alarm. So you're going to see a new box pop up in blue there and it's going to turn red and it turns blue one more time. And when it turns blue that last time it means you can hit the button, clear the alarm and then start the pump again, right? So the operator is going to get an alert and the process is basically going to take care of itself to make sure that everything's okay to restart. So how do we actually attack this with the automated utility? Well, run mousehole via the CLI to gather the important server and node information so we can understand what set points to manipulate, right? So we need to know what the max safe value is. That's called the high pressure set point. That was set to 15. We want to increase that to 100. We need to change the pump speed. The pump speed was set to 80 by default. We're going to increase that to 100. And then the last thing is closing the valve, which is the solenoid energized attribute. We're going to change that from true to false and that's going to end up closing the valve for us. So this is what it looks like on the HMI screen when you actually execute mousehole on the pipeline. So you'll see the flow speeds already at like max speed. And then solenoid valve is going to close. You're going to see the pressure go up. So it's already past it at this point. You'll see that the flow rate starts to decrease. And it's not decreasing because of pumps off. It's decreasing because the pipe is full and there's nowhere for the fluid to go. So basically what's happening here is the pump is trying to get water into the pipe but it's too high a pressure. It just can't fit it. It's like trying to stuff your full closet. So what happens when that occurs you're not able to throw pump fluid through that pipe and that's called dead heading. And this can be a problem in certain situations. The reason being is when you dead head a pipe what happens is you add heat to the fluid that's being pumped. So if you're pumping water this is okay because you're adding heat to water and all it'll do is like deteriorate seals and then it might get leaky and an engineer notices and they're like we should fix that. And not investigate any further. But if it's something combustible like mining slurry which happened in Queensland in 2017 the pump blows up. Which is obviously a problem. So this is a very simple attack I know but this is just an example of what can happen in just like a regular pipeline process. The other reason I have this example included is it's important for the rest of the talk where we're going to talk about PLC manipulation and control. The reason that this pump exploded in Queensland was that someone made a bad modification to the PLC logic that basically closed the valve and allowed the pump to pump against this closed valve. So we're going to move on to Evil Scholar which is the Code Assist version 3 utility. Command line the main things we're going to focus on here are the proxy module in Evil Scholar as well as the maintenance commands that allow me to transfer files and modify logic. So first thing to keep in mind here is that we're dealing with Code Assist. It's a device management protocol used by hundreds of vendors. Wide usage makes it a natural target and essentially it's a custom TCP implementation on top of UDP with an application layer. And basically the end of some of that is that Code Assist sucks and it's a pain in the ass. It's like pseudo routable. If you're using NAT you don't get packets back. And when you're trying to like use test a piece of malware and you're not sure how well it works, you're not sure if you're dealing with broken malware or if you're dealing with a broken network config. In particular in our case we had a lab where we could connect to the devices just fine. We had another lab where we couldn't connect to the devices at all. And this one by far took us the longest to sort of sort out. We thought it was a firmware issue and it just turned out that out of the three utilities evil scholars the dum-dum in the group. And in part because the developer made poor assumptions about Code Assist. It made assumptions about the network size, the address sizing, address sizing and Code Assist is variable length. So there's a lot of issues there. So the default version of Evil Scholar we could connect to the 241 and the 251 with no fixes if the devices are on the network of the right size. We spent about two to three weeks fixing the Python code that generates the Code Assist traffic and we were able to get it to work with regular Code Assist version three and then we were able to additionally connect to a Raspberry Pi running Code Assist as well as a Hitachi EHV plus. Provided there were any little endian architecture stuff because Code Assist also has fields that will like flip endianess depending on the target architecture. So we didn't go that far because it's not our job to develop malware or fix their shit. We're just trying to see exactly how far they had to go to make this work. So we were able to test all the plugins and so this is just a really quick example here where when we did think statically we made like a slight error and then after doing it dynamically we were like oh like this is actually what happens and in hindsight you're like well fuck of course that's what happens. So PLC proxy the idea here is that you have an internal PLC network behind an industrial firewall. The industrial firewall's job is to make sure that they configure the EWS, the engineering workstations, only talk to the PLCs for which they're responsible and no other ones. So if I want to talk to those other PLC the evil scholar tool wants to talk to all these other PLCs from wherever it happens to be. So what it does is it queries for the Code Assist gateway to grab its IP address and it selects the network that the malware happens to not be on and creates a route to that network through the target Schneider Electric PLC. So originally we thought it was actually like sending traffic to the gateway. It turns out it's just sending traffic to the network and just letting the TM251 route to wherever it wants. And that was kind of the basic idea. It ended up being kind of a minor error because all we discovered was that the proxy thing was actually just nastier than we originally thought because as you're runtime testing what we realized was that the 251 won't just proxy Modbus traffic. It'll proxy anything you give it. So you can SSH through it, HTTP commands, arbitrary other ICS protocols. It's just crazy to say the least. So yeah, I mean that's basically it. It was just it just got worse. Just not necessarily what you wanted the result to be. So we're going to move on to the logic corruption pieces of this where we at now. Okay. So Evil Scholar lets you transfer files to and from the target device. So you can pull the logic, you know, in general you can pull the logic, modify it, put it back on the controller, right? Super easy thing for us to do with this tool. And it should be trivial to modify the logic provided it's in a native assembly language and not an unknown bytecode, right? So some of these controllers have like just their own bytecodes that you have to figure out before you can do this kind of attack. In this case we were dealing with ARM. It was super easy. You see the red lights there mean the controller vector that error box there says that the controller is unable, sorry, the EWS is unable to connect to the controller, right? We were able to crash manipulate logic on the controller, create various error and output states. In both cases, comms to the controller are not possible unless you power cycle the device. If you connect to the PLC before the crash, Schneider's So Machine software will report that it knows there's something different on the PLC and only provide you with the option to override it and it won't let you pull the corrupted logic back so you can see what's going on. And this is a, this is something that's enforced by the EWS software. It's not enforced by the PLC and we know that because EWS scholar can pull the bad logic for us. So in that sense like EWS scholar like doubles as a forensics tool which is super weird. And not what I was expecting out of this sort of malware analysis journey. So the last little oddity here is that if you, essentially these controllers run an execution cycle so there's like a bunch of applications which are basically programs and it like calls each one and waits a minute for it to return, right? So on that first execution cycle if you crash the controller it just sets all the outputs to false. It doesn't set them to their like sort of default failsafe values which is what you would expect a crash to get set to because that default failsafe is like where the process should be when it's not doing anything and then you flip the right values to start everything. So it just sets everything to false which could have some unknown consequences for various processes. So that's a bit of a serious issue coming out of these controllers. So I'm not able to say too much more here because we're still working through the CV process with Schneider and CodeSys. We reported on June 22nd but isn't quite long enough a window for me to be able to show like screenshots of the code and what we manipulated and everything else. The good news is I can do that for BadOmen. So we're going to focus on the sort of logic corruption and manipulation with the BadOmen utility. So BadOmen is again another remote shell for manipulating Omron controllers and it primarily does this via an HTTP interface that's exported by the NX-NJ series controllers. The things we're going to focus on here in terms of its capabilities are the transferring of logic, transferring of files, as well as the servo module for manipulating servo drives. So real quick background on the console, it takes advantage of a hard-coded credential vulnerability so they can default login to this HTTP server that's sitting on the NX-1P2. The server has various CGI endpoints that's also used by SysMax Studio, the EWS software for configuring the controller. So in this case, it primarily uses the CPU.FCGI and the ECAT.FCGI. CPU for doing the standard downloading and transferring of files and logic and changing the controller state to run mode and program mode. And the ECAT is primarily for just wrapping up EtherCAT traffic to the NX-1P2 to send on the EtherCAT daisy chain. So same process that we did with Evil Scholar except we're just using different modules now. We took the backup module to grab the entire project file because the NX-1P2 just has a backup command. So we just ran that. We unpacked it, we took a look. It turns out it's running QNX neutrino. So the logic compiles the shared objects that gets mapped dynamically in the run time. So the entry point was super easy to find. We just had it branched to a bad address. And then we replaced the files on the controller and that resulted in another major fault. This major fault was similar to the Evil Scholar one in that we weren't able to download the, well upload the logic back I guess. We weren't able to retrieve the bad logic. It prevents it. It says there's no logic to retrieve once it crashes. I imagine this error happened multiple times even when we do some of the manipulation stuff and there's some kind of weird versioning where I think the EWS is saying you haven't done a transfer so there's nothing to pull back. Everything's still the same. So why are you asking? The other thing is it prevents you from enabling program mode and without being able to enable program mode you can't recover regularly. So we tried to restore from an SD card and that failed. But for whatever reason that allows you to set program mode again. Then a factory reset and then you restore logic to the controller. Which it sounds like a medium pain in the ass you listening to me but if you imagine like a plant engineer and this stuff happening like that's just the rest of his day. And probably a call to the vendor and then wait time and everything else. So just this simple attack would cause some downtime for a plant and depending on what it's actually processing like lots of fucking money. So moving on. The servo module testing. So real quick before we actually talk about what this is. What's a servo drive and what's a servo motor. So a servo motor is just a motor and it spins a shaft and it can be connected to any number of things. So some of the examples that I've heard about are servos control wicked gates of dams for controlling like water flow into a dam that can be in drone cameras. I think they actually help with creating base and subwoofers and things like that. Basically my general impression about servos from having analyzed this malware is that if you need something to move there's probably a servo involved. So the servo drive provides power to a motor and it also handles all the comms and back and forth to the NX1 P2 controller. The malware in particular targeted that 1S10 F ECT servo drive. But this was a 380 to 480 VAC drive. We don't have a reliable source of 480 VAC power and that's also like just super effing dangerous to just have sitting in an office. We didn't want to like electrocute the plant engineer that was so graciously helping me set all these things up. So we looked at the documentation and it turned out that there was 120 VAC model that we could hook up and 120 VAC is what regular residential and commercial real estate uses. So we hooked that up and we got the corresponding small motor and we were able to verify that the malware also worked on that smaller drive. So we did the rest of our testing on the small drive and also like just for general safety if we broke a bigger motor like bigger things could go wrong. So we stuck with like the safer option here in our lab. So this is just a quick diagram of the setup. You have the computer talking to the CPU unit which is the NX1 P2 and it's going to relay the EtherCAT commands down through the network to the actual servo drives and anything else that happens to be on that EtherCAT chain. So the actual servo module plug-in, it turns out you can't monitor the fun stuff with it. You can't just change motor velocity and like make things over spin. You can't change the direction, you can't change the direction of the motor live even though that was a parameter. So what the EtherCAT module, what the servo module does is it modifies particular parameters. Think about it as basic configuration. So there is a parameter for changing the direction of the motor, the way it spins, but you're not allowed to change that while the motor is actively running. Like everything has to be like turned off and in some weird configuration state. So there's a lot of you can do with this, but what you can do is kind of, you could be annoying, right, and it would still cause downtime. For example, there's an excessive speed detection parameter. So if you set that to one rotation per minute, right, and your process is running at 40 RPMs, the motor just stops. And then, you know, they might be able to like fix that and set it back, and then you can just go back there and set it again. So you can just keep needling them over and over again and just keep stopping all these servos, or stop a bunch of them at once. So just with that you could cause some issues, but generally like looking at this I sort of see it as a precursor to an attack where you actually screw with the motor, right? I'm gonna up the motor speed to max and I'm gonna disable all these warnings and the excessive speed detections so that you don't get like sort of a warning on the EWS that something is going wrong. And this worked out for us because we also wanted to test logic manipulation with bad omen on the NX1P2 and it would be a lot more fun if we could like overspin a servo motor, or servo servo motor, then just like setting outputs to false, right? Like that's I mean it's fun to like kind of see but like physical feedback when you're doing this is just a lot more interesting. So we know from crashing the logic that we can modify code on the device and send it back. And we also knew that the servo, the logic for the motor just sits on the same NX1P2. So we should be able to pull that and modify it. So the first thing we had to do was actually configure the servo to run and do things, right? So that's a lateral logic that's what the visual programming language that engineers use. The main thing to keep in mind is that last that last box at the bottom is a function called MC move relative and it takes four four arguments distance, velocity, acceleration and deceleration. The important one being velocity because that's how fast the motor is going to spin. So we created this program. We sent it to the controller and then we pulled the back up just like before and we decompiled the program. So the interesting thing here is again it was a shared object. You have all these symbols and like function names and there was one called POU run function block. So if you just like cross reference that like basically a function represents that particular lateral logic and it's like in order like the first call is a call to the first function and the next one's a call to the timer that I set up. So the last thing I call is move relative. You just kind of scrolled to the bottom like you're dealing with UPX and you're like oh there it is right and if you look at the bottom left there you'll see a bunch of 64 bit numbers and like 132 bit number. Those are I triple E seven five four real values. Right and so you can convert those to figure out and it turns out that the V 18 maps to the 40 degrees per second that I had set the motor velocity to. So I know degrees per second sounds weird. You don't set it in RPM so you have to do these conversions and there's some other ways to set the the velocity in the manual they call it like units of display and you have to go look up in your EWS like exactly what the units are. So we have the speed set to 40 degrees per second I wanted to spin it maximum so you just do the conversion the motors max speed is 3000 RPMs you multiply that by six you get 18,000 convert it back to real patch the instruction its arm and then reupload the program and that's my next video so this is what happens so before we keep going something to keep in mind on the right you have the ladder logic as it's running green means energized I'm just going to rewind it just a little bit here is it playing so yeah on the right ladder logic green means energized that means the program is currently running at the bottom here I think you can make it out I can't really see the screen from y'all's angle but that's a motor velocity so 40 degrees per second is anywhere from like we've seen ranges from four to seven RPMs this number isn't RPMs because the EWS software hates you I don't know like instead of spinning and up here we have the actual mode in the IP address so I'll walk you through some of this now so it's spinning at seven and now it's going into program mode because bad omen is transferring the modified logic to the servo and now it's switching back into run mode see that switch now everything's energized but there's a timer block there so it takes about five seconds and then once it's energized it just starts over spinning we're not over spinning it's spinning at max speed so that's at 3000 RPMs and all that happens automatically it was literally like I think two key strokes or something to do that we also tested spinning the servo motor like over spinning it at 5000 and it turns out the motor will only spin at max it just I don't know if that's a runtime decision like a runtime check or if that's a like a mechanical limitation of the motor we also tried sending negative values and then we actually got a proper runtime system error which we had never seen before the nice thing is with bad omen we can actually access that runtime to like look at it and see like exactly what's what's emitting that error maybe we can patch the runtime system to ignore it so there's a lot of other things we could probably do here that would be pretty nasty but also you know we had slide deadlines for DEF CON so can only do so much let's keep going here so last thing here I want to talk about safety system corruption at bad omen which is just something is pretty personal to me because one of the first things I worked on at dragoes was a trices which and we're looking at bad omen what we noticed was that you can hook up an SL 3300 on that same ether cat network to monitor the servo or anything else on that network and so what we were wondering was well if the servo's logic sits on the NX1 P2 does the safety systems logic do it or does the NX1 P2 transfer it in such a way that we can manipulate it you know pre-transfer whatever so while we were setting up the safety system what we noticed that it was very clear that it was different than manipulating the regular logic so with the regular NX1 P2 and the servo logic you would put the thing in program mode I think take it offline put it in program mode make your manipulations put it online transfer everything over and then you hit run and then everything starts to execute for the safety system transfer there was this other weird convoluted process where you select the safety system you put it in program mode different yellow popups one of which is a password that was blank and then like you hit okay like three or four times and then it goes in a debug stop and actually like transfers your your program over but that still isn't gonna run it you can't put in run mode from there from there you have to put in debug run to make sure that everything runs correctly then execute a safety validation which does I still don't know what because I I don't know I wasn't able to trigger an error in that I wasn't sure what it was checking for except for like like and after all of that then you can put the safety controller and run mode so these are all things we had never seen bad only even attempt to do there were no reference to this in the code we weren't really sure about it and so we just went to digging and the way we did this was we created a program so you'll notice that the program on the left looks a lot different than the latter logic program we saw before that's because it's another engineering language called function block diagram networks and I only I'm not sure exactly why we have both of these languages things with them but it reminds me of just like circuit programming with a bunch of ands and ors it's kind of the idea so two of the two of our services folks helped me write this program it's a simple timer and then we went through the process and steps of loading everything and we downloaded it and we started looking through the pcap to see what was going on and you'll notice highlighted red there in the sort of the payload area you'll see references to the names for the function block diagram and what what we ended up realizing was that bad omen sorry sysmax studio when it's transferring safety controller programs uses a different cgi endpoint than we had seen before so in this case it's using the nx bus dot f cgi endpoint and there is a nx send and receive data and it's using a a different protocol I will a different endpoint and probably a different protocol we dug in further and it actually looks like the safety controller is running this is kind of funny code sys safety runtime under the covers but the protocol that's actually involved there isn't code sys protocol so they're they're wrapping it a little bit differently there but we're fairly certain that we have more code sys research on our hands but just like the sort of the safety side of it so we're able to confirm that it only supports the cpu and ecad endpoints it can't manipulate the safety controllers I'd like to say that that's good news but again we don't know when how long like they're you know these tools are obviously have investment behind them and they're actively developing it at developing them and it practically speaking if you want to conduct a real world attack on say a real servo more than likely there's a safety system standing behind that servo watching it to make sure it doesn't fuck anything up and then setting it back to the right state so if you're going to conduct a real attack you have to manipulate the safety system and then manipulate the servo right somehow you have to do both and so I don't see there's no reason to believe that they aren't actively working on this type of safety system with the SL 3300 it's one more end point they've managed to get all the other end points right so I'm sure they'll be able to figure that out so caveat there unless you have a shitty network and then maybe they can mess with your servo all right so we're almost done here I talk about some of the real world usage here so the the endpoint here from running it or sort of the main takeaway here from just running it is despite the codices problems we think those things are going to get fixed right we think those are going to that's not going to be a a gap in their tool set anymore but generally speaking these utilities can cause problems now and at the very least at minimum they're going to cause financial problems if not disruptive and destructive problems for a site so what we sort of think like a theoretical attack scenario would look with any of these utilities is there would be an IT intrusion to steal important process documentation figure out what the process is what's important right where the important bits that they have to manipulate in order to actually cause a problem right maybe they've never attacked like a mining facility before and they need to figure out what all this terminology means there's a lot of research just going to have to go into it beforehand then they would deploy standard implants blah blah blah blah until they deploy pipedring and then pipedream would be used to recon the process technically what set points do I need to manipulate what kind of equipment do I actually have on the network right are there codices devices here are there Modbus devices here right and they could bring all that data back now they have the process documentation and the actual technical data put that together to develop a utility to actually like to put together a utility for causing that disruptive effect or if they're feeling lazy just type in the commands and do it on the console so what I really want to like harp on here is that this really is like and I don't like to sound like super hype because if anybody any of you will actually know me like I'm pretty fucking cynical about things and I don't like reading articles about malware analysis and this shit is serious it's like I've done this for 10 years since the first time I've opened up a piece of malware and just went off fuck right and so this does have some serious implications for the ICS industry for us in general because it's attacking critical infrastructure just in terms of the breadth of impact of this utility it is the first time that we've seen malware devs attempt to plug in framework for specific PLC's and protocols there is some precedent for this in ICS exploit and metasploit SCADA I would argue that these are a little bit different because they're a little bit more focused on the particular protocols versus sort of like a hodgepodge right you can't add like a like a random modbus exploit to bad Omen that's not gonna that's not gonna make any sense because the underlying libraries expect you to use like the the HTTP console right so it wouldn't really function that way but you can add arbitrary plugins for Omron controllers right same thing with Evil Scholar almost all the plugins are expecting to use that so you can't necessarily just like add a plugin for like you sort of can maybe but it's not as straightforward is see what I'm saying but you can very much straightforward out of plug in to do something else and code assist so the other thing to keep in mind is that this stuff is ICS process agnostic which is kind of an interesting thing if you're an adversary because you know code assist is everywhere so I imagine that more than likely now you're not limited by your target selection anymore you over there but they're both running Schneider electric controllers and they're both speaking code assist version three so it sort of made everything easier for them now the last kind of two points here I want to harp on is that again these tools can be used for a constant so the network and they can be used for research so while they can do disruptive things they are also tools that will accelerate the learning curve of the adversary right whether on your network or just in their lab at home right the next thing is these command line tools are super easy they're actually using the tools can just get like a checklist of instructions and they can sit on a keyboard and just type them all in and everything's done right so the the people using these tools don't have to be as smart as the people that are developing them right and we haven't necessarily seen that so much either with like a remotely interactive tool where they're trying to lower the bar for entry for for achieving an attack in terms of just in terms of the industry here um like really quick soapbox moment is I really think that some of these like proprietary protocols that the I.C.S. vendors are using and proprietary OSes need to be like opened up to security researchers a little bit better I think it's kind of ridiculous like for example with trices I think it's absolutely batshit that I had to analyze like a program what was it a python program and then shell code that ran an OS I'd never heard of and then extract firmware and then analyze the operating system also I could figure out one system call right that sounds fun from a research perspective but when you're under the gun trying to get advice out to operators who are seriously worried about their safety seriously annoying that project could have been so much quicker if someone had just handed me like a syscall reference you know um so and that's what needs that's what happens anytime we have every time the last two times anyway we've had to analyze these kinds of things and I think it would be a lot better because clearly the adversary is putting in the effort to learn this anyway right and we sort of need to get ahead of the game because compared to the IT world our understanding of the heterogeneous operating systems that I.C.S. uses is so much smaller right in our world in the IT world right it's what Windows, Mac, Linux I guess I.O.S. and Android right those are kind of the five major ones and that's it right and we've spent so many years studying this that we already know how they all work right for the most part with I.C.S. there's like limited research in each of these O.S. is and you end up having to pull out pull apart the firmware to really figure it out so I just think that the amount of time it takes could be made a lot easier if some of these vendors cooperated a little better with us I.C.S. to pull bad logic so that I.R. people and forensics people can like send the logic over to people like me or anyone and look at it and see oh this program is actually doing X right and in that sense I think that the the vendors are so used to being the expert and the people that get called in on the scenes and that's starting to change where I.R. teams are starting to go on the scene right Dragos goes on the scene Manny goes on the scene right but how often can they actually pull bad logic sometimes maybe they happen to find it but it's not so common or maybe they know enough about the E.W.S. that they can find a project file sitting on a device maybe they can see what went wrong right but that isn't very common either and in some sense I think the vendors need to start recognizing they are the only experts about this stuff anymore so and then the last thing the sort of fundamental problem with all of this is that like PLC just don't do integrity checking in their code that's just the most basic problem that needs to get fixed as far as I can tell from the billions of vulnerabilities that exist if there was just a modicum of integrity checking I think a lot of these things would have been mitigated to begin with so that's all I have for you just really quick so I don't get yelled at by people we do have mitigations for this so if anybody is watching online there's a link you can go check those out at that website I didn't talk about it here because it's DEF CON and I'm just like that's boring so if you're interested in that go check that out there otherwise I'm finished thank y'all for having me