 It's time to, yesterday if you were in here, we saw a Teddy Ruxpin get hacked and today we're going to do something a little bit different. We're going to do power plants. So this is going to be very exciting. Let's give Joss and Marina a big round of applause. Have a good time. Good morning. Thank you so much to all of you for coming here. So today we will be like, we will present you like, what is an effort, what it takes to design embedded exploits to actually cause physical damage in industrial facility from the beginning to the end. So briefly about ourselves, I will let Joss to present. Yeah, so my name is Joss Wetzels. I'm an independent security researcher with Midnight Blue, where I mainly focus on different kinds of embedded systems, industrial control systems, automotive, IoT. And I previously worked as a researcher at the University of Critical Infrastructure Security. So my name is Marina Krotofil. I've been doing, my major specialization is physical damage and I've been doing it for over eight years. I presented a lot of works at Black Hat and DevCon. And so how this combination of us came together is, on one hand, Joss can reverse engineer and develop like exploits and implants for any embedded system in the world. However, like what exactly do you want to do on that device? And I am my specialization, hey, here's this power plant, chemical plant or traffic light system, robotic system. How exactly you want to cause physical damage? So I will be designing engineer this exploitation scenario to cause physical damage. And then at the end, I will come up with a set of algorithms which need to be implemented and with a long list of tasks which needs to be executed on that embedded system. So with that wish list, I would go to a guy like Joss and say, hey, this is what I need to do on that device. And maybe like somewhere we will also meet in between when I look at the design of the device like, hey, the specific design features of the device can be probably used to accomplish like simultaneously several tasks. So this is where we would be working together. So this is how the combination came into place. And so this is how we will be presenting the presentation. So after the introduction and industrial control system and safety systems, by the way, how many of you have heard Triton? And I hope that everybody, good, because that was our motivation. We also wanted to show like in the mass media, this Triton exploit was like described as like so sophisticated, like only state sponsor, only a few people, and we will show you if it's really so. So after the introduction, we will go into the device exploitation because at the end, you need to obtain code execution on the device. So after you have code execution, then you start already developing, for example, implant like back doors like and payloads which actually targeted like design for physical damage. And of course, conclusions in the end. As you already probably noticed, I'm speaking very fast and there is reason for that because the topic which we're presenting to you extremely complex, but we still wanted to show you the process from the beginning to the end. And still, of course, with our deep details and it's already a lot. So it will be fast-paced talk. However, again, the motivation was to show you the full process so that you could see it from the beginning to end. And we will be posting even longer version of the slides after this talk because we have to cut out certain information. And if you're interested, you can go and review it again. So introduction. So what are the industrial control systems? Well, you probably all know, but it's important to set up a vocabulary so that all of us operate for this talk, operate on the same terms. So industrial control systems, those are computerized systems in network which control physical process. So typically, if you talk about industrial organization, there will be corporate network, there will be another network which is called SCADA or control network. And even though this is the same looking computers, one will be called information technology and another will be called operational technology or OT. Nevertheless, that is still all computing devices and the guys who deal with those systems are studied typically something like computer science. And as long as we are moving towards physical process, that is already engineering science and it's completely different. So the laws that we come into the physical process, the computing devices which actually monitor and control physical process, they look already like this and they typically know them as embedded systems. So they don't have operating system as we know it. So operating system which actually runs the device and executes the programs is called firmware. And since these systems are real time systems, the entire firmware must be loaded and executed in the memory at all times. So on one hand, the attacker needs to execute code so the attack will be executed on this embedded system. So it's still cyber domain in order to cause physical damage and basically damage, for example, the machinery and operate the machinery as needed. And so when, for example, when the attacker is going into the penetrating financial sector, his goal is to steal money in some form. So when the attacker is penetrating the industrial network, his goal is to cause some sort of form of physical damage in the physical domain. It could be also economic damage like spoiling the product. However, in the mass media, typically this physical damage is depicted as a big explosion. So I see a threat landscape has changed in the past eight years. So I started doing this type of security before it was fashionable and nobody knew about it. Well, like in IT domain, there was like crazy amount of hacking happening all the time. Like industrial control systems lived in bubbles, nobody knew about them. And then Stuxnet has happened. So it seemed to be like Stuxnet was kind of a trigger and a tipping point because we started seeing more and more publicly known as pionage attacks. And somewhere like we started seeing first publicly known like activities related also ready to recon of the operational technology system. And from 2015 it started to happen. And so there was two power grid attacks in 15 and 16 and in 17 basically it was announced at the end of 2017. Triton has happened. What it was, the attacker tried to install backdoor or remote access trojan on a safety controller. And it was a big thing because it was picked up by pretty much every mass media including Wall Street journals which said that that was very, very, very alarming situation. And why was Triton so special? Because it was targeted safety instrumented system. So what does it mean exactly for us? Physical processes are inherently hazardous. So it means that there is like toxic flammable liquids, fires and explosions, electrical hazard, moving parts in the machinery. So there are a lot of layers of safety protections to prevent harm or maybe even casualties to humans first of all. And secondly to prevent environmental damage and also machinery because it's expensive to kill machinery. So there are layers of protection starting with the design of the process. Then we have a control loops. Then we have a human operator who reacts to the alarm. But as soon as like control system and human operator are no longer able to control the process in a safety manner, we have so-called safety instrumented system. That is independent control system which reacts on the hazardous condition and tries to prevent it. So as you can see this is the last lane line of defense before hazardous accident is already happening. So and the attacker was apparently was trying to disable the system. So that is actually attack on the civilians which is not good. So previously safety instrumented system are software systems and because it is a line of defense, typically it is recommended that they would be run on the isolated and segmented network. But for ease of design and for ease of maintenance, they often are connected together to the main control system. And that allows the attacker to obtain remote access to safety instrumented system. So that was a triconix from Schneider and it is a very critical safety instrumented system of safety integrated level 3 which is only 2% of all hazardous situation for example in oil ease of that severity. So it is very critical safety systems. Triconix is everywhere for example you could find it on the swimming ships and so on. So the attacker obtain remote access to the engineering station which was connected to the safety controller so they got the ability to communicate to safety controllers. And what an attacker attempted to inject a passive backdoor or some people refer to it as a remote access Trojan which would allow attacker to read arbitrary memory right into the memory for example shell code for the attack and then they execute that code. So now even if you have the backdoor it really means nothing to you because unless you know what you want to achieve in the plant the backdoor is actually absolutely useless and harmless. So attack scenario depends on attacker goal and sometimes it means explosion but in most cases it does not because you don't need to over engineer if you just want to send a warning sign to your enemy or what not. And like some you can achieve some simple like attacks like do not press there are certain buttons on the HMI which are so descriptive like stop start you can press it and you achieve some effect but you know that is not a long lasting effect. So like yes the first cyber physical attacks were happening at the HMI then the attackers moved their exploits for example in destroyer crash override they were executing the attack already at the level of the industrial protocol but now we have Triton which is already on the embedded system. So what's going on why the attack is already moving their exploits into the embedded systems and we will explain. So industrial processes are very complicated and they actually build inherently in design to be robust and recoverable. So if the attacker wants to achieve any significant long lasting damage they actually need to obtain very specific process like very detailed process comprehension the design the dynamic behavior of the process physics and so on. So for example like what causes the pipe to explode well what causes the right pipe to explode and then what causes the right pipeline to explode at the right time and suddenly the complexity is increasing. So industrial control systems operate on the control loop principle so control systems and human operators they use sensors to observe the state on the process and then control system compute the commands to instruct actuators to control the process and bring it into the right state. So in a nutshell when the attacker is start designing or engineering damage scenario there are huge number of tasks which he has to accomplish like starting from of course manipulating the process but then the attacker needs also to obtain a feedback loop to know whether the process is moving into the right state and he is successful. I've been doing this for many many years and I've designed a lot of damaged scenarios and I can tell you that obtaining feedback loop was one of the most hardest tasks to achieve because mostly sensors which are installed there in the plant they're useful because the attack is something always weird and the plant was not designed to measure something weird going on so you have to do it indirectly and of course because you are trying to bring the process into the wrong state in the harmful state the control system and human operators and safety system they will try to fight you back and to bring process back into the normal state so the attacker also have to prevent the response and try to falls into this category preventing response because they wanted to prevent response from the safety systems so in a nutshell the cyber physical attack or damaging attack that will be a collection of clandestine control loops because the attacker is becoming a control system and you have to have the cycle of observation and manipulations to achieve a safe state so attack timing is crucial because process is not vulnerable at all times so you have to find that vulnerable time and when the attack has to be executed the attack coordination is critical as you have seen there have been a lot of tasks and for example observation is of state 8 and component B needs to trigger payload X, Y, Z so this requires very granular control across the entire process and to manage tasks quantity and timing so there was a very nice presentation even though I've been presenting on this topic a lot of times like many times so there have been a nice presentation from Jason Larsen who decided to compare different damage scenarios and he came up with this timing and state diagram where he divided all the control damage tasks into damage scenarios into different tasks where he also tried to map like hey this is a set of tasks which I need to execute how do I map them now to the different device and to different implants and so basically as soon as you as amount of tasks which you need to execute to achieve damage is becoming large you need to have an implants because like you're getting separate tasks you will not be able to coordinate them so and this also allows you actually to coordinate so you can easily then install the coordination between the implant via the communication links or you can implement the routine to detect specific state so and it's also much more stealthy because you don't have like anomalous you could not create anomalous network traffic so but before like in order to actually achieve this to implement an implant you first have to exploit the device and you better enjoy extreme programming because those devices are extremely small have very limited resources they packed with the functionality so you first have to actually find something what device does not need eliminated so that you could put your actual exploitation code so at the end Jason came up with this diagram where he has like put a score on different tasks and he compared the reliability because it's not only the reliability of the attack scenario but it's also reliability of your exploit or implant on the device and so basically it's kind of you have to find this trade of like what is the most reliable attacks implant and attack damage scenario and implant stability so now after this long introduction I am giving back to Joss and he will walk you through the device exploitation and designing like basically implementing so now I have my wish list of tasks and I'm giving it to him to implement alright thank you Marina so before we can do all this so before we can get you know to the cool stage where things are actually exploding we'll have to get to another cool stage which is exploiting the device and basically the process is as follows and I'll walk through these steps and we start by obtaining the necessary materials so we need a couple of things before we can device an exploit and an implant and the first thing we'll need is documentation and a lot of it you know developers guide planning and installation manuals all that kind of stuff and yeah in the case of TriConX it was very useful that these things are safety certified at a certain level and that means that all of the documents were available on the website of the US Nuclear Regulatory Commission so very detailed information was out there in other cases you might have to buy it but this is the first step you go about then the second step is obtaining the engineering software so these devices they're connected to a workstation running I don't know Windows XP with some software that is used to program them and this software usually contains the functionality for talking to this device and the protocols you want to take a look at so you obtain this by just going to the vendor website or asking them nicely which is the easiest route or if you're already compromised asset owner networks you might take it from their own network because you're in there already so why not grab a souvenir or you might go to the various sketchy sources on the internet like eBay or Alibaba or open directories hosting this kind of stuff so you know it's usually relatively easy to come by in the case of TriConX we found a tri-station software for the equivalent of 3 US dollars on some Chinese website so yeah that was relatively easy and then we have to obtain the device and that's a little trickier because you're not going to find that kind of stuff at a yard sale or in a corner shop or whatever it's very expensive and you might have to buy multiple copies because you might have to do a tear down or you might break a device etc etc so ideally you buy it directly from the vendor but if you're a nation state sponsored attacker you don't maybe want to directly do that so you need strawmen or you buy it at a bankruptcy auction or again eBay or Alibaba are your friend so here you can see I'm not sure if it's yeah it's all coming up it's relatively expensive in most cases a couple of thousand bucks for one of these controllers and you need multiple parts to put the whole thing together so it's not very cheap then you need to obtain the device firmware the stuff that runs on the device itself so you have various options here you can sometimes download it from the vendor websites or extract it from some update utility that's the easy approaches or you might have to extract it from the flash chip on the device itself go to hardware or hacking route and this can become very complicated because in a worst case scenario you'll have encrypted firmware you'll have chip readout protection you need to bypass it and do side channel attacks and all that kind of stuff but for triconex that was not necessary because there was no readout protection on the flash you could just desolder it, put it into an adapter and use a universal cellular programmer and you'd be good or you could just get it from the firmware update utility which also holds it all so you know the second step you have to do is usually device tear down and PCB analysis so we need info on what kind of microcontroller is on this device we need the device functional domain so what is happening where on this device we need to know about interesting interfaces like UR, JTAG, what have you and sometimes we're lucky if it's an FCC certified device you have an FCC ID and you get internal photos and documentation on the FCC website sometimes other people have done your job for you and you have public tear downs and that's very nice in the case of triconex the planning and installation guide has very detailed internal block diagrams and that helps a lot because that prevents you from or saves you the effort of opening it up but in some cases we're not so lucky and we will have to do tear downs and now people who don't come from a hardware hacking background they're usually terrified of it but it's not that complicated in most cases you know like the picture shows it's RAM flash or you Google it and there is this persistent narrative that you know especially among some OT people that you know ICS devices they're all different and operational technology it's not like the embedded devices you're used to but for our purposes it's like all the IoT devices out there that are getting hacked by the billions every day for example take these two PLCs by Schneider Electric the Modicon M238 and M340 usually you have a central processor module which does all the heavy lifting and you have a couple of input output modules which you stitch onto it on the side as you can see on the right of the slide you might have an integrated ethernet connector as you can see on the right example maybe you have a dedicated module that does the ethernet stuff and then connects to the main module over serial link and internally they typically look something like this so you have a couple of IO pins you got your serial link or ethernet link or whatever and then you have a microcontroller in the middle and typically this microcontroller will run the main firmware which has the operating system and some of the application stuff like maybe a web server or an FTP server or whatever and then you typically have a logic handling chip which might be an FPGA or another microcontroller executing the actual programs that this thing is configured and programmed in now this might differ between PLCs because this is a generalization but it roughly comes down to something like this and for TriConnex it looks like this so you have three main processors and they communicate over tribus which is an internal bus that does some voting on input values for consistency and it has a triple modular redundancy architecture to ensure that a vote of two out of three overwrites any errors that are introduced there yeah that's basically what it looks like the main processor there you have it, it runs a PowerPC chip and you have a dedicated other chip for input, output and communication stuff and we'll delve into this a little bit later when it becomes relevant so the first thing you want to look at for most ICS stuff is reverse engineering of all the protocols they talk to because in many cases these are legacy and proprietary protocols usually ports of old serial protocols that have been retrofitted onto Ethernet they control very sensitive functionality like starting and stopping the PLC updating the firmware and so on and you might find a way to get into the device itself by remote code execution here and that's what we want so the first thing we need to know when encountering a protocol we don't know is knowing the packet structure and the semantics so we can do this in a couple of ways and you know this is a very quick generalization we'll go about comparing it to functionally similar protocols that have been documented so if it's a port of an old serial protocol maybe take a look at what that looked like and what you can recognize in there test for common encoding structures like TLV, sequential identifiers checksums, any entropic analysis for fields that indicate cryptographic functionality or you might want to do differential analysis of functional batches of packets so you have like one packet that corresponds to starting the PLC in different kind of conditions and then see what kind of bit fields in the packets change and what you can make of that and like Rob Savoy said, believe it or not if you stare at the hack stones long enough you start to see the patterns and that is definitely true for pcap-only analysis as you can see here it's basically like looking at the matrix now ideally you want to assist this traffic-only analysis with binary reverse engineering because you want your reconstruction to be complete and sound, you want to write a reliable exploit not because you don't want to fuck things up but because you don't want to fuck things up for yourself so a pcap-only analysis can be incomplete, inaccurate or opaque, you know you can have undocumented or rare behavior that you don't see in the field but that's in there and you're interested in, you might guess at semantics that might not actually be what you think it is there might be encryption, compression, blah blah blah blah and most importantly pcap-only analysis damages your sanity so you want to do some binary reverse engineering which does that to a lesser extent so in the case of triconex the tristation software has a communications DLL and this has all the juicy stuff we need it's a single DLL in the engineering software and it has debug symbols so that greatly eases our life and as you can see on the slide I hope it's visible basically all these functions they have a good semantic mapping between the functionality it does like starting the PLC, stopping the PLC downloading the control logic and the function code within the protocol itself relatively easily identify all that functionality and you can see that the attacker probably went about it this way now the benefit for attackers is that you don't need to fully reverse engineer the protocol, you only need to understand a few interesting packet types because we don't want to craft a full protocol parser we want to craft an exploit so I don't care about all the rest I care about like that one packet type that does the good stuff now when it comes to vulnerability discovery which is typically the step that comes after this the next step is getting code execution and ideally this is a pre-authentication vulnerability as we'll see pre-auth is a relative concept in the ICS world and in many cases ICS vulnerabilities are often a byproduct of the reverse engineering so you won't need to go about in most cases about fuzzing or static analysis of the firmware in many cases it's insecure by default there's ancient legacy shit everywhere and you shake a stick at it and vulnerabilities fall out so let me briefly drink something so an example of this for example is the Moxa and port serial to ethernet wifi converter you plug a serial cable in converts it to ethernet it has a web interface with broken authentication so it hashes on the client side that's good once you're in you can do command injection in the ping test form so you got your code execution right there another example the opto it's used in fairly sensitive environments it's got fdp it's got a proprietary protocol without authentication now the thing is you can configure fdp with a password you can enable ip filtering all that kind of stuff but then you can use this unauthenticated proprietary protocol to disable ip filtering enable fdp and ask the credentials nicely you know and then you get them and you can use the fdp to upload a new piece of firmware and reflash the device over fdp and there's no firmware signing another example in case you don't believe me yet the modicum quantum plc that's a large plc for process applications it has fdp with hardcoded credentials which allows you to read and write configuration, firmware, what have you it has telnet with a hardcoded backdoor and that's actually a C interpreter which you know is nice it also has an unauthenticated proprietary modbus extension for starting and stopping the plc overriding the programmable logic there's basically a gazillion ways to get code execution on this thing finally to drive the point home another serial to ethernet converter by adventac it has web interface again and the nice thing is that if anyone unlocks the web interface on one pc, let's say a legitimate operator it's disabled for everyone so you know and then you have a command injection in the email alert setup form and again easy code execution right there I mean you get the idea it's like shooting fish in a barrel right and for trident it was basically the same thing it was an execute my packet police vulnerability so the vulnerability was a freebie of the protocol reverse engineering you have this safety program download functionality which is how the engineers put like the actual logic on the thing and it has no authentication and the safety logic that gets downloaded to the device has no secure signing right so you can basically skip all the way from reverse engineering to exploit development and that's neat from an attacker point of view not so neat from a defender point of view and then that brings us to exploit development here we find the suitable vulnerability and we get our code execution we need to craft and exploit to actually not execute just the logic we want but execute the instructions on the microcontroller we want and how would this look like in the case of triton well triton has safety and control applications which are developed in one of these IEC languages many of them look like graphical language like you can see on the bottom of the slide and they typically get compiled and downloaded and executed on the main processor there and that's nice because that gives you another exploit development freebie because you don't need to break out of any sandboxes you don't need to exploit any run times and because in the case of the triton xcontroller the logic was executed on the same chip that the operating system was executed on you don't need to hop across any chip perimeters you're right where you want to be so triton did have to add some additional functionality so it doesn't overwrite the original programming but it appends to it and the reason why it did this is because it allows the safety logic to continue running without interruption because once you're implanting this device you don't want everything to stop and potentially already cause a process shutdown you want to be stealthy at least at that point and our complication is the key switch on the triton xcontrollers so these devices have a physical key switch that allows you to set it to a certain mode and only if it's set into the programming mode then you can actually download new logic to it which will be relevant later when we discuss the payloads another complication of embedded exploitation in general is the heterogeneity so embedded devices are far more heterogeneous than general purpose ones you deal with a billion architectures from ARM to MIPS to PowerPC to whatever a billion kind of operating systems like VxWorks or QNICS or even custom operating systems and in the case of triton x that means that you have different architectures and different versions so version 9 had a national semiconductor chip version 10 and 11 had PowerPC and for the attacker this means that scaling the attack requires writing and modifying payloads and implants for each different version so there's some effort involved there and that brings us to the development of the implant and the OT payload that's the stage that's close to what Marina mentioned of mapping like the TSD to the implants on the devices now we have code execution we can run arbitrary PowerPC shell code so now what? Well, exploitation is just one step among many so for complicated OT payloads we will need to develop this implant and then after that craft the OT payload we have different kind of strategies here so we can directly implant the OT payload straight away after exploiting the device or we can implement a backdoor which would allow us to keep the OT payload a secret until and basically provides you with kill switch capabilities or in the jargon of some people a dormant cyber pathogen that basically would allow you at a later stage to execute any kind of payload you'd want to the second thing we'll have to decide on is whether we'd go for cross-boot persistence which would require modifying the flash and we need enough space there to insert the implant or which is what Triton did we just go fully memory resident that requires executable RAM but it does mean that on reboot the implant is gone now for safety controls this is not that relevant because these have an extremely high uptime so even if it would be gone upon reboot you know it's probably going to be there all the time anyway this has the added benefit for the attacker that it does complicate forensics the other thing you want to think about when designing an implant is the scalability of the implant so you want to target devices that are common throughout ICS so not only for one particular facility you might be interested in but across different kind of facilities different kind of industries and triconex fits this bill so there are 18,000 or more than 80,000 triconex systems in over 80 countries so having these kind of capabilities is very interesting from a strategic point of view you also want to target if you're targeting software instead of controllers you want to target the software that is common throughout ICS so used across multiple vendors so protocol and connectivity stacks are usually used by multiple vendors so having exploits for that kind of stuff really scales well and the same goes for control runtimes or the operating systems in question you basically want to construct an arsenal of exploits and implants against common devices and software stacks because that means that it is a one-time upfront investment and now there is no huge turnover in these devices these have an extremely long field life they get deployed for 10, 20 years they don't update it very often you know if there are even updates out there so if you invest once in an arsenal for let's say all safety controllers that are relevant in the market right now it's gonna be an investment that's gonna be paying off for a long time so try to make more sense as a tool in such an arsenal than a very expensive one off that was engineered for this particular attack so that brings us to the reverse engineering of the ICS firmware the first thing you'll have to do is extracting the firmware so you have to determine the firmware format and then unpack sometimes there will be firmers for multiple chips on the board or data blobs and they're glued together and you'll have to get them out of the firmware and then you'll have to do the compression and the decryption if there is any of that present this might be simple you know keys might be present in a firmware loading utility or you might have to actually do side channel attacks in the case of TriConX this was very easy because the firmware was unencrypted so you know basically the step could be skipped the step that you do after that is accessing the firmware so you need to obtain a memory map you need to know where on this chip does the ROM live, where does the RAM live, where does the external memory live, the purpose registers for interacting with peripherals all that kind of stuff and you typically get this from the datasheet so very important if you do this kind of stuff you need to learn to love reading datasheets so after you know basically the lay of the land here you go about identifying the base address so you need to know where this is loaded into memory now there are many approaches here and this is a gross oversimplification but we simply don't have the time to go into all the details but this can be as simple as you know being loaded at a chip fixed address you can reverse engineer the update utility and see where it places it or you can extract it from something like the interrupt factor table or the boot loader or self relocating coding code jump table string tables all that let's consider for example this RTU firmware piece this is a piece of firmware for a remote terminal unit that's deployed somewhere in the field it's ARM based and we have this piece of firmware and we don't know what the base address is so we load it just at address zero and then we see all these different basically branches does anyone recognize what this is a couple of people well yeah that's the interrupt factor table of an ARM chip so the first entry is a branch to the reset handler and basically if we look at the offset we have 100 as an offset here and here we have 102 if we look at that offset we see a lot of toggling with like these special purpose registers and that's very typical behavior of a reset handler so if we then deduct the thousand basically that becomes the base address we re-base the firmware image and we can see that it cleans up nicely now we know how to load it into memory so after you've done this you know where to load it you know where everything is laid out you have to reconstruct the complete code and data topology of the firmware. Firmware images are not need executable formats like like PE or ELF or MAC O I mean your mileage might vary for what qualifies as a need executable format but this definitely is not it we will have to heuristically identify the functions in there the strings, the jump tables, the structs and all that kind of stuff on the upside this is usually the bulk of the work doing all this reverse engineering the upside is that we don't need to reverse engineer the full firmware only up until readiness for this next step because we want to hunt for interesting functionality we want to reverse engineering a sniper like fashion so we want to know how does the control runtime work that actually interprets and handles this safety logic where are the protocol parsers where are the communications and peripheral IO handlers where is any security or safety related functionality there so what would this look like for the Tricon X 308 which was targeted by the Triton attack well it's firmware that's power pc based which is nice because the Hexra SD compiler is available for that and that saves you a lot of work it's not a substitute for reading this assembly but it eases your navigation across this firmware it uses a custom operating system with 27 system calls and some sparse documentation exists thank you NRC basically this is what the operating system looks like you have a scan task a communications task and a background task and we're really only interested in the scan task which fetches inputs from shared memory which is where all the analog and digital IO gets put then we do a try bus transfer for the voting on consistency and then we run the actual control logic and then we send the outputs again to the to the shared memory and basically this implements all these control loops so the targets here would have been the memory layout and management because we want to achieve this memory residency so we need to know how to do that then we need to look for consistency checks and diagnostics functionality for implant stability we don't want anything messing with our implant while we're running we need to know where the network command dispatcher functionality is because we want to be able to communicate with it over the network and achieve that we want to be able to know if there's any privilege mode management we want to do anything we can on this device so if there is privilege management we want to escalate them so and possibly finally we need to know the scan task and IO transfer stuff so that's basically what the Triton implant looks like and does so it has four stages it has an argument setter an implant installer the backdoor implant and then the missing OT payload so the argument setter it's not that interesting but it basically controls the finnid state machine of this thing which does at first if you can see that properly it does an exploit for privilege escalation and then basically it relocates the implant the third stage so the privilege escalation exploit it looks complicated but it's not it's basically a right for anywhere because of improper handling of memory privileges that allows you to write this address to this value to this address what does this mean basically what happens is that once you invoke a system call on this particular firmware it saves the machine state register and when it returns it restores it and it is stored at this particular address that anyone can write to regardless of privileges so if we overwrite it with this value we set bit 17 to supervisor privileges and escalate it and that allows us to do these next steps to install the implant the first of which is copying the payload into the firmware area of memory which allows us to achieve residents even if they wipe all the safety logic on the controller then we patch an entry in the jump table which allows us to hook a network command and then invoke our implant when this particular network command is called and then finally we patch a certain RAM check which was used for consistency between the firmware on ROM and in RAM which would otherwise mess with our stuff and this is basically the background implant like Marina mentioned it allows for reading, writing and executing arbitrary memory and basically allows you to overwrite this key switch so once you implant it it doesn't matter what position they turn the key switch into it allows you to execute anything you want in memory so you have persistence on this device so the fourth stage which is the most engineering related is the OT payload and this was missing because this carries out the meat of the attack and we can only speculate what this might have looked like and that's something we'll do so because Triton is positioned here in the attack tree the control and safety system this is what the OT payload would have been related to so basically it could have been for example an IO spoofing scenario and that would have been a scenario that Marina would have given to me and said you know we want to do spoofing of input and output values as you can see for example here you have measurement values they come as an input signal to the controller and as an output signal go to the instrumentation and then we want to mess with either the input or the output to cause some unsafe state now internally this looks like this on these devices so you have the physical IO you have the logic and there is an intermediary variable table that manages this on the triconex this is handled by an IO processor that has shared memory connection to the main microcontroller and that's where it places this variable table so we didn't have a triconex controller but we did have a WAGO PLC and I mean it basically comes down to the same thing it's an ARM Cortex PLC running real-time linux it has a vulnerability in the code sys runtime on tcp that you can explode remotely and gain root and basically what we do in our attack is we use the CPU debug registers to catch any access to this memory mapped IO and then we write a custom exception handler that gets invoked when this debug trap is invoked and we change the pin mode from output to input so any writes that would have been going to an actuator like closing a valve are now going to an input pin and they don't actually go out and we have a little demo of this that should be coming up we don't have a lot of time but alright so what you can see here is you have the WAGO PLC on the right and then you have LED which is our stand-in for a valve and it's supposed to be blinking every couple of seconds and now you can see it now you can see basically on the engineering software and the attacker well soon yeah I can't really skip anything here but there should pop up like a shell at some point now you can see the LED blinking there yeah so you know this is a root shell on the PLC we execute our payload which is basically a linux kernel module that does this thing I just mentioned about the CPU debug registers and then we stop it and you can see the LED will stop blinking while it should be consistently blinking because it you know it now thinks it's an input pin and then at some point you know we'll change it back again to an output pin and then you can see it's starting blinking again right if this was a valve it would have been having had an impact now it's a LED and you know it's just a demo so that brings us to the second possible payload which is alarm suppression so let's say we want to mess for example with a chemical process like this and our goal is catalyst deactivation now industrial processes they have alarms so if something is about to go into an unsafe state or something is out of bounds an alarm is raised somewhere in the process and these are propagated throughout the process and eventually lead to a safety shutdown and as an attacker we want to prevent this because we want the attack to continue so how do we go about this we can hide the alarms by compromising like the safety view or the DCS view or some local HMIs and this is the benefit of you know hiding them centrally but still these alarms go out as network traffic right and we don't want any kind of network inspection to see these alarms so we want to suppress them either at the field level the level of the field devices or at the level of the safety controller itself which is what TriconX is which is a PC based HMI solution that allows for management and bypasses of all these alarms and each HMI function is mapped to a TriconX logic function block that looks somewhat, that was a little bit too quick that looks something like this so you have for example a water tank level alarm here you have a water high, a water low signal and an aura of them raises an alarm so as an attacker these safety programs reside in memory as code which is how we got code execution in the first place so the OT payload here could modify the instructions to set the alarm to a fixed vols regardless of the water levels that are coming as inputs to this alarm function the attacker needs to know of course first where the program lives in memory and which instructions of the program to modify now luckily for TriconX these programs are stored as a circular linked list in memory as you can see here this is actually our implant living in memory and then you can walk this linked list and find the target program you want to modify and this is what basically the alarm looks like the core of the logic is a simple OR instruction which the attacker can then hot-patch, you know it's a risk architecture with a fixed instruction size so we just set it to a fixed vols there it's relatively easy and this is the result that comes out of it on the left you have normal functionality water is getting dangerously high the alarm is going off on the right you have the exploited functionality and the alarm is sleeping so some more speculation why did this attack fail I don't think we mentioned that but the attack was discovered because it failed in the actual field it caused a safety shutdown of the process there are a couple of possibilities you could have had a Borg payload the privilege escalation could have failed it could have targeted a different firmware version than they actually had developed the exploit for which could have caused an access violation the backdoor allows for raw reading, writing and executing of memory they could have caused some memory corruption a payload that was written wrongly they could have gotten into a fight with the watchdog so very common on embedded systems you have this watchdog timer and you need to periodically kick it to keep the counter high because if the counter expires it resets the CPU or they could have missed some additional diagnostics there's a ton of diagnostics functionality in here because they patched the RAM and ROM check doesn't mean that they patched everything rightly these are all options another option is that they got into a fight with the triple modular redundancy I mean this is just the voting on the inputs and the output but it's a very complicated patent that covers a lot of ground and it could have been that the OT payload violated something in the triple modular redundancy that in turn led to a safety shutdown that brings us to the conclusions which I'll give to Marina again we probably have just like a couple of minutes for conclusions so just to summarize the purpose was also to go through this process of development implants for embedded system and see whether it is really that complicated so what is the actual real threat is it also only for elite and we should not expect this type of attack or implants on a mass scale or it not and apparently it is not not only that exploitation of all systems relatively easy like it's cheap because there is no any exploit mitigations but also obtaining equipment relatively easy we went through the process obtaining documentation is relatively easy so you kind of have all of the needed components to quickly obtain code execution and be able to backdoor this type of devices so exploit development again easy if I will just go quickly through this implant development relatively easy so for example the case of Triton the attacker already had privileges typically on industrial control system you would not have like separation of privileges so the attacker had privileges to escalate his privileges neat so however for example in case safety system you typically have this triple redundancy sometimes it's quadruple redundancy which makes things a little bit complicated because you have to reverse engineer more so the most difficult part which we were able to establish is actually developing OTP so basically my task is harder and especially whenever we come together like for example Joss and I come together this is my wish lift and he already has an implant this is where reverse engineering started going further and that is becoming more difficult so so basically yes so the open questions which we have and this is I think Joss has covered it very nicely that it's probably what we've seen is just one tool many so basically development costs should be seen as relatively low and I think we should what we definitely should expect relatively soon is copy compared security is a fashionable industry so as soon as you like for example somebody will release a first ransomware malware everybody will be designing ransomware malware so this is what we're going probably to see so many more other like state sponsored threat actors will join this basically game and again it's not necessarily so this is what like people say on one hand yes it's easy to exploit then why don't we see a lot of those attacks because first of all you have to have a motive why would you do that and secondly it's of course like crossing this red line so on one hand it probably will be something like the nuclear weapons you will have the arsenal and maybe you will be showing the capabilities just in a slight manner like it has happened with power grid attacks in Ukraine you just show that you are capable of but you do not cause massive like damage which hurts so you basically will be having it as a like economical and political negotiation so we wanted to thank the couple of people including Felix Lindner who was like a frequent DEF CON speaker who kindly helped us to review the slides we have the slides with the toolkit which you will need and with that we at the end and thank you very much for your attention