 So, yes, I'm Jason Larson, this is Marina Cordoville and this is Rocking in the Pocket Book, Hacking Chemical Plants for Competition and Extortion. So, we're a friendly team of an academic and a hacker. And so if there's any doubt, then I'm the evil double hacker and she's the cute academic. So, we're here to play who wants to be a, who wants to be a millionaire. So we're all at DEF CON because we want to learn how to hack like in the movies. And if you're in Vegas, probably you want to be rich. And so this talk is about giving you all the tools to get all the rich and get all of the girls like in the movies. So, if you want to get rich and have some money, then maybe you want to hack some process control networks. So, industrial control systems. So, in general, industrial control systems are just a whole bunch of computers that have only one thing that's different than all the other computers in the world. They run physical stuff out in the real world. They run power grids, chemical plants, wastewater treatment facilities, all of that type of stuff. So, typically run for the benefit of mankind. So, industry means big business and big business means big dollars. So, some smart hacker starts in a coffee shop and sits down and clicks on the keyboard and then half a world away. Then, of course, all the movies will tell us that fire and destruction, everything breaks out at some factory all the way over there. But all the stuff that happens in the middle is kind of one of the big mysteries of the 21st century. It's just twiddle fingers, explosion and nothing in the middle. So, the typical understanding of skid hacking is that after an attacker breaks into a process control network and gets control of it, there's this magic big red button and you go mash the big red button and whatever your thing is. It either saves the day and shuts down the process nicely or hits the button and everything blows up. So, in reality, the attacker actually has to build the big red button. There's not one there for him. And to start with, the attacker has to actually decide what exactly he wants to do to the process. So, in general, like all attacks or impacts on the process can be divided into three groups. So, you can, for example, break damage equipment. You can go like after the production, for example, you can easily spoil the product, reduce the production rate, you can make production as such more expensive so the product will be less competitive on the market. And these two groups of damage will never make the headlines because the companies do not need to report them and they typically do not because it's bad for their reputation. So, if you want really to shame the company and make it public, then you have to make the company non-compliant. The most damaging attack will be attack on the occupational and environmental safety because it can kill humans and can damage the environment. Less damaging attack will be on the pollution. For example, if you contaminate the water or soil or exceed the, for example, heavy metals concentrations in the emissions or like, yeah, delays the production so they violate contractual agreements. And when the attacker decides like, well, okay, what exactly do I select out of this list? So, he has to like kind of follow some thinking process. So, for example, damage, equipment damage is something that comes to the hack a mine first. The disadvantage is that it is irreversible. So, you can't undo it later on. The other problem that it is difficult to understand what is the collateral damage. So, if something explodes and human is a vicinity, then it kills a human and this attack transfers into the compliance violation. The advantage of the compliance violations attack is that all of those regulations and limits and everything, it's online. So, you basically, it's public information. It's easily available. Again, the problem is that the collateral damage is unclear because, for example, if you contaminate the water, you may kill the fish. So, many people get upset with that. This type of attack must be reported. So, it's probably, it might be what you want, but if you are not unclear how well you hide it, your traces, you might not want to do like, you might be problem for you because the serious guys will be investigating the case. So, out of this consideration, the attacks on the production damage is actually more profitable, a preferable scenario because nobody needs to report them, nothing get killed. So, for the attack, it sounds like a more safe scenario. And this is actually will be the scenario which we will be considering in this talk. So, we will illustrate the attack from beginning to end, how we will cause persistent economic damage on the plant, to the plant. And the key word here is persistent. So, you won't make it hurting for a long time. That means that you have to make sure that the attack will not be attributed to cyber event and not will be eradicated. So, this scenario will be useful, for example, for the extortion attack or if you want to kick out the competitor out of the business. In any case, you can earn my name. So, process control. We're going to go over a little bit of process control basics for, this is a 101 track for anybody that doesn't understand process control terribly well. So, the basic of process control is a control loop. So, if you go and look at your typical thermostat, we have a nice cute thermostat and nest here. I actually have one in my house, but I haven't gotten around to hacking it yet. So, on the thermostat, you've got a sensing element that measures the temperature and then you have a set point. So, whatever temperature you set it to, you say, I want my house to be set to 72 degrees and keep it at 72 degrees. And so, we do this because in the 21st century, nobody likes to run up and down and manually kick the furnace on every time they wanted a little hotter and then run back up and kick it off every time they wanted a little bit colder. So, in process control terms, this is a control loop. So, in a control loop, you have the physical process, your furnace that's running on and you have some sensors, namely the temperature sensor and they feed back through the control system to the actuator and the actuators turn stuff on and off and keep everything in balance wherever the set point is set. So, in the control system, this is just another way to look at a control system. Here we have all the same processes and you've measured the actual set point is called the process variable. The observed temperature in the house right now is called the measure variable and it feeds back and then the little red X over there, we make decisions on whether or not we should kick things on and kick things off. So, if you go up to large scale, then you can't really fit all the logic that needs to run a whole big chemical plant inside of a little thermostat. So, we stick them in larger boxes, we call these programmable logic controllers. And so, they're just large specialized computers that sit there and run a whole bunch of logic that keeps your plant humming along smoothly. So, in general, the internals of PLC, you have a whole bunch of sensors that are plugged into it and one time per scan cycle, it takes all the readings from all those sensors and copies it into the input memory. Then it runs a whole bunch of logic based on that and then produces a bunch of outputs and those outputs then are turned around and run all the actuators in it. So, the PLC is really the brains of the control that are running it. So, most PLCs are still programmed graphically. This comes back from the day when everybody still ran processes from peg boards. You would actually have your input plugged in the plug, run over there, stick it in and block, take it to the output and twiggle it around. And so, we did that graphically and it's still programmed that way. So, most of the time as a computer science you scream no, just give me a real language. So, the most common control algorithm that's out there is PID, proportional integral on derivative. And this is just a complex set of equations that run to try and keep the process within certain limits. So, outside the PLC, you have all the PLC and the wires running all out into plants. So, those are usually cabled up and run into big wiring closets and these are mostly analog things. So, they do 4 to 20 milliamps, 3 to 10 volts, etc. So, this is usually not IP based technology. So, even though the PLC has all this logic and everything the engineer can throw into it, then we still need humans. So, a good friend of mine that works in nuclear says that any chemical plant you build, when you first kick it on, is it an imminent state of failure and it stays that way forever. And so, things go wrong all the time in plants. So, anybody that's ever worked in it knows that it doesn't just hum along batch after batch like a computer program. And so, the operator has to sit there and see all the alarms coming over there and figure out what to do with the alarms and shut things down and generally keep things in control. So, specifically for the stock, it is extremely important to understand what is the difference between IT hacking and OT hacking. Because what we are going to do here, it's not IT hacking, it's OT hacking operational technologies. So, to explain the thing, so this is the scenario of the stocks net. So, you see the centrifuge at the end. So, there is a system who has figured it all out for you. There are linkage of the data between cyber assets. There are workflows. And then we have the infected PLC which prevents the operator from observing the real state of the process. So, the centrifuges were breaking, the operator was not aware about that. So, after the attack was discovered, the net's admin will start figuring out how did it happen? What interrupted the flow? Was it data integrity? Was it DOS? What kind of integrity? Was it packet injection? None of that has any sense to the operator. He doesn't give a shit. Only what he says is like, I am not controlling the process. And this is what is important to understand that in OT security, the security properties which you need to observe are observability and controllability. You need to be able to observe or measure the process in order to be able to control it. If you are not measuring, if you are not observing the process, you are not controlling it. So, you are not having your process. So, an attacker that wants to do something behind, beyond simple mayhem will want to reliably control the process. So, if you run over there and you hack into a process and say, ooh, there is a burner underneath this tank. I will crank up the burner and crank it all the way up and bad things will happen. Well, this isn't actually what happens. So, usually what you are going to do is just exercise the shutdown logic of the process. Once you hook everything in the process together, they are actually related in the physics relationship. So, if you crank up the temperature, you are also going to change the pressure and the flows and everything else that is going to happen downstream. So, oops, I have lost my spot now. So, in order to actually attack the process, you need to remain in control of the process and you need to work in terms of control theory and not in terms of hacker who like SQL injections, process scripting and ROM. So, when the attack transforms in the OT domain, the attacker and the operator, they compete in for the controls over the process and the attacker wants to win. So, the security properties in the IT domain are actually controllability, observability and operability. And operability here stays something like we will say availability which describes the state of the process. So, just to remember, CIA is it for information security? CO2 for the process control security. And I know that the guys who grew up with the CIA state of mind, they will hate it, but just shake it off. So, when the attacker goes in and hits the process, he can take one of a few approaches. So, he can either take control of the process and reliably control it through the entire life cycle of the attack. And this is what we'll talk about later. He can take control of the process and control it into a state that then puts it in the, puts an intimate failure and just kind of let it go and let bad things happen. And that's what I talked about a few days ago at DEF CON. Or he can make the process unusable by just simply messing with the controls. And I'm going to talk about a little bit about that right now. So, let's consider a car and a driver. So, the attacker is hacked into the car and he's got control of the brakes. And so, if he comes over there and he grabs the left front brakes, then the driver is going to compensate and go to the left. Well, the attacker is just going to let go of the left brake and grab the right brake. And he's going to compensate back the other way until they're going back and forth and back and forth trying to keep the car under control. And so, since the attacker is a computer, then the attacker will always be able to anticipate and win this one. So, we call these actually multi-adaptive algorithms. And so, in the case of the car, the human is actually called the hidden actor in the process. And so, any subset of a process can be modeled as a hidden actor. So, if you have a refining phase or whatever else, you can consider that the hidden actor and apply multi-feedback or multi-adaptive algorithms to the feedback loops to just try and grab the feedback loops and run them back and forth and try to get things to run out of control. And so, this is actually a single set of algorithms that's based on the algorithms we use to automatically tune process control variable loops. Except we're applying them to remove the variation to add more and more variation over time. And so, you can grab those and apply them to a car or you can apply them to a boiler. And they work just as well either way. All right, so, we're ready to start. So, whenever you want, like, before you even start hacking something, you need something like a playground. You need to learn, like, the object. How do I hack that object? And also, you need something where you can test your attack. So, it's actually really possible to buy a chemical plant. No doubt, the link. The problem is that plants are extremely expensive. Also, you cannot relocate them easily. And you need more than two people to run them. Moreover, if you will hack it to the desk, you will need more money to fix it. So, hacking the real plant is actually not sustainable. And actually, in the industry, the chemical processing industry or, like, in R&D, it's more common to use, actually, the model of the processes. So, this is realistic. In this research, we've been using the realistic model of vanilla citate plant. It's a plant which produces a commodity chemical vanilla citate, which is then used to produce chemical adhesives, plastic, and so on. So, and actually, like, with this talk, we also release two models of chemical plants which were transformed into the framework for cyber physical exercises. We actually wanted to show you quickly how it looks like. Oh, we are good. So, this is a MATLAB, and this is how much far we can get to the open source. So, modeling is extremely expensive. All, all chemical, like, all processing industry models are processes, but it's a proprietary information. Yeah. So, um, yeah, um, so. Oh, no. Um, the problem is, uh, so we've been the simulant model of the vanilla citate plant. And what you see on the, uh, what size is it? It is on the left side. It's actually the same set points into the process. So, if we will kick in a little bit inside. So, this is the plant. This is will be, uh, this is the source code which actually schedules all the routine of this support in CCC processes. I have something for you. I look. How are they doing? Oh, it is not easy to get accepted as speakers at DEF CON. And you two have accomplished this. Congratulations. And thank all of you for staying, you know, late and seeing them. So, give yourselves a round of applause. Thank you. Who's the DEF CON? Oh, good job. Now, I just want to see you continue. Um, yeah. So, for example, um, the advantage of the simulant, of the simulant models is that they're easy to understand and it's extremely easy to add attacks with just few mouse clicks. So, for example, if I go here, here we integrated the blocks where we can, for example, model different attacks on the controller signals. And, uh, here's a different parameter. We can select the different attacks and so on. And here if we will go one layer inside, uh, uh, this is the controller with which handles like you, if you can see this is the, uh, attack values and here's fake signals and so on. It just to give you a feeling like what we've been working in and we've built all of this. Um, so, if we will go one layer here. So, this is the entire control structure of the plant with all of the controllers, uh, transmitters, transmitters are sensors. And this is the controllers and you can see there is a lot of parameters inside and everything what we have been introducing you in this beginning, the attacker cannot hack the process. If he doesn't understand the principle of control loops, how do you tune them and so on? Um, yeah, so that was just quick demo, just to give you a feeling of what we've been working for with. Okay, the stages of cyber physical attacks. So just like a buffer overflow needs shell code, uh, then when you're attacking a process you need a final payload, uh, that you deliver to the process. And this, uh, carries a set of instructions that are going to be carried out, uh, on the target process. And final payloads are always bespoke. I mean you can't take the final payload for one, uh, for one vinyl acetate plant and then play it on another vinyl acetate plant. But in general, uh, attackers go through, uh, several stages of, uh, of hacking. And so an attacker that's remotely attacking a process isn't immediately gifted with complete knowledge of the process and all the things he has to manipulate, uh, before he gets to deliver this final payload to the process. So, uh, in general you run through the attacks, uh, attacks in stages but your knowledge is never complete. So once you first get into a process you have the fog of war, you know, you don't know anything about it. And then you start figuring out stuff and you move through the stages of them, um, and then you have to circle back and say like, oh well, I need to know more about the process before I can start controlling the process and, uh, um, and, uh, continue on from there. So the first stage of, uh, hacking, uh, process is access. So you have the guy, uh, in the, uh, in the coffee shop in France and he's running over and hacking in the process. And in general, um, Skate to Networks, uh, have a traditional IT network where you get in, you know, just send a, send your favorite flash out exploit to some clueless user, have them click onto it, onto the, uh, uh, onto the Skate to Network, or onto the business network done. So generally process control networks have, uh, have firewalls and additional protections, uh, away from the business network. So there you've got to get, uh, get across anti-virus, uh, anti-virus is database links, patch management systems, et cetera, and into the process control network. But once you're into the process control network, then you stop having to use hacker tools. Cause most of the things in the process control network will just respond to any properly formatted, uh, command. So if you take a Modbus controller and use a, hey, go turn that pump on and you throw at the Modbus controller, it just will just, uh, take that command and then run on it. So you can move freely about the, the, the Skate to Network, um, with, with most impunity once you get inside of it. And so if you don't have a whole bunch of stuff already all stuck together, then now there's people that will, uh, sell you, uh, Skate to Exploit packs. So you can just run out into the network, apply a bunch of credit card to the problem, you're all done, you have a bunch of exploits for the Skate to Network. Um, so there is an alternative approach, the approach that, uh, was taken by Aaron Leveret, um, using, uh, using Showdown. So one of the things you can do nowadays if you don't really care who you're hacking into is you can just run out to Showdown and say show me all of the industrial controllers out on the internet, there's way more than there should be. Um, go grab a bunch of exploits, go pop, pop, pop, pop, pop, pop, and then see where they lead. Um, so, uh, a third approach you can get is that, uh, um, sensors now are getting smarter and smarter. They're getting IP stacks, they have a little CPUs, um, and, uh, and operating systems, and, uh, are all getting to be part of the Internet of Things, uh, uh, thing. So if you looked at my, uh, presentation from, um, from, uh, Espor in 2014, I described how to take a Skate to attack and miniaturize it down all the way it will fit, fit into the middle of one of the sensors. You just have to get the, uh, the sensor into the middle of the process and it can unpack itself and, uh, and keep on going. So after that, you have, uh, uh, discovery. Okay, so you are in. So how much do you know about the plant? Okay. Well, I am in some plant. Uh, you really need to understand lots of things about it. So do you know what stripper is? No. It's not exotic dancer. It's a stripping column. So you really need to understand the equipment when you try to hack a plant. So in general, that I can need to know, uh, this much of information about the plant, what and how the process is producing. So even if it's when you elicit a plant, the actual chemistry and kinetics of the process are proprietary. How the process is controlled and wired. So it means where is the location of the control valves? How the control loops are tuned? What is the control strategy applied? How the plant is built in while? Like basically how the sensors connected to the PLCs. And very important, what are the physical constraints of the plants? This stage of the attack starts very long time in advance with old-fashioned SPO-NASA reconnaissance attacks. And for example, for most, many information you will need also to hack into the third parties. Like for example, most of the equipment is designed by third parties and chemistry is developed by third parties. And actually the attack has understood the necessity of the stage long time ago. Because the SPO-NASA attacks again, the industries have started long time ago with the samples dated to 2003. And just to quote you, the description of one of the APTs is that, the goal of the attack has appeared to be collect intellectual property such as design documents, formulas and then manufacture in processes. So the attack knows what to do. So they will be looking something like for this, something like looks like chemical formulas, piping in instrumentation diagrams, instrumentation list, wiring diagrams. And after the attack figure is out, like he will understand like how the plant is building operates. He will start thinking, okay, what can I do to you to cause persistent economic damage? So we'll, at this stage, you'll already understand how vinyl acetate plant works. So in generally the easiest way to cause economic damage would be to destroy the pipe which carries the final product into the vessels. That is easy, works, but easily detectable, easily fixable. So it's not persistent. The rest of the plant can be divided into two parts, reactions and refinement. Refinement is a large part of the process. It's just approximately one kilometer long. So the attacker has a lot of opportunities to mess up with the things, but the operator also has a lot of opportunities to notice and to correct things. Moreover, you can always recycle product back and to try to refine it again. In contrast, if you will mess up with the reaction process in the reactor, you reliably produce less product. So that is a good scenario for the persistent economic damage. And important to understand that even this simple analysis is not possible without input of the experts. So after, so we resolve to attack in this, in our attack scenario, we resolve to attack the reactor unit. Then the attacker has to start, needs to figure out how the plant is building wired. So this is one of the most time-consuming and difficult part of the skater hacking. So the hacker has to figure out the relationship between tags in the PLC equipment and how that all looks on the diagrams. So most of the processes operate on their points, logic abstract layer. So everything what can be measured, basically all sensors and set, so actuators. Those are points. The plant may have 10,000 points. Where do they go? What do they do? So the attacker have to perform all of this mapping. It's a lot of work and interestingly enough, I don't remember the year, but the Havoc Smolvech has already exhibited their first attempts of the attackers to actually map and mind the equipment in the field. So the attackers are already this far. So in this stage we already could figure out all of the mapping between equipment and controllers. So we are back to the vanilla state plant. We need to figure out the location of the control loops. Once we have that, the attacker needs to understand how those control loops are tuned. This is extremely important. The good place to go for that is an instrument engineering applications because they will have everything what you need. I grabbed the screenshot from the internet. So for example you will be looking for something like, well that's a pressure transmitter and it serves the reactor control loop. We have here this will be the tag. The next we have it's actually a Kogawa. There is also a model number of the equipment. There are all the parameters how the instrument is scaled and how it is used within the process. And what is also very interesting we have also has a user who is allowed to modify this equipment. So we now know which account details we need to obtain. So with the, what ELSA attack needs to do is flows. In chemical plants the things do not nicely flow from left to right. They flow all kinds of direction. So for example in this our case in vanilla state so I set the case it flows from the from tank to the reactor section and to the refinement section. It's too bad for us because we will need to operate the valve and that means that we will have to also watch the process in the refinement section. We would not want like to do that because we need to invade more devices but well. So this is the attack I will need to, we'll find this information on the flow diagrams. So at this stage we already know everything about the control configuration and location of the control valves. So in the contents of the MATLAB model those control, the controllers are called as variables X and V. So because we later will be using it on one table. What is very difficult to understand for the IT hackers is that obtaining control is not being in control. So first of all the obtained control might not be useful to attack goal and secondly the attacker might not be able to control obtained control. And what does it mean? With that we go into the control stage of the attack. So till now the attacker was discovering the static information of the process. But physical processes have their dynamic so they change over the time and if you apply any input to it it will change. So that attack I need to understand how much he can control the process. What is his, how much he can control the process? So as I mentioned together earlier once you hook all the stuff, all the stuff together into a process they're related to each other with the physics of the process. So if we adjust a valve over here what happens to everything else downstream. So adjusting the temperature can also adjust the pressure and the flow and all the downstream effects need to be taken to account. And so one of the things we need to understand is how much of the process can we change before all the alarms and automatic shutdown routines kick in and start shutting down the process. So this is the example to illustrate you that concept. So in the red, in the red square you have the plot so I just adjusted some valve I just changed something. As you can see everything downstream has changed and it changed in different scales. The response has different shape and what is more interesting for example in one case something goes up and something goes down. This is the responses of the process to the manipulations cannot be predicted by the attack. By the attacker. And this is what he needs to figure out. And for example as you can see like why things goes different in different control loops. Not only because of the physics of the process but also that because there are millions of parameters which influence the process response. So this is a control loop which we wouldn't show before. So there are many parameters at each stage we will be influencing the response of the process to the manipulation. And the attacker needs to take into account all of them. What is more important when the attacker changes something or absorbs the input he cannot never understand whether the response is a response to the attack or it's because of the some configuration of the process. So just to give an example in my earlier works when I've been designing exploits on processes I've been working with I've been considering several of those parameters and I have to really encode into my exploit the ways how I will be dealing with them. So one of the most difficult parts of hacking continuous processes is process non-linearity. Most of the physical processes are non-linear. What does it mean? In short it means that the response of the process is not proportional to the input into the process. For example if the process is heated how the process behave when being heated from hundred forty degrees to hundred fifty degrees is completely different when it further heated to hundred and sixty degrees and if you never model the behavior of the process in that range you don't know how the process will behave. So the behavior of processes is known to controllers as well to the extent of the modeling. So if the control algorithm has never been tested and designed to deal with the process in that control in that range it will not be able to control it. The problem of the attacker he will move the process outside of the optimal envelope. So he have to deal with all of those strange behaviors of the processes and it's not easy. So just to give you an example what is non-linearity you see it's just simple response and it is reactor exit temperature. You change something and the response is definitely non-linear and all of those spikes are extremely annoying to the attacker because they cause alarms because it just hits some like maximum values. Another process which another problem like one of the issues which we have to deal while packing this particular process was that this was a very badly configured plan and all the controls were purely controlled. So whenever we've been trying to change something to our benefits many control loops had a ringing effect and if you will see this like look at the blue circle you can see well but that's like fluctuations just kind of tiny little like why do you even bother. The problem is that that tiny little fluctuation in one control loop cause enormous effect impact on the other control loops. So and unfortunately most of the control loops were causing alarms so we could not really stay like and as a raider because we did not want to cause any alarms so it took us around a couple of weeks to find the way how to deal with that. So in general we apply two types of attacks step attacks and periodic attacks. Step attacks is when you bring the process to time state and leave it there and periodic attacks when you attack the process let it recover attack the process again let it recover and so on. So we have to test all of those control loops which we've been we've identified in the reactor section we've tested them for multiple of parameters and actually at the end this was our mental model of the dynamic behavior of the process. Although I do want to know whether you measure yourself at five foot three with or without your high heels on. So that we've been like yeah like. So obviously this since this was our first attempt to you see like this is the first public talk on the complete attack from start to end so obviously this was the first attempt to understand the dynamic behavior of the process so we were not obviously optimal so we're now trying to find the way to optimize it so it would fit in a small payload like exploitable payload. So the outcome of the control stage that the attacker will need to classify the available control loops like how they useful for him how well he can control them so we basically find those factors which would be like for example we use the sense sensitivity of the control loops so highly sensitive loops are not reliably controllable so you probably don't want to use them in your final payload. Also another outcome of the control stages that alarm preparation propagation you want to know what are the marginal parameters of the attack which still do not cause the alarms. So now we know so we now know what we can control to which extend and we can actually start designing the real attacks and with that we transition to the damage stage. So in the damage stage you're trying to figure out okay I know all the stuff I can do about the process what am I actually going to do about it. So the damage stage is actually the least familiar stage to IT hackers so there's a lot of good starting points when you go and start looking at the damage stage. So one of the good basic principles is that things happened in the real world by accident you can probably make them happen by malicious intent. So you can go to all the places where these are recorded to government databases, the plant zone databases you know like the chemical safety board etc and read about all the all the things that have actually gone horribly wrong. And they're out there on the the internet for you to go and read. So the target plant may not have been designed in a hacker friendly way. So you may really really want to know what this particular value is but there may be no sensor there or no sensor close to it. And the information may be spread out across the process and the control loops could be designed to control different parameters than the ones you want to actually reach in and control is like oh I would really like to change the pressure here but there's no pressure control loop and so I won't do I won't do it. So in this case in this case so since we want to read so we decided to mess up with the production of vanilla citates so we won't actually make the plant produce less. To measure the impact of our attacks we need to actually measure how much molecules or the vanilla citates are there in the reactor exit. The concentration of chemicals is usually measured by the analyzers. There are four analyzers in the vanilla citate plant in this plant and none of them in the reactor exit. There are only flow and temperature. In order to compute the volume we need actually the concentration and the flow. The only available place where we could get those numbers is in the exit of the plant but measuring there is too late because that values are available like in eight hours. So then you will have to attack something in eight hours until you can measure it. It's certainly not sustainable. So we've been like the like we've been actually in desperation because it seems like how do we proceed? We don't have numbers we can't evaluate how effective our attacks could be. And this is at that point I remember the presentation of Jason actually from another conference when he've been talking about proxy sensors. So the concept behind that is following. So in general there are two answers to like to measuring something. It's technician and engineer, engineer answer. Technician will tell you that something is decreasing or increasing and engineer will tell you by how much. So by using proxy sensors you actually can obtain at least a technician answer. And a proxy sensor is something that changes with respect to another parameter which you are interested in. So in our case the proxy sensor is temperature in the reactor exit. So if temperature is decreasing it means that less reaction is happening in the reactor. So with that at least we can see whether our attack is effective or not. That was already a good start. But it still does not allow us to compare the effectiveness between different attacks. So we desperately needed the engineering answer. And this is where we started to think further. Actually process algorithms are extremely complex. And there are a lot of optimization applications running in each plans will try to optimize like how the performance of the plans. And there will be a lot of internal like intermediate computations. So we tried to decide to give a shot like maybe there will be some numbers computed in between which can be useful to us. So we started looking into the code and we knew that we need to look into some to find some ugly differential equation. And eventually we found the piece of the code which seems like well it seems like they're computing what we need. The problem was that the numbers which were these numbers, intermediate numbers were extremely weird. We could not do really anything with them. They did not sum up to 0 to 1 to 100. They are just too small. So we did not know what to do with that. But still the god feeling told us the right number. So we spent actually two weeks trying to play with those numbers until we actually could figure it out. So eventually we could compute the concentration of the vanilla state in the reactor exit. And with that we actually can finally measure the effect of our attacks in dollars. So the outcome of the damage state is a portfolio of attacks which you classify with respect to the damage potential. And then you just apply those attacks at the opportune time. So you would think that this is all and we already done because we have our attacks we can code it so let's just like go and hack it. But that's not all. So the final stage of hacking a control system is cleanup. So in most IT scenarios you go hack into something it's like oh I've got the databases I got all this stuff and when I'm done I erase all the logs and I'm just gone. I was I was remember really there. But in hacking process control systems if you leave the big smoking plant then somebody will investigate the big smoking plant. And so you have to convince the people that are actually investigating this that the big smoking plant was the result of operator error, misconfigure equipment all of those types of things. So having a human in the in the control loop turns this from just a purely cyber system into a socio-technical system. And so since there's real people that are out there playing with the process and real people that are going to analyze the process when it's when it's all bad and gone then we can go and attack the system. So creating a forensic footprint. So if you come over there and you create a persistent problem in the plant and the production is going down somebody's going to go notice and say like why are we not making money and to go and try and fix it. And so but you can do things like timing the attacks when a particular employee is on shift and then after a while they say like oh whenever bill is on shift then things go horribly wrong and we don't make money so let's go beat bill a little bit. He must be the problem. And so the employee can end up getting investigated instead of the process. So in this particular case we're going to show you one where we picked several ways that the temperature can be increased and so our plan is we'll just wait for the next scheduled instrument calibration. We'll perform the first attack making it inefficient and then we'll wait for the maintenance guy to be yelled at and say like oh you tuned it wrong please go tune it again and then we'll just perform the next one and then he'll go back and tune it again. We'll just pull forward the next one and the next one and the next one. So here we see four different attacks on the reactor temperature and the results and you can see they're very different. So if the lines in there and the changes are timed with the recalibrations then it really just looks like the guy's messing up and not getting the calibrations right. So if eventually they will start thinking okay probably the reactor is not performing well. They will call it actually for a chemical forensics guys to investigate. It is not possible actually to see into the reactor. So how the forensic guys investigated the compute specifics. They have a matrix to compute in which you can understand what is happening with the reactor. So the attacker has to understand how they do it in order to fool them. So this plot is just like some of the metrics which we've been computing for the reactor performance and although they look similar they all tell completely different things strangely. So then just change attack patterns according to the debugging efforts of the plant personnel and they will just keep thinking that they're not doing a good job. So in general we go through all the stages. We plan all this stuff out some of which can be totally offline and some of which is interactive with the process. But we take all this stuff and we wrap it up into what we call the final payload and that's what we're going to stick inside of our attack and send into the process and so then if all things go well then our attack works and we make lots of money and everybody is happy. Well I guess everybody on the attacker side is happy. There is a big problem right now. So like the plants really go through tremendous modernization. So most of this really old sensors are now substituted with a smart sensor which speaks IP and they're now part of the internet of things. The plants are now connected to the internet and internet can now talk to the plants. So the attackers are already in the plants and although like the argument like why we cannot improve the state of the security of the industrial control system is that like well we don't see many of those attacks. The problem of that according to the recent laws all of those accidents are not like the classified information and they're not allowed to be public but it does not mean that we have to lean on our chairs and do nothing. Cyber physical exploitation can be actually studied and we should do it. So by understanding what the attacker needs to do and how we can actually make exploitation harder, wait for the attacker where he has to go and actually create really those network monitoring solutions which works because we know what to look for and for that that was for the defenders but food for thought if you're an attacker. So most of the things when you actually get into breaking a process are about cost and so one of the reasons that we study cyber physical attack is because all the other people that are probably out making money right now are studying cyber physical attack and so for most attackers then the cost of an attack can quickly exceed the cost of the damage or the cost of the benefit they get there. So if you actually make it harder and the attacker has to hack into a large number of devices or suppress a large number of alarms and spoof a large amount of data his cost go way up and his testing goes way up and his chance of failure goes way up. And so each process is unique but if we look at a wide range of scenarios then we can start seeing all the patterns. On the other hand the opposite is true too as we figure out how to deal with all the uniqueness and all of the processes probably attackers are going to get better at their job too and figure out algorithms that can be applied a lot of different ways and so skated payloads for something that looks like a meta-supply are probably only a matter of time. So I guess in the end if you want to be the evil James Bond villain and make millions of dollars then start building your evil layer then hacking chemical and process control systems is probably not a bad place to start but it is a very complex and it's a very challenging field of hacking. So anybody that wants to come and join us and study hacking processes and physical damage then they should come and do so. There is the SES Village setup at the DEF CON so just go and check most stuff there and you can actually get some trainings there running right now as well so with that that's the end of our talk.