 Hi, I'm Montay, security architect, Foxguard Solutions, we do industrial control system security stuff. And this is what I think my job is like. No? Yeah? And your job looks like this? Everybody. And this is what my mom thinks my job is like. And this is what other people think my job is like. This is what I wish my job was like. Look carefully. And this is what my job is really like. So do some security research, look at new attacks, do some training classes, lots of conference talks, new product. And look at industry requirements in industrial control systems, right? Where bits meet atoms, where things happen. So let's get started. We're going to talk about things in light of... Anybody here working in industrial control systems? Oh yeah, we got a handful. Anybody heard the term industrial control systems before? A few more. Beautiful. Well now you're an expert. Industrial control systems is the part of security that really matters. Is where bits meet atoms, things move. This is electricity and manufacturing and your water and your power. This is where your iPhones come from. And we're going to take a look at some tools. So since I am thinking of things in terms of industrial control systems. Let's get the mic. Industrial control systems. If you're going to look at this as an industrial control system, what are the mechanics involved? It's like, well, it looks for some button pushes. It does some calculations and it displays something on the screen. That's it's I-O, right? Industrial control, reading things, thinking about things and doing things. Fun enough? As an industrial control system, what does this thing do? Do what? It opens the box. What else? Industrial control system. What? Makes holes. Yeah, but think about the operation. Bits, atoms, movement. What's it doing? Alright, so this is a miniature industrial control system that's monitoring the rate of the spin. That's one of the things that's actually controlling with the pulse width modulation output. What else is it doing? Input. What kind of input? Yeah, so there's a variable resistor here in this trigger and it's reading that value. It's reading analog value, making some calculations to the control system. Do we see anything else? Light. So we have a light on the front. It gets timed. Look, no power. Alright, we got light on the front. What about on the sides here? Yeah, so here's a display. This is a voltage display, so it's measuring voltage. It's also measuring temperature of this battery system and it's measuring temperature of MOSFETs inside. So as far as a very small representation of an industrial control system, this is it. It's measuring different kinds of inputs. It's controlling different kinds of outputs, alright? Which of these two systems do you think is more powerful? Thoughts? Comments? Yes? No? Now which one do you think is more powerful? Yeah, well, so a little bit of fun, but actually the drill is a more powerful control system. So, excuse me? Physically. Also, look, the calculator clocks are 8 to 6 megahertz and the drill, it's 8 megahertz. Besides that, the processor in the calculator takes several clock instructions, takes several clock cycles to do an instruction, so 7, 19, 14, 15, this is just sort of a random sampling of instructions. This drill basically does every operation in one instruction cycle. So computationally, it's a lot more powerful than the calculator as well. Besides it's additional functionality of sort of dedicated pulse with modulation channels and it's ability to read analog input. So if you start looking at, you know, the actual power of the system, the drill is a lot more complex system and a lot more one than I expected, right? This is, it's a battery powered drill. Who expects a computer in the middle of there? So if you were going to attack a control system, let's say this one in particular, you might look to see what's inside of it first. This is a pretty standard drill. It has a, let's see, we've got a PC board and some power connectors and a motor, a transmission. If we look at the, we look at the circuit board and we look up these MOSFETs, by the way, that controls the power to the motor that makes the drill run. They're rated at 202 amps continuous and if you do anything with electronics, that's a fair amount of power. They will also support 800 amps pulse. So, yeah, it's pretty impressive. You might also need a little help reverse engineering the system as a whole. If you're going to look at software, because the hardware provides input and it depends upon the way the drill is set up and the way the circuit board is set up. So you take a look around. I found an AT Mega 32 pin and when I first saw this 32 pin microcontroller, I'm like, you know, they make like eight pin lines and why in the world are they putting this thing in here? But if you go through and trace out all of those circuits, basically almost all of them, almost all the pins are in use. It's running front LEDs and power LEDs and two MOSFETs, MOSFET temperature, battery temperature, voltage, trigger, calm, enable pins. So it's using a lot more of the capability than you expected even in the simple device. It's using a lot of computational resources. All right, now for our little review before we... Ooh, calculator parts. Part of a review before we continue. All right, audience participation, what's RAM? Random Access Memory. Beautiful. You read right, you turn it off, it goes away. What is ROM? Read only memory. All right, so let's talk about early ROM. Early ROM is Masked ROM and it's manufactured that way at the factory. So you say, hey, I want a bunch of ROMs that have this program on here and you pay them my $100,000 and they'll set up the mask and they will make those chips with that program and they will send them to you. You have to buy a big lot to make this profitable. And now you have, I don't know, 10,000 of these chips programmed just the way you want them. What do you do with those chips if there was a bug in your software? Not bad. Put them in a landfill. Sell them and don't tell anybody. Sell them and don't tell anybody? You are my head of marketing. Yeah, yeah, yeah, or I like to say, throw them in the ocean. Because you can only write to them once, they're made at the factory and now you have to call the factory again and say, hey, for another $100,000 can you send me up another mask and run me another batch of these? So we decided that that didn't work very well and we come up with things like prom. So what's prom stand for? So this is programmable read-only memory. So you get these chips that are typically, let's say, one-time programmable. You get the chip, you can put your piece of code on it and then you can put it in your device. If that chip has vulnerabilities or problems in the software, what do you do with that chip? Yes, you throw in the landfill. But you only have the one because the next chip you get off the shelf, you can program and put it in your device, right? So that works a lot quicker and easier and it's more cost-effective. So manufacturers like that. But you go on to the next level. EEPROM, what's EEPROM? Yeah, so this is a raceable programmable read-only memory. And typically they were UV-erasable. So if you look here, this has a little window in it. A UV light will erase it, set it all back to ones and you can change some of those to zeros and end up with a program. Now, if there is something wrong with this chip, what can we do with it? Yeah, you can pull it out, you can shine some light on it and reprogram it and put it back in. You don't even have to throw anything away. But you did have to pull it out and program it. So EEPROM, what's EEPROM? Yeah, electrically-erasable programmable read-only memory. So now you can program this thing in the circuit. You don't have to pull it out. You don't have to shine UV light on it, you don't have to put it in your nails or something. And if you have a problem, you can fix it in the device. So that's quite a bit easier. Now you think as a manufacturer, that's great. If we screwed something up, we need to update something. We're all good. We can program it back in place. All right, now I'm dependent on you all. What does Flash stand for? What? You all are a quiet group. No, Flash doesn't stand for anything. It's a marketing term. So Flash is basically a version of EEPROM. Tends to be larger. You write it in larger blocks. It will sustain fewer right cycles before it gets destroyed. But it's basically a marketing term. And that's how we end up. That's a short version of how we end up with devices that you can program sort of by default. It saves you money. It saves you time. So that's a review. For the particular microcontroller in this drill, we see it has some EEPROM, which is designed to be written in smaller blocks and more time. More time. Some RAM, some Flash, and end system programming. We don't have to take it out and clip some clip on it and hook some programmer up to it. It's much easier to set up. You probably buy your hardware. You get it shipped to you. And then back at your plant, you can blow on the latest firmware when you get the device. Much easier. That allows for the updates that you may have changed or made to your firmware, your software. And in some cases, maybe account for some minor manufacturing difficulties. You might be able to adjust for them. Maybe in firmware. So we take it apart. And we're looking at how to manipulate it. And it would be great if we could find an SPI port, typically six pins. And what do we see here? Six pins. And a matter of fact, half the work is already done for us because there is a reset, a clock, and our 3.3 volts. That's a point to something else. So three of our six pins are already labeled. We only have to figure out the other three and we're in. Well, excuse me. Yeah, we need a couple. We need Mimo and MoMe. But if you turn the circuit board over, now you have MoMe and Mimo and Ground. So they labeled the other three for you. That's great. That's awesome. You need a programmer. I was in a hurry. I built one out of a teensy eye head laying around and put a 3D printed case and got some free software. But if you want to buy them, they're available in all your favorite sources across on the... They're great, $3.36. Ship to your door on the slow boat. So I pulled the software off of this device and did a little reverse engineering, maybe a little disassembly. So you end up with assembly code, all ones and zeros. It's hard to figure out, but you're going to disassemble it somehow to make it a little easier. There are a couple of free ones. AVR, OBJ, Dump. That's kind of cool. I read about this cool in RE-AVR, but it was kind of from a sketchy site from Russia and I didn't really want to install it on my computer. So I talked to my buddy and I said, can you install this software and re-run this file through it? And he was like, yeah, I'm like, sweet. I'm too paranoid for that stuff. And other commercial software. But now you end up with just sort of this side, right, the similar language instructions, which still may not mean a lot. And I worked through it quite a bit to try to figure out what these registers and what the instructions meant and what they did. This is just a brief section of it. So I made a few comments as I went along. I thought one of the interests, one of the reasons I just pulled this tiny section of code was this right here is setting up to write to the EE prom. And this is the EE prom right. So this drill has the capability of changing the EE prom, right? It can write things. It can keep track of things. I was going to ask my friend, who I was working with at the time, what do you think this thing is writing? What's it keeping up with? He says, oh, I know, I know. It's looking for warranty day plus one. Right. I'm like, maybe it is looking for, I don't know, usage time or something like that. But there's no real time clock. It doesn't get updated that way. I think it's doing calibration for the trigger, right? They can have slightly different ranges and so forth. So, oh, at this point, we are going to have a little bit of a live demo. I need a volunteer who would come up here and give me a hand. Welcome, sir. Do you have good health insurance? Let's just go. All right, let's just go. Okay, so if you would, hang on, we'll at least take the drill bit out for you. How about that? Yeah, you're standing on my line. You should stand right there. Oh, much better. Okay, can you hold this for me? Excellent, excellent. So, right, you buy them, you build them, whatever. We'll walk over this way just a little bit. We are going to... The only thing I did is I pulled those pins to the outside just to make this go a little quicker and easier. Which way is up? Turn it three times, it'll work. Yeah. You've done this before, haven't you? I can tell. All right, so it takes a line or two of code to do this, but since I can't type and talk, especially in front of people, we've got a couple of scripts. I am going to run hash comparison. So, what this is doing is it's pulling the existing firmware off the device, writing it to my hard drive, and then I'm comparing it to my known good file. And if you look at the bottom, you see it say it says files match. So, the firmware on this device hash matches my expectations. I believe this is a good device. I will be happy to use it. I also wrote some other code for this device. And now we are putting some other code on this device, and we will check the hash. If we receive the device in this state, we're reading the firmware off, we're running the hash on, comparing it to our known good, and it says those files differ. I mean, nice if it said it was malware, but it says those files differ. Oh, do you notice anything strange? There's a blinking red light along the side of the drill. Oh, that's suspicious right there. Let me tell you. Yeah. Okay, hang on. Yeah, so what is this? I thought I heard the comment up front. This is called, it has a name, it's called a Larson scanner. It's named for this guy, Glenn Larson, a TV producer and writer. He created various shows as you see. Buck Rogers in the 25th century. What are the last two? Night writer and what's the last one? Right? Right, so he's responsible for this, but what it really means is this is the start of AI, right? This is where bad AI comes from. Okay, now you have another chore to do. If you will pass me the drills for a second. It is now your job to take this microphone and hold it against the end of this drill. Okay. All right, microphone, drill, you got that? No, I hold the drill. You hold it. Yeah, you got that? All right, we figured it out. You good? Oh, you are good. All right, I like him. I'll keep him. All right, so pay attention. Hold it now. You ready? There we go. And another round of applause for my able helper. Thank you very much. All right, yeah. Darth Vader's theme, of course. What else do you play? All right. Oh, no, I don't have my Darth Vader theme t-shirt on. But let's talk about why I do something silly like this. Pretty impractical attack. This industrial control system is air gapped. There's no network connection, no financial consequences. It's an example of firmware in unexpected places. What I really want people in industrial control systems to take away is this drill is a computer running software. And if the drill is a computer running software, there are a lot of pieces in your important plant that are computers running software that you need to think about securing. Drill's probably not top on your list, but some of the other pieces of hardware in your device, media converters that you don't think of as computers running malware, could be. I mean, you could do interdiction, right? Any good intelligence agency can delay your package for a little while and add you a little special gift. But, you know, not really any financial consequences. The first one I bought looked like this, right? My budget's kind of small. I buy cheap stuff off eBay. The first one I got was broken, but the computers still work. But I noticed, you can't maybe you can see it, that these big MOSFETs, somebody had let the magic smoke out, right? They've been overheated, they've been destroyed. This is actually a thermal sensor, right? And even with that in place, it's looking for overheating. Somebody cracked these accidentally. So, you might have some financial consequences. You can blow up the drill, a couple hundred bucks. Well, lipo's aren't exactly stable. Occasionally, they catch. Who knows what, anybody know what this is? Yeah, Galaxy S7. This one, they're at a conference and somebody's laptop is on fire. And everybody's staring at it and going, oh, look at that. Man, that's bad, isn't it? If you're in the hobby of doing RC stuff, you generally charge lipo's either in a fireproof bag or something with cinder blocks around it, right? You know, so maybe if you could catch it on fire, there might be some consequences. But it's largely impractical. I did come across these things in my research, though. Research, five minutes on Google. Research these things. These two Bosch devices. These two Bosch devices. I'm trying. These two Bosch devices. One of them is called BT Exact and BT Angle Exact. Who knows what the BT stands for? All right, well, there goes part of our air gap. And if you will look at the highlighted text about it, the screwdrivers are locked when they are delivered and can be unlocked only by the access point Exact Connect. So you can't even use it till you connect it. The system can talk to and take commands from external devices like a PLC, which might be an important part of your control system. This device is going to communicate to your control system. By the way, why would you want to communicate with devices like wrenches and drills and... Maybe you wanted to... Sometimes to know where they are? What if you're making very high-tolerant stuff like vehicle suspensions on big trucks like a plant next to where I live does or turbine engines on a jet aircraft? Excuse me? Yeah, precision torque, right? You'll have a torque requirement. You need to make sure every nut is appropriately tightened and you might be checking battery life and RPM and usage. So there are some valid reasons to want to communicate with these tools. And I also saw this available at your favorite home construction store. I guess what this device comes with... All right, its own cell phone app. Because you need an app. I'm lost here. Where you can monitor the RPM, set the torques and it says it will keep track of the last place it saw the device for you, battery charge. And where do you think it stores this information? In the cloud. So now maybe you have some connection between your device and whatever. And we've seen the knowledge of service attacks blocking out parts of the internet from tons and tons of small devices. If you're an industrial control system, the network latency in messaging is very important. You might have maybe a 5 millisecond latency requirement because you are controlling a very large, dangerous piece of equipment and if you can't monitor it, feedback, change its position, see where it is every 5 milliseconds, things can go wrong. And if you're looking for a denial of service on your network, who looks at the drill first? Yeah, not even me, right? And it's not, again, it really isn't the drill. The drill is the extreme example, right? The drill is the fun part. You can see from a lot of different devices that you don't expect. It's firmware and computers in places that you don't expect. And finally, you have a chance to listen to my favorite rant. What is the difference between firmware and software? Yes, sir, in the back. Nothing! You win a prize! Dang, do I have a prize? I'll have to look for a prize. Basically, nothing. At one time, for most considerations, call it nothing. At one time, we might have said an important consideration was that firmware, for instance, you couldn't change it, right? We looked at, you know, the original ROMs, like mass ROMs and one-time programmable, you really couldn't change those. And so that might make a significant difference, sort of, for firmware in terms of security. But even think about that for a second. Let's say you have software on a device and it's software that you cannot change and you discover a vulnerability, right? Best case, you will always have that vulnerability. You can't fix it. Who believes software is perfect? You guys are way smarter than you look. So, I have a couple of... I have people that work in my company, a programming team, and I can come up to them and I say, hey, I've got two projects for you to work on. I have the software project and I have this firmware project. And what I need is you all to divide yourself out because I need all the perfect programmers over here on the firmware side and I want everybody else here on the software side. So after I have this talk, who ends up on this side? Nobody. Or, yeah, you, the liars, right? Either way, it's software. It has as many vulnerabilities or problems as anything else. So that's my rant. Never say firmware again, at least from a security perspective. There really isn't a significant difference there. It's software, it has vulnerabilities, it's written, it can damage things. People sometimes say, oh, firmware. Must be perfect. Can't change it. When we're done, we're going to go someplace else. No. And it's sort of led to my rule of thumb. I think this audience probably knows it, but industrial control systems not as much as folks working there. Is that, look, if it plugs in or if it has batteries and you bought it in the last five years, it's a computer, right? There's nothing that you could buy that plugs in or has batteries that isn't a computer. I was surprised seeing, you know, this is a battery and a switch, right? Not a more powerful computer. Oh, sorry, this is yours. Thank you very much for helping me. You're welcome. Or if not, you can give it to the guy who needs the prize in the back. All right, so it's a computer. It's a computer. And finally, my analysis after tearing down this tool is that it had very nice copper heat sinks. Copper conducts better than aluminum. I thought they were very fine. It has thermal sensors for the battery. It's looking for overload and overload on the MOSFETs. That's a good thing. The PC board has a nice conformal coating. That's a plastic kind of overspray that protects the board from harsh environments. So particularly in industrial control systems, those computers don't live in server rooms. They get bolted to things that shake and have dust and sand and are too hot. So something like, you know, this can avoid things like salt spray and dust and help protect it. I thought that was good. I thought it was good that they didn't, in some instances, you can change a fuse and the device and make the firmware right only. How much protection is that? Notice I said right only. What does that mean if I get this thing and I want to know if it's been modified somehow? I can't read it, right? So one of the things that we did was we pulled the firmware off to check that it was okay. How long would it take me to do this? You know, if I took it to the bathroom with me in a plant, 10 minutes, 5, 10, something like that, 10 years out, it takes as long as it takes, but the screws back in. So by not making the firmware right only, we can validate, verify that this piece of software running this important control system is still valid. I think that's a very good thing to be able to do that. We also have a group of people that help do patches and look up patches for industrial control systems. And I went to them and I'm like, has there been any new firmware release for this drill? And that's kind of what they do for a living and they're like, nope, can't find any. I'm like, okay, some things don't have patches. I'm not going to fault people for not delivering patches. I don't see anything incredibly wrong necessarily with it. But if I did have an ask for this manufacturer, it would be to publish the hash for the firmware that they've installed. Now, I'm taking my good firmware because I have a couple of these drills and that's why it came with, and I assumed that they matched that it was the right one. But to really be sure, I'd like to see a signed hash from the manufacturer. So that would be the one thing that they could really do better. Maybe execute signed only code, that would probably work, but the processing power and storage in this chip is pretty small, so that might be too much of an ask. But they'd at least publish the hash. And I also like that they make these headers easily available, somewhat easily available. I would almost like to see them on the outside because I'm more interested in being able to verify that my device is right. Remember we said somebody might capture it along the way, somebody might take it to the restroom, not this device in particular. There's a computer in this thing, and there are computers in these cameras and there's computers all over the place. And in an industrial plant, that might mean the difference between life and death. I want to make sure that they are, be able to check that they are okay. So if you make programming headers easily available, and often they are pretty easily available, this one didn't even have pins on it. That's fairly easy. A lot of them already have pins available in JTAG parts because they have to program them when they come out. All right. We have a couple more minutes. I want to tell you one more thing. This is mostly for my industrial control system audience, of which there are a few, is that, well, actually it's for both, how do we protect these things from a security point of view? What we talked about being able to validate the firmware. Now, let me plant one idea in your mind. This is a computer running software. So, ignoring the specifics of this device, tell me generically, how do you protect a computer running software? What tools do you have? Excuse me? Signing, if you could sign the code, right? Maybe you could sign code on it, might not have enough processing power, right? What else can we do? Antivirus, right? I might ask my vendor to provide antivirus. They might not be able to support it, they might ask, right? Who runs antivirus on this? At least one, right? I do accept this as my burner. Right, so you can run AV on these devices. So, if you can run AV, run it, and if you're buying, you know, 10,000 whatever devices it is for your plant, at least ask the manufacturer if you can run AV on it. They may not be able to provide it, but maybe. What else do we do to protect control, to protect computers running software? To protect the network, right? If this thing isn't connected, the other ones are, right? We use firewall rules, and firewall rules, and IDSs to protect the network. Anything else? Patches, right, that too. So, we're going to check to see if there are any known vulnerabilities, and we're going to patch these devices. So, the final take-aways are drills are fun, no, wait a minute, are that, everything's a computer running software. And, even if that's what you're faced with, no matter how bizarre they look, you have tools to help protect them. If you can't do exactly what you want, if you can't run AV on here, you want to be creative and do something else, like run firewall rules in your wireless, or so forth. You have to be a little more thoughtful about how to protect it. But, you have the opportunity to do that, be creative. With that, thank you very much. I'm, oh wait a minute, I took the charger apart too. Anybody, any idea what this is? Another microcontroller. Alright, well, I'm Monte. Thank you all very much. Catch me online somewhere. I'm here the rest of the week.