 Hey there. My name's James Arlen. Some people know me as Mercurial. Mike? I know. I'm talking on it. Is everybody hearing me? No. Hello? I'm not using that thing because you gotta suck on them like a rock star and I'm not a rock star today. Is it not working? Yeah, that's what I think. Is that better? Notting, smiling? Awesome. Standard disclaimer. Because it needs to be said, we're going to talk about some crazy stuff. Like I was saying, my name is James Arlen. Some of you know me as Mercurial. I'm an InfoSec geek. I've got a fair amount of experience. Almost an obscene amount in some areas. Operations in verticals such as utilities and finance. Which means I don't really understand why we do things. I understand how we make money from doing them. And I do some crazy hacker kind of stuff like game show host 9 p.m. track 4. And I'm still not an expert at anything. This is a talk not about SCADA, but about talking about SCADA. That feedback is going to drive me nuts for the next hour. There is some technical material, not a whole lot. You don't need to be an expert in anything in order to enjoy this. I am going to talk about Smart Grid. I'll talk about Stuxnet. And you're not going to end up being an expert when you walk out of today either. Anybody who tells you they can teach you how to be an expert in SCADA, anything in one talk needs to... So, about 2005, all of a sudden, the InfoSec industry noticed SCADA. That's a problem. They immediately started identifying it as a market. The simplest explanation is they ran out of markets. They'd done the Y2K stuff, they'd done the SOX 404 stuff, now what? This was before PCI came along and saved their bacon. But, you know, the simplest explanation is the easiest one. This is entirely about money. The regulators are starting to breathe heavy. And there was an opportunity for any of the major consulting firms to suddenly generate an entire cadre of experts in SCADA. Because, you know, a packet is a packet is a packet and we're just talking about packets, right? I mean, it's all just networking stuff. That's the only thing that's involved in SCADA is networks. Don't even get me started on the security religions, okay? Everybody a member of a security religion? Does everybody know the security religions? ISC squared, ISACA and SANS. Bow down to your ethics. At this point, I'm working in control system security. I'm responsible for something that they never should have left me responsible for. I spent as much time as I could pointing out these flawed responses. You know, when you're getting a crowd coming in from a Accenture or from APWC, and they're coming in and they're saying to you, hey, we have the expertise to solve your problem. IBM shows up and says we will solve your SCADA problems. A few dudes, a few dudettes, they showed up and they told me this. They said they knew how to solve the problems that I was having. They tied a nice little bow on it and they said very simply, you just need a few more blinky lights, a few more shiny things and everything will be fine. This is echoed today every time, because of the kinds of mailing lists I'm on, every time I log into my Gmail account, the little ad banner across the top invites me to buy ArcSight because it's the solution to my NERC-CIP woes. I told you we're going to talk about SCADA because there are entirely too many people talking about SCADA that are talking entirely out of their asses and we're going to fix that. Here's the long form. This language stuff is really, really important. Specificity is the kind of thing that engineers get off on and everybody who's involved in control systems is an engineer. The kind of like car people. If you use slightly the wrong term, they sort of giggle at you and send you back to the shade tree to learn how to be a mechanic. The information security profession is going to them and talking about how awesome the new synchromesh transmission from Shimano is. It's got ten speeds. It doesn't go well in your car. Not all SCADA is SCADA. Let's start with that. The media loves it. Here's what real SCADA looks like. It's big stuff. It's highly distributed systems. The kind of systems that end up being an ephemeral fog because of the size of the GR of you that's stretched across. It's power lines. It's pipelines. It's interconnected sensors and controls under central management. Get that one stuck in your head but good. SCADA is interconnected sensors and controls under central management. It's places where you need a bunch of math to happen. That's the key piece. You need a big honking server to do a bunch of extraordinarily complex math in the middle. That's supervisory control and data acquisition. A little bit of supervision at a gross level but mostly data acquisition to enable the math to be able to be completed. Control systems, on the other hand, are little things like chemical plants and manufacturing come. You find it in refineries. You find it in power generation. You find it in food. Lots of it in food. Donuts. It's a lot of individual capabilities that are in and of themselves atomic with some orchestration. That's control systems. It's programmable logic controllers which are so unbelievably awesome that they run ladder logic. Anybody remember taking ladder logic in CS101? Anybody ever seen it since? The three people that answered yes work in control systems. These things run at cycles per second and in some cases cycles per minute. Different kind of strange world. They get things bolted onto them to enable that cohesion amongst the different machines. There's this TV show that my kids and I love to watch called How It's All Made. It's the story of manufacturing. How the things that you take for granted are actually produced. It's insanely complex. The machine that folds the wrapper for a stick of bubble gum is unbelievable. That machine doesn't know a whole heck of a lot about the machine that folds the package for the bubble gum. Or the machine that slides the sticks of bubble gum into the package. Each one of those machines can operate alone. You shove paper into one of them and it spits out paper at the other end. The paper's nicely folded. It's depending on some coordination and some synchronization to pick up that piece of paper that it's about to eject and shove it into the machine that's got the bubble gum feeder attached to it. This is not terribly complex stuff. They work together towards a common process, but no manufacturing plant is a single machine. These are called industrial control systems or distributed control systems. These are smaller contained entities. You can wrap your arms around them from a geographic kind of perspective. When you try to wrap your arms around the power grid, you hug a continent. We're talking about things that fit into buildings. Even if they're really big buildings like nuclear power plants. This lack of understanding about those two things. When you start hearing the media going on and on and on about these SCADA systems and how they're going to create cyber cookies of cyber death, you've got to ask yourself, what the hell are they talking about? Because we know that SCADA is big ephemeral systems like pipelines. It doesn't matter really at one point whether you're talking about a control system or whether you're talking about true SCADA systems. The computers aren't the things that you're controlling. Fit this into your heads in the tightest possible way. There have been talks at DEF CON for years about, hey, I found this neat SCADA protocol thing and I figured out how to make it go boom. If you break the computer, you're broken a computer. The computer isn't the thing that's being controlled. This is just like every other kind of information security there is. The computers are not the reason that you're involved. If you work for a company, what does the company do? There's some widget machine in the corner that cranks and money falls out of one end, but that's not anything to do with the computers that you're trying to protect. When Edna falls into the reactant vessel, the process stops. The exact same thing happens when you break the computer, the process stops. The process stops all the time. It's a pain in the neck to clean up after. I mean, picture what a newsprint plant looks like. Everybody's seen those pictures with the huge reams of paper, slipping through all those things. Can you imagine rethreading one of those things? That would suck. But when Ed falls into it, tears his way through a piece of paper and goes in between two rollers and you got Ed pasted out the other end, you clean it up, you rethread the paper. You say, I'm sorry, to Ed's widow and you get on with the day. If you break into the computer and you manage to blow it up, the printing plant is going to end up with a big jam of paper stuck in it. Possibly some Ed paste too. So this is what the data looks like. Everybody knows how to use Wireshark, right? Anybody not know how to use Wireshark? I'm the only one who doesn't know how to use Wireshark. That's awesome. I mean, the protocols have crazy names. I mean, you've got DMP3, you've got Landis and Geary, you've got OPC, you've got ControlNet, you've got Modbus, you've got TRW9550, you've got this thing called ICCP and you want to know how buggy? I mean, when you let engineers create stuff out of a whole cloth, you end up with crap every single time. I'm not an engineer and I'm sorry to any engineers that are out there. There's this protocol called ICCP and it's not the Cisco cache protocol. This is another one called ICCP. And the best part of it is that it embeds the entirety of ASN1. Yeah. Anybody know anyone who got an ASN1 parser correct? Ever? The data just looks like data. I mean, it's, wow, that's one screen and it's a crappy screen. It's just a blob of bits. How do you get from the data to the process? That's the not trivial part. I mean, you can look up the codes in Modbus and you can say, wow, it says Coil 13 is energized. I can send the command to de-energize Coil 13. What the hell's Coil 13 attached to? Is it attached to the donut machine? Because it's the only one I care about because I want the extra sauce on the donuts. I don't have a complete sugar crash. I mean, it takes years to put together the process map. When you look at how high-speed process control is done, when you look at how quickly they can do stuff, I mean, think of everybody's seen the video of the ABB pick-and-place robots. The thing that goes down and it straightens out the sausages to put them in the thing and it picks up the pastries and stacks them. I mean, incredible high-speed stuff. Completely awesome, you know, high net is almost here kind of stuff. But how do you map from one to the other? That's the key piece. I mean, capturing the SCADA data as it flies by, capturing the process control data as it flies by, you don't know what the hell that stuff's connected to because the process is non-trivial. You know, one without the other doesn't work. I mean, okay, some of you, because, oh, did you guys know the trism of all DEF CON talks? The person on stage is not the one who knows the most about the subject. There's somebody in the audience that knows more. So one of you out there is the superstar who can deconstruct that protocol and can puzzle it all out. I mean, we know we've got serious rock star reverse protocol people. There's probably a process engineer or two out there, you know, gaining information for the other side. But if you break the computer, that's one thing. And I'm sure you can do that. Okay, this is a trism of all process control. You can break it. But can you make it do your bidding? Sure. Let's take it as given that you can do it. That you can make... The cookies have a special slice of cyber goo in the middle instead of the regular white goo that you peel off of your teeth. Because you're that good that, you know, you've managed to put bleach into there because, of course, there's always a bleach vessel attached to the cookie generation plant. You still didn't break all the controls. You forgot about safety. In every process control system, whether it's big skate of power grids, whether it's big manufacturing nuclear control, if it's an entire auto plant, or whether it's the place that makes the donuts, it doesn't matter. There's always safety systems. There's always systems that are there enough from falling into the reactant vessel. There are always systems there to say, before you run the multi-ton press that's going to punch that part out, you have to have both hands on the controls. There's a bunch of safety systems there. There's a bunch of preparation to say, we know that humans are going to screw up. And we know that machines are going to screw up because they're going to go out of alignment or they're going to fall out of synchronization. Bad things are going to happen. You screw up the process. You manage to get that bleach embedded in those cookies because of the bleach vessel that was mistakenly attached to the system by the process control people. You messed with that batch. You messed with it so bad that it fell out of specification. And the testing people caught it. And they threw the whole batch out. Maybe they didn't catch it until after they shipped it. They still throw the whole batch out. Not the 10 containers that you managed to screw up, but the entire batch, that day's run or that week's run or that month's run, they throw it out because it's too important to them. The organization needs to make sure that exactly the right products with exactly the right specifications make it out the door. And your meddling will make products that are out of specifications. And none of these systems are autonomous. Not a single one of them. They can't be. Part of it is a liability issue. But whether it's an operator or a controller or an organic mental component, it doesn't matter. We're not at the point yet where we trust the machines 100% of the time to do everything themselves. There is no 100% automated factory. There's always at least one guy who's got a mop who's there to make sure the machines don't go crazy. All right. Given, you're a super hacker. You are almost ready to star in a movie. You're Gregory Evans. You busted the system. You pwned them good. Here's the thing. Stuff breaks. No, seriously. Entropy, it happens. Stuff breaks all the time. It breaks so often that they pre-calculate what will happen when it breaks. They're called single contingency errors. They're called double contingency errors. They've done all of the math. They plan for it to happen. Two simultaneous failures in two completely different parts of the system. They know how to cope with that. They know how to shut the process down safely. So if you're making a batch of plastic, it doesn't solidify in the reactant vessel, which means you have to throw the reactant vessel out. They know how to deal with the fact that chunks of the power grid go off all the time. Small parts of cities do because squirrels and raccoons are awesome, especially when they've been fried. They know how to deal with the fact that problems happen because they happen all the time. Pipelines do this. In this case, this is what... Seriously, that's an awful picture. In this case, it's a propane facility blowing up, where they would transfer propane from one container to another. Those things go boom. Lines fall down. Number one sport anywhere within driving distance of high-tension lines is taking your rifle and shooting the insulators out. Do you know how they replace one of those insulators? I mean, we've got this picture of these guys walking around up there, but the more common way to do it is hanging out of a helicopter, wearing a Faraday cage, grounding the entire helicopter to the high-voltage line, and fixing it in the air. That's done across North America thousands of times a day. Pipelines break. Sorry, they break a lot, actually. They break badly. They break on land, they break underwater, and they get fixed. Sometimes it takes three months, but they get fixed. 999 times out of 100. Well, you know... Oh, yeah. I'm not good at math. Maybe more like 99,999 out of 100,000. You don't feel it at all. Can anybody remember the last time we had a pressure in a municipality? One, two, three. Wow. Was it during a blackout? That's kind of a blackout a little bit, isn't it? They don't just stop. You're accustomed to the squirrel that takes out the power in your neighborhood. Especially if you have overhead lines like we do. We draw a little chalk outline around the dead squirrel lying on the ground because it happens frequently enough that we get in the habit. But generally speaking, you don't notice. Your cozy little house is any less cozy. It's not like anyone's wandering the streets looking for flesh to eat. It's that you never noticed because these bad things that everybody talks about that show up in the newspaper, they don't happen. Not really. Somebody had to. Everybody knows about the Aurora demonstration? Smart guys from INL. All of whom are named Jason, by the way. A little known fact about INL and anything to do with SCADA. They proved that you can do bad things to equipment. In this case, they had a generator running, they accessed it, and they drove it over speed. Over speed generators go bang. Most likely generators run about 3,600 RPM. If you run them a little bit faster than that, you're still within the engineering spec for the bearings and for the shaft. If you run them faster than that, they tend to blow up. It was a modest effort. It wasn't a crazy amount of work. It was a very well-known attack surface. There are lots of these generator in a box kind of systems out there. You drive around. Most government buildings have a beige box out back that says Kohler or something on it, and it's generated in a box. Trying to suggest that your garden variety DEF CON hacker could do this in an afternoon is crap, frankly, and that's what the media made it out to be. You have to know which control points in the system by reading the manual, attached to which parts of the protocol, and you have to be able to decompress the protocol. Smart meters, smart meters, smart meters, smart meters. Everybody wants to talk about smart meters. Smart meters aren't SCADA. Smart meters aren't even process control. Smart meters are a financial tool. They're billing. Smart meters take care of making sure that it's easier to do the meter-made kind of job. You shouldn't have to walk around and circle the little clocks to make sure that you got the right meter reading. It's about doing easy connections and disconnections. If you live in a university town and people are cycling through fairly frequently, it means you don't have to roll a truck to do the day before meter reading and the day after meter reading when somebody moves in, moves out. They're about providing... This is what the book says anyways. They're about providing for demand-side load management. So, they can provide an economic incentive or economic disincentive to say to me, it's much better for you to do your laundry at 11 p.m. on a Tuesday night than it is for you to do your laundry that's for providing that kind of tool. There's something that is like a smart meter but isn't a smart meter and people talk about them in this commingled kind of way and it goes back to that specificity of language thing. You can arrange with your power company a special kind of thermostat that goes on your wall that can cycle your air conditioner on and off at different times. This is actually a fairly neat tool because it helps with the double peak problem. Power is utilized starting from the middle of the night. There's a peak right in the morning when people are getting ready to go to work. It drops down during the day when people are all doing their stuff. It peaks up again when people get home and turn on their ovens to 450 to make that awesome, awesome frozen pizza and then it drops down again over the course of the evening. And the marginal cost of providing power for those peaks is really expensive. So instead of being 5 or 6 cents a kilowatt hour it's more like a dollar a kilowatt hour. So trimming the tops off of those peaks rounding them in this very analog kind of way is really excellent. The system where it says, you know, we'll turn off your air conditioner for 10 minutes out of the hour so that we can balance how much load is being used in different parts of the control area. And that's all terrific. Those systems all run on flex paging. They don't run on any kind of regular things. I'm giving you all the big hints here. If you go back to 1997 and look at the loft site and pick up one of their poxag decoders you can do serious hacking of demand management for power grids today. It's not exactly a new technology. Advance, please. It hates going to the slide. I don't know why. There we go. Oh, it's busy doing the build for the next crazy slide. So, Stuxnet. Anybody heard about that? Three people. Anybody hear about the big crazy skater vulnerability last week, two weeks ago? It was in every flipping newspaper there is. Everybody had an opinion about it. Everybody wanted you to know they knew more about this than anyone else. The fundamental vulnerability is really simple. There's a Microsoft bug. No, really. There's a bug in Microsoft software. Such that when you view in Windows Explorer a directory that contains an LNK file which is your standard shortcut link it actually executes code attached to that LNK file just by virtue of viewing it. You haven't even had to click on it yet, you just view it. And some bad people, nefarious people, bad actors, nation states, who the crap knows, decided they were going to attach a stub to this code that looked for the presence of a piece of industrial automation software manufactured by Siemens. This industrial automation software is used in process control. It's not SCADA. Despite what all these headlines say. And this industrial software uses a standard database backend. It's not quite an open source project. It doesn't put ID 3 tags in the database backend. But they stole the process map. Remember I was telling you the hard part isn't the data, it's the process map. They stole the process map and they tried to upload it to a web server that fortunately doesn't exist. It was probably a proof of concept. The most interesting part about it is that it gets around by moving from USB drive to USB drive. So remember all those floppies we used to be worried about getting viruses from? Do you have scanning setup on demand for USBs? Should you really be jamming random USB drives into control systems computers? You wonder whether it was a control system vendor that carried the initial infection in? Because they would do that kind of thing. Because of course they're experts. I'm not. We're not. You're not. The real issue at the end of this was that it was suggested that you not remove the software. That you not remove this malware. Because it would endanger the operation of the system. And this isn't good for anybody. Having the vendor say wait until we've got a tool. Don't remove it. We don't know exactly what it does because nobody's done a really good breakdown of it. The breakdown didn't happen for almost two weeks. We just sort of accept that it's there. And disclosure is always an interesting kind of problem. Everybody knows about full disclosure and responsible disclosure and coordinated vulnerability disclosure. There's also vendor disclosure which nobody talks about enough. And that's that vendors are screwing this up royally because they're still letting their marketing departments and their legal departments be involved. I would encourage you to try and read that red circled piece. Those at the back of it. The nice folks in Siemens Marketing and Legal took it a little too far. They said the user login and the password for WinCCC are freely definable and there's nothing to do with access to the internal database. The internal system authentication from the program to the database is based on pre-defined access data. On everybody to swallow the term pre-defined access data. And tell me how that's different from the word password. Specificity is important in our business. If you mean password, you say password bastards. And they say you cannot change that because it's been hard coded throughout the application. So if you change it on the database and you change it all the places you can find it in the application, the application is such crappy soup that you can't get it changed in all the places that you need to change it so you break shit. You're all perfectly smart every single one of you. Roughly 50% of you had a shower today go team. You've just got to pay attention and focus in a way that legal departments marketing departments and media aren't. And you've got to do something. Since you've solved all of your organization's problems you've reviewed all of your logs. You know what every firewall rule is for starting to sound like we all suck at our jobs, doesn't it? You've got to the point where you've amassed all of the certifications necessary to make the HR department believe you know what you're doing. You've got time. You've had your feet up on that desk for a while. There's a lot of warring factions involved in control systems, whether it's little control systems or big control systems. Business function versus the asset owners. There's the traditional IT departments and the control system shadow IT departments. Everybody's got everybody's got a foot in the game and they're worried that you're trying to take their turf. I mean most control systems end up installed in places that better or worse have a bit of a union mentality and you're taking their turf. You can get past this. You just need to suck it up. You're not an expert. You sure as hell not an expert in what they're an expert in. Suck it the heck up. Buy some people some coffee. Make some friends. Because it's time to learn. You can't walk in after having been, even to a day like today where Nikita joyfully exclaimed it's the SCADA track at DEF CON. Even if you spend the entire day in this room you're still not an expert. You don't know what it is that your business does for real unless you spend days in operations. You've sat in control rooms. You've watched the machines. You've lived through cleaning up ad paste. You've done all of these things. Even though it feels disingenuous. I mean it feels stupid to stand in front of somebody and say sir please teach me please. But you've got to be the student first. After you've learned as much as you possibly can then you're in a reasonable position to go back to them and say based on what you've taught me I think this might be interesting. And because you've been smart and you've put your ego in your pocket, your giant huge room size pocket to fit your ego in it you've sucked it up enough that they trust you because you're a friend, a colleague unequal that you're the person who's going to understand their special and unique needs because every situation every control system is uniquely different and exactly different. So here's some things that I learned. The organization is completely against you. If you're going to try and make changes they are against you. You will suffer through woodshed talks on the loading dock from the union rep who tells you we got a good thing going on here don't rock the boat. That happened to me on November 2nd of a year that I can't disclose. But we're all in infosec we're a bunch of rock stars right? Anybody in here an infosec rock star? No hands. Oh, two, three. Awesome. The vast majority of the people that I know that work in the control system space would be perfectly happy with good old 8-bit computers that knew their place in the world. It is not insane to suggest that modern computers are no longer predictable in their actions. No one completely understands Windows. Not even at Microsoft. Grocking the entirety of some of these complex systems. I mean, when you install an operating system and there's 100,000 files do you think there's any one person who knows what's in all of them? Is there anyone at Canonical Command Central to explain to me where to find the file to make my Ubuntu machine not purple? It's better than being beige. The real problem though is that most of you are the age of the kids of the people who work in control systems and seriously dad doesn't like being told it to do by kid. You've got to hack your way through this problem. Human systems are the same as computer systems they're just systems that you like peeling apart technological systems to comprehend them you should be peeling apart human systems to comprehend them. You've got to hack the organization you've got to understand all of the moving pieces. You've got to look outside of the little realm of the IT department that you sleep in all the time. Sorry that you work in all the time. You've got to shadow a few people who work in operations understand these different bits and pieces it's a system like any other just get a little mitnicky. The doors start to open as soon as you've got a little humility and you've got a little bit of an ability to say to somebody I don't understand what it is that you do I don't understand how what you do fits with I do. I need to understand those things so that I can do a better job of helping you. This is the exact antithesis of the religious schools on IT security that teach you how to say no. It's time to make that change because you can learn anything fast. You're an infosec rock star if you don't know it today you'll know it tomorrow in its entirety. You won't stop at simple.wikipedia.org you can make them change you can make them come to see altruism as a way to get forward you can keep the lights on and you can hold the zombies at bay. But just for review we need to tear down a lot of preconceptions because everybody learned how to do corporate IT security from the movies. You understand that there's a specific methodology required to enable you to hack a computer. It's an awesome awesome kind of world. There's interfaces that allow you to see everything you need to know. Unix is easy to access. You're fixated on how the protocol works. The user interface thing. Do you know what these user interfaces are hard to understand I mean the media would have it as if this is the way that it is because there's all kinds of different user interfaces, different ways to access the data that you're using during work because you're reviewing log files right now and most of the time you're just doing the same old same old. In fact you're the part of the problem because you let this happen you're focused on modbus TCP and comprehending what each of those frames do. You're pinned like a bug to smart grid because you think there's something there and you're focusing in the wrong place you're not looking for where the weirdness happens the weirdness happens right at the user interface slice it happens over RDP it happens over X windows it happens inside of windows machines and you already know those things you want to break into a SCADA system you don't need to know DNP3. In fact that's a hindrance but if you know RCP or RSH you're in pretty good shape we have to sing along with the song unfortunately nobody's singing along with the song this is the way the media is putting it out it's cyber this cyber that cyber everything and it's an astonishing amount of doucheery I can't even get through it there's a hacker behind every bush they're hiding was that somebody talking to me or someone yelling because they can the same stories pop up the same people keep showing up it's something to be critically afraid of the 14 year old in mom's basement is going to get us here's the thing I vaguely remember being 14 I had a computer and I didn't want world domination I had a hormone problem I had to deal with first the conservatives especially the neocon types like we're forced to deal with in Canada want you to think of evil brown people because it's easy to point fingers the reality is it's like middle aged white guys because it's about the only thing that middle aged white guys are good at the world is generating these elite stories of soldiers there's a story in the news from a few months ago Booz Allen Hamilton was handed the contract to build the new Air Force Cyber War control center only 14.4 million dollars in taxpayer money to build a cyber war bunker sounds an awful lot like my basement actually they were handed another just to give you an idea of the scope and scale of US taxpayer dollars that are going into cyber cyber cyber cyber cyber they were handed another 20 million dollars to and I got to quote this because it's so perfect foster collaboration amongst telecommunications researchers University of Maryland faculty members and other academic institutions to improve secure networking and telecommunications and boost information assurance mailing list in a wiki we're all deadly up trade of this one aren't we yes that was up trade this is far more likely because each one of those suits contains a middle-aged white guy the internet is out to get you you can pound a cardboard tube through someone's heart alright here's the part where I'm going to tell you some interesting things this is stuff that I've personally seen doing assessments or working in SCADA operations I filed off all of the serial numbers none of this is traceable did you know that VLANs are a security separation methodology in a nuclear plant change management is for the weak physical security policy is such that you have to get past people with guns to get to the modems that are always turned on with publicly accessible phone numbers there's no security agreement with the vendor they have a VPN into your systems and a complete running copy of everything you've got they can access it at any time vendor default accounts and passwords are not meant to be changed updates and upgrades happen when you do updates and upgrades not when you need to do updates and upgrades those updates and upgrades happen every 5 to 15 years did you know that the best thing that you should install on all of your HMI systems is VNC PC anywhere you're absolutely right shared networks are the best because it makes it possible for you to get to the internet from your HMI workstation to check the ESPN you can run most processes from your BlackBerry because there are full HMI clients available for the BlackBerry OS you wouldn't believe that I've seen this as a recommendation you should not uninstall critical software such as sol.exe because it aids in managing operator boredom that's in a document the best way to do switch management in an emergency is of course telnet because all that security layers gets in the way so does antivirus dynamic ARP is always the best I have yet to see a firewall rule that didn't include the phrase any any this is 15 years of looking at this crap most PLCs have embedded web servers they're on oddball ports like 80 which makes them really hard to find once you of course access that web server on port 80 it's presumed that because you accessed it you're automatically the administrator for real and I think to myself over the course of all of these assessments I've looked at systems from chemical plants I've looked at manufacturing I've looked at crazy kill you kind of stuff I've looked at power plants of all different types I've looked at entire power grids and pipelines I think that it comes down to superheroes as the only possible solution maybe ninjas the occasional pirate but primarily it comes down to the fact that none of this is rocket science it is literally just a matter of being able to look at the instructions and do what they say and remembering that doing the minimum gets you the minimum does anybody want to live in a house that's built to minimum code standards no? because you know bad things happen do you really want your process control system or your power grid to be built to minimum standards by the lowest bidder that's what we have North America wide the machines do exactly what they're told to do and the industry can't keep up with its own awesome when you go to a control systems equivalent of the DEF CON event where you've got a bunch of vendors who are giving themselves self-congratulatory reach around, pats on the backs they can't stop telling you about how awesome they are I had a vendor try to convince me I'm not going to name them because it would embarrass me try to convince me that the best thing that I could possibly do to provide a separation an air gap between two networks was to install a router this is 1992 phoning they can't fix the basic stuff why should we trust them we're in a position where we need to figure out what we can do ourselves what you can do when you all go back and either try to break things or not you need to become an infovore I cannot possibly stress this enough if it's written down read it, if it's available on video watch it, if it's available in a podcast listen to it, devour the information devour the comprehension not of the twisty little engineering minutia that people get off on but on what the systems are like this is a systems level hacking issue this is a systems level comprehension issue read the fucking manual and if you find someone who tells you they're an expert boot them out because they're not they're probably a charlatan everybody is aware of the fact that if you think you suck at something if you think you are the worst you're probably not too bad but if you think you're an expert you probably suck hard the finance industry for example remember that project timelines are really really long implementing a new energy management system for a power grid takes years little tiny bit of stick handling at the beginning, little tiny stick handling at the third point and by the end there's not a major problem because you bought the right thing we've got the right kind of people on board you didn't capitalize your training so that training only happened once you made little tiny changes and there's some big picture stuff too the people who are putting themselves out there the experts as the mouthpieces even the ones that may have some very aged cred one time they hacked a computer if you're bullshit meters going off there's a really good reason for that you all human beings have finally tuned bullshit meters if you're bullshit meters going off there's a reason make sure other people know about it catch these people, persecute these people call a cyber douche when you see a cyber douche and you're not zero cool you're not the plague you're not neo, you're not whoever the hell John Travolta was you're not some uber-lit dude or dudette none of you I'm not, even though I act like one impress people with persuasion humility don't wear your bravado don't wear your hacker doucheery a little bit of shameless self-promotion but don't be a dick about it be a water drop water drops wear away mountains you don't have to be in their face all the time everywhere make friends with people this is crazy talk because every one of my report cards does not get along well with others make friends with procurement people so that when you come to them and you say hey I heard you're buying something big and new nudge nudge could you include this little feature in there so it's something they're doing for a friend the infosec industry itself is what 25 years old I mean the firewall industry is less than 20 so we've got a short kind of history to work from IT itself is 60 to 70 depending on how you measure it but a lot of the problems that need to be solved are already solved problems finance solved them 10 years ago because they had to because there's a fraud problem we can take those lessons pick them up, change our frame of reference looking at them from the back set them down again and learn from them it's an industry rich with comparable situations and problems once upon a time computers did what they were told to do as an industry this is something I've argued to the nth as an industry we need to get back to the point where computers do what they're told to do they're supposed to be deterministic most of my computers are no longer deterministic they will not always do exactly the same thing exactly the same way this is the same kind of suck that has given us botnets this is the same kind of suck that has made it so that every time you visit your family you spend the entire holiday doing tech support anyone who works at a major software development company who is also an engineer needs to take themselves outside and have a deep long talk anybody who works at a major software company and calls themselves an engineer but isn't one needs to go outside and have a deep long talk because this is real stuff now we've got coming off the shelf showing up everywhere it's bad when you're dealing with your own finance desktop personal information it's bad in the finance industry it's terrible in the process control industry don't even get me started on the medical industry okay this is stuff we need to pay attention to thank you all very much for listening to me rant questions I need you to pay attention to this slide this is important stuff talk to me about it later if you need to and I'll be in room 112 very shortly thank you all very much and nine o'clock tonight track 4 ten thousand cent hacker pyramid you're going to want to be there