 Alright, so what is the problem? I think one of the things that I really like so much about where DEF CON has grown up with the villages is that when you're looking at industrial control systems, one of the big challenges not only the technical intimidation of where do I even start to understand how to access this code, look at these protocols and understand this equipment, but there's also a barrier to entry of this equipment is hard to find, is very expensive and so being able to get your hands on it and to touch it and to be able to start to play with it is a barrier to being able to start that research and have an open dialogue between the security research community and of course the vendors and the corporations that use this equipment. And the villages of course have all of these things there set up for you with experts happy to walk you through, let you touch it, ask those questions so I encourage any of you who haven't been to the ICS village or any of the other villages please do so because this is that kind of opportunity where the kinds of things that we're going to talk about you get to go actually physically touch that and it's not just abstract. It's only been a second year, it was a bit more, there's more of us on this wire. We continue to see further digging and the opportunity is being taken but it needs to be encouraged we're still way beyond the end ball. And atlas aside for the most part this is not sexy so it's not the the challenges, strike the pose again please, that says industrial control systems. Thanks for dropping the beat man. It's not sexy and a lot of the challenges and a lot of the solutions and a lot of things that need to be done are not sexy. I mean when you start looking at a lot of the conversation that's happening in the research community at large, sexy, zero day being dropped, swift being compromised, sexy, don't drop it, ask. Far enough, you don't know how far I'm going to go. All right, so sexy, the swift, the various things banking and maybe well destruction of dams is pretty cool but it doesn't dive into the problem deeply enough but a number of years ago we did have some sexy parts of the grid and it was called AMI, advanced metering infrastructure. The concepts were relatively new, they were exciting, they had Aura grant money, they had backing, they had momentum and yet industrial control systems sat on the Mac burner and every IOU I spoke with, IOU and investor owned utility, every one that I spoke with just kind of shut the conversation, oh no that's something very different. We don't have money for that at the moment but we're working on it. The other part of the not sexy of course is we're people. We do what we're incentivized to do. Part of that of course is we're pursuing our individual passions but starting to take that challenge of cost and limited ability to start to do this and in the last two years we started to see the rise of the bug bounty concept and crowd sourced vulnerability disclosure programs. Where are those for this kind of community? Where is that kind of incentive? You want to touch on longevity? Yeah, I thought you were hitting longevity. I was going to but I forgot it so I'm passing it back to you. There you are, I touched on it. Longevity. We have industrial control systems. Hey, big surprise. They're built to withstand many decades. Even, okay it's not surprising because these systems control that which makes our life what it is. An attack not to be, not to be too accelerant but a certain type of attack could potentially change the way we live. So, do we get into this? Crap, I'm forgetting what's in the next slide. So forgive me if I say things twice. So what we have is a problem of systems that were created, designed 20 years ago. Put in place 15 years ago but they've got a 20 to 30 year lifespan that has been amortized by utilities and people deploying, anybody who's deploying ICS. The manufacturers have been strong armed into the lowest possible price or it will go somewhere else. They didn't build in money to maintain these systems for long enough. They were never intended to be connected to the internet but as we all know, every IP and network wants to and will eventually be connected to the internet. So the where's Waldo of industrial control systems is if you're looking at a box and you want to know whether it's an industrial control system or just a traditional computer, if it's at least 20 years old, it's an industrial control system. I mean, we're talking top of the line 30 megahertz. That's an impressive piece of equipment. And actually funny story here just since we've been talking about insecurity. We'd actually had props as a part of this presentation. Come on in. Keeping in line with OSHA, which prescribes the standards for folks that are working with dangerous things like industrial control systems and SCADA, et cetera, that we would have hard hats on and safety vests. They were stolen. So that just kind of shows you that you got to have physical security at all times as well. So God grant me serenity. What can we change? What can you change? This is an actual statistic that we made up. One of the guys who works for me, he's an expert. He may be in this room right now. 93% of all code is shit. And that is the technical jargon for it. And we made that up, but we're not wrong. And that code is code built on top of code, built on top of code. I think one of the more interesting conversations I've ever had with a manufacturer was through some level of acquisition, they had code through some little subsystem of one of their boards. Nobody knows what that code is. Nobody has any idea. Whoever wrote that is long gone through some acquisition of some time ago. We have no idea. We just kind of let it go, and we sort of know how to poke it. And occasionally it pokes back, but that's it. So we have to accept that crappy code is going to make it into the systems that protect our lives and provide for the way that we like to live. Do we give up on code? Do we say, oh, it's crap? No. Just recognize there is a natural tendency towards chaos. Entropy. Entropy. Well, can we just keep them off the internet? Sure. I'd like to see you do that. So people. People. People. Management and people on the ground don't know how to talk to each other. You can't do that. It's stupid. If you had any brains in your head, you would not have even suggested that. You can't do so many less things. Anyway. That was loud. I bet you don't want to hear me anymore. No. You almost threw off my pacemaker. That would be the next talk. That's another talk. We do medical devices. The challenge here is the classic nerd versus operations and management. Nerds speak nerd. They're kind of abusive, actually. Unchecked. We kind of beat each other up. Why? Because we were beaten up as kids. We're nerds. Now that I have some power, man. Meanwhile, management is there ensuring the function of the operation, right? They're caring about revenue. They're caring about the business and they're trying to do that. You make them sound really sexy. As an engineer, I'm kind of offended. I don't know what the interpretive dance is for dollar sign. I do actually. We're in Vegas. So management is focused on operations and the nerd guy runs down and is like, hey, boss, we have this huge problem that we just found out. There's this issue with code and it immediately starts running down into the technical bits. Management glazes it over, waits for him to leave the room. You know what happened? I don't know what happened. Stupid or not. Stupid or not, the management has a job and they have things that are very important to them. Maybe they're paycheck. Maybe they actually care about security or at least safety and the mission of their organization. Or if you have a great environment, they may actually care about you. However, what's the secret to communication? Understanding the other side. I like to say that there are no crazy people on this planet. You just don't understand the system of reference or their decision making. There are no crazy people. What's important to you? What's in it for me? Oh, you had to go there. So a little bit later, we're going to go through a little bit more granularity about the different kinds of stakeholders in this and the pieces of the supply chain. But fundamentally, you have this challenge of the two sides speaking Greek. Nerd speaking nerd, management speaking operations, and in the middle of this gap. And this is often times where security research companies like us come in between because we're there to bridge that gap between. We have the technical understanding of working with the equipment at a very granular level and understanding the implications there. And then importantly, we can tie it back to the business context and impact because that's the key. That's how you got to bridge that. When you're thinking about it's not code, it's the impact of the code. Compromise is not compromise is not compromise. Compromise or a hack or an exploit only has value within how does that vulnerability affect the business, affect the operation. That impact helps you understand prioritization. That impact helps you understand the correct remedial solution because it's not just a question of there are lots of different ways to fix problems. It's fixing it the right way for the context of how it's being used. And you don't want to take responsibility as an engineer for something that is going to get your boss fired. It's his responsibility. Your responsibility is to provide information to understand and to bring along. A very dear friend of mine last night at our party said, Atlas, we had a couple to drink. So we're all kind of reclining by the hot tub and he says, Atlas, what's important to you? How do we fix this? What are you thinking? And I didn't have this canned, but the three things that came out of my mouth were education, motivation, and partnership. And I didn't realize that that applied so well to what we're going to talk about today. Go ahead. But wait, there's more. So as we discussed, we talked about the agent incentivization problem where is the security community really incentivized to become and engaged in this. And then of course looking at where's the clear mandate coming from. Who is in fact pushing that? Because this is a top down problem. You're not going to have the average consumer which everyone in this room is, the last time I checked we all use electricity and water. Everyone in here is a consumer and a stakeholder and we as a large group are not really able to effectively for something that is a very complex problem instead of problems. This is something that needs to be top down driven. Accountability? Yeah, no, I don't. I'm not a consumer. I live in a yurt. Solar power dude. Ryan, how's that yurt coming? So there's no clear mandate. Who is responsible for fixing these problems? The way that we live affects all of us, as he just said. But who has to fix this? Is this the vendors? The people who made PLCs or to use? Is it the utilities? Is it their problem? I've seen some pretty amazing engineering going on in utilities. Is it their problem? How about regulatory agencies? Show of hands. Who wants the regulatory agencies to fix this problem for you? This is great. Go Joe. I'm sorry. Go ahead. If he's saying if it's regulatory, it'll be check the box. It's the only way that they know how to do it. And in fact, they're incentivized with both carrot and stick to make it check the box. Very good answer. But, I mean, there are a couple different agencies that we... Nah, we'll talk about that in a minute. Go ahead. So the first thing, let's address the problems that we can right now. There are things we can do. We actually want to have a 5, 10, 15 year plan. Hopefully not 15. Please Lord, let us fix this faster. But there are things we can do right now. Show of hands. Anyone want to give an offer? Any ideas? I can build my house on stilts so that when the dam breaks I'm good to go. I like the idea. I already have my house built. I'm not sure that I want to jack it all up. Anyone else? So where do we start? Well, we begin by assessing what the problem is in our own backyard. Every one of the people on the last page, even the regulators, we want to become aware of the problems that we can fix right now. For example, if we are still using Modbus, probably a good idea to get that off the wire. I don't care if you're doing cereal. Okay, there are worse things to use. But if you're using Modbus right now over Ethernet, please stop. There are other options. And I'm sure that there are business decisions that made Modbus the right thing and it takes money and time and effort. But it's something that we can fix. If the protocols that you use offer security, lab it up, make sure we don't roll out some big change and take down the grid or the water supply or back up the poop. I mean, we're talking about more than just the electric grid, right? But lab it up, test it. Migrate to something that at least offers security even if it's not perfect. DMP3 has tons of bucks, but at least it as he stepped on the chair, I was thinking he was going to call out the fact that you may be still kind of surrounded by shit, but if you can get a little higher, you can be in it less. Pick something better. If you have systems that are have replacement upgrades and you can put into place a better system, maybe a system that actually has a vendor that's still around or a vendor that you trust to partner with you, this is a good first step. There are many things that you or we, whoever is doing this assessment can sit down and identify within the next year we can do this. Let's talk to the right people, work it out, get something as part of a bigger process. To emphasize, this is the all is not lost slide. You can start somewhere and have that vision of where things can get better. Circle the wagons where you can establish control and then iterate from there. Best practices. So, a few other things that we can throw into the mix that you can do within the next year to two. Authentication is a problem we all deal with. There are many technologies available because everybody's struggling with it. To factor authentication for the right systems is a good first start. If you have to use passwords and not tokens or keys or fobs, please. The next step guys, I know that we didn't mean to plug the grid into the internet, but we've done it. We didn't design our systems and our networks to withstand the type of onslaught that a nation state is willing to offer. In fact, the state of many organizations many people here without state funding could do it. So, I'd like to lay out the idea of a multi-tiered castle at the very center with the most restrictive protections in place. You have crown jewels. You have head ends. You have things that actually change significant parts of life. Outside of that, you have slightly less trusted, less protected systems, but still very highly protected and so on and so forth. I recommend three to five layers at least. And these layers play into a potential for protection. So if you have a system that there's absolutely no way that you can have assurance that it is protected, then it falls in the outer ring. For example, pole top devices. Anyone here that's been at DEF CON you cannot possibly argue with me that a system being on top of a pole is not secure unless it's in a compound with guns. Patching policy. This is just regular Sec 101. The systems that are being used to pivot into attack your critical infrastructure, generally people's workstations check an email, surf on the web. These systems have to be maintained and protected. Now this is not going to keep everybody. I'm going to try not to throw out nation's names because I know everybody's represented here. But patching. At least the end days done. And then there are configurations in our PLCs and our RTUs that again can be used to turn on security features. Encryption and strong authentication and other things to lock down access and have available on your PLC. Now there are many who will say encryption takes time and I need point blah blah blah milliseconds response time. If your critical systems can't handle that type of way-lay and processing power those might fit on your list to replace. Monitor all the things all the comms. So this is while endpoints might not be the place that you can do that monitoring but certainly there can be transmission paths and one of the challenges with this of course is not all of the communications are going to be wired. You have to keep in fact that a lot of things are going to have RF communications and you need to start thinking of the physical security and the physical presence of those things. Please don't allow your cell phones to connect to your network and talk to your head end. There's one good rule of thumb but if you do have cell phones connected to a network it better be on something well. Hunting. Who here has heard of hunting? Okay. About one percent of you. Hunting is the act of scouring your network. No penetration testing has to go into this if you're worried about breaking things. Simply monitoring traffic patterns. Traffic can be cataloged, identified and put into a database and analyzed. For example there's no reason why workstation in the office needs to talk directly to head end. If you have this type of communication or other strange communication A, that's something you want to understand better. B, it may be the thing that tells who's on your network. Fuck Modbus. We've mentioned Modbus. We've beaten Modbus. Modbus is unauthenticated. It's unencrypted. And perfectly sane to kick the shit out of here. There are others like it, but this is Modbus. Please choose something that has some sort of protectiveness. If you have critical systems that must use an insecure protocol, maybe because you're worried about latency you should have armed guards around everywhere so that that network touches. This is the self-promotion slide so I'm just throwing that out there because this is some of the stuff that we do. But it's more, even if you don't do it with us please understand we have yet to get to the point where I need to throw binary exploitation against most control systems the way we do your laptop the way we do your servers the way we do against firewalls and routers we don't have to do that. Aka it's not hard. It's not hard enough to warrant that kind of work. But there will come a time hopefully very rapidly where that's not the case as we fix the fixable. Digging into hardware software and firmware that was sexy identifying weaknesses in code weaknesses in design of hardware weaknesses in what your vendors are shipping out like hard-coded passwords they still exist. Assessments of all three things will provide you when we finally get our poop in a group something that is defendable and if we don't start now it'll just be that much longer before we get to something that's defendable. But I've got limited budget well then prioritize if you have systems that are most widely deployed those should make pretty high on the list if you have significant systems that control large potentially irreplaceable parts of your systems that may make a very good top place you can get to the older stuff later you guys know how to think about boundaries right? We do the most important first we then back up. We don't stop until we're done but we get the best that we can for now. So while we list three device categories here of loose categories there are also three kinds of assessments white box, grey box and black box and I list those because those also require careful consideration for what you're trying to accomplish a black box assessment is saying hey you're just like somebody else out there anywhere and you are required to do all of the open systems intelligence and gathering to try to figure out what kind of information can you gain and then from there start to work that to then be able to determine how you're going to conduct your research for exploitation grey box is going to be a mix of you have access to all the documentation so there's kind of that in between and some of the factors there are going to be trying to understand how much is obfuscation and obscurity is your friend versus how much of it do you really want to just get straight to the technical research and understanding which is what white box is going to do and there's also one other thing to point out this is not check box things check boxes get you the first layer of an assessment like this because you can actually put into place a list of things that you want to check to gain knowledge about the next steps once you're past that you need people who are creatively evil and you need them to have the time to sit in their creativity and think oh hey Tim do you think that we could do blah blah blah blah yeah let's go do it oh this is my slide so as we've said before we need secure we need devices that we can deploy that have security baked in from vendors who think about security we also need we need to reassess what we're doing for comms because there are radios that have encryption enabled almost by default oh but it's really easy to enable them you simply have to choose one of 256 standard encryption keys that doesn't sound like a lot hey it's 8 ES fucking 8 with one of 256 different keys and please understand one of the things that actually one of my favorite parts of my career has been breaking into banks breaking into government buildings and doing things with cyber that aren't typical pentesting things you may think of them as red teaming if so that's great but some of our assessments have really gone oh well I can hop the fence okay here's the device you don't have to you don't have to hop the fence I know that you can do it and we've gotten our hands on a lot of attack software in fact the reason that I call out MDS is because they're deplorable in many of their at least older stuff that's still widely deployed for using being able to use one radio that you purchase off ebay to attack all the others and even brute force within 18 hours guaranteed to be on any network so the people we've talked about the people we talked about the people through multiple things where are these folks right so first of all I think that this is again what I love about DEF CON is this is the meeting of those passionate folks who really do care about this problem for various reasons either you're motivated by how we can all help humanity together or you have a particular interest of the matrix metaphor of going down the rabbit hole and seeing how far we go technically your vendors your vendors should be having security minded people in their hiring process they should be actively pursuing not just bullshit bingo security but true security and utilities they need to be able to call bullshit they need to be able to have the knowledge and skill to say that's crap or better yet get out of here these other vendors they have people who know what they're doing so go ahead I'll let you take the time how to fix for the who's who we have two regulatory agencies FERC and NERC and there is a challenge there where there's a mix between the overlap of responsibilities and a difference between suggestion and compliance and while we talked about compliance being a challenge from a checklist perspective we also talked about the fact that this is a top down mandate that needs to push and encourage this action and that is the purpose of government and legislation and that's where these organizations need to come into more of a common agreement with the vendors in the industry to help push that now they need to be light handed right now we've already got some very heavy handed things in place that actually can detract from true security they have to be careful how they engage but FERC and NERC do play a very important role our public services commission utility commissions these are the state run organizations that kind of keep our utilities in check they are the protectors of the consumers now I have seen some significant some significant investment from several PSCs into researching cyber security they want to understand they want to know how they can fix the problem and they have a big role they control how much a utility can make for a year they also allow a utility to do a rate hike which allows them to charge more per kilowatt hour or per gallon of water or whatever in the interest of maintaining a grid, a water system, a water treatment system that services our needs but our vendors they're saying where the hell is the money coming from the utilities don't want security, they're not willing to pay for it I can't make it unless they're willing to pay for it so utilities utilities are saying we don't have any vendors that provide things that we can put into place that are defensible, they have crap and this is what I have to deal with encourage educate, make the decision and in fact you have a lot of sway the unfortunate side is we have decades of utilities beating up vendors for the lowest price we have lost the partnership that is provided for our country and our world we got to get it back and so that concludes the amount of time we have for this this was a lot to condense into the short talk so I'm Bryson I'm Atlas and we're grim thank you