 Today we have, on behalf of or from Utilisec, Mr. Justin Cyril, who has spoke at a number of different conferences, including Black Hat and Def Con. Let's give him a big hand and welcome to today. All right. How's everybody doing today? All right. Good to hear. Good to hear. Okay, so at Black Hat, I did a four-hour presentation, a four-hour workshop on this material. You guys get the condensed version of it, but I still have a lot of slides. So we are going to go through the slides. We're not going to hit all the points on the slides, but we are going to have them uploaded to the Def Con site so you do have availability to all the content. Okay, so first off, I am an managing partner for an organization called Utilisec. We specialize in security services for electric utility companies. We provide penetration testing services, secure architecture design, and other types of services, including trying to represent the utilities and their interest in a lot of the standards that are out there, trying to build in security in some of the standards. In fact, some of the groups that we've worked in, we currently lead and facilitate, one of our managing partners leads and facilitates the NERC CIPs. Let's do Joe Bucciero. We also lead many of the different groups that the electric utilities have put together to try to build security or try to generate industry awareness in security issues that we have inside of the smart grid. So the purpose of this talk, a lot of times we hear a lot of different talks, Black Hat, Def Con, Shmucon, Torcon, of different types of things in the smart grid. We have a lot of good research going on, a lot of good researchers that are doing a lot of good in this industry. We also have a lot of talks that are smart grid talks that are simply there to try to generate a lot of the hype. So the purpose of this talk is really to try to give everybody a very clear picture of what the smart grid is, give you an idea of some of the issues that we're dealing with, some of the different attacks that we're seeing, and some of the different attacks that we perform as penetration testers on the smart grid. And more importantly, show you that the smart grid is more than just SCADA. The smart grid is more than just smart meters. There's a lot more out there to do. And a lot of the people inside of this audience that are professional security professionals, your skill sets are very applicable in many areas of the smart grid. You just may not know where to look and where to try to generate that business. And ultimately, one of the ultimate goals of this talk is to really try to generate more awareness, more interest in the security community, to try to get more researchers and more people in this field. Because the smart grid is not perfect. Name a single vertical industry that is secure. We need to get more security. We need more people in this. We need more expertise. We need more technical expertise in particular. So really that's some of the goals. So the first half of the presentation we're going to be focusing on trying to let you guys understand a little bit more about the smart grid architecture. And then the second half we're going to be talking about some of the different penetration attacks and the defenses that we're recommending and working with vendors and utilities to try to address these issues and mitigate them. Okay, so first off, what is the smart grid? The smart grid, anytime you hear the term smart grid, this is something very similar to hearing the word internet. It is a marketing term. It can mean anything and everything you possibly ever want. Ultimately what the goal is of the smart grid is to try to take our existing infrastructure and add additional intelligence to it. Add capabilities where in the past we had to have people sitting in the control rooms looking at different sensors coming back and having them sit and toggle the different switches, toggle the different controls to cause reactions in the grid. With this smart grid we're trying to add more infrastructure to be able to give us a better view of what's really happening in the grid. A better view of what's happening at the homes of each one of you consuming power instead of being able to see once a month how much consumption you have be able to see within a 15-minute interval of how much energy you're consuming. And hopefully this is going to be something to benefit the rest of the community as well. I don't know about you, but I personally want to know exactly how much power I'm using in 15-minute intervals in my house because I can do a lot of really cool things for my own self for that. Of course, attackers can do a lot of cool things with that as well. But ultimately that's the goal. We try to do the same exact thing with the technologies out in the substations themselves. So if you look at this diagram, this diagram goes through and shows you the different elements or different major domains in the smart grid. So we see the ones across the top, we have the markets, we have the operations and the service providers. These are a lot of the organizations and the companies that are kind of the glue holding the different processes together. Then if you look on the very bottom, we see that dotted yellow line across those four clouds on the bottom. This is the dotted yellow line represents the energy that's flowing from the bulk generation plant to our homes. All the blue lines are communication lines. We have a lot of different types of communications between these different entities and these different domains. So this is basically the same exact diagram. This is showing you just more information and more of the devices. These are the actual systems that you're going to see inside of a lot of the utility companies that are out there in their back offices. So this is all color coded. If you look at the yellow part, this is the operations. This is what most of your electric utility companies are doing. Each one of those boxes are a lot of the main control centers. These are some of the major systems that they have to control your power and to monitor your power. If we look in the... I can only go back just one slide and show you one thing. So bulk generation on the very bottom, bulk generation, I think that's pretty obvious. These are going to be the power plants, the nuclear power plants, the coal plants that are generating the power for us. That power flows over to a group of organizations called transmission operators. These transmission operators are what take this power from the bulk generators down to the distribution operators, which are more of the companies that we think of as electric utility companies because they're the ones that we're buying our power from and then that power flows back into our home. So the transmission operators and the distribution operators, they're very distinct. There are several different organizations and utility companies that act as bulk. But when we look at this diagram, we see the transmission operators in the upper left-hand corner. These are the transmission field devices, the devices that are on those big giant steel poles with the big power lines that we see across some state borders. My wife and I are falconers and we always call these the steelies because when we're looking for falcons to trap, some of your bigger falcons like the Jeers and your peregrine falcons like to sit up there in the morning and catch the sun as it's rising. So that's how we find a lot of those birds. So those are the transmission operators and their field devices out in the field. The upper right-hand corner, that's the distribution field devices. These are the devices that are put out in the field to control the power that's ultimately flowing down into each one of our houses. That's primarily done through what we call substations. Those are those big things that have the big fences around them, the security cameras, all the barbed wire. So those are substations. The transmission operator will take this bulk power, they'll drop it down to the distribution operators, usually in one of these large substations, and then from there the power is distributed out to the smaller substations that are closer to our neighborhoods and each of our neighborhoods that are connected back to those substations to be able to pull the power. We have different types of devices like feeder switches that allow the utility company to control which substation one single neighborhood is connected to. So if you ever have a certain circumstance where your power goes out and the power is only out for five to seven seconds, feeder switches and relays are going to automatically be connecting it back over to a different power source, and so that's why you drop out just for a couple of seconds and you come right back on, because they've had some automated event switch you back over to a different power source to try to avoid the power issues that you're experiencing. Now if you look on the very bottom, the things that are probably more interesting to a lot of you are going to be the devices in our home and the lines and the communications between our home and the electric utility companies. That's the bottom right hand corner with all the green. So these are the smart meters that are inside of our homes and the other devices that we ourselves bring, some of our home automation devices, those that have electric vehicles. How many people have electric vehicles in this room? Out of curiosity. Now that we're actually going to start buying them in quantity, so we have one or two. So now we can start buying them. We're going to start seeing more of them. Now look at this diagram. This kind of gives you the overall of the different components. What I'm going to do is I'm going to remove the labels and remove the nice pretty fluffy clouds and show you the communication links between each one of these devices. This will be lovingly called the spaghetti diagram. This is a diagram that I created for, I should say Darren Highfield, one of my partners and myself. The two of us created this diagram for NIST in an interagency report that we released last summer. So if you go back and check out this NIST report, you can see the reference on the very bottom for those that are interested in seeing it, 7628. This document is about, I don't know, 700 pages in length. It comes up in three different volumes. There's a lot of good information. If you're interested in learning more about some of the issues, some of the more details, some of the concerns, some of the security architectures and security controls that we're recommending. At a high level, we realize that this is high level. This doesn't go into a great amount of detail. That's a great document to be able to get that information from. Now what I'm going to do is you see this overall architecture and the communication links. I'm going to point out in different areas that we hear all the buzzwords. Everybody has heard buzzwords like SCADA, right? Everybody has heard buzzwords smart grid, or excuse me, smart meters. I'll show you exactly where each one of those different areas are. So first off, the SCADA. When we talk about SCADA, these are the types of devices that allow us to read sensor information from the field and be able to then make decisions on sensor information and send control signals back out to the field to cause reactions and changes in our real world environment. So you can see those blue sections in the upper right-hand corner that are circled. Those are going to be the sensors in the different devices, IEDs and other types of devices. Yes, IEDs does have another meaning inside of the smart grid and are to use the devices that are controlling a lot of the devices out there in the central brains that usually collect a lot of the sensor information and send it back. We're using SCADA protocols to send those back to what we call our back-end SCADA systems. So here that one single yellow box that circled is our distribution SCADA. In the smart grid we have several different protocols that we use, some of the earlier protocols like Modbus serial communications across basic serial lines. Then as history went forward we started taking that serial communication packetizing it, throwing it on TCPIP streams or UDP streams, sending it back across higher bandwidth lines back to the organization. And some of the newer protocols, DNP3 is probably one of the most commonly used SCADA protocol in the electric sector here in the United States. A lot of these protocols have very, very limited security and it's something we're trying to address. We're trying to build new protocols to replace them. DNP3 being the most popular one it's only been about two years ago that we even had encryption capabilities inside of DNP3. And we are looking at replacing DNP3 with additional protocols that have much stronger security models. So that's distribution. Transmission SCADA is very similar as well. Of course they have their own field devices out there. There's usually their own SCADA, Transmission SCADA server and a lot of those signals. So you can see that there are some communication links between the distribution and the transmission devices. And then we have last but not least the bulk generation. Generation plants, nuclear power plants, all those guys, yes they have more infrastructure than what we show with this one single box. The reason why it's only one box is that was for the most part out of scope other than the lines of communication between those bulk generation and the utilities themselves. For the work that we were doing for NIST. So that's why you only see that one box. But realize that there's a lot more technology and a lot more devices and communication links inside of the bulk generation. The next buzzword we want to talk about is the electric vehicles or the PEVs. A lot of us are to bring these back to our house. Right now they're fairly simple in their communications model modules. For the most part when you get a PEV you are going to be talking to your electric utility company and they'll usually either issue you a separate electric meter for your house that's specific towards your electric vehicle. Because each one of these electric vehicles on average consume about the same amount of power to charge its batteries as your whole entire house uses. So there's a lot of power going on there. Each utilities are trying to find different ways and different strategies to try to deal with additional load inside of our infrastructure. So part of that with separate meters sometimes they'll have you just plug into your normal links inside of your house but then you usually end up getting charged more. Most of the utilities will give you a price break by having the separate meter in there. Right now like I said there's little to no communications. There's a lot of work going on with communications to be able to allow your PEV to communicate back to the electric utility vehicle. It'll be interesting to see whatever happens with this but part one of the initiatives and some of the vendors are trying to get to the point where the electric vehicle can self-identify itself and when you plug power in or plug your vehicle into either your employer or your neighbor's house or some family member's house a lot of people in the industry have this idea that they want to be able to charge track where your vehicle goes and it still gets paid on your bill no matter if you're plugging into your neighbor's house or your family's house or somewhere else. Of course this becomes a security nightmare trying to tie this together with the electric utility companies. For most of us in the states it's fairly easy to reach each city or each town tied to one single electric utility vendor it's not quite as easy a case with those people down in that little state called Texas. They're a little bit different beast for most of us in the electric sector. The next area, syncophasers. Syncophasers are another technology. Now for those people in the room who understands the difference between a digital multimeter and an oscilloscope okay so right now if you want to think about it the way that in general once again this is over simplification but in general the way that the electric utilities right now are controlling and measuring the power inside of the grid is more or less with a whole bunch of really smart digital multimeters that are in all the different substations that are taking readings on average about every two seconds to find out what's going on how much power is being used and all the other pertinent measurements that they need they realize that while that's necessary well that's good they have some good information it's not quite as fine at as they need and so they're trying to employ additional technologies to give them something that's more visibility like you would see with an oscilloscope instead of just getting a digital number telling you what your what the power is they're putting these devices out that will more or less let them recreate that sine wave or what the power is really doing they actually call these phasors and they do phase angles to try to figure that out these measurements, these synchrophasors are making readings of the power at minimal 30 times per second at maximum a lot of the vendors are doing up to about 120 and there are some discussions about pushing it all the way up to 240 times per second down the future in the future but right now I'd say probably 30 to 60 seconds or 30 to 60 times per second are what most of the utilities that are employing these synchrophasors are trying to get to to give them a little bit better idea on what the power looks like at one end at one state to the other state across their whole domain because in general that sine wave should be nearly identical across the whole entire grid the United States is actually broken into three separate grids we have the east, we have the west and we have Texas like I said they're kind of their own entity and their own beast and they have a little bit different issues to deal with as well when it comes to power so that's what the synchrophasors are the synchrophasors are primarily being used by your transmission operators the people that are actually pushing the power long distances this information is also being sent back up to what we call regional coordinators regional coordinators are entities in the United States that try to work with the transmission operators and help the transmission operators balance out and make sure that we have a nice stable grid so there's about 15 different regional coordinators inside of the United States that help try to manage this power the next the buzzword we usually hear about are the smart meters themselves while these are definitely fun devices to play around on this is something that is a relatively minor a minor issue that we have compared to a lot of the other issues that we're facing with the smart grid so down here you can see that the smart meter in that bottom right hand corner the one right in the middle that's going to be an interface that is usually deployed on the meter itself it's not necessarily its own device in most circumstances for residential people when we get into larger deployments for corporations and more so even in industrial there's going to be a separate interface to be able to control and manage all the different meter readings that are deployed out there so this is where the meter is there's lines of communications between the smart meters that are being deployed back to the electric utility company to what we call a head end that's in the back of the electric utility company with this infrastructure traditionally for most of the people inside of this room most of you do not have the new smart meters on the sides of your house if you're interested to find out what type of meters you have on the side of your house just simply go out there look to see what the manufacturer is look to see what the model is it's usually very visible right on the front of the meter once that into Google and you can get spec sheets on any of these meters to tell you information about what capabilities it supports a lot of you will be able to go to the meters and you'll find that information you'll also see this little acronym that's ERT on the front of the face of the meter what ERT is is this is a protocol that probably the majority of us inside of this room have inside of our meters this is not the new smart meter technology this is a one way broadcast it's a protocol that the meters will go through in protocol and about every two minutes what their data consumption is so one way there's no way to be able to use this communication protocol to talk back to the meter and it's broadcast out using a 900 megahertz protocol that does frequency hopping well doesn't even do frequency hopping it randomly chooses when it comes up to it's time it's allotment every two seconds it randomly chooses one of 40 channels to broadcast that information on so that way it tries to avoid stepping on top of anybody else inside of the neighborhood that's what we have with meter readers they used to have to come up and read the meter themselves now they can just do drive-bys because they're collecting these ERT signals that are being sent back out there are other protocols besides ERT there's also some that transmit and communicate over the power lines themselves but the ERT is probably the most commonly deployed precursor or semi-intelligent meter that are being deployed right now in fact we call these meters AMR meters these ones with the one-direction communications that are out there as opposed to AMI meters AMI meaning the more intelligent ones that have bi-directional communications the AMI meters, the new smart meters their deployment is relatively small depending on the research that you see you'll see that deployments in New York from 10 to 25% across the US once again there's a lot of debate on the exact number of those smart meters out there and exactly what's the difference between some of the smart meters and what are not some of the smart meters so to dig a little bit deeper into the smart meters themselves that generates a lot of interest in general if we take all the detail information out and abstract this to a generic architectural view this is true for most of the field devices in general that we deploy in the smart grid just differences of protocols in the exact terms of the devices but when we look at the AMI meters themselves we see on that far left hand side of this diagram the electric utility company themselves and all their back end systems that's reading this information the very first device that's part of the AMI network these servers that are purchased with the meters are going to be these head ends these head ends usually are protected by some type of a firewall or some type of perimeter that they set up before we get out to the field devices then of course we'll hit some routers and the routers will then put it back out for the most part to the ISPs and the the telco companies that are connecting to them most of the meter data is going across cellular connections so you can see that the different links we have there the good majority is through cellular connections out to these meters we do have some proprietary third party offered RF towers we also had certain circumstances especially in the case of industrial industrial customers will have leased lines out to each one of these communications for the meters go down then these control signals when they tell the shut the power off at your house we'll go down to what we call the drop point or the aggregator for the network these aggregators are devices that are deployed out in the field either as a pull top device up on top of a pole or as a meter kind of a more intelligent meter that's placed on the side of one house inside of your neighborhood you can tell when you have these take out points these aggregator meters when they're on the side of somebody's house because in general these will usually be sticking out a few inches further than all the other ones inside of the neighborhood because they have to make room for that additional communications namely they have to make room for the cellular communication module that's inside of the device beyond that in some deployments in some of the vendors that have the smart meter products that are being sold they will have some of them will be deployed with cellular modems in every single one of the meters but I would say that the vast majority of the vendors that are selling here in the United States instead will have a meshing technology set up so that all the meters inside of the neighborhood will set up a communications mesh to get that data and assemble that data back to the take out point or the aggregator to push that back up to the utility company itself of those devices that have this mesh network it's rather interesting every single one of the vendors are out there while they may be using meshing technology and almost every single one of them is using a 900 megahertz frequency to do that meshing technology every single one of the meshing technologies and frequency hopping patterns of those 900 megahertz spectrum communications is different generally what the vendors are doing is they're choosing chips from TI and the other different types of communication chips that are out there that are generic off the shelf commodity hard chips for their boards they're taking these they are taking the configuration file that dictates the frequency hopping they'll build their own frequency hopping algorithm and the frequency hopping patterns based on a number of different factors by changing the frequency hopping they can change the amount of bandwidth and the amount of distance they can get out of some of the devices they'll tweak that to get the magic numbers that that vendor is interested in and then on top of that they'll build the meshing protocol and every single meter vendor out there that has a mesh network has their own proprietary mesh network there's no shared technology between the meshing technologies above that meshing technology we'll see a combination of vendors I would say probably about 50% of them are using a standard protocol for meter communication called C1222 each one of them of those 50% that are using C1222 even their implementations are different while they meet the specification to the letter they don't necessarily meet it to the point where it's interoperable with anybody else because of a number of different factors the other 50% of the vendors out there for communications they're going to be using a TCP IP connection either IPv4 or IPv6 depending on the vendor and then pushing their own proprietary protocol across it either raw C1219 tables which is C1219 tables or how each of the meters actually store their data inside of the meters themselves or they are going to be using some standard protocols they might be using some web services across it many vendors out there doing web services or XML data streams and exchanges on those meters but there's a couple that are doing that here in the United States of this area of the mesh when we think mesh wireless mesh at 900MHz most of us automatically think ZigBee at least in the field as I wanted to clarify I did mention that every single vendor is using their own proprietary one that means that nobody is using mesh for this meter to meter network neighborhood every single one uses a proprietary and most of these devices are using ZigBee which most of them do have ZigBee modules in them are between the meter down to the devices inside of our house so that's where the ZigBee communications come into play and so if you can get these slides later you can go through and see some of the protocols and dig a little bit more into it if we look at the payloads that are being sent across to give you a high level idea of what happens in the type of communications that pass back and forth between these devices notice that the first block of payloads primarily are communications between the meters themselves and the head end the one below it, the three lines below it at the very bottom of the slide these are pass through communications so these are going to be the communications that come from the ZigBee network inside of our home area network the devices inside of our house passing back to the utility company these will usually either be tunneled across the connection in some situations they will usually be data inside of some of the C-1219 tables that are being passed with whatever communication protocols are there the head end will simply pull out that input from that table, pass it back on to whatever devices need it on the back end for the most part a lot of the utility companies aren't doing a lot of those pass through communications yet it's there for future use some of the goals that they have is with demand response programs this is where a utility would give to the power that you're paying in order to allow them to have some limited control of some of your high consumption devices in your house like your AC unit during times of peak flow so for instance I live in Salt Lake City, Utah a lot of people up there the electric utility company Rocky Mountain Power give this discount if you have this device called Cool Keeper put on your house between your AC unit and the control unit of your house and they'll lie when it's the highest temperatures when it becomes a threat to the power stability inside of Salt Lake because we're having a bit of a power issue over the last couple of years they can go through and they can power cycle for small intervals the AC units inside of the neighborhoods and they'll actually do this in a coordinated fashion they'll take approximately one fifth to one eighth of the AC units inside of the neighborhood they'll cycle them down for five minutes or seven minutes and then they'll go ahead and let them cycle down the next fifth for that amount of time so that way they're at least trying to decrease some of the load inside of the grid and they've had a lot of success with doing this and as customers we appreciate that to some degree except for on the hottest days when they actually start turning the cycling on but otherwise it's these are some of the different types of programs they have expect to see additional demand response demand response programs here in the future on the meter communications themselves we have some basics of course we have the consumption data that's coming back to the head ends themselves we have control signals to be able to turn off power at the house part of the reason why they have these communications in there is when people move out it's always been a big problem to have people going and squatting inside of the houses and getting free power so now when you call up and say hey I'm moving can you go ahead and remove from the bill they can immediately just shut down power at your house when you move in you can call up and say hey can you re-enable power they can do that while you're right on the phone with them that one piece of functionality that remote disconnect is probably the greatest threat from a security perspective of these meters I would say the other major threat from these meters themselves is realize that these meters are collecting information and sending them back to a controlling server on the back end and any time we have data passing input data from a meter back to a controlling server on the back end there isn't a chance for an attack there I've never seen any good proof of concept code to be able to attack any of these head ends there's always a concern in my book and something that I think personally deserves a little bit more research that are out there seeing if you can take the few input fields that those head ends are accepting from the meters and seeing if you can get a buffer overflow or some other type of attack to gain control of that head end because ultimately that would give you the most control over these meters themselves there are other attacks out there with vulnerabilities if you can find the right combination of vulnerabilities you might be able to get in and try to control some of the other meters remotely you might be able to communicate on these mesh networks or if you repurpose one of the meters and try to make it so it communicates on it if you can get the right combination of vulnerabilities you can do things and attack the meters remotely the good thing is is most of these meter manufacturers are on their third and fourth generation products right now if you go back and look at the first generation products it was a huge nightmare and very very much a possibility to be able to perform these types of attacks with the current models right now with these third and fourth generation devices that are currently being deployed it's a much harder attack surface to be able to find that right combination of vulnerabilities to attack that infrastructure most of them are doing a fairly decent job now with their security infrastructure and security architectures okay so attacks and defenses that gives you a little bit of idea of what the architecture looks like so let's talk a little bit more about some of the attacks that we perform as penetration testers and some of the defenses okay so this is an oversimplified chart realize this is not what a utility company really looks like this is highly oversimplified this just goes through to show that we have client side attacks, server side attacks network attacks just like any other industry we do have that additional vector of network attacks in the field devices and the hardware attacks on these devices as well so really fast client side attacks it is a threat honestly out of all the attacks are out there inside of the smart grid all the attacks that we can do to the meters on our homes all the attacks we can do to the big iron products in the transmission operator substations that are out there at the end of the day these are the things that personally keep me up at night the things that I'm most worried about because electric utility companies are just like any other organization out there they have internet links they have clients that are surfing the web and while they're for the most part they're control center technicians and control center operators they don't have direct internet access on their workstation they do have connectivity to services and different devices inside of the organization so if an attacker comes in through the front door and sees a presence inside of the organization it's only a matter of time before he can get his way and work his way to the right workstations and the right subnet to be able to gain access to some of these servers so that's what keeps me up at night I personally think the day when we see the biggest incident from attackers that affect us from the smart grid I personally think that the front doors that launch this attack will be coming through a client side attacks server side attacks nothing different here same vulnerabilities realize that all these control servers that I showed you the ones that are controlling all the communications for the SCADA networks and for the AMI meters this is commodity operating systems on commodity hardware so the things that everybody in this room does for penetration testing and devices your knowledge is directly applicable here for the server side attacks and just like any other industry we have a lot of the controlling interfaces are moving from that client type control interfaces and serial and terminal they're moving to web-based interfaces as well so things like cross-hidery press forgery immediately become problematic and for those of you that do web-pen testing you know cross-hidery press forgery is out there and probably 95-98% of every single web interface you ever touch so it's been a huge issue and this is something that's a very valid attack angle for us network attacks in the network attacks when we're looking at the field devices inside the organization it's just like anything else most of the protocols that you saw in the earlier architecture diagrams that have communications most of those are using web services and other types of very common protocols that we're used to dealing with when we get to the field devices we get a lot of proprietary one-off protocols especially when we get to the substations because for a lot of the devices that were deployed in the substations in the last 20 to 30 years a lot of them were custom built for each one of those utilities and so they're very, very customized proprietary languages even when we have standard languages like I told you C-1222 for the meters each of the vendors are doing customizations of each one of those protocols as well so when we look at network attacks for the most part when we do penetration tests if we have enough time and enough budget we will go through and try to reverse engineering the proprietary protocols that are out there but a lot of times just like it always is the case when we do pen testing we always want to at least do a very, very simple check of the network communication protocol for some basics some of the base that we're going to be checking for is going through and checking for the cryptography that they're using checking for encryption just trying to see if they actually have encryption enabled at all other thing we often see is for the wireless communications is a misconception that frequency hopping is one of the security controls frequency hopping is not a security control with the right equipment and enough time you can backtrack and trace what that frequency hopping algorithm is and bypass that it's more of a method for obscuring than anything and ultimately the reason why we do frequency hopping isn't for security it's ultimately be able to provide higher quality of communications we also see problems with communications with cryptography because a lot of the protocols that we're used to dealing with cryptographic protocols in normal IT simply are too heavy handed and the embedded devices that we're deploying don't have that capability so we have very limited capabilities for some of these devices so a lot of times we're going to be messing with symmetrical encryption because it's a lot easier to have these embedded devices use it we do have some asymmetrical protocols out there but there's very few asymmetrical options that we have at our disposal so when we're dealing with these protocols one of the first things we're interested in is how they do their key updates and distribution because that's one of the biggest weaknesses in Achilles heels to symmetrical encryption when we get a proprietary protocol that we've never messed with before never seen before the first thing we're going to do is if we're going to try to determine whether it's an encrypted protocol or not and if it is encrypted try to determine how encrypted it is so here's some charts just showing you different ways to determine you capture the packets, you strip out the header information, take just the payload we build a histogram checking the entropy of the data inside of the payloads themselves the graph on the chart the graph on top is going to be this is the prime model that we want to see when we want to see encryption something that's very evenly distributed out the one on the bottom is still encrypted but it's not as good as in this encryption mechanism because we don't have quite the even number of balance and if you ever see when you're doing this histogram in a comparison a large jump right in the very middle and it right back down usually means you've come across an ASCII-based protocol because you hit the big jump right in the middle of the ASCII characters and very very few representation anywhere outside of those common ASCII characters so another issue we have is in the web world we have products, vendors that are selling web-based products that say yes we're secure because we use SSL and TLS we have the same exact problem that we have inside of the embedded world we have vendors coming up and saying yes we're secure because we use 8 yes so one of the first questions we're always going to ask is what types of psychromotor are you using for your 8 yes and trying to start digging down into the architecture there of the hardware attacks the hardware attacks does represent not just the meters themselves this can be performed by any of the pull top devices like the aggregation points or the feeder switches this also takes effect of the RTUs and the different devices in the substations themselves of those hardware attacks most of these are susceptible to different types of physical attacks themselves either just getting in and trying to get information back out or making some modifications in the hardware themselves a lot of the vendors that are out there have minimal capabilities or minimal controls around trying to detect when people are tampering with these devices but for the most part they're fairly fairly weak when we are trying to attack each one of these hardware the purpose for us to perform a penetration test on one single piece of hardware isn't to compromise that hardware just like everybody in this room knows when you have physical access to the computer we assume everything on that computer is compromised with some minor exceptions and some edge cases same thing with the meters themselves and any of the hardware field the embedded hardware devices deployed to the field we get our hands on these devices not to show that we can attack these devices and get information off these devices and try to get information to enable us to attack the other devices inside of the infrastructure some of the things I'm going to be looking for are the cryptography keys the asymmetrical or the asymmetrical cryptography keys that are stored on these devices because with access to this information I can go through and I can attack the other devices inside of the infrastructure either directly let's say I can get the infrared password because all the meters have that little infrared interface it's an infrared interface so I can go through and just very quickly launch attacks on each one of the other meters inside of the neighborhood but what I'm more interested in is getting the keys for the wireless mesh communication protocols so if I can get the asymmetrical or the symmetrical keys for that and I can find a vulnerability in the way that they've been implemented I might be able to launch other types of attacks like impersonating another meter or impersonating the head end and sending control signals down to the other meters from my device the other thing we have is this is something we've had forever it's not a new thing ever since they've had power people have been stealing power from the meters themselves there's a lot of information any of these terms you'll be able to get some details information about how people have done this in the past and some of the attacks this is basic stuff simply trying to steal power that's not all that interesting to us but it is kind of fun to talk about especially when you see really cool pictures like this of people stealing power piggybacking multiple meters together ripping the meter out just getting in and actually hard wiring communications back to one single meter or usually outside of that one single meter so it's not being read at all we also have physical bypass a lot of these devices are being tried to protect with different controls like locked cases and lock boxes on them and fences and perimeters and security cameras and all these things of course locks we can pick the locks fences we can climb the fences cameras well number one if they have cameras number two if they have cameras if they do monitor them how long does it take to roll something out if you get a good attack and you can actually replicate this attack and make it very easy to go through and launch you can go to each of the different substations launch your little attack get back out within a 10-15 minute at least give you some remote presence and wireless remote presence inside of that substation to be able to go through and continue your attack down the road so these are things that we're trying to combat some of the things that we're trying to address inside of the smart grid when we're working with the hardware itself told you we're looking for keys in the hardware we want to go into these keys and try to extract these keys out two methods that we use for extracting these keys out is either going through on these hardware devices and identifying where the data is stored at rest so the eProms the RAM chips the flash memory inside the chips the onboard storage of the system on the chips and the microprocessors trying to gain access to those and dump the data back off of them another thing that we'll do is we'll go in and identify communications between the microprocessor and other keys like the wire or other chips on the board like the wireless board and we'll jump in and we'll sniff the bus communications off those devices trying to find those key exchanges this is something that we'll occasionally find is in order to set up the encryption if a lot of the encryption is being done by the microprocessor and the RF chip itself is just a very very dumb chip building the median for it to communicate on we might not be able to get any data off because the data is already coming encrypted but if that RF chip has more intelligence and is doing the encryption itself quite often the data is being passed plain text in a serial link across that bus to the chip and quite often that key is also being sent clear text over that chip as well because the key is usually stored on the microprocessor itself so if we can capture that key back off we can get access to the cryptography keys for the communications channel so these are just some pictures of going in showing how we jump in on some of the different devices with different wires to it using syringe is that one thing on the left hand side that's a big syringe that we use it makes it very easy to use a syringe to go in and get some of the small surface mounted pins that are on the devices themselves occasionally we have to go through and make some minor modifications to try to get chips out of the way from interfering with our communications and try to stop the attacks that we're doing we avoid this one possible it's not always possible here's just a little screen shot of us going through usually using either the I2C or the SPI communication buses on these chips themselves so basically bypass the microprocessors, get our own hardware and straight into that e-prom or the flash module and dump the contents out here's an example of bus snooping identify the bus, jump in between the two chips that we're trying to get to capture the information from and capture that information so we can go back and do analysis later once we capture that information we have to identify the information that we want out of the side of this dump because a lot of times these dumps are very very cryptic two different ways to do this if we have symmetrical encryption keys we do attacks very similar to the attacks that they did with the Blu-ray disks where we systematically go through and take the exact length of the key that we know that they're using and we systematically step through the memory dump until we find a key that successfully decrypts whatever traffic we're trying to decrypt if you see here in the very bottom this is a combination of Travis Goodfet's Goodfet tool and Josh Wright's Killer B protocol that we do this attack in when we're dealing with ZigBee so dump the memory contents off of a ZigBee chip or grab it straight out of RAM we then use that binary dump to step through it until we can successfully decrypt a ZigBee packet asymmetrical is even easier asymmetrical keys are randomly pseudo-randomly generated so all we do is we go through an entropy analysis of the dump we have and as long as the rest of the data on the device is not encrypted you can very easily decide exactly where that asymmetrical key is so you see that one spike right in the middle that was the asymmetrical key inside of this dump so they're fairly easy to identify once we go through and we have this information once again the goal is to try to leverage this for other purposes some of the defenses we're recommending for the utilities is try to use system on a chip as one of the best defenses they have as well as try to limit the cryptographic keys that they're deploying out to the other devices another item that we're going to be using or looking for besides the key itself is the firmware because with the keys while we can get the keys as all great fine and dandy if we can get the firmware itself the firmware gives us capability and greater insight to these devices and you can go through and use these or binary binary decompilation and go through and analyze the binary itself or do source code review on the device those are two different options if you have copy of the source code if you are going to be doing binary analysis on the flash it does become a little more problematic than most of you do in binary analysis on common everyday computers because each one of these embedded devices has different microprocessors with different instruction sets so you quite often will have to go through and build decompilers to try to simply gain access to the instructions in the first place so that's a huge obstacle that we come across and for those that like writing decompilers we could definitely use a lot more of them out there for the embedded chip space so great huge opportunity for you to get into other than that for conclusion yes the smart grid is out there the smart grid does have security issues out there no industry is perfect but at the end of the day I think we are moving forward we have a lot of people looking at these devices and we can always use more for those that are running security shops out there and providing security services realize that there is a huge potential inside of the smart grid working with the vendors of the smart grid as well as the utility companies that are buying these products your skills are directly applicable in many of the different areas and for those that want to learn new skills there is a lot of new skills out there you can pick up with some of the hardware hacking techniques and some of the proprietary networking protocols that are out there so a lot of good information out there if you want more information you can go through and check out a couple different resources throughout the slide deck I try to give resources wherever I possibly could if you want more information about some of the attacks specific on my previous employer in Guardians they do have on their website an attack methodology that goes a little more into detail about some of the hardware attacks that are out there that's a great source of course I did mention the NIST interagency report that was released last summer you can go through and grab that the URL is up there and then also if you go to smart grid at pedia it's a great place to get information and specifically on smart grid at pedia there is ASAPSG one of the work groups that I've been working on for the last two years all the products that we create and all the documents that we create to try to help the utility secure their infrastructure so that they can be publicly on this website you can gain access to any of these and get more information about any of the specific domains that are spoken about today other than that, thank you very much I will go ahead and take questions on Q&A but thank you, thank you