 Hello and welcome to the DEFCON 29 ICS Village where today I'm going to talk about crippling the grid and examination of dependencies and cyber vulnerabilities and just what that means. So first, who am I? Well, my name is Joe Sloak. I'm currently the CTI and detections lead for network security monitoring company Gigamon. Previously I performed cyber threat intelligence research for domain tools and Dregos. And prior to that I've held multiple roles in the US government, including at Los Alamos National Laboratory in the US Navy. But enough about me. What's our agenda for today. So today we're going to first talk about understanding what the grid actually means as a concept as a really a machine, and then with that high level understanding will then dive into some of the particulars of how the electric system actually operates and potential these and injection points that could lead to cyber nexus attack routes in order to cause disruption and disruption in a way different than what we see with most headline events that have taken place over the last 10 years. And finally we'll conclude on a more uplifting note in talking about what is actually required to defend against attacks of this nature. So first, when we talk about the grid. I have to realize that really not talking about one thing from a North American perspective really talking about several things within the United States alone, we have the western Texas and eastern interconnects as well as a little bit out similar sections covering Alaska and Quebec. So we really have different sections different geographic regions that, while not completely independent of one another, certainly do create segmentation when it comes to the overall transmission and distribution of electricity, which actually gets us to another point that the electric sector is not a monolithic entity but really consists of entities spanning several distinct phases from generation what people typically think of when they talk about the production of electricity, but that generation has to be transmitted across substantial distances in order to reach consumers where final sort of last mile distribution takes place by other entities. And these entities can, and often are distinct from one another, representing different companies and it's important to note that if we look at historical events and impacts. Attacks take place at different phases of the electric system. So the two Ukraine events the first in 2015 targeted distribution, the second in 2016 targeted transmission. And we're not quite aware of any actual in the wild impact on generation, at least not at this time. It's not to say it hasn't happened it just means that no one's really talked about it or proven that it is the case, although there certainly have been some tests like the Aurora generator test that have touched on the subject. But to close this section up, there's no single grid, we have multiple phases of operation throughout the process of delivering electricity to end users, and different organizations often operate at different phases of the this operation. That already gets us a pretty complex attack surface between different regions different organizations operating within that region, and how they might interact with one another, but there's a lot more going on than just this high level view of the electric system. Because but wait, there is more. Because there are additional system characteristics that tie into how the electric system operates as a system of systems as one continent spanning machines, so to speak, that gets us into items such as requirements for generation. The stability of the system in order to operate and operate in a way that's safe, as well as the actual delivery of services to consumers. We're going to touch on this last one that much but it is something to bear in mind, especially as we see increasing adoption of things like smart metering and other ways of transmitting more control and more interaction, all the way to end users that previously just didn't exist before. That's an entirely different topic that we can do an entire presentation on on its own. But we'll talk about though, are generating sources and fuel. So, while it would be ideal that electric generation was more renewable in nature or even nuclear I would argue, primary ways of generating electricity across much of the world right now consists of fossil fuels principally coal and natural gas. And because we're talking about burning a substance in order to generate steam that then turns a turbine. We need fuel in order to make this process work. Well, that's a pretty important vulnerability and how these systems work because if you can somehow cut off or disrupt the supply of fuel that can be problematic very quickly. So we have coal storage yards that can be quite large and have maybe several days worth of supply, but natural gas is typically a just in time, very low storage at least on site proposition. So disruption in these sorts of markets could be quite catastrophic quite quickly. So we have balancing authorities when it comes to things like system stability. So we already looked at the grid is consisting of at least three high level distinct entities, but within those entities. We have a further set of responsibilities for balancing the frequency phase and similar characteristics as well as matching the actual amount of electricity generated with demand at any given point in time. In order to make sure that the system functions electricity has to be consumed essentially as it's created and has to do so within fairly well defined tolerances were to do so safely without damaging equipment. And that's what these authorities are set up to do to organize these systems and make sure that everyone is playing along the right set of rules. So the thing is, is that if we start getting a unstable or an unmanaged grid within one of the balancing authority geographies as well as through the entire interconnect bad things can happen. So for example in North America, we generate around 60 Hertz in terms of frequency corresponding to really just literally how a generator is turning and how many revolutions per unit of time. So that is we start getting away from that frequency, even by relatively small amounts, we induce the possibility for either errors in the system but also set the stage for outright equipment damage that needs to be mitigated through various controls. The same applies for things such as the frequency around which the, we talked about frequency the phase around which electric generation takes place and how this is then communicated throughout the system on to transmission and even distribution levels with other physical equipment balancing authorities step in to make sure that everyone participating in these systems is doing so in the right fashion and doing so in a way that allows the system to function normally. Now, I've just talked a lot of very, very high level user complex topics and we can talk about these for quite an extended period of time, but you didn't come here to talk about power engineering. You came here to talk about cyber and we haven't really touched on cyber at all just yet. But the thing is is that cyber actually has a role to play in this and it's one that's increasing, and that leads to several risks. For example, just recently, a few months ago, we had the colonial pipeline ransomware incident. This impact all the electric sector, particularly this was a petroleum pipelines that we're talking about vehicle fuel aviation fuel and similar. This does show that pipelines can be fragile to a certain extent now in this case, the disruption to colonial pipeline, never really resulted in ransomware induced shortages, mostly panic buying and quite frankly human activity produced the impact in this scenario, but it does show that pipelines can be impacted in ways that have physical repercussions through something like relatively indiscriminate ransomware operations. Well, what does that mean we start talking about natural gas and not patrolling natural gas, the sort of natural gas revolution in power generation in North America is astound. And we start here that where natural gas used to be sort of a lower level contributor to the electric generation mix of North America. It is rapidly grown to become the number one source of electricity within the United States. And the thing is, is that while that is fueled by things like the shale gas boom and other sort of operations still requires the movement of natural gas from its point of extraction, typically places in Texas, and the northern Midwest and other areas to get to generating facilities typically typically closer to populations. And as you can see from this map, there are some pretty interesting choke points and we're starting to look at this as a system. We see around major gas development regions again like Texas and the Gulf Coast, we see lots and lots of pipelines and lots of reliability or at least fall potential fall failovers within the system. But as we start getting into areas like the Western United States, the Eastern seaboard, the Florida peninsula and other areas, start seeing fewer and fewer redundancy within the system. So if we were able to induce some sort of disruption to pipeline systems on the gas sector. Well, the thing is we already seen some of these events take place but they did so at a fairly small level. So in 2019, there was a notice put up by the United States Coast Guard of all entities that a relatively small pipeline facility was impacted through a Ryuk ransomware event that resulted in a disruption. It was a minor event given the size and scale of the facility involved, but it shows that we can get disruption of petrochemical including natural gas infrastructure through something like ransomware. One thing to keep in mind is that just recently CISA announced, or at least attributed and made publicly available a report that existed for some time. A widespread intrusion campaign from 2011 to 2013 linked to the People's Republic of China, that was targeting pipelines at a very thorough, very widespread sort of basis. These areas are either sort of unintentionally from the ransomware perspective, or very deliberately through very long duration intrusion operations involving pipelines exploring this field to what end you may ask. Well, if I can identify a critical pipeline or segment, maybe not getting quite down to an individual compressor station but at least identifying the control center that would control multiple compressor stations and similar infrastructure for a given pipeline. That could potentially disrupt that control center or similar functionality through a cyber event and resulted in a disruption of pipeline operations. That would restrict the flow of gas to follow on generation stations as well as things like home heating and other items that could start to create a real physical impact. Now certainly some of these systems are well segregated and there are defenses in place but given the trend line away from very local very hands on sort of management to be systems, and instead to having distributed management from a single or handful of remote control centers, this sort of scenario starts become significantly more impactful and realistic than it may have been even just five or 10 years ago. In addition to things like feedstocks we also have stabilization impacts. So these are two examples that took place in early 2019, ironically both on the same day in 11th of January of 2019 of frequency deviation events one in the United States, the other in the European Union. In both cases there were essentially unexplained at least at the time, significant frequency deviations that resulted in non trivial impacts to the overall electric system, shedding load of people not having power, taking plants offline in order to prevent physical damage and other things. While some explanations have emerged for why these events took place it's still very strange when you start looking at either and so he or this project from Oak Ridge National University of Tennessee that tracks the sort of anomalous events that these are quite common but systems can step in in order to prevent the worst case scenarios from developing as a result of these grid wide phenomenon that take place. So examples of those sorts of impacts include just simple operational efficiency as well as wear and tear. So having things like protective relays stepping in to take a system off or put a system back on. There are significant frequency deviations does generate wear and tear and that equipment. When things get a little bit more dicey and sort of what we saw in the European example, we can start seeing asset assets become isolated so moving equipment from the system, or shedding load in such a fashion that consumers experience a blackout in order to make sure that the system remains stable. Finally, we could even get into potential physical damage scenarios. So the Aurora generator test that I spoke about earlier, involved a manipulation in the electro the protective relay system for a generating asset, and by manipulating that in just the right way so that you resync the generating asset out of sync to frequency or phase you can cause it to basically shudder and do that enough times or with a large enough deviation, you can actually induce physical damage. Now we haven't really seen anything like that take place, at least not to a breathtakingly destructive fashion because there are many systems that step in in order to prevent these sorts of scenarios from occurring. But it is something that the grid has been designed around because it's a very real risk to equipment. So from a cyber perspective though we're starting to see adversaries become more interested either incidentally or very deliberately to the types of systems that step in from a balancing instability perspective when it comes to electric system operations. So for example within the last two years, really actually just within the last year this took place last year, there were two events of interest. The first involved an intrusion at the European entity and so he, which essentially links the transmission operators on the European continent together to allow for better communication and whatnot. So and so he doesn't really control anything itself but it's an awfully nice springboard to getting access to the TSO organizations that are underneath it to facilitate follow Not what you would typically expect for an adversary to really go after for an intelligence value, or anything else and there was a ransomware in this case, but a very intriguing target for what it might enable for follow on operations. Around the same time, there was also a ransomware event that targeted a entity in the United Kingdom called Alexa that's responsible for essentially market operations to match electric generation with electricity demand. While it's very debatable that taking down Alexa would result in any actual physical damage to the underlying electric system. It certainly would increase the amount of friction in play in trying to operate deregulated electricity markets. And so we can't quite ignore that either. If we go back a little bit further to and it's not on this slide. We could look at the 2016 Ukraine power event as another example of at least approaching some aspects of this frequency and stability question when the attack resulted in manipulation and actually disabling of the protective relays on the victim power of transmission station as part of that incident. So adversaries are starting to learn that not only are these systems present and represent a sort of soft underbelly to the overall electric system, but they can also be reached targeted and potentially manipulated through computer network operations. And we have examples of this taking place, albeit at very low and at least not yet, or at least not not known destructive levels for over five years if we go back to the 2016 Ukraine incident. So possibilities for an adversary in this sort of grid stability attack scenario range from denial, walking access to system wide resources and removing system wide safety and other sort of checks that allow for efficient and very integrity focused system to disruption where we can outright disrupt operations as they shut down in order to preserve safety in an unstable or just uncertain operating environment to outright destruction. So like we talked about with things like the Aurora test and something like the 2016 Ukraine event there are possibilities to induce physical damage. This would be really, really, really flipping hard to try to actually execute and practice because it would require setting off the event so to speak by creating the network wide instability, but also removing protections and other controls that are in line to prevent worst case scenarios from developing so an adversary would have to do a number of things to make this possible. However, the increasingly networked nature of a lot of our grid operations systems as well as individual portions of the electric system as it may exist, mean that there are increasing possibilities for adversaries to reach out and touch some of these elements. So we can't quite ignore it, even if this represents a very distant right or left tail possibility compared to some of these other scenarios. So that's all very concerning and present some problems, but as London would say, what is to be done. Well, the first thing is we have to really remember and recognize that the grid is a big complex project, whether we're talking in Europe in North America, etc. And not only is that, you know, in the classic case of like oh there are actually three grids in the United States, but also comes to understanding those dependencies and other items that allow for the grid to function in the way that it was designed to. That means that we need to invest in security visibility and redundancy across all phases of operation in order to really be able to respond to or counter high end threats to this environment. Part of that means making sure that we're increasing our visibility to try and match threats, we have that increased visibility we can then have greater assurance of deploying root cause analysis on any incidents that may emerge like those frequency deviation events we talked about earlier, and by applying this continuous analysis of our own environment, we could then marry that to continuous analysis of the threat environment, what lessons can we learn from what adversaries are doing out in the wild, such as the targeting of entities like E, or Alexa and pipeline operations, etc. Based on that understanding fusing internal and external perspectives, we can adjust defenses in light of that knowledge, and ideally, begin sharing that knowledge, not just with others within our sector, such as other power generators are similar, but also sharing partners and other entities that make this system function at a massive scale, omitting the balancing authorities and omitting fuel suppliers and transportation functionality means that we're essentially extending vulnerabilities out beyond where we can really control or see them and allowing for sort of round ways of impacting electric utility operations. Another part of this and since we are at DEF CON it's important to bring this up is that we also need to be a little bit more creative when it comes to our security testing and red team. Just saying like oh the bad actors can't get to our generator controls that we're safe omits ideas like some of the stability attacks and other items where we're looking at attacks that originate outside of our own perimeter. We can identify and test for these scenarios, such that we can identify when controls exist and what controls need to be improved, as well as for evaluating things like our business continuity and disaster recovery plans, should one of these scenarios manifest so that service can be restored in a timely fashion. So this is a very short talk and we only touched on things at you know the 10,000 foot foot view. I hope this was very insightful and really identifying that from a defense of electric utility operations. It's not just about the utilities it's not even just about transmission generation and distribution. There's an awful lot that we need to be concerned about and making sure that we have eyes on. So with that, I hope that this has been enlightening and leading to some interesting thoughts about where to go forward. This few resources available on this topic I've also written and presented on these topics elsewhere so plenty of other things out there that can be referenced. But with that, if there's time, we will try to take some questions. So thank you.