 solutions for cyber security for the smart grid, so again, Jeff, Colin, Galen, Chris, thank you so much for being here today and help educate us on this important topic. First a few words about myself and what I do, I'm in the Computer Science Laboratory at SRI International, which is an independent non-profit research organization. And my group does research in security for critical infrastructures. We also do R&D portfolio management for our clients, in particular for the Department of Homeland Security and their Science and Technology Directorate. So at SRI we say that we work on important problems, not just interesting ones, that's something we take pride in to address what's really important and we believe that cyber security for critical infrastructures is one of those really important problems. And this has a lot to do with the possible consequences of a cyber attack on an infrastructure. There are lots of interdependencies between our infrastructures, between oil and gas, telecoms, electricity and so forth, but everything in our modern society is dependent on the electric grid. So therefore, dependability, resilience of the electric grid is something that's very important. And this is because the consequences of an attack could indeed be catastrophic. So when we make the new infrastructure, when we put the smart grid in place, put smart devices, computing devices, data communication in place everywhere, we have to make sure that smart doesn't mean vulnerable or hackable. And unfortunately, that can be challenging because security is really not an easy topic. If cyber security was easy, we wouldn't have to be discussing it, we wouldn't have to be working on it, we wouldn't have to read about it in the newspapers every day about cyber security problems. For many reasons, many of which we'll hear about from our panelists today, security is a difficult problem. And also, it's not necessarily a topic of interest to many people. I'm very happy that you are here today, but it's a fairly sparse audience today and it's always interesting to come to these energy conferences, for example, everyone's excited about new technology and what we can do with it. And then we security people come and say, oh wait a minute, not so fast. But what I want us to hear about today is how we can actually use technology to also make security better and make sure that we become less vulnerable, not more vulnerable. So end of my introduction here and I'll ask each panelist to come up here, give five minutes of an introduction. So we'll start with Jeff. Please go ahead. Great, thanks very much. I better start my clock or I'll go 25 minutes. Okay, so I'm Jeff Gooding with Southern California Edison. Southern California Edison is a utility just to the south of here. We cover 50,000 square miles and serve about 14 million customers. I am the manager of smart grid systems engineering, so I come at cyber security engineering from a little bit of a different perspective. Prior to doing that, I was the chief architect of our smart meter implementation, 5 million smart meters. So if you have any tough smart meter questions, feel free to shoot them at me. Not literally, right? I know we're in Northern California. Anyway, so from my perspective, you know, we are aggressively pursuing sustainable energy goals. And what that means is we're introducing a whole range of new technologies, energy technologies to the electric grid that don't have the typical characteristics of generators that we've had in the past. So for example, right now, the grid remains stable largely through the laws of physics, large synchronous rotating mass propagates a robust waveform that goes all the way through to the end customer. And that allows the grid to ride through transient events, lightning strikes, all sorts of things that might happen out there to wires on the grid, all sorts of faults you might see, you know, most of the time. And that's what makes it highly reliable. What we're seeing with the introduction of wind, solar, power electronics, even some of the distributed generation is you're seeing inertia, which is generated by that large rotating synchronous mass, dissipate across the system. And the effect of that is you see frequency stability, which is really the job of the California ISO and the WEC to kind of manage frequency and voltage become more fragile, managing voltage is the job of the distribution company, which is my company. So what that means is the time scales that we have operationally to actually respond to these emerging events so that they don't cascade into larger problems across the system are becoming much, much smaller. So the way that we actually address that problem is with technology, right? Wright Brothers Plain, governed by the laws of physics, had a very specific mission. B2 Bomber, massive amounts of capability, inherently unstable in flight, and technology is used to keep it moving straight and deliver all those extra capabilities. So that analogy doesn't stand up 100%, you know, with the electric grid, but it's a good way to think about how we're going to use technology to keep the grid stable in the future. So when am I going to get to cybersecurity in my long-winded five-minute talk? Well, basically now. So the introduction of all of this automation, all of this connected equipment, does facilitate vulnerabilities. And those vulnerabilities actually expose us to threats that need to be mitigated by security requirements and fed back into the design, hence the system engineering approach. So what we're doing about that in Southern California, Edison, you know, similar to what we did on AMI, when we came into AMI and we said, okay, we're going to put a ZigBee radio in this, we're going to put a disconnect switch in it, and we're going to put a communications network that goes all the way back to the utility. How do we make sure that the network cannot be attacked from a single and seized from a single endpoint? And so we looked across other industries and we started talking to the Department of Defense and specifically the NSA about using some of their SweetBee technology, and that derivative, a derivative of that, you know, was used on AMI. Coming out of AMI though, we realized there were some weaknesses there, very vendor-specific solution. We worked across a bunch of standards bodies to try and get everybody lined up on requirements, but there was no interoperability between solutions because of the cybersecurity. This cannot happen in a smart grid because all of these multi-vendor devices have to be able to talk to each other and communicate, and the relays have to be able to very quickly send communications back and forth on the state of the system. So we started working again more with the federal government to transfer technology, and this is a HAPI's derivative if you're familiar with the cybersecurity standards in the defense and intelligence industry, but this is a HAPI's derivative technology, and we redid our architecture in the grid control center to make a common service. So a common cybersecurity service where any device at the end could access cybersecurity services in the grid control center, and we had a way to distribute that logic. So if we lost communications, there would still be a set of policies and a set of keys out there. So that's the work we've been doing. Now I'm looking down and it says zero. So I'll wind it up here. Anyway, that's the work we're doing at Edison. It's in Department of Energy, smart grid demonstration right now, running on a lot of radios, and we hope to be rolling this security out in response to NERC's version five requirements in the utility industry over the next maybe year and a half. So thanks. Good afternoon everybody. I'm Colm Lennon, and I work for Honeywell Building Solutions, and I'm the global offering leader for our service and energy lines of business. At Honeywell, I can tell you that we're obsessed with cybersecurity, and it has become an obsession because we do believe at all levels of our organization how big this threat is. I don't know if anybody has seen the recent report that came out from the Department of Homeland Security, but the first six months of this year, it's been 111 attacks on industrial control systems on the electric power grid. It's actually, it wasn't the electric part as much as the natural gas distribution network that some of these hackers were really going after. That's 111 in the first half of this year, and it was only 88 in all of 2012. So there is a significant increase, and so our obsession with cybersecurity is related to this, not only the traditional threats of hackers doing it for fun, doing it for fame, doing it for fortune, but now it's some of the emerging threats. Cyber security has become something that we have to worry about in terms of cyber terrorism, something we have to worry about in terms of cyber warfare, and so you can think about how important it is to build security right into everything that we do in Honeywell, because what we do is we build out the control systems for buildings like this, but also industrial controls that go into refineries and petrochemicals and power distribution networks and things like that. We have demand response automation solutions that we sell to the utilities to help them manage their demand response programs. So you can think about the potential impact that a hacker could have if they took over one of our systems. So that's where the obsession comes from, and we've gotten industry recognition. We are on the top 10 of the Security 500 benchmark, and our chairman and CEO has been at multiple executive summits at the White House on cyber security, is very much helping the federal government with setting policy and making sure that we're thinking through what needs to be done to protect our critical assets. And so throughout our entire organization, it is everybody's job. It's not just the job of our security folks, not just the job of our product engineers, not just the job of the people that go out and make sure that we're patching and updating our systems all the time. It's everybody's job in Honeywell. And what we want to do is we want to make it more of an ecosystem responsibility. Jeff just mentioned it very well, is you got the smart grid where you have all of these different technology players that are inventing technology and they all have to communicate with one another. So it's important for us as an ecosystem to make sure that we're addressing the cyber security problem and working together because the hackers are working together. The hackers are investing billions upon billions of dollars to get into these networks and wreak havoc. And so we have to make sure that we're working together and bringing the best and the brightest together, sharing our knowledge and our best practices so that we can get always a step ahead of our quote unquote competition, which is a very nasty group of people. So with that, thank you. All right, good afternoon, everyone. My name is Gail and Rasha. I'm a technical executive with the Electric Power Research Institute. And I manage our cyber security program for our power delivery systems there. And so if you're not familiar with EPRI, we're a non-profit independent research institute based here in Palo Alto. We actually do a collaborative applied research institute for the electric sector. And our research programs are funded by utilities from around the world. And cyber security is a very, very important issue to them. So why is that? Well, they view cyber security as being critical for protecting their current systems, but also as an enabler for deploying the next generation grid technologies. Because the fact is, if you don't implement good cyber security, you really can't or at least shouldn't deploy a smarter grid. So I'm going to ask a quick question here. Who here works in cyber security? So one person quickly dropped his hand. Anyone else? All right, well, hopefully you'll leave today with Ben understanding some of the basics of cyber security and how they apply to this market. And it's a very, very challenging area. If it was easy, it would have been solved already. But there's a lot of differences between IT systems and operational systems that make cyber security more challenging. And you also have issues of legacy systems and other equipment that's in the field that doesn't support cyber security very easily, that can make it very challenging. But I think the main thing to keep in mind today, though, is that even though cyber security can be challenging at times, it is still a key component of grid resiliency. So just keep that in mind. Like I said, it can seem like a showstopper sometimes if you try to bolt it in at the end. But if you plan it correctly, it can really needs to be that key component to help maintain a resilient grid. So we've heard about some of the cool technology that SCE is deploying with their CCS architecture, very advanced security architecture and somewhat Honeywell is doing. I decided to talk a little bit about a challenge area that's not quite as exciting, but it gets left off the discussion sometimes. And that's assessing and monitoring risk. So the fact is, right now, there's not a commonly used industry accepted methods for assessing and monitoring risk in the electric sector. There's been some good work that's been done recently on creating more common ways to do security assessments for utilities. So for example, in 2012, there was a White House initiative where they worked very closely with DOE and DHS to develop the electricity subsector cyber security capabilities maturity model, which it's a bit of a mouthful, but it goes by the acronym ESC2M2. And what that did is it provided utilities with a common set of guidelines to measure how mature their cyber security processes are in key domains, such as configuration management, asset management, identity and access management, threatened vulnerability management, situational awareness, all great areas. And so you could take that and see how mature your utility was in these different areas and compare that from one utility to another. But what it doesn't do necessarily is actually tell you how to assess your risk and monitor your security posture in real time. So I think that's a very big gap right now for the industry and one that we need to be working on as a research community and applied research community as well as working with utilities in that process. But one issue there that holds that back is also not having a common set of security metrics to be able to measure what your security posture is. Now, utilities have in general taken different processes from the IT space and adapted it to their particular situation. But once again, there's not this common way of measuring what that security posture is across the electric sector. But if you had that ability, then it'd be possible for the executives of utility to know where do they stand right now. And it'd be possible to have a much better idea for an electric sector as a whole. Where do we stand as a critical infrastructure on our security posture? And right now that's a very difficult thing to measure. So if you don't know where you stand, you can also still create and deploy great architectures and good incident monitoring technology. But it's hard to know exactly how secure you are without that. Good afternoon. I'm Chris Viller out of the California Public Utilities Commission. Thank you all for coming and inviting me to be on this panel. So before I begin, here's my usual disclaimer. Since I do work for a state agency, any opinions that I may express are my own and not those of the commission of which I represent. So on cybersecurity, the commission really got really involved, or the commission staff at least got more involved in cybersecurity beginning last September when the staff of the PUC issued a white paper outlining steps that a state commission, in particular California State Commission, may choose to do in ensuring the security of the grid going forward. And it outlined several reasons for doing that, as well as several challenges on how to make that happen. So I'm just going to talk about the challenges side of it. So the first thing to remember is that the PUC only regulates the distribution aspect of the electric utilities business. So all the wires you see driving down on the streets, those are the ones that we regulate. The big ones you see on the highway, we don't regulate. That's regulated by the Federal Energy Regulatory Commission. So right off the bat, we have a jurisdictional issue. On the transmission side, there are longstanding cybersecurity standards promulgated by NERC. There has not been anything on the distribution side done by any state regulatory body ever. So this is one of the big challenges is as things start to show up on the distribution grid that are smarter, that interconnect with other things, the challenge from the state perspective is, how do we make sure that those new things that Utility wants to put in are secure? What steps should a commission do to make sure that the Utility is considering cybersecurity on these new investments? One of the ways that we do that is as Jeff was talking about all of his fun little projects he wants to do is that we have to approve the budgets for those fun projects that Jeff wants to do. So if Jeff wants to do a fun little project and it gets rolled into the right case and we review it and we say that looks good, but it's too much money at this time of the day, this time of year. So that money you want, we're going to cut it in half. Use that money and go do your job. So as we start thinking about how do we understand what the fun project that Jeff wants to do, we have to start having an understanding of what does the impact of that? How do we know that the programs that Jeff wants to do, the risks that he wants to take and the way he wants to measure is how do we know that the money that the PUC has to fund for those programs is worth it? How do we measure the success of those things? And Gail is talking about metrics and that's one of the things that we are still struggling with here at the state level is how do we make sure the money that's being spent is appropriate? How do we know it's not gold plated? How do we know it's not enough? How do we know we're just lucky? And so that's one of the challenges that at least on the state side we still have going forward. Another one is of course staff expertise. The PUCs headquartered in San Francisco and we basically paid Sacramento money. So it becomes very difficult to retain or even hire the requisite experts that are needed to go into Edison, review their audits, make sure that their cyber security program is appropriate. We don't have that technical staff that can do that. For example, the three people sitting up here as well as Ulf, if you read their bios they have very long education backgrounds and long history of work. If you read mine I have a BA in history. So here I am a history major talking about cyber security. That's just an example of sort of the challenges that many states have. As going forward, so we have our white paper that talks about the electric grid and for the most part we're talking about electric grid. But the PUC regulates more industries than just the electric grid. We regulate the natural gas companies, investor owned natural gas companies. We regulate the landline telecommunication carriers and finally we regulate the water companies. So as we start thinking about this C2M2 or the risk management protocol that was also developed by DOE and we are focusing on electric, let's not forget the interdependencies amongst the industries. So without electricity does water get pumped and without water can many power plants run effectively. So how does cyber security policies that are developed, for example at the state level, how does that impact and how can that be implemented across other industries that are seeing similar risks. Water company has SCADA networks just as the electric utility does. Water company may be about 20 years behind on the technology investment but that's just, that's the perfect time to do it. You know we're talking about building security in to the investments. This is the exact time we want to do that. As we do smart grid, as utilities eventually, as the water utility especially eventually start catching up on technology, making sure the security is built in provides more cost effective solution for the rate payers, for the customers and for the utilities that the commission regulates. The last thing I just want to bring up is California despite having our paper and being in the press all the time is not the only state doing cyber security. There are many other states that are struggling with the same issue and investigating the same questions. State of Missouri has a rulemaking opened up last July asking the same questions. Now one of the challenges that we have is how do we have a public record for determining these policies. The Missouri PSC issued a series of literally 70 questions to their utilities to answer and that filing was all filed under seal which is good for the PSC staff and utilities were together but not very helpful for the public or the other experts that are really necessary for developing good policies on this. So as we go forward you know California the PUC is doing a lot on cyber security and we're getting a lot of good press but there are other states doing just as much work if not slightly more than us that as an industry we should all be aware that what happens to Southern California or PG&E is very likely to happen to other utilities and the ability of sharing that information across utilities with one another and across PUC is going to be a very valuable tool going forward and one that we should be working together to support. Thanks. Now we're going to go over to the discussion portion of the panel and I'm going to start by asking our panelists some questions and we'll have the one who feels most compelled answer the respective questions and we'll get a discussion going. So we heard in your introduction your respective introductions that the cyber security for the smart grid is facing many challenges. So I want to start by asking what are the most important things that your respective organizations and your partners are doing to make sure that the smart grid is as secure as it as it needs to be both looking at today's problem and also looking ahead into the future. So who wants to take on that? Jeff? Yeah I guess I can start. I think you know a lot of times utility companies approach problems like cyber security by looking at the minimum set of requirements they need to meet in order to just comply with a policy or a regulation. It's not just the utilities. Yeah and so it's like an epidemic across our industry. Anyway so you know I think when we start looking at how to solve these problems and how to bake a solution into and how to bake a cyber security solution into a new capability or a new solution we're developing we're really not just looking at meeting the minimum set of requirements anymore. I think the most important thing we're doing is trying to look ahead and figuring out how to design systems and develop architectures that are flexible enough to accommodate new requirements in the future that we don't really know. And that means from a practical standpoint what that means is a lot of the solutions we're developing are more like platforms that host policies. They're policy based types of solutions that allow us to respond to the changing threat environment. So aside from the investment and aside from all the money that Chris is going to send us I think you know really taking that focus that strong system engineering look ahead and making the most of every dollar that comes our way so that we don't have to rip things out and replace them in the future is really critical in meeting these challenges. Yeah I think I think that's key Jeff and one of the things that we always struggle with is our product engineers always thought about securities and afterthought so all of a sudden we'd be you know well down the product development path and somebody would say you know what we haven't taken a look at the security of this thing yet let's have a let's have a look at what's under the covers and then they'd get angry when we'd find 5 000 vulnerabilities on it and back to the drawing board and it was the security guys fault for finding all the vulnerabilities right so so over the years we've had to really teach our guys that security isn't an afterthought security is an upfront investment and that architecture is so critical and so it's it's kind of like building a house or building a commercial building I mean that foundation is necessary and absolutely critical from our perspective up about getting off to a good start and not making security an afterthought so with that solid foundation then you can build products and services that are going to be secure in their nature because you started off on the right foot and then after the fact one of the important things is just making sure that you've got process and governance in place so that you can check back and make sure that that security architecture has been implemented correctly because all it takes is one engineer to go down the wrong path and to take their own version of the architecture that you've set forth and you can open yourselves up to a lot of vulnerability so it's really important that everybody is engaged everybody understands what that architecture is and how it needs to be implemented and then having the process and the procedures and the governance in place to check that everybody's doing their work appropriately it's a very good point and since most of you in the audience perhaps do not work in cyber security but maybe you develop new technologies you oversee them you fund them you support them you promote them it's a very important point to make sure security is there from the beginning because we've seen so many times where something is readily developed and now let's look for security problems and guess what we usually find a lot of it all right and I'll add on to the end or the second part of the security cycle then so so we're talking about how you can build security in and protect protect devices but another key area that we've been working in is how to manage incidents and so that's how you detect respond and recover from cyber incidents because I think even with really good security protections in place there are going to be cyber security incidents there are going to be security breaches and that's mainly because it'd be very very expensive to implement all the security controls necessary to make every single system be impervious to any type of cyber incident it's just not cost effective to do that so we've been working with our utility members on this since 2012 and this year we've been focusing more on transmission distribution system and enterprise security as well and right now I'm even focusing on how to help them develop an integrated security operation center and probably nobody here has been inside of a security operations center but think of how to describe one if you may have seen a scene in a on a tv show in a movie where if you were going to this room with have maybe 20 monitors on the wall there's usually a picture of a world map on there somewhere and all these operators with four four screens in front of them with all these alarms and logs and things you know going by and things flash red on there you know and they're supposed to go do something about it then so that's kind of a gives you a visual image of what a secure operations center is or how it how it functions now we're not advocating something as grand as a global global secure operations center for everybody but but right now one of the challenges is that it can be very difficult to correlate incidents within a single utility that occur on the corporate systems and also on their ot systems so what's happening the control centers on their transmission systems their distribution systems their am i systems their physical security systems right now these are usually handled by different power systems groups and so they can be siloed there and so if there is an incident it can be difficult or take some time to go track down the information you might need from each of those domains and so some of our members have on their technology roadmaps now to build out an integrated security operations center and so then if you have that you can have this a much much better situational awareness for what's actually going on within the entire utility and it's not just connecting these systems if you think about how physically spread apart also you know the power systems are i mean you know you're really looking at systems over a very large geographic area so again let me let me stop you there and move to to a related topic of and that chris also mentioned about jurisdiction and the many stakeholders so in security for the smart grid we're talking about the federal government state governments the utilities the industry groups system vendors like honeywell here research labs like epri and sri and many other stakeholders so i'd like to hear somehow we we look at the respective roles of these stakeholders and also you can describe how you're collaborating with with some of these other parties today and also if you have ideas for how we can get better at collaboration information sharing and so forth sure i'll start so the usual mission statement of a regulatory body is is we oversee the utilities to make sure they provide safe reliable service at a reasonable cost one of things the white paper proposed is adding the word secure so we regulate the utilities to make sure that the power supply is safe reliable and secure at a reasonable cost for indeed if it's not secure your power can either be safe nor reliable um so that's one of the cultural challenges or issues that that the commission is being is that we're being addressed so what do we do is that we have we do work with the utilities i come to conferences like these i get on phone calls with a number of national standards development efforts that people talk over my head and i just focus on the policy side and how it implicates the utilities we regulate um that that runs the gamut from all sorts of things and the funny thing is you know we're talking about how we're at a technical conference like today with all fun all sorts of tech vendors out there and we talk about securities the guys in the room that always be that no one wants to talk about we'll also do privacy and if we think securities are the what blank you should try privacy talk in the big data people um the other thing is you know jeff was talking about flexibility and rates i hate talking about rates i'll be this i'll do this very quickly utilities come and ask for funding every three to four years so we have to budget for the utilities on a three to four year rate cycle i don't believe people hack utility networks on a three to four year rate cycle so how do we let the utilities and work together with utilities to make sure that that they have the appropriate flexibility to respond to these threats accordingly as they evolve over time because we can't wait three to four years for them to come back say oh we had something that happened three years ago we need money to do something to respond to that that is way too late and the pace of technological change is going to be one of these challenges i think we're all going to have to address at some point so chris you mentioned timescales and jeff you talked about that too and we see this challenge where it technology evolves so fast but we're we're looking at infrastructures where technology investment has to last for a very long time and even things like the am i the smart meters if they cost a hundred dollars each and you put them out to 10 million customers that's a that's a fair investment that you're probably not going to do every two three years when you need to upgrade them so this is a keeping up with all the big changes in technology and vulnerabilities in attacks in security technology and so forth that's a real challenge for for these kinds of of industries where you can't just throw out the server and put in a new one so what any of you like to comment on that yeah so i i think you know good am i lesson learn for me was you do want to try and keep the utility technology that's deployed and selected a fair distance away from fast cycle technology meaning fast cycle technology is dominated by consumer adoption of new technology and when you look at where those two things interface you want to make sure that you don't as a as a utility company pick a take take a really hard stand on a technology and i think that was a good lessons learned from selecting zigby and putting them in the meters now the world is going to be dominated by wifi zigby is not going to be the home automation standard we have five million zigby radios out deployed so what are we going to do we're going to put gateways out and it's going to be more expensive than chris and i had originally thought about to actually get this market off the ground and you need to understand that when it comes to fast cycle technology and consumer adopted technology you know the utility companies really are unable to participate in influencing those markets on the other side of the meter so what that means is we need to get comfortable using any communications pathway we need to develop company competencies that allow us to connect to the customer and their devices in a way in whatever way is available so if that's through the gm on star communications if that's through the internet to get to an inverter to get services that allow us to you know send a price signals down so that or some sort of signal down so that an inverter can behave to contribute to the overall health of the system or at least not do harm to the low voltage transformers then that's you know sort of the approach that we we have come to settle on i don't know if that exactly answers that question but no it does it's a hard problem yeah it is anyone else want to comment on that the only thing the only thing i'd add to that too is the the challenge is all over the place right it's in controls it's in you know critical infrastructure it's in servers it's in networks it's now everything and the pace of change for technology the pace of change in the security community in terms of their capability and the amount of money that goes into their capability is astronomical so it goes back to again what what i talked about earlier is that the ecosystem has to work together and combine forces more and more to stay a step ahead what that means is standard bodies are going to take more of a critical role in making sure that we're staying a step ahead uh companies like honeywell are going to have to work with some of our technology partners and our customers to make sure that we're implementing technology that isn't going to require a significant upgrade so that you can get more security benefit from it that we're making these systems in this infrastructure that's patchable and updatable so that we can stay ahead so it's it goes back to all of us need to be working together all of us need to be putting the best and brightest on solving the problem because the problem gets worse every day it doesn't get easier unfortunately i'm sounding very doomy gloomy today it's been that kind of a travel week okay we can switch to something positive well the funny thing about security though is that it's it's not all about technology and people related issues play a big role and sometimes they play a bigger role than technology issues we're all kind of technologists but would you like to comment on the on the people related aspects of security perhaps on the consumer facing things or what you do for for awareness inside and outside your organizations right well i think jeff could probably talk to this um very directly as a utility but you know one of the trends that's been happening um across the electric sector is um a couple of staffing problems so one is that i think in the next five years there's some very large percentage of of electric utility staff they're expected to retire you know so we're actually losing a lot of the um capabilities just on the power system there um but then also there are not very many people that have a very good solid understanding of how the power systems work how communication systems work how networks work how cyber security works you know that there's not a lot of people that that walk around with good capabilities in each of those areas and so i know that do we has been investing a lot in various programs to focus on workforce development for the electric sector there and i think one effort this out there is sponsored by the national board of information security examiners and they were looking at what are the security capabilities that you want operators to have and different roles within an organization to have and how can we um test for those capabilities you might have something like this cis sp which we have folks more on it systems what is the equivalent of that you know for an operator for the electric sector and so that's been an ongoing effort um but it's not not complete yet so what i'll say is that we have there there are three the three points i want to make on this one one is um you know the chat the people challenges you know as we've heard all day today in california there are many of the policies on the energy side is pushing more and more things on the distribution grid on to things beyond side customers homes so as we think about the people's issue the utility is going to have less and less control over this so how are the people setting up their distribution grid or their uh dg units their storage units things that are on the side of their house or inside their home that utility hasn't control over so utilities have to handle that uh two is an issues as again we're talking about people we have supply chain questions still you know the utilities and honeywell and they have to are relying on vendors to provide them with things and how sure are we that those things that they're buying are secure how secure is this supply chain of those vendors um so how we determine that level of comfort is going to be a challenge and the third thing um the commission had a thought leader's area this year and what was repeated over and over in this conference is that the number one cyber security threat is still phishing getting an email and someone clicking on something they're not supposed to be clicking on so the phishing and then allowing the malware to to warm their way into the utility networks just through email is still what we're told is the number is still the most likely successful means to hack into a utility network it's amazing what people will click on yeah but um one more thing to add to that about security awareness uh in honeywell we've implemented a security awareness program that's very comprehensive and every employee is educated on a very consistent basis about not only things that impact honeywell but things that impact their personal lives like how do you protect yourself from phishing attempts how do you protect your your identity and and how do you safeguard your your family and your children from security issues and so that combination of uh training related to security in the job or in the workplace with making it personal and providing them some value in their home and in their everyday lives has really gotten people's attention I think and focus and awareness of every day that security is something we need to be thinking about so that's one of the things that I would recommend in all your organizations is to think about how you make it personal not an email that you ignore but an email with a newsletter or an email with a link to a webcast that people really think about going to because there's some value in their personal lives right very good so I hope everyone here knows what what phishing is about it's when you when you get that email that says you won the lottery or it's from your bank or your airline click here just enter your social security number and everything and your your debit card pin and you'll be fine don't click on those all right so we're going to open up uh the questions from the audience just before we do that uh is there any uh sort of key takeaway messages from the the pattern for the audience today that you want to make sure that we state before we go into the open question session I mean I guess I can give a quick one just on the last question it takes me so when I hire an IT cybersecurity student it usually takes two to three years to get them trained up on how the electric grid works and teach them about embedded systems I have a much easier time hiring electrical engineers and training them up on how to figure out how to secure a lot of these systems so somewhere along the line cybersecurity got off into its own silo and and from an education standpoint it would be great to see people that had multi-disciplinary capabilities in systems engineering you know add it into the cybersecurity mix because it it does take a long time and a lot of investment on our part to bring those folks up to speed good point and I think it's an excellent point I would say the same is is to make that don't make security its own thing don't make it something separate make sure that security is everyone's challenge and everyone's opportunity to keep our systems in our in our network safe and to make sure that not only within your organization but outside your organization that you're working with your ecosystem partners to make sure we're sharing those best practices making sure that we're developing standards that are going to help our overall industry and keep us safe all right um I've just had um that you know sounds like there are a lot of challenges still in cybersecurity and there are one to emphasize so that there's been a lot of work that's been going on in this area since I'd say around 2007 2008 or so you know so there's a lot of good reference material out there and methodologies out there like the you could find the nister 7628 which is cybersecurity guidelines for the smart grid you know so if you find yourself in a position of needing to look at cybersecurity for the smart grid don't start from scratch there's a lot of good material out there make sure to make sure to use it so the the one thing I would make sure you all are aware of is that the relationship between the pvcs and in this instance electric utilities are actually really good on this topic it's not inconceivable for me to say that two even two years ago jeff's regulatory affairs people wouldn't say no way will you be on a cybersecurity panel the regulatory representatives so the fact that we are that we are able to and routinely now beyond panels together talking about cybersecurity I think is a good showing that we that the utilities and the regulatory bodies are aware that this is a solution that we are both having to work together to to implement and to address excellent great makes us all feel better thank you all right so if you have any questions please walk up to the microphone here so everyone one can hear you uh it would help some security conferences people are anonymous but my name is paul grant since 2006 I've been an independent energy technology consultant here in the valley prior to that though I had a 40 year that's 40 career with IBM and that was followed by 12 year career at upree as a science fellow actually well listening to this I remember in the 1950s as a 21 year old programmer for IBM working on the norad system that we actually did some encryption of the teletype communications between this is way before the internet by the way between the radar stations and central computers now this is on the cyber security question I'm going to ask now uh it's physical security and after 2000 after 9 11 at upree we put together a informal red team and our red team launched an attack on the transmission substations in california 10 of the largest we armed ourselves with hunting rifles and we knew where to shoot at the 300 me the uh transformers and we calculated we could bring down the state of california for months so my question is what's the condition of the physical security of the infrastructure in california and the united states given the fact just a few minutes just a few months ago some amateurs were able to compromise metcalfe does everybody know what metcalfe is okay to the extent that you can answer this yeah i mean it's 500 kv substation uh in pg and east territory uh where somebody shot the uh luckily they didn't know exactly what they were doing like in your simulation um but a lot of mineral oil was was spilled out of the transformers and they were able to get it back up and running you know so from 10 years ago from that study that epri did that you participated in we had the large transformers sitting out in the yard the spares some of them take three five years to build uh and from that study uh we got rail cars out there and and at least moved the spares uh off to a more secure location so they weren't all sitting together in the yard just waiting for you know if one fails then you you take the line you just hook up to the the next one um you know i have to say because the infrastructure is so distributed and big we still do not have adequate defense against somebody with a hundred more rifle who actually knows where to hit uh a transformer other than those spares uh and um we do have fiber optics now we do have communications we do have alarming um however as far as protecting against the attack itself physically um you do not see the really high walls or anything like that now maybe maybe because of this situation in pg and e uh we're going to get ordered to uh uh do something more physically secure we haven't a lot of the security problems we've had in the past 10 years have been people stealing copper and sometimes not so successfully unfortunately for them um and so we have a lot of thermal imaging and that sort of stuff out there video cameras but nothing like a security force uh you know that that's protecting the broadly distributed stuff just one more question in the study the red team study we did at apri one of the issues was cascading failures if you took out exactly substations now is that been treated now with better communications yes actually um so we we do have the the RAS schemes remedial action schemes that are in place at the substations uh have been there's projects underway to centralize some of them so you do have better protection phaser measurement units we have about 20 of them uh in which actually allows us to see whether we're suffering uh from a wide area uh problem or whether it's localized so a lot better simulation and modeling is going on inside the utility company right now so and then we're pushing that down to these automated systems so i'm feeling a lot better about our automated protections than even five years ago um but again these areas are still you know under development some of them and not as broadly deployed as we would like and we continue to actually you know move forward on that front so all your points are well taken and it sounds to me like these are areas where the new technology actually can lead to better security uh we heard about about the power electronics the secretary too was talking about this morning uh with that they replace the big physical transformers they would be easier to to protect uh and with the increased monitoring you don't have to wait for a phone call from a customer to figure out how the grid is doing you can actually see that in real time it's true but there's still very remote locations with very adverse weather conditions where the physical security as good as you can make it is still not good enough we've got a customer that has a wonderful perimeter fence and some great security cameras but when they got 10 feet of snow you walk right over the top of the fence on top of the snow i won't say where it is i won't say what the customer was but let's just say you can get yourself into areas of the grid across the united states and some very adverse conditions that you're not as well protected as you'd like to be so i'm glad to hear that something california Edison is in good shape but let's just say that the answer to your question is it depends all right next question hi i'm gary farhama phd student and management science engineering here stanford um guess my question is how will cyber security change the business model for utilities and if i was asking the cpuc i would say how should it change the business model for utilities and i'm gonna give two two sub branches for the question one is that are we if we're if we look five or ten years from today are we gonna look into more centralized more regulated more controlled kind of regime or are we gonna look at more deregulated where we have some more competition and thus some more market incentive to drive innovation into the utilities this is on one side and then on the other side is that does it does cyber security um or should cyber security be addressed by the utilities themselves or should we allow external parties to address the cyber security issue of the of delivering electricity so pjne can just deliver the electrons and we can count on somebody like google to make sure that our lines are secure chris that's all yours um thanks i have to admit i don't know how to answer that question that there's so much wrapped in that question that it impacts not only the cyber security policies that we would regulate over but just a simple market delivery of electricity um um so the way i would say it right now is that this is gonna be a very legal statement unfortunately is that current legislation does not allow the situation that you're talking about right now the utilities are the provider electricity unless you happen to live in a cca area the community choice aggregation area so the the delivery of electricity services itself is something that is subject to state law and we would uh let me strike that last statement uh it's it's subject to state law and that's sort of what our restrictions are right now now as we think about the business model how cyber security changes the business model i don't know my suspicion is that as things become more distributed i'll use that word um the utility challenges change and the ability of the utilities end to recover appropriate rates to fund those new services they have to or the new programs they have to implement becomes very challenging if there is not a sufficient rate base to support those investments so that becomes that becomes a very different issue and one that you know i think the utilities are all facing as we hear i believe secretary to yourself said that the shrinking rate base becomes a problem as more and more customers come off of the utility off the off the utility quick clarification on the legal issues so rather than you know the utility going and hiding somebody and training them for three years to become really experts in cyber security is are you saying that we cannot basically have a 30 like pgne contract with a company like google to actually take care of the cyber security for them well so maybe i can just answer what i've seen in other industries there's some aspects of security that can you can outsource for example monitoring even the security operations center but if you're an organization that's responsible for operating critical infrastructure you have to have some responsibility of the security for that that cannot be outsourced well part of our monopoly charter right as utility companies is to ensure the reliability of the grid so while we might be able to hire um you know the best brains in the world which we often look for in order to help us solve some of the these problems we would not abdicate our responsibility for maintaining the reliability of the grid because that is our charter as a business the grid wise architecture council actually is doing a lot of research on something called transactive energy so you might want to take a look at that as far as distributed energy resource integration with the grid because that that might be an area or a model for what the future of you know the business model migrates to in the utility sector so actually i have a very brief comment but before i get to the question you can outsource to google because the hardware software integration is is intimate period but i when i was listening to you now i speak to stanford professor um i had a previous job where i had to know a lot about cyber security i mean we guard the nukes we have national labs um and my question to the panel is how many of you have actually have security clearances and have gotten briefings anybody okay so um let me let me just say that it's a lot scary and you think and then there are various levels of security reasons okay even the lowest level security brief from wolf i will you will be convinced it is a lot scary and you think uh and the higher up you go the more scared you get so audience if you weren't scared enough enough already uh so so so um i'm and when we were working with homeland security in trying books we have a lot of the very high tech computer stuff again you know some of our firewalls have to be all the stuff having to do the nuclear weapons has to be very very good uh and as we start working with them and trying to figure these things out this has come up that uh because you know the financial guys have are very concerned because you know this is billions of dollars switching by a keystroke but the energy infrastructure should have at least equal concern and so i'm you know and so this is something where we were trying to figure out how to get the utilities to get a security clearance type of thing that you begin at least get a glimpse of it when google got a glimpse of what was really happening and surrogate bren in particular he just oh my god it's it scared the jesus out of google so okay yeah i mean this is serious there's nothing so the most frustrating meeting to your point the most frustrating meeting i had in the last year was when my ceo came to talk to us who were working in cyber security in the research department about the briefing the classified briefing he went to he couldn't tell us anything except just vague generalities about foreign militaries attacking investor-owned utilities and i'm like well we cannot if a foreign military government with all their resources decides to attack an iou infrastructure we need help from our own military so this is this is a work in progress because homeland security uh and you know the administration not this particular administration any administration really has to begin to work with the utilities because most chances the utilities won't have to wear with all i guarantee you yeah no and yet and there are certain things that you know but and but then it's once you know what's possible it also makes it easier for people to get you know it's got a double-edged sword to this but but i would say not to make anyone in this audience nervous but you should be nervous on that note i hope you'll all sleep well after hearing this panel let's thank the panelists