 Well, thanks for the introduction. It's interesting how you found out some background on me. And yes, Leland Stanford came from the same area where roughly where I grew up. It was also an area where George Westinghouse once lived and grew up. So I guess I have the electricity in my blood from a number of years ago. The topic of our talk is resilience revolution. And the idea here is really to focus on them, the idea of resilience and how it's challenging the traditional aspects of reliability. As Leland was saying, my name's Gil Bendewalt. I'm involved in research on the grid side for over 20 years. With me is Stephen Walsh. Stephen. Hi everybody, very glad to be here. Looking forward to the presentation and some really good question and answer session at the very end. Thanks Stephen. So if we can go to the next slide please. So we thought we'd couple different topics that might be of interest. The first one was really focusing a little bit on the DOE work and resilience. I'm gonna try to cover some of the portfolio that I've been involved in both within OE and in the past. The other, and then Stephen's gonna pick up and really talk about applying that in terms of both the hurricane recovery as well as sort of some of the priority setting. So hopefully you'll get sort of both aspects, some of the theory, some of the understanding of where some of the research is going, as well as also I'll say direct experience in terms of how resilience can be applied both at a local level and also in coordination across federal agencies. And so please we look forward to your questions in the end and next slide. So the first thing I wanna do is talk about how reliability is defined. So for those of you who are less familiar with the grid, NERC or the North American Electrical Reliability Corporation is the one that's basically charged with maintaining system reliability across the U.S. and much of North America. And they defined sort of the reliable bulk power system as one that essentially meets the electricity needs of end-use customers, even when unexpected equipment failures or other factors reduce the amount of available electricity. And what's really important to know is there are two aspects, one's adequacy, one's security. The first one really deals with about having sufficient resources to get it to where you need it, when you need it and the amounts that you need. The second is security. And the reason why I'm touching on that just briefly is because that's in the language that mirrors a little bit of the sounds like resilience, which is the ability to sort of withstand sudden and unexpected disturbances. But much of the focus, as I mentioned here as well for decades is really about the, what I'll say, natural causes, unanticipated loss of system elements like a short circuit, a conductor and being able to see how that would affect it. So you're building in redundancy or aspects into the planning process to help operations to ensure that if you did have something, what I'll say, a single point of failure, that that was not affecting the overall ability to get electricity to where it needed to flow at the bulk power system level. Over the last, I would say decade or more, I'd say there's increased interest in terms of being able to not only look at these natural causes, these uncorrelated events, but really starting to be able to also look at things like man-made physical or cyber attacks. And I think what that does is it really starts to change the understanding of what risk is and how that affects the ability to keep the lights on within the broader sector. Next slide, please. So then how is resilience defined? So at a federal level, the termination, the definition or terminology is based upon this directive. It's called a Presidential Policy Directive 21, came out in 2013 and it's defined resilience is basically the ability to prepare for and adapt to changing conditions and withstand and recover rapidly from disruptions. So resilience includes the ability to withstand and recover from both deliberate attacks, accidents, or naturally occurring threats or incidents. So hurricanes, ice storms, tornadoes, those all fit within the realm here. But again, also physical or cyber attacks would be something that would also, the system has to be able to withstand or be able to recover rapidly from. In 2017, the National Academy's released a report, pictures on the right. For those of you who really are interested in this particular topic around resilience of the nation's electric system, I would really encourage you to download a copy. DOE was the sponsor of this, but it brought together thought leaders from across the country. It really, I think articulates well, a lot of the challenges and opportunities as we think about what resilience means and how we can be able to enhance it and promote it broadly. Next slide please. So these are a few graphics that I put together. They are illustrative, I want to emphasize that. I was concerned I'd get all these Q and A questions about how these curves were derived, but really I wanted to put out there just a couple things. The first one on the left is really trying to capture the idea of what is resilience. And again, you see in terms of a disruptive event, you're going along, you have an event drops down. The whole point is to be able to either reduce the magnitude of that disruption, bring you up to that dotted line or improve the speed of recovery. So you're mitigating the effects that an event would have on the system. The graphic to the right, I would say is my attempt at sort of describing the difference between reliability and resilience. Reliability is really driven about trying to understand how to keep the lights on, right? How to keep under a variety of circumstances to get that adequacy and security piece. When I think of resilience, it really starts to play more into the tales of this curve. Really understand the nature of risk, how that affects the system and how you can assess and make adjustments in terms of decision-making about how to improve that ability to withstand and recover. Challenge with that as you look at the tales are these are low probability events, so often at least, and so they're subjective. There's little or no empirical data. So unlike reliability, where you may have treasure troves of data that you can do statistical analysis around and understand what reliability implications are, the resilience piece is really event-dependent. And the other challenge with this is the further out you go is typically the incremental cost in terms of managing risk is something that goes up. So it's this balance between how do you focus a portfolio of risk or investments related to reduce the risk in terms of improving resilience in a way that tries to, I would say, cover as much of the spaces as possible. The traditional way for reliability, of course, is redundancy, et cetera. I'm using the term manage risk here in particular because it's almost impossible to eliminate risk. Our system is amazingly reliable, 99.99 or they're about the percent reliable. It's the challenge is the, I think is, and I think that's a testament to what the operators and designers have put into that. But I think, again, there's more recognition now or attention around sort of the tails. And by understanding those tails, it helps to better understand how to maintain, well, I'll say it's a healthy system and sort of how that plays out. Next slide, please. So as we look back at that, PBB, the Presidential Policy Directive 21, PPD 21, what are some of the words that came out? Well, the first is this ability to prepare for, right? That really gets into the notion of planning capability. The second piece is withstand. So when you see an event, how are you able to withstand it? A lot of that deals with strengthening, hardening, looking at infrastructure approaches that allow you in itself, the system to be able to tolerate and withstand events that I may see. Recover rapidly. I mean, that really gets into things like redundancies and building in efficiencies in that regard. Adapting to changing conditions, again, this isn't a single point, right? You're not optimizing for a single scenario, single set of conditions. The grid really is in no same state twice. It's always changing. So how do you adjust for these multiple equilibria? And then the idea of the multiple hazard piece, right? The deliberate attacks, accidents are naturally occurring. So that gets really into the box at the bottom, which is that a lot of people think of resilience as in the moment, right? I'm resilient. I've seen an event. I'm able to bounce back. But really what only makes that happen is the idea of having robust planning, preparation, and food investment. So the time and effort that you put into it upfront to be able to be well-prepared for when you need it to adapt is what is critical in terms of the ability to achieve the resilience that's desired. Next slide, please. So how is OE focusing on the resilience? Well, I sort of broke it into some themes and then I wanted to just break down some of the areas of both research and some of the coordination. The first is sort of understanding the context. Again, we're trying to understand the tale. These are determined a lot by particular events. So that gets into things like the threat and event characterization. But it also really comes down to understanding the demand and behavior profile, the needs. I mean, one of the things that's coming out of the current COVID situation is the ability to better understand how loads are affected, which has consequence in terms of operational challenges, whether it's managing voltage or power flows or even controlling and operating the system. That all ties into understanding the context of what you're trying to look at. The second piece across our portfolio within OE is sort of looking at how, I would say, analyzing existing flexibility. So some of that is very data-driven, right? You have a system. You're trying to understand that. So sensors, being able to have a higher level of fidelity in terms of the different dynamics that are occurring. Resilience metrics, being able to understand how resilience is measured and evaluated. And then there's things like maturity models, which again, different entities and stakeholders will value resilience in different ways. And they may be at different positions in time of where they are in that maturity. So how do you balance resilience maturity models in terms of the technologies you're in and the goals and attributes that you would have? Resilience, in a lot of ways, is very much at a local and regional level. And so some of what we've talked about, there's some projects that we're working on with the National Labs, one of them particularly at Sandia. In that particular project, they emphasize this balance between community resilience, visibility needs versus utility reliability. And I think that really gets the heart there. The third piece is really identifying gaps and solutions. So part of identification of the gaps comes down to modeling and ability to have a framework that allows you to assess and compare some of these options. And then there's also the investment in solutions themselves. When you identify the gaps, what do you do? And so that's work in areas like energy storage and microgrids, as well as more advanced software architectures and other approaches. And then finally, I think this idea of coordinating objectives and facilitating decision making. There's a portion of our office that's focused on technical assistance. And so it's working with the states and other decision makers about how to consider and value these. And that also plays in some work on the architecture side. Next slide, please. So I mentioned modeling. And I think, again, if we go back to that one, we don't have to go back in the, if we think back to the graphic with on the right hand side where it talks about where resilience was and looking at the tails. I mentioned there that there really isn't a lot of what I'll say data. Again, if these are low probability events, you're not necessarily data rich. So how do you understand both the data? How do you simulate potential events? It comes down to modeling. And so these give some examples of five areas that I thought of and highlighted in terms of some of the work and reason why we're focusing on grid modeling as a key part of our overall research portfolio, not only within the office of electricity, but also looking how to also characterize and model areas like solar and DER and wind technology or even hydro resources. It's really helping models become a platform for helping to understand complex data, which the grid is a complex system. And then it's also being able to understand the physics that govern the dynamics of the system. It also comes into understanding parameters and values that you don't have measurements for, right? You need to estimate those. I mentioned the simulations. And then I think finally, I think models give you a platform to evaluate options. So if you're trying to understand what resilience means and you're trying to, for example, understand where storage may play, a model gives you an ability to look at those and be able to understand where a likely place would be, either to maybe do a demonstration or to deploy a technology to have a system effect that's desired, excellent place. So a key area that's developed under our assistant secretary in coordination across the department is something called the North American Energy Resilience Model. And this is an area that is leveraging a number of laboratories across the United States, including labs like Livermore in California and Oak Ridge, PNNL and others. It really is about, I mentioned models and a lot of those seem like singular domain models. But when you talk about resilience, resilience doesn't stop at the generator or the wires, right? You need to be able to understand that if you're talking about the resilience of the broader power system, the energy system, it goes into the generators and it goes into understanding the nature of the fuels that feed those generators, that then flow through the electricity, the options that are on the load side. So the NARM or North American Energy Resilience Model is really focused on these interdependencies. It's being able to understand, for example, how natural gas and electricity flow together or communications. It's at the bulk power system and it's deploying, what I would say is engineering class modeling. First at the planning level, but it's also geared towards understanding how near real-time data can also help both situational awareness and decision-making as well. Next slide, please. So the workflow, again, this sort of is an overview, but it really begins with understanding the threats. I talked about that or the events themselves, how you characterize them, feeds into this sort of integrated modeling environment. And then really on the end, it's all the different ways that the models can inform decision-making. So understanding outages or impacts, understanding resilience in terms of being able to assess it. We talked about metrics and economics, what might be most appropriate for a particular region or type of investment that's needed. Operational planning options or, again, near real-time situational awareness just to know what's going on. And next slide, please. And then the unique part of what NARM is, it's really about an interactive modular approach. So it's not a single model. It's about being able to know the same way that events are evolving and threats themselves. It's being able to have a framework that can also scale, extend, and expand in ways that continue to address the needs that may evolve. And so that's where, again, we're looking at how different pieces are coupled together and being able to leverage those capabilities to the fullest extent. So this interaction and interoperability, this interdependence piece is really what NARM is about. It's not a singular model. It's about being able to couple these. And through that, also, I think understanding how the various domains also relate to one another and the stakeholders that are involved in those. And then I think, next slide, I think it's the breakpoint. It is. So, Liang, I'll turn this back to you. Thank you, Gil. Very exciting. So for audience, if you have any simple and the clarification question, please click the button, which is raise hand in the button, then I'm going to unmute you and allow you to ask question. So we have one question from Martin Morph. You are unmuted. Martin Morph, you are unmuted. Okay, I think he mute himself. Okay, any other questions? Okay, I think we can move on, Gil. That sounds great. I think Steven, you're up. Okay, and everyone can hear me. Good. Yes, it's good. Okay, good, thank you. So thank you again for helping me. Gil did a wonderful job of trying to give an overview of all of the varied efforts that we have in resilience. And I will take the baton from where he left off in identifying NARM as a basket of modeling tools that we can apply to learn more about resilience and talk about how we use some of those tools for Hurricane Irma and Ria recovery work and what we've learned along the way. What you see here, this slide is essentially a citation network of peer reviewed papers on resilience from the early part of the 21st century. And you can see one that there's a lot of different disciplines that have talked about resilience in peer reviewed literature. So that's a good thing. What you see is environmental and psychology being the two biggest bubbles. I know the ecology peer reviewed lit on resilience goes back to at least the early 70s. And psychology talks about resilience of the individual and how it copes with different stressors. When you see in a faint circle on the bottom is the engineering and other hard sciences talking about resilience. It's the much smaller dots and they tend to connect back to psychology or environmental psychology or environmental and ecology. And next slide please. You can see on the chart on the right here how much the literature has increased on resilience in the past few years. And you can also see a few more definitions of resilience. And we're not gonna go through all of these. It's just to point out that there are many extant definitions, working definitions of resilience. As feds, we use PVD 21, which Gail went over in depth. And it does a pretty good job of defining what resilience means for us. And in the chart here, there's just some definitions that reflect the different perspectives on resilience from the previous citation network chart. When you see this, I think these slides will be distributed. You can read these at your leisure and go to these other papers and find other definitions of what resilience means in different contexts, and particularly with regard to engineering. Next slide please. One thing that kind of jumps out once you look at all of the different disciplines and think about it through an infrastructure perspective, these common elements of resilience emerge. Gail talked about some of this, so I won't go into it too much in depth, but there's three temporal phases, usually pre-events, in other words, what are we doing to prepare for on-tail events, these low-frequency, high-impact events? Then of course, there's during an event that's actually happening, and immediately afterwards. And one distinction here that I think psychology and ecology have with regard to resilience is that immediate post-event is not quite long enough in most of the lit. They tend to think about things in years following a particular event and the stressors and how those dynamics play out over years, whereas most of the literature on the infrastructure and engineering side is really looking at how quickly can things go back to normal. In order to become resilient or have a resilient infrastructure system, these other capacities that you see here are either explicit or implied by the definition. Two things that I think are interesting to note are and have some impact on our hurricane recovery work, that the resilience definitions imply that there's some sort of pre-event capacity to address the concerns that are out there. So it's one thing to have a bunch of rich, high-quality data and to feed that into a world-class modeling capability and get outputs that are intelligible, that are mathematically sound, and also can translate into real-world action. That's all well and good. If you can't, though, act on those insights, then resilience will be just outside of your grasp. So one thing that's important to keep in mind is that a financial capacity to act on resilience gaps or needs is important. Another thing that's implied is that resilience tends to be viewed as just a bit more than asset performance. Gil talked about that as well. We've got the planning capacity beforehand. We also have the ability, how quickly can you bounce back? But one thing that gave me a little bit of trouble with that limiting resilience to just that framework, kind of, it started bothering me a little bit as I was doing the recovery work because if it's just infrastructure performance, then that's not a full explanation for what we were seeing. Next slide, please. What you see here are some of the software tools and models that we were able to apply to recovery work following hurricanes and Maria. You see they range from the bulk power system and the fuel system all the way down to distribution and edge. A little bit later, there's a link to a website where if you're curious about any of this work, you can go and get a bit more detail. Our email addresses are at the end. So if you have specific questions about this, either Q&A or if it comes to you later, you'll have a way to get in touch. The point of this slide is to show the breadth, I think, of the tools that we have available to understand what resilience needs an infrastructure system might have and the different scales and levels at which those models provide insights. Of course, they each need different datasets. Luckily, there's some overlap, but there's a lot of unique data needs for each of these tools. The tools don't have a really clean handshake between the outputs of one and the inputs of another, which I think Gil and the NARM team have been working really hard at. But for now, I mean, when we're doing this recovery work, we're running each of these almost completely separately. Next slide, please. It's just a little bit more detail on the tools that we deployed. There are links to some of these. You can do a little bit of deeper dive on your own time. If you're interested, the one thing that I would like to draw everyone's attention to that Gil mentioned in his portion is the resilient community work that Sandia does. Then the name of that tool that they use in-house is RENCAT, so that's the third one down that you see there. Another element from this is it's not just about the existing assets. It's also looking at what other tools and what other technologies can be brought to bear to make the system more resilient. What can be done to make sure that next time the infrastructure performs better and it's a lower stress on the community than what we saw with Hurricane Maria. So if anyone's been following the news there, they may have seen that Puerto Rico passed a 100% renewable energy target, which is very ambitious. And some of these tools, particularly PVWOTs and Festive and Mafrid there at the end from the National Renewable Energy Lab in Colorado, will be really helpful in making sure that different capacity, different technologies that generate electricity all play well together and will actually keep the system up and running and keep the lights on. So we have everything from making sure at the top, making sure that there's gonna be enough natural gas, that understanding with the head-out tool, for example, what the different failure probabilities of different assets under different wind speeds, all the way down to what will a 5kW PV system do on a residential rooftop. It's a pretty impressive array of tools. Next slide, please. And we've been able to produce a number of deliverables. Many of these are official use only, but what you'll see there on the right, left, yes, left, is a link that I can make available afterwards, but where there's a bit more detail on what's public, the information that's publicly available. Since one implication also of resilience work that we often don't think of, is that compiling all of this data together actually creates a bit of a sensitivity with regards to how that information is shared and who has access to it. Each of the data sets on their own might not be all that sensitive, but when you put it together and you get a very complex, rich picture of how the different infrastructure systems interact, you might have a data set, outputs that aren't safe to just be broadcast publicly and you wanna be a little bit more careful about who you share things with. But as we were doing all of this work, next slide, please. It occurred to me that a lot of what we were looking at was how quickly it took an infrastructure system to return to normal following an event. And that I think is at its core what many resilience definitions in the engineering and infrastructure space look to do. Take a system as it is before something bad happens, examine it, its performance or model its performance as something bad is happening and get a sense of how quickly it will return to normal operating conditions and normal parameters and how that might be improved. Ultimately, when you simplify it down and distill it, you end up with something that as Gil showed earlier, looks like a resilience triangle. And there's a paper here, Ayub, that I think does a very good job of describing this concept and I highly recommend everyone take a look at it. There's a full citation on the next slide once we get there. But this is at most simple or most straightforward, what many resilience analyses look to do is describe this triangle. Systems operating normally at point A and then B, event happens. How long does it take to get to D? Next slide, please. And these can get very complicated as you see here. I think this paper does a very good job of making a complex resilience triangle. It provides for pre-event mitigation. It provides for different performance depending on the threat that you're looking at. And it also allows for improvements back to an equilibrium that's different than the one where you started. And of course, looks at cost and time. This is a wonderful model and a great paper, I highly recommend and I think this is, this might be in some ways the height of what the resilience triangle can do. Next slide, please. We see here some of the strengths and weaknesses. The approach is consistent with the definition that GIL laid out. It allows for comparison of system performance in the three phases of a resilience event that we talked about earlier, pre-event, during the event and mitigation. Different failure rates and modes. So you can optimize based on different constraints and it provides non-negative outputs, which for those of you deeper into the math will appreciate that it will actually converge and give you a unit that makes sense. On the other hand though, it's only as good at the system level logic used to define the components of the system. And what does that mean? Essentially it's at its core that's junk in, junk out. So if you don't define the system well, you won't get useful outputs. Or if you define the system well, but you don't characterize the component of performance in a given threat well, you won't get useful outputs. Which means that as you get more and more useful modeling specifications and outputs, you then have more and more specific models and outputs, which we kind of talked about earlier with the different buckets. Each of those tools great on their own, useful, give you very good insights into whether it's the bulk power system or micro grids or storage. But in order to be good and useful, they have to be specific. And we know when we're thinking about resilience, as Gil mentioned, it's multi-threat, it's multi-equilibria, and it's low frequency event. So we don't know exactly what's gonna happen, what the event will look like, what it will do. So we have very specific models that are giving us great outputs for specific conditions that may or may not be the ones that we actually need with the time comes. And then of course the last one is a charge that you could probably levy at any mathematical model that attempts to look at things in the monetary way, but that essentially, the cost of making a system resilient might not actually be a useful thing to measure at all. And that's sort of hit home for me doing the Hurricane Maria recovery work. I mean, there was very little pre-event financial capacity to mitigate. The storm was worse than people were expecting. And while there was a great amount of suffering and the infrastructure performed fairly poorly, the community itself continued to survive and will thrive again. And so the investigations of resilience that we had been undertaking sort of rang a little bit hollow in the moment. Next slide please. And so it made me try to find another way to think about resilience. And there's some good literature talking about resilience as an emergent property. Mathematically, that's essentially there are changing boundary conditions. And so you're never quite sure in one moment what the next boundary condition might be. And so the path from the pre-event to post-event isn't exactly, isn't very easy to calculate. But it also means that it's hard to characterize. I mean, when we think about resilience in the context of an individual, there are many different stressors that you might imagine and someone goes through whether small or large, frequent or not, that we go through and we can say and feel that we're resilient for having gone through them and responded in the way that we did. It wasn't about how much money we had at our disposal, although that often helps. And it certainly wasn't just the amount of time it took us. It's something more than that. And I think that's why I'm kind of settling on this idea of resilience as an emergent property. It still leaves us with the task of assigning, defining the system that is to be resilient. Is it a power grid? Is it the distribution level? Is it the bulk power system? Is it interdependent infrastructures like you was talking about? Gas electric communications? Or is it something that includes a bit more of the human element? How quickly can small businesses reopen? How quickly can a population displaced by the event return to normal life? Or to a new normal like in New Orleans? Things certainly changed there due to Katrina, but the city is thriving. I don't think anyone would argue with that, but it's also very different. So we can say that New Orleans is resilient, but it certainly wasn't, it's not the same as it was. And so it's more than just measuring how long it took this place or this community to get back to normal because normal means something else. And those last two bullets are really about how the federal role is impacted. States traditionally exercise police power. So the things that we think about as absolutely necessary during an event, fire, hospitals, police, all of that's controlled at the state level. Of course, the feds have influence over certain things, but that's really state domain. Next slide, please. And so through the course of the Hurricane Marine recovery work that we were doing, we combined all of these tools, specifically a team at Pacific Northwest National Lab, combined tools in the way that you see here. And this, I think you can characterize this as an adaptive robust optimization, where you have on the head out at the beginning, you get assets with different probability distributions for a given threat. Those are maps and also looked at, put through a time sequence based on actual events that have happened over essentially hurricanes that have happened over the past century. Then you do an optimization Monte Carlo analysis sampling of what assets would survive under a given threat. And then it looks at dynamic contingency analysis, looking at what things will fail, whether there's a cascade failure based on the assets that are damaged as identified as damage based on the analysis up to that point. And then it kind of cycles through at the end until the event ends. Very good work, publications forthcoming, but for anyone who's curious, if they look for adaptive robust optimization, you'll see some fairly recent literature on that. I think this approach has good legs. It's just very data intensive. It's time intensive. It takes a lot of computing power. And we're still at the stage where it's a single threat and a fairly small foot geographical footprint. But it's very promising. I think there's some good potential with this approach. Next slide, please. Which takes us to where the federal rule is. And I know Gil talked about some of these things. Even though we're not, the federal government doesn't have the police power that the states do, we can still help build local capacity in a number of ways. One of those is data collection and standardizing data sets so that baseline system performance can be measured and compared from one place to another. Building that planning capacity so that people can take the data that they have and view it in a way that makes sense and is robust given their circumstances. We can also help with an accurate prognosis capability like NARM that Gil was talking about. In other words, when something is happening, the ability to understand what has happened and what the near-term consequences of that might be very helpful for identifying what needs to be done to mitigate those consequences. And in another way, if something bad is happening, you need to know exactly what those impacts are gonna be with specificity so that you can avoid them or solve them quickly. And then of course, through FEMA and other agencies, we can complement the local response capacity. And the final bullet, I think is something that is there's increasing awareness of this need for the federal government to finance resilience assets. There are probably others as well and I encourage you all to think about them because this is a complex space. There's a lot of work that is ongoing but there's also a lot of work to do. And I think that includes everything from understanding what resilience means in a given context and how to help communities be more resilient. So all of your brains will be very helpful in helping us do this well. And with that, next slide. Happy to answer any questions along with Gil and thank you all again. Thank you very much, Stephen and Gil, very informative talks and a lot of useful information, especially for the students who have not think about this area or may plan to jump into conduct research in this area, especially a lot of free access data sets and tools are extremely valuable to the researchers. So with that, I would say if anyone has question, please submit your question through the Q&A portal or you can use the raise hand function, then I'm going to unmute you so that you can ask question. We have about 10 minutes for Q&A and I would start with a question about the last slide Stephen mentioned, so the data collection. So at the previous couple of slides, you talk a lot about third party accessible or free accessible tools like the Sandias resilience community and the microgrid tool and others. So how about the data sets? Because Stanford students, I talked about last week, 80 or 90% of them took the class which is machine learning or AI 101 class and they are struggling about any use for data they can play around. They're very curious about any data sets available for the resilience. Can you elaborate a little bit more about available data sets related to resilience? Yeah, it's a great question. I'll start and then I know Gil will have thoughts on this as well. Several federal agencies collect data that would be of relevance. The energy information administration collects a lot of data about the energy system in the United States. That's electricity and fuels, location, volumes, prices, et cetera. There's a lot of good information from them. FERC, the Federal Energy Regulatory Commission also has some data sets available. They might not be obvious but you can ask them for certain things if there's a need. You can't necessarily code an API and have it scraped but there's some good stuff there. We're also working on essentially like a data collection primer where we identify high value data sets for this kind of analysis and who usually owns that data. That's not quite ready yet but hopefully it'll be done soon and it would be out there. And I know that's... Because when it gets right down to it, a lot of times these data sets are scattered in different places and there's really no, unfortunately, no replacement for the legwork of going around and getting them. But Gil, please chime in. I know you've thought about this harder than I have. Gil, you probably mute yourself. I am, Liang. Can you hear me now? Yes. Okay, so that was my redundancy. I muted my screen and my phone so I had to stop or unmute them both. So I was just gonna agree with Steven. Couple things that I was gonna say is I think you highlighted an issue that's really important. This is one that has come up in multiple seminars and briefings especially under the Advanced Good Modeling Program as we coordinated with the Office of Science. As you know, things like open data, as you pointed out, provide an opportunity to be able to do analysis and research and there's no second to them. So as a, I would say across several different offices including OE and particularly RPE, the Advanced Research Project Agency, we've started to try to create some open data sets that would anonymize what I will say specific power system characteristics but allow, for example, the ability to do some fundamental analysis. So under the RPE office, they had the Grid Data Initiative which allowed some approaches to the way different algorithms would allow unit commitment or security constrained up in Power Flow or other different types of approaches to the way you would do analysis on a system. Within our Office OE a couple of years ago, we actually had a funding opportunity announcement that allowed the provision of some open data specifically targeting machine learning and AI. Basically recognizing that there's a need to be able both to have data sets to train those algorithms and then broader deployment and demonstration around those. And it's one that we're trying to continue to grow. So I know all things like natural gas domain and others were looking for what that appropriate balance is to support the innovation and to support the research that you're talking about, Leon, balanced against again, the sensitivities around some of the system-specific data. But we're working hard to try to be able to create some opportunities there for being able to help students and others specifically or particularly in academia be able to do what they need to do to help the industry and the sector move forward and transform the place. One question from the Q&A portal, I think it's a little bit vague. Let me try to interpret it in my own way. A lot of useful information you presented here and how can a local town office of emergency service and to know this kind of information and to have access to this kind of information? Steven, did you wanna start with this one or do you want me to? Please go ahead. So I think one of the things from a department standpoint, if I understand the question, Leon, it's really about how do you access this? That gets into my earlier comment about technical assistance. We talked about it at the state level. But there've also been projects and demonstrations that we worked with the, well, say individual communities. Steven mentioned New Orleans. There are other communities that we've worked with at a local level to balance and understand, again, community resilience versus utility reliability, for example. We've worked in the past with some organizations that were focused specifically on just like a hundred resilient cities-type initiatives, but again, and I think it's an area of a good point. It's about recognizing that a lot of these decisions and priorities are made at the local level and being able to figure out ways that this information is accessible is something that we're continuing to do. Part of our hope is the things like the models, not only are a platform to do the analysis, but they actually can be linkages that show how coordination across those different layers, fed state, local, can be improved and coordinate the insights that help all of those get the appropriate stage decision-making to support resilience and reliability. And I'll just add that it's a tough problem that we've got with this space. There are a few thousands local governments around the United States and it's next to impossible to reach all of them in an efficient way. But FEMA works with state counterpart, state emergency management agencies who have, and the states have varying degrees of successful relationships with their local governments, whether it's city, county, township, et cetera. So we try to get what we've learned out through that network. And then we have some people dedicated to technical assistance whose job it is to try to find states who need this information and help them learn how to use these tools. But it's a problem that we'll never fully solve. And it's just, it's another one like data collection that ultimately boils down to hard work. Thank you. We have one guest raise a hand. I'm trying to clean up as many questions as I can. So I'm going to unmute Nori Rudravesh. Hi. Thank you. Can you hear me? Yes. Okay. So my question was about the metrics for resilience, right? I think Gil mentioned in his section that one of the efforts that Office of Electricity is doing is also working on new metrics for resilience and I was wondering if you could speak to that a little bit. And a follow-up was that we know that we have well-established metrics for resilience and Nor can fork are responsible for implementing these metrics. Is there any effort to do similar implementation for resilience? So that doesn't be a question. Thank you. Yeah. You're going to see us? I know, yeah. Go ahead. Yeah. So I'll take a start on this one and then maybe Steven can highlight some of his thoughts. So the report, there's actually a series of reports questions well timed. Just came out last week. They were released. They're part of something called the Grid Modernization Laboratory Collaborative or consortium. It's sort of if there are a series and I can send this link maybe to Yuliang or if somebody specifically wants my emails in there. But they include an assessment of the different metrics not only in terms of resilience but they include reliability as well as other attributes and try to put together, we'll say decision frameworks on what they may mean to different localities and communities and help them think about what might be most appropriately structured for their particular interests. But yeah, it's been a, I'll say a very labor intensive process, one that's acquired a lot of coordination but I think it's a good synopsis of a lot of the discussions and work that's been, I will say cutting edge in terms of being able to think about and assess these. I think one of the, I'll say the challenges is how you start coupling these different attributes together. How do you weight them? Because that often becomes a very individualized thing. So part of what we're I think looking at in terms of some of these projects called the Resilient RDS Projects under GMLC but Resilient Distribution System Projects is exactly that. How do you look at resilience but look at other attributes such as affordability and reliability or security and balance these in terms of the priorities and investments that may need to be made. Steven, I'll turn it over to you if you have additional thoughts. Yeah, I'll just add that. I recommend the report that Gil identified. There's some really good work represented in those papers worth a read for sure. I think you also touched on the heart of the issue with resilience with regard to metrics is that the threats and risks from in one place aren't going to be the same as they are in another. The risks in Miami aren't the same as Omaha, Nebraska. They both face very real risks and threats but they have different risk profiles, infrastructure needs, community sizes, and so on. So it's hard to identify ex-ante what metric should be used to measure or indicate resilience that can be used in both places. So right now, I think we're kind of stuck on the idea that in order to do something measurable, you have to have a local context in which to apply some of these frameworks. Terrific. Thank you again, Gil and Steven for your presentation. I really appreciate it. Thank you everyone for attending today's webinar and Gil and Steven, when time is allowed, I will definitely invite you back to Stanford. We have an amazing museum here. You'll be able to see something that's from your old town.