 Good afternoon everybody welcome to the NRC's Rick technical session on making effective decisions and using defense in-depth safety margins and risk The session is a joint effort by the NRC's Office of Nuclear Regulatory Research and the Office of Nuclear Reactor Regulation Before I start I just want to acknowledge and support our friends in the Ukraine and their valiant efforts to defend their country We want you to know that we stand with you in this time of challenge Next slide, please Just wanted to go over the agenda quickly. I'm going to start with some quick introductions and then our panel members will present and discuss the effective application of risk-informed decision making They're going to talk about some practical examples, best practices and lessons learned, successes, challenges and other considerations and how risk has helped us keep our focus on safety. This morning Chairman Hansen touched on the importance of risk in decisions and the wide range of views. This session will facilitate understanding of what drives risk and the differences of perspectives on risk. For example, we'll cover misconceptions about the role of risk considerations and defense in-depth and how we integrate the use of risk-informed decision making with deterministic reviews for effective decision making. These are historically some of the most challenging issues and today you're lucky because you have the experts here to assist with untangling the ins and outs of risk-informed decision making. Additionally, the NRC and our external stakeholders will highlight both the positive benefits and potential pitfalls of using risk-informed decision making. And it will provide you a great opportunity to engage with the panelists who are experts in this very important area on the issues during the panel discussion. We have about an hour and a half for our discussion and we plan to spend about half of that time in discussion, so we'll ask that you hold your questions until all the speakers have finished with their presentations. Just a few acknowledgements before we start. I want to thank everyone for their preparation and participation in this session. In particular, I want to thank the session speakers for agreeing to participate in the session and for their effort taken to prepare for the session. And a special thanks to Lundy Presley and Matt Humbersoed of the NRC for their work in coordinating the session and of course our AB staff that's making sure that everything goes very smoothly. Thank you. A quick overview, and I think this has already been said before, but the journey to become a more modern risk-informed regulator is a really important one and it will help us to more effectively accomplish our mission to protect people in the environment by helping us focus on safety significant aspects of our work. This is an important topic as risk-informed decision making is at the heart of sound regulatory practices and for the NRC it's a critical part of our regulatory transformation. The importance of this topic I think is reflected in the number of people who registered for this session. At last count, I think it was over 800. That's quite impressive. I just want to start by introducing our panelists. First, we're going to hear from Doug Tru. He's the Nuclear Energy Institute Chief Nuclear Officer and Senior Vice President of Generation and Suppliers at the NEI. He has more than 40 years of nuclear industry experience in nuclear safety and prior to joining NEI he contributed to many of the major milestones in risk-informed regulation and he was responsible for one of the largest specialty engineering organizations in the nuclear industry including being president of the largest nuclear PRA firm in the world. Secondly, we're going to hear from Shmain Yalloui. He's a senior probabilistic safety assessment technical specialist with the Canadian Nuclear Safety Commission. Mr. Yalloui has a master's in nuclear engineering. He specialized in probabilistic risk assessment and he joined the Canadian Nuclear Safety Commission in 2008. Internationally, Mr. Yalloui contributed to the development of IAEA safety report series and he's a member of the NEA working group on risk. He specifically participated in the IAEA and WG risk projects on multi-unit and site-level PSA. Mr. Yalloui took part in the international seismic probabilistic seismic assessment peer reviews. Our third presenter will be Mark Baggert. He serves as the Deputy Director of the Division of Risk Analysis here at the NRC in the Office of Nuclear Reactor Research. Mr. Thaggert joined the NRC in 1989 and in 2009 he was selected for the Senior Executive Service serving in the Office of Federal and State Materials and Environmental Management Office and also in the Office of Nuclear Security and Incident Response and most recently in the Office of Research as a Director of the Division of Risk Analysis. Our last presenter will be Mike Franovich and he is the Director of the Division of Risk Assessment in the Office of Nuclear Reactor Regulation. He has over 30 years of nuclear experience. Mr. Franovich is a member of the Senior Executive Service and he previously served as Deputy and Acting Director of the Fukushima Lessons Learn Division and enhancing defenses against extreme natural events. He currently leads the Division of Risk Assessment which conducts probabilistic risk analysis and establishes regulatory standards for risk-informed nuclear reactor licensing oversight, accident consequence analysis, health physics, and fire protection engineering. And without further ado I'm going to turn the panel session over to our first presenter Mr. Doug Tru. Thank you Andrea. I'm glad to be here today and thanks for inviting me. Look forward to the discussion today. I thought of my presentation today risk-informed decision-making greater than the sum of its parts because I really believe that that's true. In fact I think it was over 25 years ago I wrote a paper that talked about how when you use PRA information it's important to consider defense and deaths and safety margins and the performance of equipment in those considerations and that was a few years even before REC-117-4 was invented and the term risk-informed was coined. So this is a subject that's near and dear to my heart and I really believe that there are some misconceptions and some misunderstandings about this. I'm going to try and talk her way through. And I want to start with a discussion on safety versus risk. A lot of times we think of those as being different things. I actually believe they are intimately related. So let's go to the next slide. So as we all know our regulations with the current plans are based on deterministic requirements to provide that foundation for ensuring the safety of our nuclear power plants. Risk analysis gives us a tool that allows us to assess the risks that remain when the regulations are met and that risk is never zero. The risk analysis not only gives us a chance to make those residual risks or a level of safety quantitative level of safety but it also lets us understand what contributes to them and through that we can understand also changes in requirements either additional requirements or relaxations of requirements and how they impact that risk that residual risk. And in the case of the STP process under the ROP what non-compliances look like and what the significance of those are. This means that really what we do in the safety side is tied directly to what we measure on the risk. And I want to turn the next slide to an example of that. So if we think about the relationship between risk and safety you can have different levels of safety requirements across the bottom of this graph and different levels of risk on the left axis. Here we're looking at redundancy. You can have no safety system then very high risk one or two or three or four safety systems and have a lower and lower risk. You can add a whole bunch of safety systems but there'll always be a residual risk. And what we're trying to do in understanding what that residual risk looks like is to understand what contributes to it how in this case redundancy applies but it works in all kinds of different directions not just in the sense of redundancy it could talk about how much safety margin we need in our containment how much shielding you need for for radiation safety. It's understanding that residual that's left after we've decided what level of deterministic requirement we're going to have. The benefit of that understanding of what contributes allows us to make risk-informed decisions. Let's go to the next slide. And that understanding allows us to actually improve safety. It gives us the ability to focus on what's truly safety significant. It allows us to allocate resources in the manner that most effectively improve safety so we can focus on the things that are the biggest contributors to risk and maybe minimize the amount of effort we put on things that are less important to make sure we're staying focused on the things that are most important. Risk-informing also incentivizes the licensees to focus on what's important to safety. If they understand what's important to safety they know where they're on the risk with the regulator of having a significant safety issue. And as I said it also allows us to know where we can spend less time on things of less importance. Overall we've seen across the last 25 or 30 years that this focus on safety significance that risk allows us to do actually stimulates a net improvement in safety. What you focus on actually improves and therefore risk goes down. You've seen me and others at NEI present that curve of CDF versus time that shows how we've driven risk down. That curve happens to apply just to internal events but the same thing is true of external events. All the work we did in NFT-805 greatly reduced the fire risk at those plants, other enhancements we've made and plans have reduced other contributors to risk. But what's important when we do this is that we're using PRA as a tool is that we understand its limitations. As much as I'm a PRA practitioner and I've been one for now 40 odd years it's a tool that can only be used within its limitations and it has to be used appropriately. It's neither omnipotent nor omniscient. It doesn't always come out with an answer that can be trusted as a flat number. You got to understand what contributes to it. But when I say that about PRA you can the same thing applies to deterministic approaches. There's nothing perfect about a deterministic approach either. It has its own limitations that we've learned over the years which is one of the reasons why we brought risk into our decision-making process. So let's move to the next slide and talk about that. The Red Guide 117-4 outlined the risk-informed decision-making process that's shown here on the right hand side of this graphic. The PRA results are one input to that process and they're a product of a model that certainly contains uncertainties. Uncertainties exist whether you're using a deterministic approach or you're using a probabilistic approach. Just the PRA allows us to quantify though and those and illuminate them in a way that is very difficult to do if you're dealing with strictly a deterministic basis. The brilliance of Red Guide 117-4 was that it outlined an integrated decision-making process. The process said use PRA as one piece but also asked us to consider events and depths and safety margins in that decision-making process. And it's important that they use the term integrated. It's not that PRA is a gate, that defense and depth is a gate and safety margins is a gate that you have to pass through each of those. They need to be considered in an integrated manner where you understand what the risk analysis is telling you, what the defense and depth considerations are telling you and understand how those fit together. Sometimes PRA isn't very good at modeling things. Sometimes it's quite good at it and that should be taken into account. And that goes to the point of uncertainties. And what's important about using PRA in a decision making process is understanding the uncertainties that are important to that decision. Typically, when we're looking at a particular use of PRA, we're looking at one particular slice, whether it's a piece of equipment that had failed and had a performance deficiency associated with or it's a change in antithetical specification or some other plant configuration change. We need to understand what the role that change makes in the overall risk profile and which uncertainties contribute to that. If we do that, then we can understand how that plays into the defense and depth safety and safety margins considerations. And a good example of this is when we were looking at what to do after Fukushima. We could have gone after that with a strictly numerical approach and tried to divide some method to say once again we get below 10 to minus x, we're good. But instead we said there's a lot of uncertainty here. These black swan events like happened at Fukushima are rare. It's better for us to think about this in a defense and depth posture. And then we ended up, that's why we ended up with flex. So PRA has its role and certain things, it's very good. In other cases, we need to think about what the uncertainties are and put more emphasis on other pieces of this decision-making process. And I just said the bottom there took a quote right directly out of regular one with them for that talks about how important it is to take all these pieces together in an integrated manner so that we understand the relationships between the risks, safety margins, defense and depth, how we can monitor that performance and what we're talking about in terms of the regulation. I'll finish up with one last slide and next one, the next one. I talked about this in some examples. So from an regulatory perspective, we've had some fairly significant decisions that use the risk insights to drive that. ATWIS rule for sure was driven by our understanding of ATWIS events back at the time it was promulgated. A station blackout rule where we actually identified sort of a hole in the whole defense and depth process where we realized that fossil fuels by power events were more likely and we needed more than just a limited amount of redundancy. We actually needed an ability to cope and in some cases we needed an alternate AC power supply in order to be able to mitigate those risks. That insight wouldn't have come about from a strictly deterministic understanding. It came about because we understood the likelihood and the consequences and we understood the defense and depth we had. And another good example is in the Fukushima period where we installed through accident events in the water addition to via DR mark 1s and mark 2s. In that process we looked at a lot of different options but we learned that without the water addition having a severe accident event wasn't going to help us much because the containment was ultimately going to fail and that vent pathway that we thought was going to give us benefit would actually be compromised. And that understanding came about by understanding the nature of the events that occur in the Syracan regime and understanding how to mitigate those and the uncertainties associated with them. And lastly I won't go through all these in detail but utilities have made and licensees have made a lot of voluntary changes. I can't count the number of plans who found things that were compliant with the regulations but from a risk perspective were driving risk results and they made enhancements to their plan to address those. A good example is fire protection piping that and a number of plans were found that a flooding risk from that piping couldn't leave the impacts on AC and DC power. Plants weren't comfortable with that and they made changes to mitigate that flooding risk and reduce their overall risk profile because of that. Plants have installed non-safety equipment to perform functions that are important but lots of procedural changes to use their systems in different ways and fed back the insights from their PRAs into training so that the human actions are in a context that fits with that residual risk. So they make sure we're training our operators on the actions that are really going to be necessary in order to address some of the most important things. So as you can tell I'm very bullish on the risk informed concept. I think if we do it well we can really enhance and get a better value out of our regulations by getting the best out of the risk insights, the defense and depth, and safety margin understandings that we already have. That Andrea Andrea I'll turn it back to you. Thanks Doug you gave us some really good insights based on your experience so I know I appreciate that I'm sure the audience did too. I think it's time for our first polling question so that's kind of exciting. So these are going to be a couple questions about your the audience's experience in using risk informed decision making and how extensively you've used it and has it benefited your work. So we're going to ask these questions and we're going to save the answers to the polling questions to the end of the panel session so you can wait in anticipation for those and then just a note in order to access the polling questions you can go to the right of your screen and toggle from the Q&A space into the polling questions so that you can answer those. Okay I can't see the polling questions can anybody else on the panel see them? No okay so I think what we'll do is maybe move on to our next speaker and maybe we'll catch up with the polling questions after the second speaker. So our second speaker will be Shmaine Yolaoui and he's going to be talking to us about risk informed decision making in Canada so I'm going to turn it over to Shmaine. Thank you very much Mrs. Jair thank you for having me in this technical session. So my presentation is about the risk informed decision making in Canada. If we go next slide please. Yeah so this is the outline of my presentation. First I will discuss the RIDM within the CNS regulatory framework. I will describe the key principles of the RIDM. I'll provide a brief history of RIDM in Canada. I will discuss shortly the staff procedure for risk evaluation and estimation and evaluation. I will talk also about the PSAUs to support the RIDM. I will then elaborate on the CNS series handbook tool that we have developed at the CNSC. I'll talk about the emergency mitigating equipment consideration known as flex in the US. Next I will discuss the benefits and pitfalls of PSA using RIDM and then I'll finish with a short summary of my presentation. Next slide please. So for the RIDM in the regulatory framework here are some highlights. We have a regulatory document RECDoc 3.5.3 which is regulatory fundamentals. This regulatory document describes the risk informed approach to licensing and compliance activities. It emphasizes that the focus is on issues of higher risk. There is also an internal RIDM procedure at the CNSC that further elaborates on situations where staff can apply a risk informed approach for regulatory requirements and guidance and for regulatory decisions as well. This CNSC RIDM approach emphasizes that PSA can be used to complement the DSA and other RIDM key principles with due consideration of uncertainties. We can go to the next slide please. This slide shows the overall approach of risk informed decision making which integrates insights from the deterministic safety analysis, the PSA operating experience and mandatory requirements. The chart if you just click on the next it will show the chart that describes the different elements of the RIDM. The type of decisions and the candidates for RIDM in Canada is the same as in other countries and the regulatory bodies. Here I just named a few of them and these include for example the design, siting, licensing, periodic safety reviews and life extension projects and decommissioning. Next, yeah this is the chart I was talking about I was referring to showing the different elements that goes within the RIDM and same was presented by the by our first presenter, Dr. Doctor. Next slide please. So here I'm showing the RIDM key principles as introduced in the chart from the previous slide. The key principles include first the demonstration that relevant legislation and requirements are met. The defense in-depth is maintained as the second key principle and this generally this principle is assessed without invoking the PSA but we know that the PSA can offer some insights like the cut sets can be informed by revealing how many failures may occur before we can get the core damage frequency or a large risk frequency. The third key element or the key principle is about the safety margins should be maintained and there are also instances where the PSA can be used to show that the safety margins are maintained. The fourth principle is acceptable risk impact and here PSA can provide the calculation of incremental risk such as the delta CDF and delta LRF and the last principle is the to monitor the performance. However the challenge that we face with these key principles I think is just to find is there is there any balanced way for the consideration of these key principles or is there any weighting factors that we can use to consider the five elements. Next slide please. So in this slide I'll talk about the provider brief history for IDM in Canada. So traditionally decision making is heavily relied on defense in-depth and export judgment in the past but the last decades we show increasing use of PSA in Canada use of PSA in our IDM and this is mainly because back in 2005 we issued that the CNSC two regulatory documents one on PSA and the other one is on the liability program for nuclear power plants and in parallel the the staff were developing a procedure our IDM procedure based on the CSA standard which is called risk management. In 2018 the CNSC issued this regulatory documentary Regdoc 3.5.3 and in 2019 about three years ago the CSA standards issued the CSA 290.19 which is called our IDM for MPPs building on staff procedures and right now there is currently a CSA is conducting a survey on the use of this standard and to which extent it helped the industry and the regulators in applying this standard. Next I think I covered all this border. Yeah I think I covered all these borders. Next. Next. Next. Next. Yeah. So now on this slide they I'm showing can you just press the next please next so yeah and next and no another time okay perfect so in this slide we I'm I'm showing the staff procedure for risk evaluation this is based on on the risk tolerability scale for determining the risk significance levels it's it's almost I should say deterministic and the risk evaluation is based on using matrices as you can see in the second chart which shows the consequence and likelihoods and we defined the risk significance levels dependent on the likelihood and the consequences of each so this procedure which was based on risk tolerability was successfully used in the past for the reclassification of what we call conduit generic safety issues and I'm providing just in the blue box some examples like we for the reclassification of generic issue of pressure tube failure coincident with the moderator heatsink failure which we call local loma loss of cool accident and loss of moderator safety improvements for steam line breaks in multi-unit nuclear power plants and the large local reclassification for certain break sizes to to be on design-based accident next step please so in this slide we the PSA used to support the RIDM as a direct use of the PSA and the PSA result that we get from the licenses CNS staff have developed what we call a risk handbook tool which is a web-based application in this tool the PSA and reliability program results and insights are used to risk inform the licensing and compliance verification activities other PSA used to support our RIDM include the risk management for outage planning for online maintenance for example the what we call risk monitor or risk watch we use it also for life extension project PSA can help identify safety improvement opportunities if a plant needs to go through the life extension we also use it for SMG severe accident management guidelines development emergency preparedness drills and exercises because all the prognostic and diagnostic analysis are done using the PSA and scenarios derived from the level 2 PSA next slide please so in this slide i'm going to develop a little bit further on the CNS risk handbook first the purpose of this handbook is to support the regulatory compliance program focusing primarily on applications for inspections supporting us which means that we this is mainly developed to support site inspectors this tool is used for to optimize the inspection planning and improve efficiency for example we get request for inspections of let's say inspections for electrical mechanical or whatever says me qualification inspections then we use the PSA insights and results to provide a focused set of equipment or human actions or specific hazard information for the inspection purposes we do also use the PSA to evaluate inspection results and for example the site inspector may have a sense of the risk increment incremental risk if a piece of equipment is taken out of service it also helps in the determination of safety significance of operational events if some event happened to have some sort of idea what is the incremental core damage frequency or largely frequency next slide please this slide is about the emergency mitigating equipment credits in the PSA emergency mitigating equipment once again is known as the flex in the US emergency mitigating equipment functions that we have is the first to prevent a severe accident second objective is to repower instrumentation and monitoring critical safety parameters the third objective is to prevent the severe core damage the fourth one is the universal retention of collapsed core and the fifth objective is to repower containment supporting functions next next so if we can see in the in the in the graph that I showed here the third the three first objectives they are applicable to level one PSA and the two last objectives they are applicable to level two PSA next slide please so the prerequisites for EME credits in the PSA of course as any everything that you created in the PSA there should be a clear guidance to deploy emergency mitigating equipment and decisions are also made within the main control room or secondary control room by authorized staff because when the operation shifts to the emergency operating center probably the to deal with the human actions is more cumbersome in this situation the challenges that we face with crediting the EMEs in the PSA is the the first one is the use of PSA models with or without EME for different applications let's see if we want to come to compare against safety goals do we consider the improvements from EMEs yes or no the second question which is also discussed now at the CNSE with the industry is the identification and classification of systems important to safety so if we take into question should we consider emergency mitigating equipment as systems important to safety with all reliability program that should be applied to it yes or no yeah this is we have also the issue of EME credits in multi-unit PSA sequential EMEs or the flex equipment for different units going into accidents and there is also the challenge of surveillance requirements for EME because this most of the time they are just industry grade equipment next slide please in this slide I will discuss the benefits and pitfalls of PSA I think Dr. Dr. was provided very insightful about the benefits of PSA the benefits as we know that PSA can provide the rigorous and reproducible assessment of incremental risk delta CDF delta RAF compared to risk significance levels which are based on on subjective judgment let's say oh we think that consequences are high and they're likely it is low PSA will provide the very very good quantitative assessment of the incremental risk I'm just giving an example here for example in our operating policies and procedures which is the same as tech specs in the in the US we have in the traditional deterministic approach if the system redundancy is reduced the repairs shall be made promptly or other actions taken to ensure adequate system reliability and capability so if we want to interpret this this requirement what do we mean by promptly is it me does it mean minutes hours days so whereas the PSA we can calculate what's the if we have a time at risk or the completion time we can calculate exactly what's the incremental risk and compare it to the guidelines the pitfall the pitfalls of the PSA is most of the PSA sometimes we there is a lack of cause and effect relationship in some cases for example if emergency cold cooling valve is passing this is not modeled in the PSA if we want to change the methodology for trips at points determination PSA may not be a useful tool for piping inspections you need to do some work on PSA to to help you use the PSA there is also the issue of uncertainties in the PSA in the PSA and we always caution about the over reliance on PSA to address all safety concerns PSA is not the solution for every every issue that we may have next slide please this is the summary the summary of my presentation I think in summary I would say that guidance is needed on how to assess impacts on defense in depth and safety margins as well as on benefit cost analysis if we don't have clear guidance on how to assess or how PSA can can help assess the impacts on defense in depth and safety margins we may not have a clear and reproducible scheme for our IDM and I think we all understand that PSA is a valuable tool to complement deterministic and export judgment in in the our IDM process uncertainties of course should be accounted for for especially when the PSA results are close to the acceptance guidelines and I believe that the development of our IDM guidance will allow a transparent and reproducible process for regulatory decisions next so this is this is the end of my presentation thank you for your attention thank you Schmin okay I think now we are going to go to polling questions so we're going to do we have a total of four questions and we're going to do the first two now to kind of catch up so the first question is about your experiences and using risk-informed decision-making how much have you used it and how has it benefited your work so I'd really like to hear from you and then maybe during the discussion at the end we can get some feedback on this so oh there's the results already have you used risk-informed decision-making as part of an official technical process the vast majority of you have and just a few 13% have not so it'd be interesting to hear experiences for those of you have how you how you got to the point of using those risk-informed decision-making tools that you talked about and if you have it maybe we'll hear a little bit in the discussion session about why some folks have not used it so that that's kind of an interesting result there and I see it's fluctuating a little bit we can go on to polling question two so polling question two is going to have to do with how risk-informed decision-making has benefited your work and I think we we've already heard that there are differences of opinions on the benefit of risk-informed decision-making so it'd be interesting to see the results here okay so we're still fluctuating a little bit but it looks like most of the folks in the audience have the opinion that risk-informed decision-making has benefited their work and again a smaller percentage says it hasn't so it'd be interesting to hear as part of the discussion at the end where perhaps it has significantly benefited our work and if it hasn't why not or maybe there are certain areas where it's not as beneficial so that that could be an interesting result we're going to move on now to two the second polling question so there's an A and B aspect to those as well the next two questions have to do with your opinion about how both the industry and the NRC or other regulatory agencies are using risk-informed decision-making two A here is a question about whether industry has doing enough with regard to risk-informed decision-making and the results are fluctuating just a little bit at this point so it looks like majority opinion here is that there's more to do on risk-informed decision-making which really isn't surprising because I think we've already heard it's a complex topic and there's about 20% of you think that industry has done just the right amount there okay and then the last polling question for now is going to have to do with how regulators like the NRC used risk-informed decision-making and whether there's been enough there so a kind of interesting the same result here for regulatory agencies as we saw for the industry the vast consensus seems to be that there's more to do and just to me personally that's not surprising I think we've made great progress but you know given some of the uncertainties and challenges that both Doug and Shame just talked about it's not really surprising to hear there's more to do okay so that's interesting and maybe we can build off the answers to some of those questions as we move along in the presentations but at this point we're going to go to our next speaker who is Mark Sager who's going to be talking about the safety marker study Mark good afternoon the NRC has a long history of incorporating risk insights into its decision-making some of that you're going to hear about from Mike in his in the following presentation how the focus of my presentation is looking at the potential impacts of safety from the use of risk-informed decision-making I plan to go over some insights from a fairly comprehensive assessment conducted by the NRC staff recently to look at safety trends within the U.S. nuclear industry the staff looked at a broad range of measures and markers to see if safety was trending in a positive direction negative direction remaining steady or were indeterminate keep in mind that this was a trending assessment and that the focus was in looking for trends there was no attempt to make a comparison against a particular performance metric the staff also made no attempt to ascertain or infer compliance or non-compliance but our regulations next slide NRC maintains numerous safety activities and programs to monitor assess and reinforce safety for the staff assessment as noted the staff looked at a broad range of these activities to get a comprehensive safety picture of U.S. commercial nuclear power plants over the last 20 to 30 years a time period that included issuance of the commission's probabilistic probabilistic assessment policy statement issued in the 1990s some of the measures and activities reviewed include looking at the numbers scrams the number of license event report inspection findings performance indicators risk measures and insights from studies such as the state of the state of the art consequence analysis of soccer just to name a few the staff assessment focused only on safety and did not look at other possible trends such as economics or electric output next slide a key consideration in the staff's assessment is that there are several aspects of trends that can affect the overall conclusion one clear example is the time frame you consider some performance measures might show improvements over the last 30 years but may not show an improvement if you only look that a 20 period another consideration is the advancements in our technical and modeling capabilities for some performance measures improvements might be a result of our advancement in these areas and not a natural safety improvement considerations such as these require caring making a definitive conclusion on the training direction of some activities accordingly for a number of activities the conclusions was indeterminate even when the staff could see a trend for number of the safety measures measuring performance came down to a collective engineering judgment next slide one of the considerations discussed in the last slide is the time period chosen impacts the overall conclusions looking at the period between 1988 to 2000 the staff saw several performance measures that clearly show safety improvements this period saw many changes in the u.s. nuclear industry including implementing the station blackout rule the maintenance rule and individual plant examination program next slide this slide highlights an example of a performance measure that showed clear safety improvements during the 1988 to 2000 time period if you look at the chart on the left it shows scrams while critical per unit over time we show a clear positive trend over this time period please note that the reduction in the number of scrams over time is viewed as positive in this context the chart on the right reflects the difficulty in ascertaining safety improvements even when the data clearly indicates that things are moving in the right direction it shows significant events per unit over time which also shows a clear positive trend over this period again the reduction in the number of events over time is viewed as positive even with the apparent positive trend the staff couldn't definitively determine that this was reflective of safety improvements because the agency changed this reportability guidance during the period which could have affected the trend however taken together the two graphs helped support an overall conclusion that nuclear plant safety improved during this period for both of these performance measures the positive trend after 2000 is not as clear next slide it stated when looking at the last 20 year time period the performance measures do not show the same level of safety improvements there's still a trend in the right direction but we do not see the same level as seen during the 90s there are several possible reasons for this one is the safety improvements from the 90s may have made further safety impacts less obvious that is the law of diminishing returns this last 20 year period does include several changes that have a positive safety impact on the industry including the react to oversight program the b5b security compensatory measures requirements use of floods and design enhancements such as to react to coolant pump shutdown seals therefore another possible reason that we see less improvements in the in the last 20 year time period is that safety significant changes may not be fully realized next slide there are several different ways to look at all the performance measures these are six categories of performance measures used by staff as you can see from the different category activities some were more easier to ascertain a quantitative trend next slide over all the staff looked at roughly 50 performance measures these roughly 50 performance measures eight clearly show a positive trend let's look at each of these the number of scrams has dropped to historically low levels the total number of scrams and scrams per unit show 20 year decrease in trends while the number of plant plants with zero scrams show a positive 20 year trend the number of the inspections has been trending steadily downward when looking at two different time periods the occurrence rate of all precursors exhibits a statistically significant decrease in trend for the 2000 2020 time period the collective radiation dose per unit shows a 20 year decrease in trend the number of greater than green finally show a decrease in trend since 2014 a micro level fleet average internal event core damage frequency or cdf based upon the agency's risk models is trending downward installation of the reactor coolant pump seals in western house pwr nuclear power plants has resulted in a reduction in both cdf and large early relief in our risk models lastly there is a highly statistically significant decrease in trend in the overall in the frequency of overall loss of slight power only one performance measure showed a clearly negative trend which is loss of all slight power recovery time this marker showed a statistically significant increase in trend next slide other roughly 40 some remaining measures looked at by staff nine were viewed as positive but the assessment was more qualitative in the interest of time i won't go over the nine which are listed on the slide the remaining 30 plus measures were either steady a staff could not say definitively one way or the other the training direction an example where staff could not make a determination was the number of license event reports even though it showed a positive trend as i previously noted there were a number of the 30 plus performance measures like this that the staff included as indeterminate next slide several points noted by the staff in the assessment were that there is a reduction in risk over 30 year time period as measured by calculated average core damage frequency however they noted this only considers the contribution of risk from internal events external events has its could in some cases significantly add to plant risk the staff also noted a reduction in performance issues based upon the staff's assessment there's some margin between calculated risk and nrc safety goes however it is important to note that both uncertainty and external hazards needs to be considered in determining the amount of the margin next slide in conclusion based on the performance measures looked at by the staff the user risk insights in the decision making at the agency is having an overall positive impact on the safety of the industry notwithstanding the need for consideration of the influence of external hazards in our assessment and accounting for uncertainties there's been clear safety improvements the performance measures show a more prominent trend in the 1990s with a more gradual trend after 2000 next slide this concludes this concludes my presentation we'll save questions for the end i turn it back to you andrea thank you mark i just want to put a plug in for everybody to get their questions in we have one more speaker to go but uh so you have a little bit of time but if you have questions go ahead and submit those and we'll cover those during the discussion period now we're going to go to our third polling question which also has two subparts and these questions have to do with your opinion on the state of industry safety nuclear safety today versus 10 years ago and the role of pra so first question in your opinion is the nuclear industry safer than it was 10 years ago it looks like there's less fluctuation now in the numbers but a vast majority of you feel like the nuclear industry is safer than it was 10 years ago and then there's a small percentage of folk who are either in the no or depends category okay and then uh 3b uh polling question 3b is about uh the role of pra with regard to uh industry safety in whether pra has played a role so in your opinion has the use of pra and risk informed decision making made the nuclear industry safer today than it was 10 years ago okay those results look like they uh kind of stop fluctuating so looks like these results follow closely the question right before this on the state of uh nuclear safety and that the vast majority of you feel that pra and risk informed decision making has contributed to safety and then there's a smaller percentage who either answered no or maybe okay so that that was interesting um we're going to move on to our last speaker mike franovich and he's going to talk about safety improvements using risk insights so on to you mike thank you andrea if we can have my first slide please next slide all right i'll try to get us back on track a little bit we're a little running a little bit behind schedule here uh so what i'm going to cover in my discussion is to give some context a little bit of historical background of issues or policies that are active today and still affecting and shaping the way we do our business i also want to give you a few tangible more contemporary examples of where advancements in risk assessment and other engineering analysis has actually helped enable us to make better decisions some additional work going on in particular with uh weather events uh heaf events and also new reactors but more so what i'm talking about is how it's shaping our work in the operating reactor side in particular licensing and then a shameless plug here for an agency initiative that i have a i'm very strong proponent of and that is the b-risk smart framework it's an initiative that is looking to try to unify and provide more uniform application of risk concepts in all of our work not just the technical work but some of the more of the corporate support work and using the risk triplet and looking at the rewards that may come out of doing various projects just not the downsides of them so if we could go to the next slide okay meeting the challenge of becoming a modern risk a more modern risk informed regulator we could spend a lot of time on that topic alone the journey is continuing a couple of insights i want to share with you is that the technology in terms of risk assessment does continue to mature risk are dynamic we do have to have effective risk management by maintaining our vigilance and assessment of operating experience the technology and insights do complement the traditional defense and depth framework that we have worked used for many many decades to achieve an acceptable level residual risk not zero risk but residual risk and this journey does continue so if we can go to the next slide now i mentioned the b-risk smart framework and a couple other things that are going on in the agency really what we have seen is a great push here in the last few years to try to come up with a more uniform application of guidance that cross cuts different offices and business lines really looking at the risk proposition not just the negative side of what are the risks of certain decisions and not just all the plants themselves but also different projects but what are the rewards what are the gains that we may get go gain by pursuing certain activities in the agency and trying to do it in a way that shows that we are getting a good return on our investment and if we are applying these approaches in particular the b-risk smart framework it should enable us to become more agile and adaptive when we look at a different projects within an office or across offices in the nrc so i'm going to give you a flavor a few of those things that are going on that that might provide more insight what's going on inside the nrc you can go to next slide now mark mentioned that i would talk about some of the major policies that are affecting our thinking in the agency my point of this slide is not to go into a deep dive of these particular policies but to to share with you that they are active they are live they do shape the way our work is going on today and periodically we need to need to go back and refresh our memories as to why we have these policies in particular starting in the 80s with the severe accident policy statement on how we treat severe accidents also some anchoring guidance and expectations out of the severe out of the safety goal policy statement in terms of the qualitative safety goals as well as the qhos and then moving forward in time to the pra policy statement where there's one particular aspect i want to highlight out of that policy statement that's still alive and well and that is we should be using pr8 technology to the extent increase in all regulatory matters as supported by the state of the art and it goes on to say the compliment that of defense and depth philosophy that we use in the agency and my next few examples are going to illustrate a little bit more how the state of the art or what we would say today state of the practice has actually advanced quite a bit and then lastly i want to note a very important commission decision that was made in 1999 based on a commission paper a white paper set up in 1998 and that is an overarching framework for how we are to use risk and form performance based regulation not just in operating reactors if you take a close look at the paper in the srm it actually speaks to our work in the materials area as well as the fuel facilities area that paper is alive and well and is germane to a topic i'll talk you hear about when i get toward the end of my discussion next slide if you can advance it to the next slide please okay my screen is frozen so i don't know if you all are seeing a the next slide but if not i will try to use my local copy okay it looks like we have a little bit of a technical problem going on at the moment that all the slides are frozen so i'm just going to speak to what would be on the next slide that you would be seeing and that is the integrated decision making principles that Doug spoke of and that there are five key principles not really intending to go into depth of each of those principles of defense and depth and safety margins and performance monitoring i will highlight that the the objective also is that the risk changes and for facilities should be small and within the safety goals but really it to me it's something more important about how we go about making these integrated decisions you can take each one of those decision making principles and do them in isolation and that's been a real challenge in the agency to to try to break down some of the silos and do more integrating through what we call integrated review teams where you start a project with the principles in mind not trying to bring them together through a long review process and seeing how they all fit and that they really are not mutually exclusive of each other to me the integrated decision making process really is a place for critical thinking what's going on in each one of those decision making boxes in terms of defense and depth the plants are not static they have changed many capital improvements have been made where are the margins both in physical margins and analytical margins so it's important that we have a culture that drives for having those conversations and i apologize for whatever the technical difficulties are but we're not i'm not able to see the slides either uh on on my end so i i will be speaking verbally hope you all can still hear me uh i will just speak to describing my uh for sake of time um there's another process oh here we go thank you very much looks like the technical problem has been resolved if we could go to the next slide and these integrated decision making principles out of red guide 1174 we're on rev three by the way even though this red guide came out in the 1990s we do use it to influence uh and help uh guide a number of other processes we have and one of those processes i'm going to talk to you about lick 504 that is an agency or n or r process if you recall the davis bestie event from 2002 we're actually i believe close to the 20th anniversary um important lessons learned and you need to have a process for not only making integrated decisions but also documenting the basis for your decisions so we had a number of corrective actions in the agency we took to create such guidance this particular guidance is used very uh much in the agency today it's a mature process we're currently on revision five in the latest revision we did add other considerations for example if if there are actions that are warranted would what would be some of the risk offsets for example any additional exposure to radiation workers those should be active considerations and what we are looking at in terms of options um and also we included a section on risk uh decision or rather decision making biases there are some areas where you can get into group thank for example and we need to be conscious about those when you're in that decision-making box or curve that you're not susceptible to those biases or you try to minimize them it is a two-step process where we look at immediate safety issues first uh if we need to take proper regulatory action and then if if not we'll look at the longer term actions uh in a second step process and we are using the be risk smart framework which is in a new reg that came out new reg k m 16 i encourage you all to read it it's a very easy read again it's not unique to pra but it's promoting more of the risk triplet across the work that we do in the agency can you go to the next slide all right the first example i want to give us back in august 10th of 2020 we had a derecho event that hit the dwayne arnold plant uh very powerful line of wind uh storm that came through the plant and the plant performed as expected and so did the operator so it's a very good news uh but we did uh identify a few um unexpected conditions that happened even though there was loss of outside power which is not unexpected there was uh an issue of potential combined event and that the essential service water system did have some degradation from debris that was on the on the river that came into the intake system so we did use the lick 504 process a few important takeaways came out of that and looking at similarly situated plants and those similarly situated plants had done some upgrades for example alternate cooling for diesel generators if they're dependent on essential service water they have alternate means we also found now with greater capability to quantify the benefits of flex that uh that is actually quite a difference maker in terms of risk impact depending on the nature of the plant and its location and its site specific hazards it is quite influential on the risk results and we applied the best be risk smart framework and the big takeaway there was even though we recommended a generic communication in terms of information notice um we also used the t in the teaching elements um to get the word out and conduct a a webinar and a multi organizational panel to discuss the insights that came out of the study of various sites that we did take a look at that that have some susceptibility to derecho um we did also find that the risks were not trivial uh but nor were they in a matter of a level where we needed to do some type of mandatory backfit analysis uh so they were they did have import and we thought it was important to share that information with industry and the rest of the nrc we do have significant turnover uh a turnover in the industry as well as in the nrc so it's the t can also be viewed as knowledge transfer we could go to the next example Doug had mentioned the number of capital improvements that plants have made in particular for nfp 805 implementation with alternate seal injection backup diesel generators uh and and so forth uh one of the hazards the plants did do additional measures for is to provide additional protection for high energy arcing faults we more recently have come to discover there's a little bit more aluminum in these electrical components than originally envisioned and that aluminum might be a little bit more reactive uh than copper and so what does that actually mean if the hazards are slightly different in terms of risks so we're currently undergoing a lick 504 evaluation and in that first step which we did complete we used more of a defense in depth qualitative risk uh thought with the defense in depth in mind looking at the plants already have a level protection for heave but they also have protection from post 9-11 measures for dealing with large fires and explosions so these are other qualitative considerations we need to bring into our decisions the work is ongoing and there's a massive effort actually been going on between the nrc and epri for years to build more consensus models to to help bring in the more advanced fire modeling capability as well as the advances in the pra technology and in my last example if we can go to that slide is on new reactors uh i will not go through all this uh on new scale specific uh you can read that at your leisure we had a very important uh takeaway in a commission direction uh which had to do with the single failure criterion and the uh the treatment and use of risk and those kinds of decisions but more importantly there was a insight our direction from the commission as a gentle reminder that um we are to apply risk informed principles in our decisions and it actually looped right back to the 1990s policies that i mentioned that are still active so we were kindly reminded as a staff you need to continue to be applying these principles in your work and and so that's what we're doing we're continuing on that journey and if we can go to my last slide i'll wrap it up a couple takeaways uh i want to leave you with a couple of thoughts and we continue to support risk informed decision making through our programs risk is permeating a lot of our work uh it touches a lot of areas in the agency we are also using um in a manner that still complements defense and depth and safety margins and we're also including consideration for enterprise risk management which is a federal government wide requirement and how we manage our projects and activities we do also take that into consideration as well and we're also leveraging this new be risk smart framework with pra technology and also looking at other business lines to see how we can help out and with that i will turn it back to andrea mic that was a really great summary the history of risk informed decision making at the nrc and some other things we have going on so thanks for that so we're running a little bit short on time so we're going to quickly go to the fourth polling question for a and for b and then we're going to jump right into question so we have plenty of time for the discussion section so for a and b i have to do with the use of pra and risk informed decision making um in our work and the first question is pra and rhythm create work efficiencies and aid in correctly focusing priorities through or false so that looks pretty steady it looks like the vast majority of you feel that pra and risk informed decision making do create work efficiencies and help us to focus our priority would be really interesting to hear from some of the folks who are saying maybe it depends i would be interested to hear about that maybe we can get to that during the q and a okay and for b for b is pra and risk informed decision making create efficiency benefits when interacting with regulators so it looks like we're evening out there a majority of you say sometimes it'd be interesting to hear when those sometimes are what the considerations are and whether pra and risk informed decision making create efficiency when interacting with your regulator so that's interesting and then there's a smaller percentage of people say always in a very small percentage say never okay so with that i think we're going to go right into the discussion um section of the panel session we have a little over 15 minutes left and so the first question i have is actually for uh i heard it was dr true uh so first question for dr true no dr here no dr okay i thought somebody called you doctor earlier so just Doug plain was just being very nice oh okay great okay so first question for um Doug in recent years there has been an impression that pra and risk informed decision making are too complicated and costly do you believe the benefits of risk informed decision making have been exhausted in the current framework and if so what needs to be fixed and while i start with Doug i'd also invite other panel members to jump in um after Doug finishes thanks thanks and i got a number of thoughts on this and i think it actually ties somewhat back to some of the answers to the questions we were asking so i think people generally felt like there was more that could be done and i think that's certainly i certainly personally believe that's certainly true but i think also the last question i think it was that talked about efficiencies sometimes we do get kind of bogged down um and i have a few thoughts about that i think part of this is we're on a journey here as the nrc tries to become a more modern risk informed regulator and as mike pointed out there's a lot of work going on to try and bring people up to the same understanding of what risk informing really means this is true on the industry side as well as nrc it's not unique to the nrc that this industry was founded on sort of deterministic approaches to things but bringing risk into this is a foreign concept to a lot of people that have never been exposed to it so i think there's more work to be done to educate people to understand what risk is and what it isn't understand what its limitations are and what its effective uses are and that's something that that we all need to undertake i do think secondly on this subject that there's been the tendency sometimes to focus way too much on the numbers we get all bogged down on decision thresholds and and and i think that it's really should be more about how we understand the plant from the pr perspective numbers are important but there are uncertainties in the numbers so getting too bogged down in the numbers is not not helpful and can often stretch things out unnecessarily and third you know they're doing some of these applications that are more ambitious for example risk informed collision times require a more extensive and expansive dra because you're asking a more difficult broad question and that makes a little bit more costly and i think that's been a challenge too but we still we still continue to see places where we think there's opportunity for more risk informed thinking we're in the midst of a discussion about the tornado impacts on cash loadings that we think has a low safety significance and merits being disposition that way we we also have you know other issues going on in the industry where the we've successfully used the low safety significance process to this business so i think there's more opportunity and i think education that we will a long way to help us move in moving forward on that just invite any of the other panelists to weigh in on that question but i would say i agree i agree with Doug completely in his assessment i would add that i think you see a little bit of a lagging effect going on because there's an ongoing shift to move to the more advanced risk management programs when i'm speaking of more specifically it started with the surveillance frequency control program we also have 50 69 and the risk categorization of ssc's and then the risk informed completion times for the surveillance frequency control program i think we're right now at almost 100 percent of the plants have been authorized to use that program which started with the first plan i think believe it was limerick in 2008 as a pilot um but we are making significant progress in approving the 50 69 uh applications that are coming in uh i think we're well over 20 now as well as the completion time program and as those reviews go on eventually they get authorized and they'll get implemented but it's just and it's in a staggered way uh and so when they get authorized and we see more broad use of them i think you're going to feel more of the work that's going on now that doesn't maybe resonate or feel like we're doing enough in the way of risk informing but there is a lot more room that can be done and a lot of other domains um one of them that we're actually trying to do a little bit is in in the space of treatment of the alternate source term um can we use some great approaches and thinking in that space since we've collected a lot of engineering experience in particular for seismic and seismic PRAs um that's just one small area but there's a lot of other domains where i think we can actually do much more and then there's the whole area of the new and advanced reactors which is a whole another topic area okay i'm going to move on to the second question um the second question was for schmaine but again i'll invite other panelists to jump in on this one um first of all a question is a statement great presentation schmaine so thanks thank you and then the question in your experience what is the fundamental difference between defense and depth and safety margins in traditional engineering and risk informed processes um is it the same or is it balanced in a different way uh thank you very much i think this is a very good question uh i think we all know that defense in depth and safety margins are part of the traditional deterministic approach and these are fundamental concepts but is this how this is balanced in our idm once again this is a very tricky questions and there's no straightforward um uh answer to that i think it's a very uh difficult like to say as i in one of my slides i say there's no weighing factors to balance the different key principles of the r idm uh but once again how this uh how this is balanced in our idm that did and defense in depth is something that decision makers and makers need to consider uh it's not always easy to say for example to which extent or how much safety margins erosion is acceptable these are not easy questions i think we go uh with different case on a case by case basis and they need to have all the specialists from safety deterministic safety psa engineering judgment and other other specialists to have to to have a good picture for the risk decision makers so i leave it to other planners if they have any other thoughts to provide the third question um the third question was for mark um but again others can jump in how do the trends um that you describe mark compare with the overall consideration of defense in depth and safety margin even if some trends go up is defense in depth and safety margin being impacted well so so doing the doing the training assessment we we didn't consider the defense in depth and safety safety mark mark mark safety margin um i think one of the things i started when i started presentation i i mean i i made a comparison to risk informed decision making primarily because the defense in depth and safety margins are kind of like fundamental they're always there and so the changes that we've been ignited over the years been primarily in instituting risk informed decision making uh so so i don't think the trends uh i don't think that if i'm understanding the question correctly i don't think the training assessment that we looked at i don't think it had took any consideration in terms of safety margin and defense in depth because those those are kind of like baseline they're always there obviously if some of if some of the trends go in their own direction it's going to impact the the amount of margin we have um so you know but uh but we didn't we didn't fundamentally look at that with the exception of looking at risk we we did a comparison looking at the the amount of margin in terms of the risk numbers that we're looking at now obviously as if that risk number if that goes up then you you will see an impact on the safety safety margin if i'm if i'm gonna send a question correctly just ask if any other panel members have anything else to add yeah i i just add that i think mark in your presentation you talked about how the v5b and flex uh were considered those are new levels of defense in depth that we didn't have back 10 years ago 20 years ago and our reflection of actually increases in safe in defense in depth uh and uh safety margin is a little bit harder to measure uh you know well done p r a can often do that but sometimes it even has difficulty doing that as well but i think um we haven't seen significant reductions in defense in depth in the applications we've done so i think it's increased net or where we were 10 or 20 years ago might might take anything else on that question if if not we're going to move on to the fourth question um mike this one was for you but i really think it applies i think any of the panelists may have reflections on this one so i'll start with you and then just let everybody else um add so the fourth question is it is true that risk informed decision making has expanded however in recent years it seems that either nrc or industry have pulled off in this expansion on areas like physical security and others do you think there's a slowdown due to a blockage that needs to be overcome or is the low hanging fruit no longer available that's a that's a very interesting question i think the i would say there's still active work going on in deploying reviewing and deploying approving the programs that are coming in that i mentioned are ready to three advanced risk management programs and so it may not look like there's a lot of work going on but i would say there's a lot of production work and when you look at what's probably taking the oxygen out of the room uh on a lot of the operating reactors work it's that of the work of the advanced reactors and the licensing modernization project which is really strong um more pra technology dependent framework there's been some shift in that area but on the operating fleet you see more of the production we have had inter uh actions quite a bit with the owners groups in particular in a few projects that are striking some interest one of them is in particular is there a possibility of looking at how a licensing modernization project framework could be applied to the operating reactors the lmp to operating reactors and leveraging some of the insights that are coming out of our level three pra project which is ongoing work there is some interest there uh how that might shape and a level review and work in chapter 15 space uh that's still sort of in its infancy as a new uh concept we do have other work going on for example the uh risk informed process for evaluations ripe we do have the first submittal in-house uh that's being actively worked on right now um you know i think when you just start a program like that you see where it goes uh but it that too is sort of at the at the beginning phases so there's still a zeal i definitely see a zeal there um and if i listen to my colleagues over in the security arena they have some done some recent work in terms of off-site response capability leveraging flex and risk insights there's still work there but i would say it may not garner the visibility uh that it once had i mean our attentions have moved i'm not judging this one way or another i just observe these kind of dynamics in the shift and a lot of work with with the advanced new reactor designs that maybe just appears that things aren't as active in the risk side but i think they're in my perspective they're very active hey other perspectives on that one and i think um this will probably be our last questions of others have perspectives so i just like that one thing in terms of the the the insights about the security um so i think that the the the comment about the low hanging fruits it's probably a good good good analogy if you if you in order to quantify the risk for security i do think it's it's a bit more challenging because trying to quantify the initiating event is it's you know it's difficult and so i think the ideal that maybe it's more difficult to to to look at risk from security from that perspective it's you know and so that that's probably a good point so i would agree with it i would agree with the question that the question that that what they're raising so their perspectives on that last question yeah i'll jump in um yeah i largely agree with mike there is opportunity there's still more a lot more activities going on i think that the implementation of the very low safety significance is your resolution process and the right process to try and help us to spend with issues of low safety significance uh quickly and allow our resources to be put back on the things that are more significant are are really important and and still in the early stages of implementation uh and there are a number of utilities that are continuing to actively pursue uh some of the more ambitious applications like 50 69 and the risk informed completion times and we'll see those uh coming over the coming years on the physical security side i agree principal with mark that it's very difficult um and yeah i think beyond the state of the of the art to quantify safety or security risks um and um but that doesn't mean we can't use risk insights and and a lot of the things we've learned on the risk informed completion or the risk informed activities to inform our security practices and i think we've started that with um looking at offside response we've started it with looking at other dimensions of the security response that i think can be made more realistic with a complete understanding of the overall plant and its and its capabilities so i think there are some places to get some benefit there and then uh even outside of the reactors i think there's that risk informed thinking can be beneficial oh and one last thing we're starting a new initiative on uh using risk insights not necessarily puree but risk insights uh in the aging management programs to focus on the things that are of safety significance and put less focus on those that are less safety significance than the overall aging management program and uh working with nrc on that and expect to see um significant benefits there in the in the long-term operations i'm moving to some closing remarks we have only one minute left um just want to take the opportunity to thank everybody who supported today's panel session um really appreciate all the panelists time um i appreciate those of you who are in the audience and participated in the discussion as rich the questions were great so thank you for that um just some some highlights that i picked up um from the presentations and some common themes um i i think we got out of today's session that risk is is graded as a graded concept it's multifaceted um and that can bring challenges um and i think we heard that risk informed decision making does allow us to focus on the issues that are most important to safety i heard today about a lot of tools that we can use to help us make these decisions dog talk about pr a schmaine talk about um processes and procedures and mike talked about integrated decision making to me those are all tools that we can use to help us make risk informed decisions um and use the tools that we have in the best way possible um i heard from your response to the polling questions and the discussion that we have made progress but there's more to do um and how can we move forward to address those issues where there's still challenges um you know i think building understanding through through discussions and understanding different perspectives i heard about issuing guidance to help uh folks um understand how to make those decisions and areas that are challenging like defense in depth um and i think we heard that overall plant safety has improved and pr a and risk informed decision making have been a part of that but however there are still uncertainties that exist we need to kind of step back and look at the big picture so in closing if you can bring up the context slide oh you did thank you very much there were a few unanswered questions um so the contact information for our session coordinators is up on the slide um so any unanswered questions will be sent to these coordinators i encourage you to follow up um with them uh to get the answers to any questions that you have that were not answered today and with that i'm going to close the session and tell you to have a wonderful evening and we'll see you tomorrow