 Good afternoon. I'm Ben Fick, Deputy Chief Financial Officer at the U.S. NRC. I'm very happy to welcome you to this virtual RIC session on Be Risk Smart. Real examples, real impact. I have the honor to serve as the chair of this panel. As one of the three original executive sponsors for the Be Risk Smart initiative, representing the corporate perspective, it's wonderful for me to see the significant amount of progress we've made over the last two years. We've developed the Be Risk Smart framework and issued guidance last March that is publicly available. We also developed Be Risk Smart training for NRC staff, which over half of the NRC staff have completed. We also have numerous success stories around across the agency of staff applying the Be Risk Smart framework and we're excited to share some of those with you today. Please submit your questions as we go along and we will answer as many as possible after our panelists presentations. We'll be asking you two poll questions. So as they pop up, please answer those and we can share the results as we have them with you. Please click on the word poll on the right side of the screen. And we actually have our first question, which is, have you heard about the NRC's Be Risk Smart framework for decision making? We'll give you a moment to answer that question and then we'll share the results. So I think it just takes a second here for it to come up. So thank you for your patience. So it looks like 63% of you have heard about the NRC's Be Risk Smart framework and about 39% no. All right, well that helps us as we go into our panel. So I appreciate that. So slide two, just take a second for that to come up. Okay, so we have a wonderful panel assembled of NRC staff members. Reed Anzalone will describe the Be Risk Smart framework and share reactor examples. Mr. Anzalone is a data scientist for the Embark Venture Studio in the NRC's Office of Nuclear Reactor Regulation. He's the product manager for the NRC's Mission Analytics Portal, a suite of data analytics tools designed to facilitate access to data and enhance data-driven decision making at the NRC. He's also a member of the Be Risk Smart Corking. Mr. Anzalone holds a bachelor's degree in mechanical engineering from the Virginia Tech University and also a master's degree in mechanical engineering from the University of Texas at Austin. He's recently gotten into 3D printing and is known for having a dry sense of humor. We also have Mira Bell Shoemaker, international safeguards analyst in our Office of Nuclear Materials Safety and Safeguards, who will be sharing materials program examples. Ms. Shoemaker began supporting the agency's Be Risk Smart team in 2020 due to her curiosity about how to apply the framework to her niche area of international safeguards and material control and accounting for special nuclear material. She's become a Be Risk Smart ambassador for NMSS, supporting multiple working groups as they apply the framework to activities across the NMSS business lines. She's a notorious house plant killer, but is a proud parent to three thriving children ages 6, 3, and 1. She is a dual BAPS from George Washington University in International Affairs and Political Science, and she has an MA in International Affairs from Catholic University. We also have Billy Dixon, branch chief in Region 3. He will be sharing inspection examples. Mr. Dixon joined the NRC in 1996. He started his NRC career as a reactor engineer in Region 3, and he's held positions as a resident inspector at Dresden, senior resident at both Clinton Power Station and Braywood Nuclear Station, and branch chief of the Health, Physics, and Incident Response Branch in the Division of Reactor Safety in Region 3. Mr. Dixon received a BS degree in Nuclear Engineering from Mississippi State University, and he loves fishing. Rounding out our panel is Stacey Prasad, security risk analyst, Nuclear Security and Incident Response, who will be sharing security examples. Ms. Prasad was technically trained as a computer and electrical engineer at Drexel University before beginning her career at the NRC in 2003. She's worked in the regional office supporting the resident inspector program and regional inspections before moving to NRC headquarters to support electrical inspections for vendors supplying components to operating reactors, aircraft impact inspections, and implementations of the NRC's enforcement program. And in answer, she became one of the ambassadors for the BeRisk Smart Initiative. Stacey indicated that she's using the three extra hours a day that she has saved by not commuting during the pandemic by picking up the violin again. Her husband is grateful to the BeRisk Smart framework as he was able to utilize the framework to promote a different quieter alternative to practicing a musical instrument by determining the risk to his hearing was not justified. At this point, I'll turn it over to Reid. If that makes sense. So if we can go to the next slide. So I'll be walking you all through the BeRisk Smart framework and before I get started with that, I want to acknowledge that BeRisk Smart is kind of a cutesy name. But you shouldn't be fooled by that it's the robust framework for integrated risk informed decision making. Next, please. The first step is to be clear about the problem. This step frames the scope of what you're trying to address, which helps you a lot down the line as you begin to identify risks and how to manage them. And we actually found that without this step, it's really difficult to make sure you're appropriately constraining the scope of the decision that you're trying to make. Next, then you spot what can go wrong or right what are the consequences and how likely is it. And for those of you who might be familiar with probabilistic risk assessment. This is essentially just identifying elements of the risk triplet risk is likelihood times consequences with a couple of minor differences from how we normally do it. The first is that we want to explicitly consider benefits in the decision, because those are an important factor. And as we work through the framework we found oftentimes that it was easier to consider consequences before probability. But that's not a hard and fast rule. Then you manage what you can. This step gives you a space to think about the risks and benefits that you can affect and how you would manage them to attenuate the risks and accentuate the benefits. Next, next step is to act on a decision. Once you've got all the risks and management strategies figured out, you make a decision while considering your risk appetite for this particular decision and that can vary depending on the kind of decision that you're trying to make. If it's a decision that's just for you, it might be your own personal risk appetite. If it's a decision for your organization, you might need to factor in your organizational risk appetite. Next, the next step is to realize the result and this seems to kind of trip some people up, but really it's just the step where you take the decision that you've made and you go forth and implement it. And then while you do that, you manage what you need to to make the risk susceptible or the benefits more appealing and you monitor your performance and progress as you go through. Next, the next step is to teach others what you learned and this step is really important for us at the NRC to create a record of our decision, but it's also really important for knowledge management and to help spread the word of how the risk smart help them the decision making process. And finally next. So at the bottom of our logo there is a little arrow underneath and that emphasizes the iterative nature of the process. As you make one decision you may need to roll right in the next one or in some cases if the conditions that went into your be risk smart evaluation change, you might want to take another look at it. So that's the process. And now I'm going to go through some, some examples from the reactor area. The next slide please. The first impact that I'm going to talk about for that be risk smart is helping have in our in the reactor area is making decisions in the presence of uncertainty. So the be risk smart helps you identify and monitor conditions that are uncertain so you can manage them appropriately before they become an issue particularly using the manage what you can and realize the results that the framework. And in general, having a structured framework like the risk smart, it helps give you confidence that you're adequately considering what you need to even when there's a lot of uncertainty about what could happen or how likely it is. So the first example I'm going to talk to on this slide is the Vogel 10 CFR 52.103 g finding. So, for those of you who might not be aware, Vogel units three and four are Westinghouse AP 1000 reactors that are under construction in Georgia. And these plants can't officially operate until the NRC makes a finding under 10 CFR 52.103 g that all the acceptance criteria have been met. And there are a lot of inspection test analysis acceptance criteria leading up to that finding. Excuse me. This means there are a wide variety of scenarios that might occur in the late stages of the construction process that could delay and otherwise timely regulatory finding if we're not prepared for them. The staff use the be risk smart framework to identify all the possible risks and outcomes and put in place plans to manage those risks where necessary. This is included training tabletop exercises and a new office instruction to provide support to the staff. And one element that we put in place is shown on the screen and that's a dashboard to help us track completion of activities related to Vogel three and four. And while we haven't made a finding under 52.103 g yet, we have confidence that we'll be able to handle whatever comes our way because we've thought it through. Another example of how the risk smart helps us make decisions in the presence of uncertainty is related to CRDM funnel issue that happened at a pressurized water reactor. So during the refueling outage, there was a pressurized water reactor licensee that identified that a CRDM funnel had detached controller and drive mechanism. Sorry. And the NRC had to figure out the safety significance of this issue and what level of oversight was appropriate at the licensee work to repair the problem. And a lot could potentially go wrong if the control rods don't work correctly. But the NRC staff evaluated the issue years in the be risk smart framework and found that the risks that the control rods weren't inserted correctly was managed appropriately during startup with multiple tests. And the foreign material associated with the funnel repair were managed through the licensee's foreign material exclusion program. The NRC inspectors at the site decided to focus on the foreign material exclusion program, and only nominally inspect the other aspects of the event because they were routine at the site. There were no issues at startup and we were able to minimize the use of NRC resources and focus on only the most safety significant issues. And the final example for this slide is related to regulatory guide 1.99 which provides a model for radiation embrittlement of reactor vessels. So there are some indications that this model may be non conservative relative to actual test data at high neutron fluences that would occur at extended plant lifetimes but the data is limited because we don't have a lot of operating experience in that regime. A model that's been developed by the international community appears like it represents the high fluence data better than red guide 1.99 but again the data is limited. Staff considered all possible outcomes that could result from a non conservative red guide 1.99 model and staff also did a probabilistic fracture mechanics evaluation that found the risk of brittle failure to be low even if the model is non conservative. They factored this into a BRISC smart evaluation that included all of these considerations and found that the model was adequate for now. But we're going to continue monitoring material samples collected under 10 CFR appendix H and operating experience related to pressure vessel embrittlement to ensure that the model continues to be adequate. Alright, next slide. So the second impact is making use of all available evidence to make a decision. And this has been alluded to a couple of times in different plenary sessions. I know the commissioner or the chairman is very keyed into making sure that we are using data and evidence needed to make decisions and be risk smart is one way that we can help bring all of the available information to bear, including risks into our decision. So the example I'm going to talk about on this slide is about operator react operator licensing for Vogel unit for as I mentioned that's one of the units that's under construction down in Georgia. Now typically for licensed operators comparable units are added to the operators license using the waiver process under 10 CFR 55.47 to avoid the need for additional examination. However, this waiver requires the operator to have extensive experience on the original unit and for operators license to Vogel three, the waiver process for Vogel four is complicated by the fact that Vogel three is still under construction. The staff evaluated this issue using be risk smart and found that there were both legal and enterprise risks with attempting to use the waiver provision in spite of these challenges. The staff also didn't think it would be necessary or efficient to have a license operators take another exam for Vogel unit for. So what to do, using the risks and benefits uncovered by be risk smart staff decided that exemptions would be both more appropriate and more legally defensible than using waivers. I've also decided that in the long term rulemaking is needed to address this issue going forward for other multi unit sites under construction. Now I'm going to turn it over to Mirabelle to talk about some examples from the materials area. Thanks read. So something I've experienced firsthand when using the be risk smart framework is discovering access to the greater toolbox for risk assessment and risk management and decision making. Being able to identify the entire applicable toolbox for risk informed decision making is especially important in an office like NMSS, which works across multiple business lines and disciplines. It's also incredibly valuable for decision making that considers qualitative information. Whether I've worked in a matrix team or within my own working unit be risk smart has enabled our team to identify the guidance and tools that will aid in decision making at all levels. So the first bullet states in using be risk smart staff doesn't need to reinvent the wheel. Instead the framework provides a common approach to identify and assess risk across various disciplines. A good example of this is when NMSS was faced with two activities, a rulemaking discontinuation and a pilot program for graded approach staff face the question of whether they could discontinue the rulemaking before completing the pilot and determining it was successful. There came the problem statement for both working groups, and by using the framework in parallel, the staff identified that there could be risks related to public confidence and effective regulation, if the activities were not sequenced appropriately. So using the be risk smart framework the staff spotted these risks and identified what could go right or wrong for both rulemaking and licensing activities. Using this common approach to spot, manage and then act on risk insights, staff determined that publishing the rulemaking discontinuation before completing the pilot would likely produce concerns of public perception that the pilot was not fully considered before the rulemaking discontinuation. So staff assessed that waiting to publish the rulemaking after the pilot would allow the pilot to reach level of maturity that would give enough information to demonstrate a risk informed approach and uphold principles of good regulation. So in this example, we sequence both packages for concurrence. When the pilot on the graded approach was confirmed with no significant adverse comments received the rulemaking discontinuation package could reference the completion of the pilot. So in this example, two groups responsible for different disciplines were able to identify and manage risks by coordinating their efforts through be risk smart. Now there are times where disciplines converge and a matrix team is needed to support and MSS activities, contracting renewals for example require coordination with corporate and technical staff before we can renew a contract. In these cases where matrix teams are assembled team members come to the table focused on their expertise areas, equipped with the tools that allow them to identify and assess risk. In the second bullet states be risk smart is beneficial to matrix teams because when the team has access to the full, the full toolbox for risk informed decision making. They can have a coordinated and systematic approach when considering multiple risk factors was varying uncertainties. So consider a licensing for fuel facility. The application includes safety and security experts environmental technical reviewers, legal staff and of course management. And the be risk smart framework supports a collaborative approach to identifying the problem statement and spotting risk and balancing those considerations and multiple risk areas. During a recent license renewal for a fuel facility staff conducted the environmental review in accordance with the need for requirements, they published a draft environmental assessment and a finding of no significant impact for public review and comment. And during the comment period new information arose and produce public concern over groundwater contamination and other potential hazards in the area. This recently applied the framework to the problem statement. How does this new information affect current the current environmental assessment and the finding of no significant impact and should the NRC change its review to an environmental impact statement, considering this new information. So by using the framework the staff balanced risks related to safety and public perception, and this in this example they took a more conservative approach by proceeding with an environmental impact statement. The first time an environmental review for fuel facility resulted in a not in an EA Fonsi decision. So management was incorporated in the consideration well in advance through the use of the be risk smart framework. It was critical for communicating the staff's risk assessment and management plan and to avoid significant delays to the project schedule by commencing the EIS as soon as possible. So to summarize the key messages on the slide, be risk smart framework serves like an umbrella and houses all the available risk tools under it. One division NMSS developed an info graphic that I included on this slide, and it articulates the steps of decision making highlighting be risk smart at the beginning. The elements of be risk smart are conducive for early and comprehensive identification of risk and a coordinated application of quantitative or qualitative risk assessment tools. And the goal is communicating the decision achieving alignment and acting on that decision. Next slide. Now I alluded to a benefit of using be risk smart. And I want to dive into that a little bit more with examples on this slide. So using be risk smart staff is able to achieve early alignment on risks with decision makers. Specifically, early alignment on potential risks, allow the NRC to develop a management plan and environmental reviews. I mentioned a license renewal for fuel facility a moment ago, where alignment in action was needed before staff could proceed with an environmental impact statement. Early on early alignment enables earlier action, whether that's allocating resources, or developing a communication plan and earlier action can reduce impacts to the project timeline. So let me highlight another example. The be risk smart has been the be risk smart approach has been used pretty faithfully in environmental reviews within NMSS, especially when issues involve public stakeholder engagement. So be risk smart allows staff to incorporate public perception risks alongside safety and security issues. And a recent example is when be risk smart was applied when considering whether the environmental impact statement comment period should be extended based on public requests. In accordance with NEPA NMSS prepared an EIS, and the typical comment period lasts about 45 to 60 days. Public stakeholders can submit requests to extend the public comment period, and there's no limit on the number of requests that may be submitted, or the length of time for the extension. Be risk smart has been helpful in guiding these environmental teams through consideration of these requests. The framework gives weight to the value of public participation in environmental reviews. In the spotting and managing steps, teams will consider potential risks if public participation is not maximized or doubt arises in public competence. Using be risk smart, the staff has been able to efficiently disposition requests for public comment period extensions to minimize impacts the project timelines, while supporting full public participation. We also saw the benefits of early alignment, very beneficial with the onset of the COVID-19 public health emergency. Identifying risks early allowed the NRC to focus on risk significance during a time when we were in mandatory telework. I'd like to highlight a specific example before turning it to my colleague Billy who will give another example of be risk smart during COVID-19. So during the public health emergency, when we were in mandatory telework, like the rest of the government, we were faced with a challenge in continuing onsite inspections at facilities. We had planned for an onsite inspection at an independent spent fuel storage installation site. And the purpose of the inspection was to observe the loading campaign, the spent fuel loading campaign. Isphasy inspections of this nature are conducted every two years and this particular site was due for completion in 2021. So postponing would result in a missed opportunity to evaluate cast loading, which is the most risk significant activity at an isphasy. Understandably, this problem was of interest across management and office levels. It was a prime opportunity to deploy be risk smart and gain alignment early, especially with the small window of opportunity to observe a cast loading. So following the elements of be risk smart, the staff identified and weighed the risks of what could go right or wrong in conducting this inspection, and then the likelihood and then the balancing those considerations of safety in the traditional sense of isphasy operation, as well as the public health and safety considerations in the pandemic. The decision makers agreed with the recommendation for remote inspection with support from the licensee cameras and remote monitoring to allow us to observe cast loading in real time. A direct line of communication was maintained with the site and remote administrative reviews facilitated observing the loading campaign. The result that was realized was early alignment enabled efficiency in decision making during a narrow window of opportunity. It also produced a secondary benefit of developing innovative solutions to deploy remote monitoring, which resulted in longer observations of multiple simultaneously occurring activities with reduced accumulated dose to the inspectors. My colleague in the region has personal experience using be risk smart during COVID-19. So thank you for your time and Billy I look forward to hearing your example next. Thank you. So use of the be risk smart process has allowed regional organizations to identify, evaluate and treat risk to different level different degrees of formality. In some cases, it's helped to limit the scope of problems to individual branches divisions or facilities and other is help expand solutions to the enterprise level. It's provided means for that evaluating assessing and managing risk, whether it's regulatory risk, operational risk or reputational risk in areas where traditional standards have been established, but don't explicitly address these types of risk. One of the more significant impacts of the use of be risk smart process, allow for the adaptation and use of new and existing technology, specifically during the onset and prior to just prior to the declaration of a national health emergency. When there were several unknowns and uncertainties. It was essential that the inspection staff was protected against this threat, while maintaining the agency's essential need to have situational awareness. The be risk part methodology was used to evaluate the feasibility of using licensee provided the computer equipment off site to substitute for the presence of the resident inspector staff for specific inspection activities. The risk, the be risk part process was also used at the inspection program implementation level to find acceptable methods to perform key ROP inspections such as the baseline, such as the problem identification of resolution inspection and the cybersecurity inspections remotely or partly remotely. This process has resulted in organizations being more strategic about how to manage and deal with the unexpected, resulting in a more agile organization organization. Next slide please. So regional organizations utilize to be risk mark process to provide for greater flexibility and the ability to adapt to changing conditions when faced with certain decisions within well defined processes. For example, in one of the regional office, a decision had been made to commence a special inspection to assess a series of failures of the diesel generator at a site. During the preparation for the for the start of this special inspection, what was described as a complicated reactor trip occurred at that site. When evaluated using the management directive 8.3 process, the event landed in the overlap region between the baseline inspection and conducting a special inspection. The be risk mark process was used to assess a range of options related to the region's follow up response to this event. Ultimately, through options derived during the use of the spot phase of the be risk mark methodology, regions the regions decided to combine both the diesel generator issues, along with issues observed during the complicated reactor trip into a single special inspection. The exercise of answering the questions of what could go right or wrong, or what are the consequences help inform decisions such as a team composition and communications with the licensing and the public. It also helped develop guideposts and warning signs for regional management related to when that decision to combine the special inspection needed to be reassessed. In another example, the be risk mark process was used to evaluate a suggestion from the region three risk informed decision making team to assign the dedicated inspector stricler to evaluate the licensees management of risk during the outages. So refilling outages are part of the baseline inspection program and then inspection procedure contained a variety of inspection requirements. Depending on what activities are planned during the refilling outage, not all of the inspection requirements are equal risk importance, and some such as the containment walk walkdowns require inspector attention regardless of other on ongoing outage activities. Over the years, outage durations have significantly decreased, making it challenging for the resident inspectors to complete the inspection requirements while focusing on the most safety significant issues. According to resident inspectors asked when they're asked when the outage schedules change the sequencing and risk assessment of activities and the related contingencies can consume a great amount of time in inspector focus. The regions regions have supplemented the resident inspectors with other inspectors during refilling outages, but the resources were not always specifically focused on assessment and management of others risk during periods of elevated risk. The challenge is to complete the inspection, use risk insight to focus the additional inspectors assigned and remain within the inspection baseline sample procedure samples in estimated hours. Using the be risk smart process region three implemented the practice of supplementing the resident inspectors doing carefully selected refilled outages with staff that specifically focus on licensees assessment and management of others risk during periods of elevated risk. By having the dedicated inspectors, those activities were able to be assessed at a level of permission with risk, rather than the availability of the resident inspector staff. The dedicated inspector position permitted the resident inspector staff to remain aware of all outage activities and the associated risk, while the dedicated inspector was able to closely inspect specific activities and associated contingencies based on their detail review of the licensees shutdown plan. So support the realize and teach phase of the be risk smart process. The dedicated inspector provided the coordinator this effort with notes regarding lessons learns associated with implementing this activities. These lessons learns were summarized and have been shared with the regional inspection staff during knowledge management sessions. And the regional staff at right has provided feedback to the inspection program office in our regarding ways to risk rank or prioritize this specific inspection procedure requirements within within the refilling outage inspection procedure. I believe that this effort will help improve the effectiveness of the baseline inspection process and program. During my participation applying some of these examples discussed today, and my studying of other examples from other regions, I have concluded that the be risk mark process has served as a source of improved risk communications. At each phase of the be risk smart methodology, the impact from of uncertainties from all these types of all the types of risk must be discussed, evaluated and managed. And in my opinion, the be risk mark process has held the best ideal that risk communications are not just a separate component of a specific process such as the ROP significance determination process. I've concluded the be risk part process must be integrated in all aspects of decision making, both at all levels, both informally informed informally informally. So next Stacy, we'll discuss other examples where the be risk mark methodology was used to use to further refine for the processes Stacy. Thanks Billy. Processes are the way the public knows they can trust us when we make safety decisions as highlighted by the chairman during a speech that kicked off the brick this year. So lucky you you're going to get a couple more slides of how the framework was used in our processes. In the first case of the three cases on this slide, we use be risk mark to take a look at the process the issue national special security event related security advisors. The purpose of these advisories is to communicate urgent time sensitive operational information that directly relates to the security and common defense of national infrastructure. They're operational in nature and are issued in response to urgent situations or recently identified vulnerabilities. Using the framework help identify a way to remove the view timelines and previously approved what foil put languages use. So this basically means we're now able to expedite issuance without compromising the review process. This example provides a good reminder that just because the process has worked in the past. It does not mean it can't be challenged and that the viewer smart tools cannot be used to identify with the further risk informed processes. The second example focuses on an inspection procedure process. While revising the procedure staff use the form cybersecurity inspections staff utilize the viewer smart framework to determine whether performance metrics met by licensees could be used to reform cybersecurity inspections more efficiently. In this case there was a particular direction in mind from the staff in the headquarters office for how the procedure should be updated. Working through the process included communication with regional offices that helped identify additional what can go right, what can go wrong criteria that were not initially identified. So this helped to further risk informed updates and ultimately change the procedure a little bit. The headquarters staff learned that early engagement is beneficial for procedural development processes. In the last example on this slide it discusses a current process the NRC uses provide alerts and warnings for due to magnetic disturbances. Staff gives the framework to determine if alerts or warning should also be provided for possible or future events kind of like a pre notification. So this was evaluated and it was determined that the process should remain as is with the winner safety consequences related to not making these pre notifications. This last example is particularly important to show that the framework is being used by staff to ask questions to identify what can go right and wrong assess the consequence and make an informed decision that does not necessarily always result in a change. Next slide. Switching gears little kind of going back to the public health emergency. As the public health emergency was first emerging, the agency had to make some tough decisions on how to best implement our inspection program I consider both the safety of those being inspected and the staff going out to perform those inspections. One of the security inspections we perform are the force on force inspections and they verify effective implementation of a licensee physical protection program. These inspections involve a significant amount of people that can be in close proximity in order to properly control observe and perform the inspection. These are very large inspections compared to most of the NRC inspections. Before sliding on the best way to conduct these inspections, the NRC assess multiple options assess the risks of each option using the viewer smart framework for made a final decision on how to best and most importantly safely perform the inspection. For this example we created a matrix where we weigh principles of good regulation to inform our decision. We've been able to reuse this matrix to help us inform our decisions in the areas of efficiency, openness, clarity, reliability and independence. In this case, using the framework helps to identify other options that all they're not the best option during the public health emergency can be used and considered to better inform the future of our security program. Continue with the theme of how the framework is being used to inform decisions during COVID. The NRC identified the risk of the impact to lay billing for applicants and licensees during the public health emergency and its potential impact on NRC meeting our statutory collection requirements. The public health emergency costs financial impacts NRC licensees disrupting their operations and challenging their ability to pay NRC fees. Using the framework and decision was made to suspend billing fees for 90 day period and 2020 to defer all mitigated the risk of significant unpaid invoices. The action of deferring payment to mitigate the financial impact filter at the NRC can be flexible while continue to comply with statutory requirements. Now I'll hand it over to Ben to give you guys a little break from all the process talk. Thank you Stacy. Just to remind everybody, please feel free to put in your questions at any time in the chat box in the right hand side. We're happy to answer your questions after we get through our presentation. So I would like to share a corporate example now related to advancing strategic workforce planning. So choosing this example as I know we're not unique at NRC with this challenge and that many of you may be likely experiencing similar challenges with an uptick in retirements in your organizations. Our Office of the Chief Human Capital Officer applied the risk smart framework to the NRC summer hire program to mitigate the risk of missing the best candidates by making selections too late in the year. By utilizing the using the be risk smart framework they decided to shift their recruitment to start earlier and to use a tiger team approach of supervisors across NRC to speed the process and to foster greater collaboration. Those changes have improved the timeliness quality and diversity of that important entry level feeder pool for the agency. The Office of Research was able to leverage that summer hire program, along with other hiring innovations in FY 2021 to fully utilize their staffing levels going into FY 22. Specifically, they applied the be risk smart framework to address this risk and implemented proactive strategies including aggressive use of summer hires and conversions of those summer hires to co-op positions. External hiring at lower grades and critical skill areas, including strategic over hires and they leveraged the university grants list. This chart shows the staffing utilization over three years. The agency is in orange and research is in blue. You can see that the Office of Research achieved 100% utilization going into FY 22. The lessons they learned will be very important as the agency works to address this enterprise risk this year across all offices consistent with the EDOs vision. Next slide. I'm going to pause for a second and ask you to complete our next poll. So again, on the right hand side, please click poll and you should see a question. Have you seen impacts from NRC using the risk smart. So I will take a moment to let you fill that out and we will share the results. Okay, it looks like the data is starting to stabilize. Thank you very much for your feedback. We have about 43% of you guys saying that I have not seen any impacts. I mean, 34% where you've seen impacts and they specifically impacted you. And yes, you've seen impacts 22%, but they haven't specifically impacted you. All right, well, that's very helpful. We appreciate the feedback as we continue to improve the roll out of the be risk smart throughout the agency. Going back to my slide. So I just wanted to conclude by saying that these examples demonstrate that we shared with you the impact that we had so far of applying the risk smart across all parts of NRC, programmatic, legal and corporate. On the right of the panel, the application of the risk smart has had multiple impacts, including improving confidence and making decisions in the face of uncertainties, helping to achieve early identification and alignment on potential risks, enhancing decision making and risk communications in the region, helping staff assess and select the best option, and also advancing strategic workforce planning. Next, we have a series of resources for you to use. If you're interested off the NRC homepage, you just click on transformation on the left hand side of NRC's homepage and then click on risk. And you'll see the be risk smart guidance, training video on be risk smart more case studies, and also a link to our transformation survey for NRC's external stakeholders that includes that risk informed decision making questions. We'd like to acknowledge Mirella Gravellis, PhD and director of the Office of nuclear security and incident response for doing an amazing job leading the be risk smart team. And now we're going to turn it to questions. So to get the panel started, we wanted to ask the question, whether or not you've seen be risk smart make a difference if you have any data to support whether or not you believe that it has. So read I wanted to start off with you and then I can share some information as well. Sure. So, internally, we have data from some polling that we've done show for the staff, showing a noticeable positive shift in attitudes toward risk informed decision making over the last few years. We've also conducted surveys as part of be risk smart to better understand who's driving risk informed decision making at the NRC. And most of the staff that we've pulled think that both management and individual individual contributors are about equally involved in driving forward risk informed decision making. So, we think that that's been that at least partially a result of the implementation of be risk smart. And I would say, sort of more anecdotally, we're having more conversations about risk, and how to manage risk and I would say also, in large part thanks to the pandemic people are starting to have more conversations about what risks there are and how to manage them. And I think the risk mark has given staff a common sense framework to think about risk, as these conversations are occurring. That's just for internal. You wanted to tackle external then. Yeah, unless any other panel members wanted to add to what read it said before I do that. Okay, so we do have a goal of at least 75% of our external stakeholders that we're currently surveying in this this year, agreeing to the fact that risk information and data has improved the timeliness and quality of agency decision making. And while we don't have a lot of people responding right yet, and we're still collecting data, the ones who have responded so far, 50% of the respondents agree that the use of risk information is approved. The quality of decision making over the past year, and that 41% agree that the use of risk information and data has improved the timeliness of decision making. So, I do think that we are making progress, but there's more work to be done. And I think that somewhat correlates with what we were seeing internally read that you were sharing. Okay, so let's see, I think we have another question here. And one is to read and to Billy is be risk smart or risk informed decision making at the point where we're able to say PRAs are worth it. I can go first. So, you know, we, there are a lot of licensees who have an RC reviewed PR is I want to say it's probably around around half of them. And be risk smart is is one of the tools that we have that can incorporate that into our decision making process. There are other ones, like the risk informed process for evaluations or the very low safety significant issue resolution effort, and the other lots of other risk informed decision making processes that are all, I would say more or less consistent with be risk smart. As to whether they're worth it or not I think that is something that individual life needs will have to figure out for themselves but you know we are trying to do more to be able to take advantage of that. And the risk that we get from PRA is an other risk analysis tool and incorporate those better into our decision making processes because we think that that's important data that we can use to make better decisions and and make sure that we're focusing on the most significant things. Thank you read. And from my perspective as a manager of inspection inspection group. I believe that the PRA is worth it. I mean it allows our inspectors to focus on the issues that matter. When we're doing an assessment of issues of the PRA results help us key in on those those those risk important night risk important issues that we need to key in on to be an effective regulator so yes I do believe the PRA is worth it. We have another question. This is to read and I think Mirabelle you said that you would be able to add in after read answers as well. What role does considering benefit play in your be risk smart decision making. I think that benefits are there to make sure that you are. When when we were originally coming up with the framework, we wanted to make sure that you would acknowledge that there might be a benefit that you would miss out on if you chose a decision one way versus the other. So for instance, let me think of a good example. If you if you decided to do something tackle a problem one way there might be a benefit that you would miss out on that could be gained from another way of approaching it. So that was one of the reasons we wanted to include benefits. So we wanted to make sure that we were really factoring in all of the available information you know benefits are benefits and risks kind of go hand in hand. Yeah I guess that's what I would say to that. What I wanted to highlight about benefits is that when you look at the risk triplet there's the opportunity to consider what can go wrong and what can go right and what can go right as an opportunity there's a yielded benefit potentially from one decision making to another. I think I highlighted in the example of the coven 19 public health emergency when we were considering whether or not to conduct inspections remotely. We came up with our decision to do a remote monitoring, and there was the added benefit and another area to be able to use technology, observe the inspection in multiple ways. There were also activities that were occurring at the same time. And we actually ended up with I would say greater risk informed decisions so benefit in the early stages of spotting what can go right and identifying an opportunity allows you to make the comprehensive risk informed decision. So we have another question. I think this is for Mirabelle and Stacy is the agency thinking about extending the very low safety significance issue resolution process to other business lines and activities. I'll take the question first. So with respect to very low safety significance issue resolution. NMSS has created a working group. One was set up in 2020 to consider whether we could apply NRRs be lists or concept in NMSS. And actually the working group created a digital exhibit that's available at this Rick. And to this digital exhibit which provides additional information and resources you can look to online. There, the working group created screening guidance to be incorporated in the technical assistance review process to aid staff and identifying issues that might be candidates for the application of the lesser. This is mostly working with supporting staff that it's in the process of updating inspection manual chapters to see how the lesser principles can be consistently incorporated into these revisions of the chapters. The working group is also supporting engagement with industry. Recently a topic on the issue of short short term operations that is to seize was identified by general license holders as maybe a potentially appropriate topic. So the lesser principles have been pulled in to those public meetings for consideration. I mean something that's currently underway so I can't speak to it too much. But we're soliciting from industry, you know potential topics where the lesser can be applied. The working group will be looking to create a public facing web page after the Rick to be able to provide updates on the application of the lesser on fuel cycle decommissioning and as the seas and material users. So, we welcome that kind of feedback after the Rick. I'll just add a little bit for security here so the lesser is part of the process we use when we determine how to process an issue of concern, and that includes security concerns. So when any region comes to us they have a security issue. We have a security forum or input is received from all staffs and regions kind of like a little power to get everyone's opinion make sure being consistent with the application of our regulations. And then this we include assessing the significance of security performance deficiencies. So we have been using I think it came out in 2019 or beginning to see staff question and more and more as time goes on. Alright, so we have another question for the entire panel. How do you consider multiple risks while making a decision. Who wants to take that first. Go ahead and take a first crack at it. So, I'm speaking for myself personally and I think we talk about this in the be risk smart guidance, especially because this is a tool that is commonly used in data science and engineering. A heat map is a really useful tool for being able to compare multiple risks. Whether they're quantitative or qualitative you know here, if they're quantitative you can have likelihood and consequences on the X and Y axes if they're, if they're qualitative you can have likelihood and consequences on the X and Y axes but instead of numbers you've got low, medium, high and being able to put those together on a single chart and sort of consider everything together all at once is really helpful. And it also helps, and I kind of alluded to this when I was walking through the framework. It helps you understand, you know, maybe you draw a line where your risk appetite is in terms of consequences and probability so maybe you, maybe you really don't like high consequence low probability event so you know, keep those outside the envelope of where you're okay with the risk appetite and then you can sort of factor in multiple different risks all at the same time. I'll go ahead and add to what Reid just mentioned. The, the issue of risk appetite is really important there. When we rolled the idea of be risk smart to NMSS we started with discussion about risk appetite and what that looks like when you're weighing multiple risks in a decision. There's a conversation going, especially for qualitative data, what methodology is going to be deployed to weigh risks and managing those risks. So, NMSS benefited from learning how NRR conducts the heat map and allowed us to take an approach to apply qualitative data in a heat map. What's your ability, did you want to anything. Okay, that's something real quick. I agree the heat map is very useful for what we're talking about here. Sometimes, personally when I use the be risk smart framework, and you get into an area that's kind of in depth they're very technical and you find out if just going everywhere there's so much in the NMSS. So, what I've seen a couple times is, there are certain ways to look at the heat map and look at the appetite for it, but you might also determine that this is an opportunity to have a different be risk smart and ask a different question that could be more broadly applicable to your program and I think that's helped with a variety of cases. And I actually believe that tracking and trending of all of the risk factors, understanding the brackets in which you have risk tolerance. All is all important, all important to understanding all of the impacts of the different risk and we try to do that, implementing the be risk smart process itself. So this questions for read sorry read seems like a lot are for you. The risk informed process for evaluations right is and how it's been applied to date. Sure, so right is the is the risk informed process for evaluations that you as you said, and it's a process for evaluating low safety significance issues that are known to be within the plant licensing basis. It's possible to resolve those kinds of issues for plants that are already known to have technically acceptable pr a so once that the R.C. has has looked at typically through one of the tech spec travelers. And who have also established an integrated decision making panel and there's guidance out there for how to do that. And so, if a licensee has established a technically acceptable pr a and an IDP and the licensee can characterize whatever their safety issue is as low safety significance. So they can request some kind of licensing activity by the by the NRC or that can be reviewed by the NRC using a streamlined process. But currently it's not available to make changes to plant technical specifications but there's a revision in the works that is working on that that we're hoping to roll out later this year. And we got our first application late last year or early this year and accepted it. So this questions for the entire panel. What is the difference between be risk smart and risk informed. So be risk smart is a risk informed approach and it kind of lies between risk based and deterministic approaches. So risk informing ensures that risk insights are considered together with other factors to establish requirements can measure whether important public health and safety. So what we're really saying here is we'll look at at risk but it's being looked at in conjunction with safety margin defense and depth performance monitoring compliance with regulation stuff like that. So this is one of the most explicit places where we assess these areas to make the most informed decisions. So I guess the bottom line to the question is your smart is a risk informed approach. Did anybody want to add anything. So the next question and this is for the entire panel, whoever feels comfortable, what is the risk metric used for risk metrics used when risk informing the NRC decisions and processes. Is there anything other than core damage frequency and large early frequency or is it more qualitative with respect to the risk triples. Oh, I can, again, I guess take a stab at that. So, fundamentally, I mean for for reactor safety questions. Yeah, the metric is core damage frequency and large early release frequency. When you have multiple options for decision making that all are at least good enough. There are other metrics that can come into play. You start talking about reputational risk you start talking about which which might incur the greater legal risk. And those are obviously more qualitative. I hope that answers the question. It's sort of a combination, you might have both quantitative and qualitative that you need to bring together and be risk smart is a great way to accomplish that. Anybody want to add anything else. Yeah, I think read what you were saying was in reference to the heat map you talked about earlier and be risk smart facilitating an early discussion about the methodology for identifying the risks and then giving weight to them, which is particularly helpful in a qualitative setting. And question for Billy and the rest of the panel is be risk smart use to cut resources. Not in my experience. In fact, what be risk smart has been used for in the regions or to help gain efficiencies and with the current research staff that we have. We've experienced it's we've experienced some some losses due to retirement and retention and we've used be risk smart to deal with to do with those. Those resource issues, but in my experience to answer a question directly no, I haven't seen be risk smart use to cut, cut staff. I'd like to weigh into that a little bit so. You know, sort of from a broader standpoint we've tried to structure be risk smart to be neutral. It doesn't really say you should, you know, use less resources. If it's supposed to tell you that you should be using the amount of effort that's commensurate with the level of risk associated with something right so what could be agnostic to whether something is going to consume more or less resources and you know we do see examples and I think several of those were highlighted here by the panel earlier today. We see examples where people have used more resources or at least not the least amount of resources and resolving something that they evaluated with the risk smart. We also align the resources that we're using with the needs and can support increasing the resources to areas that are more risk significant. And sort of more, more broadly, also I think we need to. Make sure that we're not just using it to cut resources we as an agency and this is my opinion, need to be using the risk smart and more decisions and be more disciplined and applying it, not only in areas where we want to drive efficiencies we want to be able to say, we're we're focusing the resources where they need to be and so that means we need to be using risk in more of our decision. Does anybody else want to add anything. I'm just having a conversation with my office director yesterday about this and kind of aligns with you know are we seeing results from be risk smart in our area. And an example, similar to what Billy mentioned was the decommissioning reactors oversight program us be risk smart when reviewing the program and it didn't yield the decrease or increase of resources that simply minimize them based on the risk areas associated with the program and allow the inspectors to sharpen their focus areas when they go in. So we're seeing, you know, a benefit there and we're seeing the risk smart realize with positive results, and not necessarily with the eye towards you know changing resources in the process. So just a reminder, please continue to put in your questions into the chat and we will try to answer them put them in the queue. So I have another question. This one's for Billy. Is there alignment between be risk smart and the use of probability probabilistic risk assessment. Would you say that be risk smart positively or negatively impacted the NRC's use of PRA. So, I'm very confident that the be risk mark has not harm the PRE process. We use the be we use insights game from the PRS PRA process to apply to the be risk mark process. So we're using them in conjunction with each other. It's very important vehicle to risk informed decision making during the development of the framework. The agency focused on not making on on making it consistent not making it inconsistent with pre existing processes and again I'm talking about the be risk smart process. We wanted to make sure that they were not inconsistent with processes like the PRA. I have another question. Does the risk informed decision making process allow to say how safe is safe enough. Who would like to take that. I can take a stab at it. So, I mean fundamentally the standard that we hold to for safe enough is reasonable assurance of adequate protection. That's defined through, you know, compliance to the regulations and a bunch of different other things and going back to the answer that I gave before about, you know, what the risk metrics are there. You know, specific risk metrics for core damage frequency and larger relief for for reactor sites, but those don't really apply across the board. I wouldn't say that rhythm itself defines how safe is safe enough there are. There are lots of different factors that play into that. I don't know if anybody else wants to add to that. I agree with what you said. And I just wanted to add it really is a risk appetite and the specific question that you're asking. So sometimes you may be willing to be a little more risky in an area maybe if there's a little less defense and depth because as we said this is a risk informed process right. So we're looking at risk and we're also looking at other deterministic factors that's all going to play into the decision you're going to make and we look at those. So we have another question, and I think Mirabel this might be one for you but other panel members are certainly welcome to add to it. How, how does be risk smart fit into the vision of energy being a modern risk informed regulator what does that really mean. Sure, so I think be a risk smart in terms of a timeline was really born out of the idea for that, you know, buzz term of modern risk informed regulator. I remember shortly after hearing management talk about modern risk informed regulator. There was that push to create a framework where we could apply a risk informed approach to decision making. So really it's chicken egg sort of the question. Where be risk smart fits in as it enables us to answer those questions about, you know, this, where we now live in a digital era where we didn't foresee sharing information through certain mediums or receiving it a certain way. We didn't have insights about what the nuclear landscape look like 20 years ago when regulations were written, and we, you know, advanced reactors are accident tolerant fuel these are developments that are, you know, we're living through them and we need ways to assess them for licensing and be risk smart allows us to do that because it allows new information to be considered alongside the traditional risks that we've assessed and make a risk informed decision to ensure that we're meeting our mission. Does anybody want to add anything. Okay. We do have another question and this is a really important question. How would the NRC know that it's transformation efforts on the use of risk and decision making is reaching all levels of staff. I know that many of you have served as ambassadors for the be risk smart and also part of the core team. You know, we've gotten, you know, some survey results and, you know, the vision the ultimate vision, the, you know, ideal state would be having everybody across NRC factoring and risk information in a common way across the enterprise, so that we could have meaningful discussions around risk and make the best decisions in the most timely decisions so what would you say on that. Sorry, where you have to go Stacey. I'll go after you I'm sure yours will be a great answer and I'll just come to you. Well I was just going to say, you know, take a look at the cross section of the staff that we've got here today, we've got people from all different offices we've got. We've got, you know, line supervisors, staff, senior manager, all represented here. I think it is happening. We are making, we are reaching everybody at the NRC and across the business lines and extending to, you know, not the, not the technical areas as well, you know, into the corporate and legal areas too. So I think we're doing a great job of that. And, yeah, so. I was just going to add. So, people are resistant to change and anything new comes out and I remember when this first got rolled out people are like oh another risk informed initiative. And it was kind of like that cooler talk no one wanted to actually use the framework. That's how it started that's how a lot of things started when people are seeing something now. Now when I go to meetings or we have teams where we can chat privately with other employees. So more often than not, than not people are even bringing up in those discussions. Oh I use the framework for this. What do you think of this heat map and it's becoming part of the normal conversation when a big decision is made. Management. I'm an answer some under morale into the big support of the be your smart frameworks will be use it all the time and management and answer but all levels down I can't think of a meeting that I've been to, at least in the past couple months where the be your smart framework wasn't otherwise. Not tell you from my perspective, many of the examples that I studied and even presented today, we're born out of the suggestions for staff. And during the application of the response in their presentation to myself and to management. And it makes me believe that the staff has embraced the process, and they're using it for their benefit, you know, just last week I listened to a be with a discussion on be risk mark from the inspectors and health physics, health physics branch to talk about the part 37 and documentation and it really exemplified the staff's use and acceptance of the be risk mark process so yes I've seen it and I think it's taken hold at all levels of other staff. So then I'll run down with the cynical voice that hopefully has a little positive ending at the end of it. As you mentioned in my bio I came to the working group with some skepticism about application to be risk smart specifically to my working area. And that's something I had really heard throughout NMSS. Without the use of a PRA how are we going to deploy this framework, you know for corporate for technical for legal decisions where, how do you consider risk in these different disciplines. So that was a challenge for me coming into it. And I think it's through that practice approach of be risk smart and teaching and talking to other staff that it's sort of become second nature for me. At the same time, I think there's still a lot that can be done, especially by the working group to continue to reach staff at all levels. I'll get a little bit into the weeds on what we do in NMSS and hopefully it sparks some feedback that we can consider. But in our office, we have a working group that is constantly talking to staff and collecting examples of be risk smart wins. And we really are emphasizing the teaching element of be risk smart. So we're making sure it gets into nucleopedia, which is a resource for teaching examples and lessons learned on various high, you know, high profile topics. We created a SharePoint where we have been hosting pop up seminars, where staff talk about their use of be risk smart in a scenario. And then we put together recordings and put them available on our SharePoint so staff can go and watch them and hopefully identify with the area that's being covered. You know, we've covered corporate examples, things about contract renewal. We've talked about engagement with stakeholders. We've talked about safety analysis. We're trying to cover all areas because especially in NMSS where there are multiple business lines and multiple disciplines, we want to make sure that we're reaching all those levels. And so it's a big part of our progress right now for us. Reid, did you want to add something. Yeah. So, you know, I when I joined the agency, however many years ago it was, my background is in deterministic safety analysis work and you know I wanted to kind of echo what Mirbell said though my my cynicism and reluctance, I would say predate even the the advent of risk smart. And, you know, when I joined the agency, I was sort of like, Oh, well, PRA is the risk is the thing for those PRA people to do. And I don't need to worry about it I'll focus on my deterministic safety analysis. And I think that working with those people who are in our division of risk assessment and sort of opening by mind to the possibilities of what you could consider with risk and understanding how it's used in the, in the inspection program, and elsewhere in the agency really helps me see its value and utility in making decisions. And I think that a lot of people are starting to come right even in, you know, the staunchest parts of the agency where people would have that opinion of well that's something for somebody else to worry about. And I think that they're coming around to that, because they're actually seeing it make a difference. So I'll just add that from a corporate perspective we have enterprise risk management, which is part of our internal controls and strategic plan and performance management system, and the senior leaders quarterly talk about enterprise risks. We don't just want senior management driving risk discussion we want to have discussion happen everywhere within the agency. So that, you know, people who are best positioned to identify a risk or managing that risk and then engaging with the senior leadership as appropriate. But you know, I mean I can reflect on some of those discussions and we've managed so many risks I can't even, you know, I don't know how many but I mean we have a whole very systematic process where quarterly we look at what are the major risks before the agency and I think, you know anybody who's been sitting listening to the commissioners and the EDO and chairman has heard over and over this hiring risk because I think when you looked back and saw the data, we hadn't been hiring enough externally, we've been moving people more internally and that's why we're shifting that focus in FY, in FY this fiscal year. We're really trying to shift the agency's focus on bringing in more people externally while still allowing people to, you know, grow, we definitely want our internal staff to have opportunities to get promoted as well. So that's that's a something that is an example of an enterprise risk and a lot of that was driven by some corporate concerns, and now it's become like an agency focus. So it's actually really exciting to see we actually put together a steering group and a corporate workgroup to really focus on it and once we put the workgroup together. We identified that there were multiple risks, there's the onboarding risk, I mean we can't just hire people we have to onboard them well. And so we really want to retain people we want to onboard them, I mean we just want the whole process to be to be addressed. So anyway, a lot of that was born out of the thinking about risks. So, read a question for you. Can you talk more specifically about the mission analytics portal. How is it helping to improve decisions. I'm happy to tackle that I'm not sure it's a serious smart question, but it's a spot piece so if you could connect. Yeah, right I will use of data and decision making and that's definitely part of it so. You know, I think that one of the big things that that map is doing at least in NRR is helping make information just readily accessible to make decisions so we used to have a meeting where we would go through all of the projects that we're working on, and and talk about you know which ones might be at risk of blowing through metrics and whether those needed more management attention on them to make sure that they were getting done, and we were going to meet the metric. That meeting has gotten streamlined and condensed thanks to the use of dashboards and data to where like it basically doesn't even need to happen anymore. So that's a risk that we're managing, or we're managing risks on each one of those individual projects. By being able to see exactly where things are in the process at any given point in time. So that's, that's one example. I'm sure there are others and other business lines. I know that we recently just implemented a change or a new set of dashboards to help facilitate the end of cycle meetings in the regions for their inspection cycle. I think though they're starting to make a difference there. So the next questions for the entire panel, do you have any indication how many people are using this framework. So I'll start off with, in terms of data and a specific number. I couldn't ballpark that for you but what I can say is that for NMSS to get an idea. We have one of our objectives and key results being use of the be risk smart framework. In, with that respect, we wanted to, you know, encourage staff to use be risk smart in their decision making and document it so that there is that opportunity to teach down the line but also the ability to track that metric. And how we're helping staff is identifying the types of meetings where they can expect that be risk smart will be asked and give them some tips for preparing for it. When we're talking about external stakeholder engagement, you can expect be risk smart will be inquired by the upper levels of management so we have staff thinking about that an earlier point in time. When we're talking about assignment and alignment meetings. That's a great opportunity to use the risk smart to ensure that we're aligning on the risk significant issues that are under consideration so I don't have an exact number what I can say is that it's very it's being promoted to us which areas absolutely should be considering be risk smart and capturing documentation of that in our repository so that we can continue to teach. I will say that right now, our objective and key result is set around. We want to see about 75% of decision making not only is using be risk smart but it's also focusing on that data element kind of what read was alluding to and looking at more ways to incorporate that aspect into the decision making and capture it so that's something we're working towards. If I can add on to a little bit what Mirabelle was just saying so the. Also in NRR have have an okay are related to use of be risk smart in decisions, but I would also say that, you know, not all decisions that people might use concepts from be risk smart are ones that we would track or get a sense of, if people are, you know, talking about very low level things decisions in their own everyday work that that they're trying to decide they they wouldn't document that and pass it along to somebody else or put it into a dashboard someplace. And I think that that's the kind of ground swell of interest in risk informed decision making that we're seeing is kind of at that level. So I think it's hard to objectively measure exactly who's using be risk smart and what senses I mean definitely what what Mirabelle is talking about we are doing that at the at the office level but there are lots of lower level decisions that happen all the time that are a lot harder to track. And hopefully we're making an impact there. And you just reminded me, we are when I we've had conversations with staff we find out they are using be risk smart, they just didn't know that that's the name of it. So part of our effort to engage with staff on be risk smart is to help them realize that they're already using this umbrella framework, and it could be applied at day to day, everyday activities, and just making it feel more organic and part of our regular toolbox. So, here's a good question. How do we make sure that the risk smart does not become just another fat. I think Stacy you wanted to weigh in first and maybe others want to add after that. I can start up with this. So the group that put together this framework that a commendable job setting up a solid framework that really has all the elements that you need to succeed. Not being said could it still be a sad, maybe, but to keep the framework relevant we purposely added that little arrow at the end of the process that allows you to reassess. And I'm sure we are using that process to reassess as needed so we're keeping the pace with regulatory environment transformation. I mean it's all part of the transformation process we have it filled in. So far it's looking good and we're hoping they continue down that path, but only time will tell. Yeah, I'll add to what Stacy said when I was talking to my office director about this the other day. I think it's a good point that, you know, be a risk smart is at a certain level of maturity where we're going to start seeing the benefits of some actions as we go through that realize step now. So, I don't think it's going to be a fad in a sense of disappearing, you know, in a year or two because there are projects that are underway that still need to go through the realize elements to understand you know where we need to a correct course and, you know, go through the iterative process like Stacy said, there are ways that we are ensuring we're ingraining the framework into current guidance. For example I mentioned the the inspection manual chapters that are under review. In many cases the text of those documents are incorporating be risk smart language and terminology, just to help make sure that we're ingraining in the guidance that staff can put their hands on. So the one thing that I would just mention is this is very correlated with enterprise risk management. The whole vision of enterprise risk management is really to engage staff. So I see them all very connected. You know it's just going through a systematic methodology where you're really thinking about these things and it's common sense and it's giving us some language where corporate people like myself can talk to technical people like you, as well as legal people. And I've also, you know, found myself pushing back on legal sometimes when they just say, here's one option I'm like well that can't be the only option. I mean we need to generate other options we need to look at the risks associated with each option. We need to look at the benefits and we also need to look at what problem we're trying to solve. So I'm like constantly saying these things over and over and over and I remember Mike Weber I don't know if you guys remember Mike Weber but he used to say, when you're exhausted of communicating. That's when you're finally reaching people. So I don't see this as a fad because it's just common sense, you know, and I think that it's adding value and I think that that's what will make it thrive and live and anyway, I just I wanted to add that. I wanted to chime in with one more thing and I think this kind of goes back to what Mirabelle and I said on the last question. I think where we get to not fad status is people are using it, they're comfortable with it it becomes part of their regular everyday work. And then it's not a fad because it's just something that you do. And it doesn't go away because it's just part of the organizational culture. And it's something that you know you learn when you come to the nrc is this is something that you do. That's, that's where I think real successes and and, you know, not being a fad. And I would say, we've done, like I like I mentioned in my talk we did some polling internally to talk about who's driving risk informed decision making. And right now we think it's equal, you know, we, we want to see that continue over time to be sure that we're being successful. So just give you an opportunity for any final thoughts before I turn to closing remarks for almost out of time. Billy did you want to make a closing statement of any kind. No, as I stated, well, I'll just mention as I stated during the presentation. And we've talked about this, we've talked about this through this throughout this question and answer session. I believe that the staff is intimately engaged in the be very smart process. You know, as, as I've managed daily, I constantly have staff presenting issues to me and they're presented them in the be very smart framework. And I think from that perspective it's been a it's been a real success. I've seen decisions that have created effectiveness and efficiency that I think are have been invaluable over the last two years or so. So, again, I believe that be very smart process should is a process that should be incorporated at all levels of agency and in ways found to incorporate it in all of our decision making processes so. Anyone else would like to make a final statement. Okay, well you guys are just an amazing panel. That's pretty much all the time that we have. This has been a wonderful discussion. And I would like to call up this email on this slide, you can send any remaining comments or questions that you have that did not get answered for you. And we can ensure follow up. I want to thank the panel members for their wonderful presentations. I also want to thank Katie McCurry, Elizabeth Bowen and Morelle Gavrilis and other be risk smart team members for the critical roles that they played in making this panel a success. I just want to thank the audience for your engagement and your fantastic questions, and I do want to remind you about the stakeholder survey on transformation please take that if you haven't already. That concludes our be risk smart panel and thank you for attending.