 There are QR codes provided both for the Q&A portion of the session and for our live polling questions. For those of you in the session room, please take a moment to scan the QR code displayed on the screen to your device so you can participate in the Q&A and the live polling during this session. For those of you joining virtually, you will find there is a box to the right of your screen with tabs for electronic Q&A, live polling, and feedback. We are starting off this morning with two live polling questions to reflect on the past year and how effectively we have leveraged risk-informed decision-making, both at the NRC and in your organizations. Please go to the polls tab on the virtual conference platform, or I guess we need to bring up the first two live polling questions? Perfect. So please go to the polls tab on the virtual conference platform or text the code shown on the screen. We'll look at those results in just a few minutes. If you did not already see it, I recommend viewing our BeRisk Smart in-rand video playing out in the four-year area. It illustrates the BeRisk Smart steps highlighting its usefulness as an intuitive tool that the agency is using for decision-making across the breadth of NRC's activities. I want to give a shout-out to the creative team, Andres Rowe, Zachary Helgart, and DeLon Atkinson, and also to those that supported its development, including Jeffrey Miller, Mirabelle Shoemaker, and the NMSS BeRisk Smart team, as well as a thanks to the RIC team, Lorna Kipfer, and the RIC AV team. I am ceaselessly impressed with the creativity and innovation our new folks bring to the agency. BeRisk Smart is a vital tool for the staff to integrate risk insights into NRC decisions and the agency culture. We could go back to the title slide. I am honored to be chairing today's session. Over the past two days, the importance of risk-informed decision-making has been a key topic in a number of sessions. It was featured during the chairs and the EDO plenary presentations, and was a topic of several questions during the regional session yesterday afternoon. As you can see from the agenda, we will have brief presentations from the panelists followed by a roundtable panel discussion, and then transition to the Q&As. Let me introduce our distinguished panel members, and then we'll check in on the results from the live polling questions. The first speaker is Mr. Mike Pranovic. Mr. Pranovic joined the NRC in 1990 as a general engineer in the Office of Nuclear Reactor Regulation, or N-R-R. Since joining the NRC, he has held positions of increasing responsibility, including project manager in N-R-R, nuclear engineer in the Office of Nuclear Materials Safety and Safeguards, resident inspector at McGuire Nuclear Station, and senior reliability and risk analyst in N-R-R, Chief Probabilistic Risk Assessment and Operational Support and Maintenance Branch in N-R-R, an Executive Technical Assistant in the Office of the Executive Director for Operations, Assistant for Reactors for Former Commissioner Austin Dorff, and Chief of the Programmatic Oversight Regional Support Branch in NMSS. He was also the Deputy Director of the Japan Lessons-Learned Division in N-R-R, and Deputy Director in the Division of Safety Systems, Risk Assessment and Advanced Reactors in our Office of New Reactors. In January 2018, he assumed his current position as Director in the Division of Risk Assessment in N-R-R. Mr. Franovich received a Bachelor's Degree in Nuclear Engineering from the University of Florida and a Master's Degree in Reliability Engineering from the University of Maryland. He is a graduate of NRC's SES Candidate Development Program. Our second presenter today will be Mr. Doug Tru. Mr. Tru is NEI's Chief Nuclear Officer and a Senior Vice President of Generation and Suppliers. Mr. Tru is a veteran of the nuclear industry with more than 35 years of experience in nuclear regulatory policy and implementation, playing a substantial role in U.S. policies and regulatory activities, ranging from industry response to challenges like Fukushima and 9-11 to major contributions to the formation and implementation of risk-informed regulation. Prior to joining NEI, he served as Executive Vice President of the Power Services Group at Jensen Hughes. In this position, he was responsible for one of the largest specialty engineering organizations in the nuclear industry, covering a broad range of analysis, testing, design, and research consulting services. Mr. Tru previously served on the NEI Board of Directors and has been a contributor to NEI, Electric Power Research Institute, Institute of Nuclear Power Operations, and Federal Government Guidance on a broad spectrum of technical and regulatory issues. Mr. Tru earned a Bachelor's of Science in Chemical Engineering from the University of California Berkeley. Our third presenter is Mr. Cody Schnaufer. Mr. Schnaufer serves as Division Manager, Safety Risk Management Division, the Flight Standards Service at the Federal Aviation Administration. Mr. Schnaufer began his career at the Federal Aviation Administration in 2014 as a flight test engineer in the Aircraft Certification Service responsible for testing and certification of new aircraft models. He transferred to the Flight Standards Service, where he preserved in progressively more responsible positions, including Manager of Evaluation Programs and Manager of Systems Assessments. In his current position, he leads his team responsible for implementing a safety management system within flight standards, developing the Flight Standards Workforce Safety Management System Competency and supporting the identification, mitigation, and monitoring of risk. Before joining the FAA, Mr. Schnaufer began his career in flight tests with the U.S. Navy and recently completed his time as an officer in the U.S. Navy Reserve. He is a graduate of the United States Naval Test Pilot School. He holds dual master's degrees in Aerospace Engineering and Engineering Management from the Florida Institute of Technology and a bachelor's degree in Aerospace Engineering from Emory Riddle Aeronautical University. Our final presenter today will be Mr. Mark Foy. Mr. Foy was appointed Chief Executive and Chief Nuclear Inspector of the United Kingdom's Office for Nuclear Regulation, or ONR, on June 1st of 2021. Having been ONR's Chief Nuclear Inspector since 2017. He is an executive board member of the ONR and is the regulatory head of the organization providing independent and authoritative expert advice on nuclear safety, security, and safeguards to the ONR board, ministers, and parliament. Mr. Foy has 35 years of experience of the U.K.'s civil and defense nuclear industry combined with extensive international experience, having undertaken various senior roles within the international regulatory community. He provides authoritative leadership and insight on nuclear safety and its regulation. Mr. Foy is a chartered fellow of the U.K.'s Nuclear Institute and has an honor's degree in mechanical engineering. I look forward to hearing from each of our panelists. Please join me in a welcoming round of applause for them as we bring up the results of the first live polling. I apologize, I have to look. So, looking back over the past year, it looks like we've made progress but still have some work to do to both fully leverage risk-informed decision-making and to communicate our successes and raise awareness of how risk insights are enhancing the NRC's decision-making process. I'm pleased to see the progress we're making incorporating BeRisk Smart into agency culture and I look forward to the journey continuing. Looking outside of the NRC, could you please bring up the results of the second live polling question? And it looks like the trends are about the same. There is still some road to travel on the journey to create a risk-informed culture broadly. I look forward to hearing from today's panelists about their journey and the lessons learned along the way. With that, let me welcome our first speaker, Mr. Mike Franovich. Thank you, Michelle. Good morning, folks. I hope everyone's had their coffee and tea or whatever your beverage is. Welcome to day three of the RIC. I know it's been a long week but hopefully we have a good conversation today. We are doing a little different style panel, a few formal remarks perhaps from each of the speakers so we will not go into great depth but I do have a few thoughts to share with you all. If I could advance the slides. First of all, the title of our session has the key word in a trust. If we go back and think about trust in decision-making, and it's something the chairman mentioned in his opening remarks on the first day of the RIC, and risk-informed decision-making is sort of the four pillars of a culture and organization that he's striving for. I go back and I think of Stephen Covey's work, the late Stephen Covey on the speed of trust, and Stephen Covey had great insights about collaboration and one of his quotes was, collaboration is the foundation of the standard of living we enjoy today. Trust is the glue. Now that was a statement over 10 years ago, 12 years ago and it would be interesting to see if Stephen Covey, what he would think today, with our challenges before us as a society, but I think those insights are very pertinent to the type of work that we're doing today. And in his approach, one of the key steps toward building informed trust is to have straight talk. And so in our panel discussion today, I think we're going to be striving for that as an outcome. And so I will be a little bit more direct, perhaps, a little less unvarnished. Before I go into details about what we've done over the last year, I do need to set the stage a little bit from an NRC perspective. Some of you are familiar with this and some of you are not. So those of you that are new, I want to spend a few minutes about things that happened about four to five years ago that shape what we're doing today. There were several shaping factors affecting what we do today. The two in particular that were quite specific. One was targeted toward the Office of Nuclear Ractor Regulation about 2017 to come up with what's called a Ritim Action Plan, an action directed by the Commission. And NRR at that time was an office focused on the operating fleet. And so we did develop an action plan to try and build upon the successes that we had in the past, but also deal with the challenges to bake in Ritim into our culture. We did accomplish that work. One of the key outcomes was to do more integrated work as a team. Integrated review teams was a key take away and to institutionalize some of our practices and more durable guidance. But that said, at a broader level, agency level, there was a transformation effort also going on. And there were many directives and a commitment by our leadership to do more in the way of becoming a modern risk informed regulator. And Ritim and risk concepts were baked in there. One of the outcomes of that effort was the B-Risk Smart effort. And I need to give acknowledgement to Marilla Gavrillas, who I don't believe is in the room, but online today, about all her work to develop with the outstanding team the practices that we use today in the B-Risk Smart framework. But that framework crosscuts more than nuclear safety. It goes into the things that Andrea Vail spoke to yesterday during the EDO plenary about other areas where we can use risk. So these two actions going on the agency are driving some of the things that you see today. And so what I'm going to share is a few successes in particular. I wanted to set the stage. So first, B-Risk Smart, of course, I've mentioned that. I'm encouraged we have shared this activity and the guidance of B-Risk Smart with our stakeholders. And I've actually seen some of our stakeholders use it in the dialogue with the agency. And so it makes a more productive dialogue when we're using a similar framework. Digital INC, there's been a long road to the treatment of common cause failures and much progress has been made in the deterministic area. However, there was a huge breakthrough over a year ago when the commission revised the policy to allow for alternative path using risk insights and the risk approach. We're making progress in that area with Branch Technical Position 719 that is now integrating the thoughts of our risk framework that we use in Regulatory God 1174. And so we're on track to deliver revision nine of that branch technical position. The commission had some urgency and they said get it done in a year and we are delivering. Turning the topic a little bit to the new reactor fleet. We have been using risk insights a lot for the light water SMR effort but could we do a better job? And so we did have an opportunity with the new scale US 460 standard design application that came in to apply risk insights on the front end of the review but target it more to shape the level of review and our resource estimates. So we did that very early on and that's an ongoing dialogue because there were refinements in that design to hear the insights from the vendor about the trade offs and how they were leveraging risk in refining that design to help make it more deployable and to improve its safety posture. Accident tolerant fuel increased enrichment rulemaking. There was a wonderful session on this yesterday I won't go into detail about it but I can tell you that some areas that have had more traditional defense and depth look such as the treatment control room operator standards or dose and emergency response we are taking a very hard look at using more risk informed approaches in that area. And so we've had three productive workshops actually in exchanges with the industry to see in a lot of brainstorming how we can bring these concepts together just like we did with the branch technical position 719 and digital INC. There's some similarities in thinking there. Risk informed process for evaluations the right process there's an example of how we're trying to capitalize on other programs that we've already approved for licensees or processes like integrated decision making panels that licensees use. This is to drive efficiency in our process and I'm pleased to say not only have we approved the first one some time ago but we have several applications in progress in using exercising this process. And lastly I'll note SPAR- the NRC has its own PRA models as you know for all the operating plants we also have some for the new reactor designs in fact our latest SPAR model is for the new scale US-460 design this gives us our independent capability to do some assessments and also not to burden our applicants with some questions that we can handle internally but we created a dashboard to help promulgate or put out more information to the non-PRA practitioners that's a huge audience for us where we can actually help enable sharing of risk insights so those are some of the things I wanted to note we can talk more in the Q&A about it so I mentioned that NRR had a rhythm action plan about five years ago but that was focused on the at that time as an office before we merged with the new reactor office and one of the learnings we had is to integrate better we needed to probably think more about how we can use the and our rhythm plan and the practices that were in the new reactor office and make more decisions in a conversational dialogue way where we're actually talking about the five principles of risk-informed decision making as the NRC defines it at that time we were also living under direction from the commission in SRM 1936 better known as the new scale block valve SRM to actually apply risk principles in our decision making so here is a tool that we're actually piloting on the new scale review activity and what you see on the left side and its objective is really to enhance the dialogue amongst the reviewers on the team to lay out the five principles and actually have folks write down their thoughts where are we on these principles are we okay, are there concerns or are there some red flags that we really need to focus on putting pen to paper or typing out your thoughts is very helpful in backing up the conversations to say where should we focus how can we mitigate some of these issues in our exchanges with the applicant ideally what we could do with this type of approach is roll up the issues and put it into more of a project risk approach where we map out the actual project risks and conformity with the written principles and that's what you see on the right side this is a management tool to help us focus on things that might be in the red zone that need much more attention they may be critical path there may be safety technical issues that need to be resolved and so this is another way to sort of reduce the volume of information to something that's very more useful for the management team and the review team working together and then lastly I'll go to let me go to the next slide so mission excellence and being a modern risk and form regulator it is a marathon a continuous journey and we are committed to increase rhythm in our activities recognize there are places where we still need to do more work I think you see that in the polling results that's not lost upon me the positive change management building on the successes having the difficult conversations about areas that may not be working to the degree that we would like them to work is very important also over time things that you assume are working well or institutionalized may not be working as well as you think so we have an effort called rhythm 2.0 that we're just starting to take a closer look how well are we implementing these integrated review teams some places it's called a core team some places it's called interdisciplinary review team but the bottom line how are we functioning is a team in integrating and using risk insights and the other principles to us it's a conversation it's important to create an environment where we can actually have critical thinking to bring the entire team and views together and to make timely decisions at some point you do have to make your decision and move on but it's important that all our folks and the seat at the table their views are heard and that we learn from each other so we're making better decisions at the end of the day and with that I will turn it over to Doug great thanks Mike and there's certainly a lot going on these days in the risk informed decision making area I want to reflect a little bit on the journey we've been on and then it's kind of where we're heading to bring that into this conversation and so my talk has described this evolving role of risk informed decision making I want to start with the current operating fleet and the progress that we've made over the last frankly 40 years of evaluating risk at the operating fleets this curve I think we first made back in 2005 shows the internal events the average internal events core damage frequency for the operating fleet over from since 1992 when the original PRAs were done to today we recently had APRI update this and for the first time we began to see the influence of flex in the results we got another factor of two decrease in the overall internal events core damage frequency from the implementation of flex flex you would note was really oriented towards protecting against the black swan kind of external hazards but implementation of that and some of the features that were included in that had a direct impact on internal events we don't have a curve for the other hazards we haven't done the studies consistently over that time we're looking at whether we can do something similar for fire because I think there's a very similar trend since we started on working on fire in the early 2000s we brought this into time periods when you go and look at a plant using PRA that was designed using deterministic methods there are a lot of insights that come out those insights at some plants led to relatively high core damage frequencies that were addressed by those companies that led to that early phase of steep production in the average as we moved past the initial IPs we went into the maintenance rule and the sort of early phases of risk informed decision making justifying longer AOTs and things utilities began to look at how do we understand what this means, how does our performance tie into this and we saw another period of downward trend as we got into the early 2000s it began more about operational efficiency the RLP was moving forward a lot of focus on how do I establish more margin in the STP process and provide RLP margin and so we saw another reduction and then most recently the implementation of flex in the mid teens of then years ago or so showed another drop in the risk that drops pretty precipitous there in over a one year period that's because the updates of the PRAs that occurred in that kind of time frame so we see a significant drop rather than a smooth kind of a drop we saw in some of the other time frames but I think the industry, I think the NRC for focusing on risk and having us use this as a metric in our regulatory dialogue really deserve a lot of credit for reducing overall risk by over a factor of 20 using risk informed methods that came about in a process where we had largely deterministic methods that we used to set the design basis for the facilities we brought PRA along, looked at those results and we made adjustments and we called that risk informed as we look forward to the new reactors that are coming we're sort of flipping the roles of those and we're starting with the PRA having a much more prominent role in understanding what the important scenarios are what equipment is going to be important and then adding defense and depth and safety margins in the sort of traditional deterministic considerations on top of that that turns the process around a little bit and puts us into frankly a new era of risk informed decision making the next slide is an eye chart I'm not going to walk through it you can download it if you'd like the main message of this is that we've had 30 years of using PRA for light water reactors we're quite comfortable with that we've seen great strides in improving the safety of our reactors the challenge as we move to the next reactors is that a lot of what we depended upon as sort of the basis for how we make decisions for the current fleet, large light water reactors doesn't totally apply as we go forward the contributors are different the data that underlies it is less deep and we just have a lot of new factors we need to bring into the decision making process so as we turn this corner and move in the direction of the new plants that are coming we need to take that into account as we move forward there's not an argument against moving in this direction it's just we need to do this with our eyes wide open and understanding the strengths and limitations of PRA because that's what's gotten us that reduction of a factor of 20 in internal events risk which is highly notable but we did that with our eyes wide open understanding the tools and what they were telling us so risk informed decision making and then using PRA for the operating fleet has been a really valuable regulatory tool and operational tool for the industry it's helped us focus on safety risk informing improved safety by focusing on what's important to risk you focus on the things that are important to the residual that's left over after you've designed and regulated this plant and that makes the plant safer it also helps us improve efficiency because we're not wasting resources on things that are less valuable we're entering into an era here with the operating fleet where we're taking a lot of the more ambitious applications of 5069 risk informed completion times we're taking PRA to kind of a new level in the plants I think there's a lot of opportunity for the industry and the NRC to become more understanding of those approaches I think we have a lot of training to do on both sides to become comfortable with that and I think there's more opportunity to see that focus on what's important lead us to greater reductions in risk and improvements in safety as we, what we know today though it will change as we move to advanced reactors and I think it's important that we move into that phase with an understanding of the strengths of PRA which are many it's a very valuable tool but also it's limitations and as we enter into this era and I don't let our technical hubris for thinking we know more than we know influence where we end up because we need to use this tool within its limits and make sure that we apply our risk informed overall risk informed thinking process that Mike laid out five principles in a consistent manner with that I'll turn it over to Cody Okay thanks Doug, good morning everyone my slide is a little bit less technical a little bit more perspective in context I might be asking yourself why is somebody from the FAA here so completely different industries but the framework and the thought process is exactly the same in fact as we look to grow our risk informed capabilities both in internally to the agency and in the industry we oversee we often look at the nuclear industry the risk tolerance of the general public is very similar the ability to keep pace with the growing industry is another challenge that both the NRC and the FAA have in common and there's a lot of lessons learned we can get from each other so just a couple thoughts for context I am a member of the flight standard service in the FAA that's relevant in the sense that we are the second largest component of the FAA outside of the air traffic controllers so they're about half of the FAA you take them out where the next big is part of the organization we are in the aviation safety world we are responsible for oversight of air carriers, airmen mechanics, all of those things so we have embraced as I say on the slide there a risk informed decision making when I say embrace what I really mean is we are scaling up how we do this risk informed thought process has always been a part of aviation it goes back to kind of the earliest days of aviation training if somebody is going out to get their private pilot license right now the first thing they'll do with their instructors they'll walk around the airplane look at everything examine the various parts they'll look at the weather they'll plan their flight and then they'll make a decision based on those factors on do we fly do we not fly so that's something inherent in our industry and the way we operate and now it's really about taking that and being able to scale that up to the larger organizational enterprise type issues I already mentioned the growing industry but if you have even a passing knowledge of the changes going on the aviation industry they're huge right we've got commercial space flight we've got urban air mobility is a big growth area drones unmanned systems have been for the last 10 years or so a booming industry with a lot of value and a lot of ability to actually do things safer right the idea in the drone world of taking a drone having that fly and a grid pattern over a field is inherently safer than having somebody in a crop duster they're able to do it lower more control but it's different and that's the part that as a regulator we're trying to keep pace with understanding those changes and like Doug said it's really about being able to apply our resources to those areas of highest risk just a couple other notes it's an administrator level strategic objective to we call it risk based decision making I actually like the term risk informed a little bit better I think it better encapsulates what we're really after we have looked into our rulemaking our certification both to apply to the areas of highest risk but also look at how do we do our regulations with a changing industry and what's the best way to do that it's looking at things like performance based regulations understanding that it's about attaining a certain level of safety rather than very prescriptive in how you attain that and then our oversight so there's a couple things that I should have spelled out SAS, RPAT, BSRP and ASAP so what those are just various programs we have as an agency to collect data and to use that data to make those risk informed so SAS is our safety assurance system we look across all the operators from the big carriers that you're familiar with down to the little mom and pop crop dusting shops collect that information and look for those systemic trends of what's working well and what isn't RPAT is just another tool that's a risk based ability to target our resources so we have this data about an operator we as a government agency have a high night number of folks so how do we best apply those resources to make sure we're looking at those areas of highest risk while also keeping an eye on those smaller operators those might not be the ones that jump out but they operate a lot more and it's a different level of risk we actually talk about as a safety continuum so at one end you have the commercial flights the one I flew here yesterday so right that world is a huge part that's what the public really looks at but how do we scale that safety and how do we scale our regulations down to you know that 16 17 year old student pilot who is about to do their first solo flight and then the last two those are big I don't have a fancy chart like that but big game changes in the aviation industry are actually voluntary reporting one of the biggest drivers we saw coming out of the 1970s and 1980s was by taking a less adversarial oversight approach and a more open non reprisal type approach we're able to get more information of things that weren't going well right times when someone didn't follow a procedure or times when things didn't always go according to plan previously in the industry that was the kind of thing you kept very quiet you didn't share we have made a hundred and eighty degree turn with our industry we are very open most of the information that is reported it'll be available to the general public NASA does a great job of putting out the information on aviation safety incidents and we've really re-oriented that to let's share the information as a regulator we're not going to see everything so let's work together to identify those areas of highest risk and with that I will turn it over to Mark thank you Cody good morning everyone it's a privilege to be here this morning to share with you our journey on risk informed targeted regulation in the United Kingdom as the UK's nuclear regulator all in all has considered itself a risk informed regulator for many years actually over many decades so it's not new to us operating within a performance based goal setting regime allows us to regulate on a sampling basis which we do seek to undertake in a targeted and proportionate way in relation to risk our legal and regulatory framework is clear that license season duty holders are the ones that are required to manage and control the risk that their activities actually give rise to over the last few years stakeholder feedback and internal assurance reviews within our own organisation have highlighted potential challenges with our application of a risk informed approach it's been revealing insight for us hence we've embarked on right our risk informed targeted engagement initiative just over a year ago the range of considerations associated with the risk informed approach and for the UK this includes the health and safety law within UK requires risk to be reduced so far as it's reasonably practicable consequently risk has always been a factor in O&R and in its decision making but we've not been consistent in how we deploy our resources and focus our regulatory attention a key factor also is that the level of knowledge and experience of inspectors within the office for nuclear regulation has changed over years our demographic is changing and our inspectorate now has less experience in addition we need to look at the environment a resource known needs to be deployed across the existing facilities some of which are seeking lifetime extensions but we're now also having new areas that we've got to regulate in an increasingly ambitious nuclear sector so that's a broader portfolio for our regulation which is reflecting the UK's strategic priorities our right approach will allow us to prioritize consistently across our safety security, safeguards and transport regulatory purposes but also enable individual inspectors to properly target their attention and allocation of time meeting the demands of the sector but regulating effectively right will ensure we target effort primary on those activities that give rise to the more serious risks whether hazards or the vulnerabilities are least well controlled or where ongoing compliance with the law needs to be established it also considers how we behave as a regulator how we interact and collaborate with each other within the organization so that our overall approach and decisions across each one of our regulatory purposes or areas of focus is consistently risk informed key is deploying our highly skilled and experienced integrated regulatory staff in an efficient manner importantly our inspectors as part of the right initiative will be supported throughout with effective coaching and tools to help them use risk insights effectively when deciding how we utilize our resource right requires anything that we will do as a regulatory body to be anchored in one or more of three underpinning pillars of risk informed regulation regardless of whether it's about nuclear safety, security or safeguards transport or indeed industrial safety which we regulate those are what we term of five regulatory purpose and they've been considered they've been considered separately in the organization up until now the three pillars are in relation to the potential for harm secondly how well the hazard and risk is actually controlled the robustness of the engineering human factors contributions and the organizational maturity of the duty provider it will also consider intelligence led decisions utilizing the experience we have as a regulator on the licensees previous performance the approach will consider the different ways risk can manifest across the industry that we regulate and it will be evidenced and we will consider that evidence in risk assessments safety cases, security plans and safeguard cases as well prioritizing our strategies resource deployment, balancing those competing requirements across a diverse nuclear state in the United Kingdom while still ensuring we regulate activities in a way that support those that we regulate not only to comply but also meet their own ambitions we will be publishing our right guidance document in early October this has been developed as I say over the last 12 months but we also recognize it's not just about guidance it's a cultural implementation in the organization and that is a much longer agenda than just publishing the document next month that cultural challenge of embedding this in the fabric of O&R and the mindset of our inspectors is a significant one and will require ongoing commitment across our leadership and at all levels in the organization but as part of that implementation we'll be looking to embed right coaches in the organization provide structured training and coaching which have already identified what we're going to develop risk profiles for the sites that we regulate that will consider all aspects of the purposes also enhancing our processes and guidance but again another important factor is establishing metrics to monitor and measure the impact of the approach on the organization and its success so just to conclude risk informed regulation must be adopted to guide our decision making as I said before it improves the safety it promotes efficiency while still ensuring effective regulation of the nuclear industry we have to bone the regulatory work appropriately we cannot consider everything nor should we be expected to a risk informed approach avoids undue burden on industry and regulatory bodies it allows for time review of submissions and targeted regulatory scrutiny and approvals while also enabling appropriately informed decision making to ensure safe and secure operations thank you thank you thank you Mark and thank you to all of our panelists I'd appreciate if we could bring up the third live polling question on the screen there we go so to respond again please go to the polls tab on the virtual conference platform or text the code shown on the screen our response options are somewhat limited in the live polling format but I look forward to your thoughts on what is most needed to support a culture that embraces risk informed decision making we've heard some great ideas from the panelists and I look forward to engaging more directly on this question in our panel discussion this question is intended to gain some insights from you our audience today on the focus areas and tools that you believe would be most beneficial hopefully everyone is able to go ahead and weigh in it looks like our results are coming up and I see a lot of support for the positive change management initiatives I think that's certainly you know trying to move our culture that's a critical element um I hope the panel can reflect on those results and incorporate them into our panel discussion and I guess we can move right into it so first question for the panel to make risk informed decision making truly a part of organizational culture you have to bring all of the people in your organization along on the journey what best practices have you identified to build organizational support for risk informed decision making I think Mark you just were mentioning the steps you guys are taking if you'd like to go ahead and start yeah thank you I think it's a reinforcement of what I was just saying a second ago senior leaders in the organization have to be strong advocates and visible within the organization supportive of a risk informed approach it's no point your leadership offering different views perhaps undermining that approach and they then work with their own teams to make sure that the teams beneath them understand why a risk informed approach is important and then that should hopefully cascade and manifest itself in the organization that's one of the learning curves for us in the UK some of the feedback we have got from stakeholder surveys that we've undertaken has given us some questions around our proportionality in relation to targeting our effort and when we've actually peeled back the cover the feedback has been that the senior leadership and the management are espousing the risk informed approach but it's not always put into practice in the organization so it needs to be identified all those areas are try and understand why that's the case but then work with them to get them onto the same page of adopting that risk informed approach thank you so more broadly to the panel I think that really raises what is the role of leadership in driving risk informed decision making within your organizations any takers whack at it so I think Mark is definitely right on when he says it's got to be a top down driven and then that cascading is really important in an organization as complex as ours and sometimes compartmentalized there are significant disconnects sometimes between maybe not the intent of that message but the words being used that it's not readily apparent that we're all saying the same thing one of the the greatest benefits we have in our agency is we have a very dedicated, a very mission focused workforce you could ask anyone that works for the FAA and they will tell you our mission is to ensure the safest, most efficient aerospace system in the world they don't need to think about it, they are ready to do that where that becomes a hindrance sometimes though is we have established the safest aerospace system so the question becomes why would we do things differently if I change the way I do my day-to-day job am I maintaining that level of safety and that is where the role of leaders is really important to help first off acknowledge that that's a huge driver and it's an important part of it but to really tie our messaging into that larger mission and we have quite honestly this is a growth area for us leaders we haven't always done a great job of making that connection for our folks yes absolutely what we did got us to this point here is why we need to change so that why is a huge part of it and really for us it's the changing industry it's the new technologies and we need to be able to do things differently to keep that level of safety we've already attained so it's the book titles what got you here won't get you there that's part of the mindset for us of to continue being the safest in the world we need to do something different and like Mark said that definitely has to be driven top down has to be a consistent message throughout the organization thanks so much Cody Mike or Doug any additional thoughts you'd like to add I think I'll throw in something I think there's sometimes a misconception that safety and compliance and risk are different things I'm not sure leadership this is something that came to me late in my career I'm not sure leadership always makes that connection that actually there are just two sides of the same coin that risk is giving you an indication of what the residual risk is after you've complied and after you've demonstrated you are meeting requirements understanding that residual is valuable on both sides it helps you understand if there's a non-compliance how big a deal was it, was it big or little if it helps you understand if I want to reduce that residual what are the contributors to that and what additional things could I do and I think too often we've allowed this idea that there are kind of different ways to do safety be pervasive in the organization and actually they're totally connected and we couldn't do PRA without the deterministic basis and we couldn't do we wouldn't be sure we were safe as we found when we did the IPs we wouldn't be sure we were safe without having that understanding of where that residual risk resides so I think doing a better job what I've found in trying to communicate this is to connect those and then that helps explain how things can form improved safety because you're looking at what that residual is at the end of the day A lot of great thoughts a lot to work with commitment from leadership is that's essential proclamations, statements, objectives it doesn't translate into change, it doesn't happen by osmosis it requires this continuous dialogue and engagement at all levels and so very much what Mark was saying it's that reinforcement of that expectation that we hold each of us accountable to that commitment and that's just not senior leaders again it's throughout an entire organization but on the ground level there's also a different perspective that we have to and that we've been very sensitive to and that is that we don't create a mindset of its risk versus the traditional engineering they are blended together and respecting that the traditional engineering approaches are reflected in the risk assessments this is something that that is a little bit of a mystique about risk that it's somehow a statistical, just a statistical approach and it's far more than that if you look at the information that includes risk assessments a lot of the engineering analyses done for the plants are part of that suite of information that is reflected in the results and the other part I would say is that the conversation piece is there, what we've learned we conducted what we're called risk cafes to actually outreach to the various communities about how they perceived risk and felt about what the messages were coming down regarding applying risk and tools, guidance were very important to develop but also a reflection that risk is more than a quantitative effort which is very important but there are also qualitative aspects to it which is part of traditional engineering so bringing those together and making sure that folks are heard was one of the key takeaways for us organizationally I really appreciate the insights from the panel that was very informative and next I'd sort of like to hone in on an area that has had a lot of discussion at this week's RIC for the development of advanced reactors and new designs within the nuclear industry there has been a focus on international collaboration and seeking more uniformity in regulatory requirements across different countries do you have any insights on how those trends may benefit from or impact risk-informed decision making Doug or certainly Mark thank you Given that we've just signed the agreement with the CNSC and the NRC I think it will be quite appropriate for me to comment I think international collaboration and the benefits of it are really clear in terms of improved efficiencies but also in terms of learning and sharing across the various collaborative parties and you end up then I believe having more timely evaluations of new technologies but the cooperation at the moment is very much centered on light water reactors I think one of the speakers said earlier about advanced nuclear technologies they are less well defined there's lots of claims on their performance and the levels of safety that needs examination across international boundaries is essential to get a consistent but certain picture for the industry about how the technologies will be received that shift towards passive safety systems new and novel technologies and such will lead a shift in the emphasis on the analysis, validation and verification that will be required beyond those in traditional engineering substantiations today I appreciate it Mark another vantage point is when you look at the themes of the RIC regarding adaptiveness and the demands on looking at advanced reactors, new reactors and sustaining the light water fleet you quickly realize to meet all these expectations in a timely manner is you don't have all the resources you would like to have something that Cody mentioned specifically about how you do resource management and so the international collaboration affords us that opportunity in these pockets of expertise that are needed to actually take the opportunity there to reach out and learn from each other and not just learn but there are organizationally at times potential for blind spots so having another regulator working with them sometimes does reveal things and we've had a few examples of that so it's a more efficient way if it's done on a scale like we talk about with a trilateral agreement and then also just targeting which sets of experts you could leverage to really get those additional insights because there are some areas of expertise you simply cannot go out and hire suddenly and have an instant transformation to not only have the knowledge but experience that goes with it so it is a very optimized kind of way of doing work I'll just jump in briefly I think that the way that the trilateral arrangement has been set up is smart I think that a lot of people talk about harmonization of regulation worldwide I think that's a nirvana maybe but I don't see it being particularly practical I think the collaborating and using the resources and reducing the amount of re-review focused is a much better way to proceed and I think that Mark and the NRC and CNSC really deserve to be congratulated for a thoughtful approach to move us down that road some day we may get to more true harmonization with a lower case H but I think that this collaboration thing is really good and I think the risk of bringing risk informing into that will be I think a learning experience for everybody Just one additional pointer in top of what Doug has said there I think the view of myself of Chair Hanson and President Jamal is that from small acorns you grow large oak trees so this initial collaboration was between the US and Canada it now involves the UK collaboration will go beyond those three parties and involve other regulators and indeed other vendors of technologies as well Thank you all and I think you know the benefits of that cooperation you know go way beyond just the initial projects that we work on but really building that community is so important for us we've talked a lot about you know some successes of implementation of risk informed decision making and the importance of leaders I think there's also real importance to those first followers in an organization and having that sort of building momentum that there always do seem to be some pockets of resistance and so reflecting on the how risk informed decision making the journey within your own organizations what activities have your organizations and reach those sort of folks who aren't convinced what do you think? So I'll go first on this one kind of similar to my earlier comments about the strength of our mission focus that does lend itself to those pockets of resistance and I wish I could sit here and say we had it all figured out and here's the answers and you follow you know nice step by step and everyone will get on board it is a growth area for us but I also think those pockets of resistance can be looked at as an advantage in some ways if you work with those folks in the right way I am a big fan of diversity particularly diversity of thought I saw Ted talk recently they talked about it from an economic standpoint they've proven statistically that you get better decisions the more diverse thought you have in the room people coming from different perspectives so those pockets of resistance there are reasons that they resist there are reasons that they see it different being able to leverage that and we use the term joint design within the agency bringing them in to help understand what their concerns are and try to make a better product whether that is a regulation whether that is our risk informed decision making that's a cultural thought we have I think the other important part of trying to approach pockets of resistance is really knowing your workforce for us we've got a huge array of backgrounds that come into the agency we've got air traffic controllers, we've got engineers flight standards, we're mostly pilots and mechanics and they all have their own set of cultural norms their own things that they bring with them their own way of looking at the world understanding that if I'm going to talk to engineers or I'm going to talk to pilots it's a very different conversation the pilot mechanic world tends to be very process driven, process oriented what's my checklist, what steps do I follow I know this is safe so trying to do that differently there's a lot of concern where our engineering workforce tends to be a little more open to doing that, the flip side of the coin for them is they're all very smart and they have their own way of doing it so trying to get consensus around this is the way collectively we're going to do it so those are just a couple thoughts things to keep in mind like I said I don't have all the answers but I would just say reorient that and rather than look at it as pockets of resistance those are pockets of information that you need to tap into so initially similar to Cody there the opportunity to bring mixed groups of individuals together within your organisation and have discussions look to dispel myths that people may have about adopting a risk informed approach but also as Cody has said try and understand the diverse views some of them will be important and will be valuable so you need to listen and hear what is being said but also the importance of using peer influence to get that mindset changed but also you could work with your staff to empower them to support what good looks like in turn of risk informed decision making and use them as conduits into your organisation but what I would say is ultimately you're going to have to tackle the issue head on where risk informed approaches aren't being adopted and that is what you have decided as an organisation it's in some instances views being expressed where well we've always done it this way we're going to continue doing it this way and that shouldn't always prevail at all you have to have a growth mindset and be a learning organisation so I had just a few thoughts so we outreach was very important for us in our rhythm action plan as well as in the be risk smart framework one take away we had was in order to be have a more inclusive approach there's a perception that training is important but it's more of being told what to do in a one way communication and that a more complimentary approach would be to have forums where you have open discussion and we have sponsored several forums internally in the NRC as well as we had one last fall externally to hear from not just the nuclear sector but we also had the FAA we had NASA management permeate so many high tech industries but to hear the different strategies and approaches and not just PRA folks talking to each other because there's actually a larger community that you need to have the conversation with that's just one constituency and so we found forums are we usually get very high attendance in forums when you have something more of a dialogue and perspective exchange that seems to be a very healthy approach to getting an airing of use and getting more buy-in into an approach that people can see their work is reflected in the ultimate decisions that are made Thanks Mike and noting the time I think we can shift into some of our audience Q&A and Mike this first question is really for you and sort of builds upon the discussion we were just having risk-informed decision-making requires some amount of experience and expertise to be effectively implemented how do you view the dichotomy between new and dynamic nature of the advanced reactor industry and the desire to have enough information to implement risk-informed decision-making I could go back to Doug had actually I think his framework laid out a number of areas where if you don't have enough operating experience how do you build a bridge to address that gap what I like to say in the new reactors and maybe a little in advanced reactors I work mostly in the new reactors which in the NRC lingo is more of the light water advanced designs advanced passive designs water-cooled SMRs so you can have a risk assessment that's aspirational from a design standpoint but understand that it's aspirational in other words it can be built on information from operating experience that is available and it can have uncertainty associated with that's reflected in the risk assessment based on the data that's available on the other hand what's the story behind making those estimates come true their estimates design PRAs are used to help provide insights not only the vendor itself but also the regulator but there has to be a story a framework that surrounds the making those target estimates come true because the programs are not going to be like the traditional plants have today and that's what a lot of data are built around those traditional programs so if you're using more non-safety related equipment of course we have policies and programs that address that but what is the proposal from the applicant that will help sustain that more aspirational target values and so it takes a very comprehensive type of approach that looks at everything from product development to qualification to testing, initial tests and then the controlling measures on availability, reliability and quality in a different way than the traditional fleet has actually experienced so it's the story that's behind the estimates in part Thanks so much Mike our next question from the audience is really looking back a little bit regarding your portrayal of risk over time how is industry managing factors that would tend to increase risk over time such as aging structure systems and components or licensed operator demographics for example replacement of experienced operators with inexperienced new operators within the current fleet Well obviously there are aging management programs that are put in place as the plants age to monitor those passive systems that aren't being threat tested and refurbished through the course of the life of the plant and monitoring that and as plants update their pure arrays periodically they look to information they've learned to feed that back into the process I think the human side of things is a trickier one I think that we have methods for assessing human performance and translating that into the pure array use every operator that comes in is well trained and I think that is a foundation for those methods and I don't know that we have seen evidence that there are gaps in that training that would cause us to think that performance is degraded certainly experience in the moment is beneficial but that's not really kind of the way that the risk assessment is founded it's more based on what are the expectations for the knowledge and training and procedures for the operator we have a question for you Cody you talked about the shift in oversight approach at FAA from one of reprisal to open exchange of information how did you effectively lead this cultural change within FAA to address concerns regarding reducing independent oversight that's a that's a great question it was not an easy process it was not a quick process I came into the agency in 2014 here we are 10 years later and we're still on that journey one of the right one of the biggest concerns culturally was that here's how we've always done it why would this be any better it's really about one reorienting the conversation to why now why do we need to do something different but also getting enough traction so you can start seeing results from that to be able to right it goes from in theory this will be better because to based on this data we are showing an increase particularly what does resonate with our workforce it's not about doing less oversight it's about as an individual inspector you may be doing the same amount of oversight you've always done but it's about targeting it appropriately because we don't it's not that we don't have as many inspectors as we used to it's that the industry has grown significantly and trying to keep pace as a federal organization we just can't operate the way we always have so the big thing to help with the cultural shift was really getting enough of that in practice and showing that we were making improvements that we were safer the other thing that really helped with that is from an enforcement standpoint if someone's not complying to go through that process is a very long process right involves legal involves the courts so that could drag out for years where the idea behind the compliance philosophy is okay here is something that's not compliant what is your plan to get back on track to get back into compliance and well it's not quite voluntary it is a lot more flexible a lot more cooperative and how do we get there so rather than in the enforcement days a lot of I'm going to appeal this so I'm not going to change anything until the appeal goes through we're able to more rapidly make changes to make changes with our operators to get them back into compliance with that yeah just on that theme and that open exchange of information the comment I'll make in a second stands fast for the question earlier on around advanced nuclear technology and advanced nuclear technologies and the maturity of the designs and aspirations that Mike was highlighting but also in relation to aging and the impacts on risk management there should be an open exchange of information between the industry and regulators as a matter of course there's no hindrance to that at all in the United Kingdom as a regulator I spoke on a panel yesterday about the enabling regulatory approach that we adopt in the United Kingdom and that's about working with the industry I don't shy away from using the terms of cooperation and collaboration to achieve the outcomes at the end of the day those outcomes have got to be achieved safely and securely but you can work together in achieving those so that exchange of information has got to happen there shouldn't be any consequences and you should be working on considering the challenges together in a mutually respectful relationship that's what's got to pervade otherwise my view is you're not succeeding in working effectively and efficiently either as an industry or as a regulator in coming together just a last thought just reflecting on Cody and Mark's comment we some of you know lived through a transformation all changing episode in the history of our agency thanks to Congress and looking at a different way of doing our inspection program and the reactor oversight process and the key takeaway there is a collaborative stakeholder approach not just with industry actually was a fascinating approach that some of you still recall may recall we used a federal advisory committee that also included some of the non-government organizations that had a more I would say more focused concern about safety and they were included on the actual FACA panel that helped roll out develop and roll out our oversight process which is actually technically a voluntary program and that because there was buy in we were able to obtain a process that helped with free flow of exchange of information a whole suite of new performance indicators but also to help stabilize what at that time was used largely is an inconsistently applied enforcement approach compliance based approach so that we have lessons from the past that we can leverage moving forward in that sort of more open exchange and have a more stakeholder all stakeholder viewpoint approach to these technologies thanks Mike and we'll just keep the mic moving here because this next question is directed to you but I'm sure others on our panel will want to also weigh in the question how do we best bring together the range of probabilistic and qualitative risk insights to help decision making particularly where the probabilistic element is less mature okay well certainly not one magic formula it's case by case in my opinion you do have to look from our perspective from the rhythm principles how much margins defense and depth perhaps taking more of a performance based approach in terms of trying to come up with the ultimate outcome so if you're don't have that much information available to support the quantitative piece they are compensatory type of approaches you can take if you're going to lean more through newer technology trying to leverage more of a probabilistic quantitative approach this goes back to the topic of uncertainties so looking at margins and things like that so they are offsets that you'd have to have on a case by case kind of evaluation but looking at those sort of pillars does help more sound decision at the end of the day that's another thing I'd add is that if you're talking about a situation where there is pure information and may not be perfect other qualitative considerations a lot more in the pure model than just the number that comes out of it a lot of times we like to boil this down to oh it's x point y times it's n to the minus whatever and say that's good or bad and the models actually can give you a lot more context for helping make the decision understanding the scenarios that are involved understanding the connected system or human actions that are involved and actually using that can help you interpret some of the qualitative information in a better way not using the numbers but understanding how the qualitative considerations actually fit into the safety picture because VRA gives the integrated view of the plant that we don't get through most other technically stove pipe sort of oriented views that we take to nuclear safety so I think digging in a little bit beyond just the number that comes out of any risk assessment is essential and I think we need to do that in those cases thank you I have to tell you Cody while our industries are very different the ability to gain insights from another regulator is just sparking a lot of questions here so I'm going to sort of consolidate a couple but these are focused for you Cody the first is really about is there a cautionary tale in overreliance on risk in regulatory decision making there are some examples most recently I think the 737 MAX it was a high profile story in the news but we're interested in your thoughts on the role of the FAA and the risk based oversight that FAA performs so as I gather my thoughts on that I think there is an assumption in that question that the certification of any particular aircraft or certification of a new operator is done solely based on risk it's not really where the risk informed piece comes in is in our we call it continual operational safety so something is out there and now it is how do we best manage our resources to look at those things that are emerging issues to like to say be preventative but a lot of our oversight really is driven based on data so it's a little bit more reactionary that is one of our goals to grow into more proactive than maybe even predictive as the data systems mature but from a certification standpoint that's actually one of the areas we've identified as highest risk so whether it is a new private jet a large transport carrier a helicopter for emergency medical services those regulations are set based on achieving safety in the NASA and complying with those we don't pick and choose so to speak it is an across the board you need to comply with all of these type of regulations so I think it's less about sorry can you repeat the question again and make sure I address it appropriately that question was really you know how it was a are there any cautionary thoughts about over reliance on risk and how it's supplied in your programs? so if I kind of decouple that now to the world where we really do focus on that risk based approach are there any cautionary tales? I think the biggest thing would be kind of to the previous question if we were to disregard some of those qualitative things like we have some inspectors who have 20 years of experience at an operator in the industry now 20 years experience in the agency that's 40 years of experience and if we try to boil it down to right this is the number the number says go look here and not rely on their experience I heard somebody say once kind of that gut reaction that instinct isn't is actually one of the most data driven things we as humans do it's usually unconscious there's something in our experience that's raising that concern so by being able to take advantage of that experience as we're doing oversight to look at things that we may not have been driven towards with the numbers so I don't know that's a cautionary tale but that would be the concern for me as if we stopped relying on that and went strictly to hear what the numbers say. I was just going to build on what Cody said and also what some of you may not be aware of there's a great deal of exchange of information going on amongst FAA, NASA and NRC actually through the National Academy of Sciences and there was a mandate to actually help update their risk manual and techniques and methods so there's been a healthy exchange and set of recommendations that came out of National Academies that actually there's actually a board of experts of consultants that are assisting and providing perspectives to the FAA to help build upon their experiences and their recommendations so some I see one or two out there that are we like me on the board of consultants to assist but these exchanges help we understand that what for example one take away I learned is there's a lot of data FAA collects. It has tremendous banks of information on pilot performance software issues but perhaps areas of the techniques used in risk assessment are a little different and whereas in the nuclear industry maybe we have a lot of data but perhaps not so much in the operator or human performance area so it's been a very eye-opening experience but there is a very good exchange going on so the take away for us is the learnings amongst the different agencies that have very similar types of regulatory functions just a couple of short comments from myself what we're asking our regulators or inspectors to do is to make a judgment at the end of the day and I think most of us on the panel have said risk-informed decision making so it is risk-informed we have to take a holistic view the risk analysis helps to inform where you target your effort but you have to look at the overall design on what is contributing in that aspect so it's about being risk-informed not overly driven Thank you and in our last couple of minutes here I think this question it's a little bit of a shift but I would really love hearing from all of the panel members if you'd like to so the question is really about what would be your suggestions for moving an organization forward once a decision is made that not everyone agrees with just an easy one to wrap us up here send it to the boss Chairman correct yes I think from my perspective you've got to keep working on it if there is a disagreement there will be a reason why there is a disagreement but at the end of the day you've got to make a decision and joking aside the ultimate decision maker in my organization is me so we have a framework arrangements for trying to deal with any particular conflict in a regulatory decision if you can't come to an agreement and it will mean that potentially someone is upset at the end of the day you've got to make a decision it's best interest of you as a regulator but also the industry with you regulator Cody do you have any thoughts from FAA's experience well I think Mark's answer was fantastic I think I mentioned earlier I think there is a lot of space a lot that can be learned from and revising or reconsidering certain things but at some point a decision has been made and an industry that changes as quickly as ours doing nothing is a decision but it's not necessarily an intentional decision so that's the one we want to avoid there are reasons to do things differently reasons to change and those should be informed but once the decision is made there is that need to go forward there does need to be that top down driven approach especially our organization we are so diverse that if it is not that unified message those pockets of resistance continue to exist they continue to be a part of it so I think the other thing is transparency I think is really important being able to show we are getting better this is achieving the intended result that's something an organization we necessarily haven't always been great at traditionally we put a regulation in place to prevent this thing and as long as that thing doesn't happen again we assume everything is working great so that is a growth area for us is really trying to understand why we make a decision why we change course and establishing some kind of criteria with which we can evaluate ourselves so being transparent with that what's going well where there are growth areas is a huge part of getting people to come along but at the end of the day like Mark said that is the the role of leaders in an organization to set the direction to make informed decisions but then to drive the change Doug you lead a different diverse set of individual every leader is faced with this whether you're talking about making a regulatory decision in a risk-informed environment or personnel decisions or anything else I think that Mark and Cody have hit on the main issues you got to understand where the person is coming from or a set of people ventilate that contrast it to what the other position is and ultimately we need to make a decision and there needs to be a process to make sure the decision maker sees that information can ultimately make the decision I think that needs to be done in a timely manner so that we can move on and Cody's talked several times about the rate of change that they've been in we're in an inflection point right now for this industry and I think we're going to be confronted with more of this and we're going to have to be able to process through these quickly and ultimately it is going to come down to management having to make that call be quick, own the decision be clear about who the decision maker is it's something that's baked in in our B-risk smart framework be clear up front be inclusive of the folks views listen carefully there may be a lot of emotion tied behind that conversation if you don't listen carefully and set aside the perhaps more emotional response that can happen from time to time you may be missing out on some insight there and setting yourself up for failure the other factor is that while decisions need to be made timely there's also feedback and operating experience they're not immutable to change if something significant does arise we have processes we have monitoring expectations to react in the moment to emergent issues this is something we have learned time and time again in the nuclear sector and so re-emphasizing that yes there will be oversight and monitoring to make sure that the outcome of that decision is still correct and maintained appropriately so I just want to throw that in there those are really fantastic answers I really appreciate all of the input from the panel what a great way to draw to a close for our audience you are invited to scan the QR code and share any thoughts or insight she may have pertaining to this session or the conference in general I'd like to thank each of the panelists for sharing your time with us today I want to thank all of our audience members for participating in the live polling questions and for the insightful and thought-provoking questions provided during the Q&A I would also like to thank the session coordinators Chen Pan, Daniel Jew and Daniel Silverstein for their exceptional support in preparing for this session and just a note for those of you in the room if you have follow-up questions that you would like to talk with any of the panelists please move to the exits the panelists will be right outside the ballroom shortly so we can make way for the next session and with that I would like to just give a round of applause to all of our panel members thank you again so much