 Here we go. Hello and welcome. My name is Shannon Kemp and I'm the Chief Digital Manager of DataVercity. We'd like to thank you for joining this DataVercity webinar, AI Governance, Drive Compliance, Efficiency, and Outcomes from your AI Life Cycle sponsored today by IBM. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we will be collecting them via the Q&A in the bottom right-hand corner of your screen. Or if you'd like to tweet, we encourage you to share our highlights or questions by a Twitter using hashtag DataVercity. And if you'd like to chat with us or with each other, we certainly encourage you to do so. Just click the chat icon in the bottom right-hand corner of your screen for that feature. And as always, we will send a follow-up email within two business days containing links to the slides, the recording of the session, and additional information requested throughout the webinar. Now, let me introduce to you our speaker for today, Scott Buckles. Scott is the North America Business Unit Executive for IBM's DataOps and Data Science Solutions. Scott has over 20 years of industry experience as a developer, consultant, and sales executive in numerous technology areas and industries. His focus remains on helping customers achieve their business goals by leveraging technology, process optimization, and people within an organization. Scott's passion for customers and technology has helped deliver results from the early e-commerce days to today's burgeoning world of advanced analytics and artificial intelligence. And with that, I will get out of the way and turn the webinar over to Scott to get us started. Hello and welcome. Thank you, Shannon. And apparently, 20 years in this business or over 20 years still means that you have a hard time getting into a WebEx. So, my apologies for the delay, everybody, and thank you for bearing with us today. And thank you, Shannon, for all the help and getting that figured out. So, I guess it's just further proof that technology is still hard from time to time. So, hopefully, everybody's having a great afternoon and a great last couple of weeks of this probably the craziest year of our lives. And really appreciate you joining us here today, which should be an interesting topic. And I look forward to taking you through this and then also taking you through some questions and answers at the end. So, let's just start with a picture. So, this is a wonderful picture. I've actually seen this picture in a number of my colleagues' presentations and decided to use it because it's very stark. It's right in your face. And that is a bunch of garbage. And when I think about this picture, it reminds me of the old adage, and we say this a lot, that garbage in, garbage out. So, we say it in other ways to our kids, what you put into it, you get out of it, right, whether that's your schoolwork or your sports or whatever. And data is a lot like this. I mean, it applies to data maybe more than any other place in the world or just as much as it does anywhere other place in the world or any other activity that we do. Data is imperative that we understand it, that we know where it came from, that we are able to trust it, and that we are then able to use it. And I think in this world today, we have all this great technology in our hands, and we sort of lean on it like a crutch. And we think because we have all these great systems in place that we can, that those systems are going to make our lives easier and we can get away from some of the fundamentals. There's a former IBM executive, his name is Steve Mills, who is maybe the founder of really what became IBM's software group. And I remember hearing him talk one time, he said that there are things that we argue about in technology that we would never argue about in the physical world. And the example he gave in this particular presentation was he said, if you had to get 300 people from Los Angeles to Las Vegas, then you had to do it in under five hours. Would you be, A, better off putting those people in 300 different cars or maybe 150 cars and having people travel and choose on the roadway, on the highway from Los Angeles to Las Vegas? Or would you be better off putting them on a rather large plane and get them there within that period of time? And a lot of times we argue like that would be really hard. If you had four hours to get people from Los Angeles to Las Vegas, there's really only a couple ways to do it and get it into that time frame and meet those requirements. But a lot of times in the technology world, we would argue that there's a better way to do it or there's a different way to do it in whatever problem we were trying to solve. And you just, there's times where the fundamentals still apply. The best part about being in technology and all of our jobs that we do day to day is that there's so much innovation. There's so many different things that are evolving and changing and there's people coming in and they revolutionize the way we do things and the way it impacts the way we run our businesses or even live our lives. And that part's exciting. But you can't get away from some of these fundamental truths. And that is that you still have to put the work in. You still have to do, if you put garbage in, you're going to get garbage out. If you don't put a lot of effort into it, you're not going to get a lot out of it. Your data is the exact same way. One final analogy I'll give you, and I think about this because I was Christmas shopping last night and three young men, 12, 10, and 7, and they all play hockey and they all want new hockey sticks. And if I go out to the different manufacturers of these sticks, Bow or CCM and others, and I see these latest and greatest technologies and they talk about how it's going to make your shot harder. It's going to make it more accurate. It's going to make your passing more crisp. And I think, well, those are all great. And there's a ton of technology that goes into these sticks and a ton of engineering work. But if those kids or any other kid doesn't go out and shoot pucks in the garage and doesn't go to practice a couple of days a week and work on their shot or work on their stick handling, then they're not going to get the advantages or the technology that that stick brings out of it. It's going to just sit there. It's basically not going to be there. They wouldn't know the difference between a $2 stick and a $200 stick at that point. And if you think about the folks at any level, a high level of any profession, whether it's an artist or an athlete or a business person, the amount of time that they work on fundamentals to become that successful. And if you think about a hockey player, the amount of training that they do, the amount of shooting the pucks and drill that they execute on the ice, off the ice, day in and day out, that's how they're able to take advantage of the technology that comes in, whether it's a stick or a skate or their other protective equipment. That's how they do it. It doesn't just happen magically. And I think a lot of times we think that that's going to happen. And so when we think about AI governance, and we'll get into it a little bit, but it's really an evolution of some of the things that we've seen around data governance, or it is an evolution of data governance at its core. And data governance is something that really started out in a couple of different ways. The most notable way that we've all been exposed to data governance is through compliance. And whether it's an old banking regulation that's 20 or 30 years old or HIPAA here in the United States, or some of the newer regulations that are around data privacy, like GDPR, CCPA, and other forthcoming statutes on data privacy from other states and other countries. Those are all driven in a reactive sense or in a compliant sense that we're trying to enforce or ensure that our organizations meet those regulatory standards. And those could also be corporate standards or corporate mandates as well. They don't have just to be government statutes. And then the second piece of it is really around governance or insights. And this gets into more of a proactive use case. This is where when I think about the change that's happened in the last four or five years in our industry where we've seen the role of the CDO take off. If you go back six or seven years ago, there might have only been a handful of CDOs, chief data officers around. And now almost every major institution has a chief data officer. And they're tasked with getting more out of their data. They're tasked with not just the compliance part, but monetizing that data, delivering true business value with that data. And that's where we've seen some of this governance or insights really start to expand. So let's talk a little bit about the journey of how this happened. We did a presentation a couple weeks ago and we talked a little bit about the departmental data governance journey. And there's been a lot of changes over this in this space over the last few years. Really, when 10 years ago, the amount of effort both in terms of labor and technology that it took to govern your data made it limiting to where the compliance use case was really the only use case that you could be successful with because it was too hard. It was too manual to be able to govern your data at scale. And so you saw a lot of departmental efforts, a lot of siloed efforts, whether that was in operations or finance or in sales. There was a lot of siloed efforts. And that yielded some value. It helped change the mindset of people to say, I've got to find a way to get data quality injected throughout my department. It's got to be on the forefront of my mind. And it helped generate a mindset around the data has to be clean, the data has to be trusted in order to ensure that compliance. And then that helped fuel more of those reactive that the use case is more of that compliance driven use cases. But even then it's not to the point where we're going to get a ton of ROI out of it. So then we start doing in these proactive use cases. These are the governance for insights use cases. And now we're starting to get some momentum. And then when we take that across an enterprise, you know, starting from one department to an enterprise level offer. Now we're starting to maximize our return on our investment. Now we're getting the thought process and the mindset that our data quality, having business ready data is part of our core operations. It's just like going back to the analogy and around a hockey player. It's now getting into I have to go train every day. I need to go work on my skating. I need to go work on my on my stick handling in my in my shot in order to start taking advantage of the technology available to me. That becomes ingrained in our thought process. And that's where we start to get some really great benefits that really help ensure compliance in a much more automated automated way. Excuse me. And then also start helping separate our business or our organization from our competition by delivering better customer service or a better product, better product quality, a variety of different things, better profitability by understanding our different processes and maximizing our cost efficiency around what we do. And now that's all great. And we've done a, I would say that that the cool thing is, is in the last three or four years, the attitude attitude achieving towards data governance has gone from, oh, why do I have to do that to, oh my gosh, when I do that, I start to see some benefits. Sort of like, you know, if you're going again, I'll just carry this analogy throughout this. If you take a kid who doesn't want to go and do that work, and then you get him to do that work, even if they begrudgingly did it to begin with. But then they start to see the results and they start scoring more goals or having more fun playing the game. It becomes contagious. It becomes ingrained in their approach to how they take every day or every hockey game to go out and prepare for that. Data governance is just the same thing. And as we get into the AI world, it makes it that much more important because data governance is evolving and it's expanding. And AI is forcing us to, for that expansion, because there's more considerations that come into play that we have to have those fundamentals and then build upon those fundamentals around data governance and started applying them to, to AI. So, if, if we're not involved in an initiative yet across our organizations, the likelihood is that we were very well soon will be. Our very soon will be if I could talk. And this is Gartner. So, in 2021, so this is, you know, starting next year, we're expecting to see 2.9 trillion in business value generated and recovering 6.2 billion hours of worker productivity. So, and this uses the word augmentation, which I'm big on, and, and I know that it's one of the prouder things that I love about IBM these days. And in terms of our stance on the ethics of, of AI is in our approach to AI, we're not looking for this to replace jobs. We're looking for this to augment jobs. And I think that recovered productivity and that additional business value is really about helping our experts, whether that the data engineer or data steward business analyst, data scientist, a sales executive and operations executive, somebody, whoever it is across that organization, helping them have more insights to make a better decision, not to replace them or replace the decisions that they need to make, but help them have the insight to make better decisions. And the basis of that is having good clean data. But as we will see as we go through the evolution into AI governance, it's also about understanding how we govern our AI models and our AI processes as well. But it all starts going back to fundamentals and I'm big on fundamentalism. I love, I love being able to focus on it. I think that they're so important just in life, like learning those, those good skills that you have to do, like to work out and to study hard and learning those traits. And if you get into a sport or a job, like even a sales job, there's certain fundamentals that you have to have. And in technology and in data governance, that's what the AI ladder is about. If you've heard anything from IBM in the last couple of years, you've heard us talk about this, but this is a roadmap or a ladder of the fundamentals around data that are imperative to being able to make your organization AI ready to be able to collect all of that data. Understanding where it is, making it simple and making it accessible. Being able to organize and that really comes down to what we're talking about in data governance and data ops is an even more modern term for data governance that really starts to apply the methodology around the people and the processes along with the technology to get more out of our data and our data pipeline. Making sure that we have analytics, we're able to analyze that deal data and do it at scale. That's important as well, along with the trust and the transparency and then being able to infuse operational AI throughout the process. And if you can't do the first three things, it makes it really hard to start infusing operational AI throughout all of it and have success. Yes, you can do it, but can you do it and have success? That's much more challenging. So let's look at how governance for AI really starts to evolve beyond just governance. So we started with the governance for compliance use case and how that evolved into governance for insights use case. Now we're talking about governance for AI. And the thing that is really transformed of here isn't just in, it's not a technology thing, but it's understanding how those AI capabilities perform appropriately, ethically, morally and legally to mitigate market and social risk while benefiting business objectives. So it goes beyond just understanding where your data came from and understanding the policies, the data policies that you have as an organization or an entity and applying the right quality rules and forcing those data privacy standards. Now we get into, okay, we created a model. Do we understand where the model came from? Do we understand how that model was governed? Was it done ethically? Was it done morally? Was it in compliance with any legal standards out there? And these are all evolving components of AI, but they're very fundamental to the AI journey. And we'll show some examples here shortly of where AI governance has really helped companies understand where they may have been exposures and then be able to correct the course much faster than they would have if they hadn't had that. So what is AI governance? So if you believe that AI strategy is strategic imperatives, use cases, competencies, technologies, then let's start with that as a foundation. And by definition, AI is hardcore computer science. It's not magic. It's not something, it's not science fiction. It is hardcore computer science that we need to build on. AI governance is then model management. It's digital ethics. It's compliance. It's monitoring. It's ensuring quality across all of those models. And then you also have explainable AI. And this really comes down to if I'm leveraging an AI model to help understand whether or not I'm approved for a loan. And I was denied that loan helping me understand why and being able to explain it to me in a fair and equitable way. And so that if that decision was traced back, those things would hold true. How important is AI governance in this world? Well, by a little over a year from now, the belief from IDC and the research that they've done is that 65% of the enterprise will task CIOs or CDOs to transform and modernize governance policies to confront these risks. So 65% in just about a year's time. That's pretty astounding. And I guess if you looked at it out to 2023 or 2024, that number probably goes from 65 to 90 or 95% if you ask the same question in the same survey. So we have to make sure that we have compliance in it, that we align our strategy with the regulations and legal requirements, that we maintain our trust, our customer staff, our brand value and transparency to that. Those are going to be big. So we've talked about no trust, use your data for a long time in the data governance world. And those same principles really carry forward a lot into AI governance because trust is so important. And if you don't have that trust or that transparency, especially with AI, it can be very detrimental to our brands, to our customers, to our overall business. And a lot of that starts with understanding not just where your models came from but also where your data came from that are helping fill those models and with the fuel that they need to be able to generate the AI. We also have to have efficiency. Speed is still a problem. Whether it's data governance or data officer, AI governance, efficiency is still a challenge for most of us. And we need to be better about doing that at scale. Luckily, there's a lot of technology out there today where we are automating discovery of data. We're automating policy enforcement. So we set policies as an organization and an entity and we are able to go out and enforce that automatically within our data, you know, even to the point of who controls or excuse me, who has access to the data and making sure that it's delivered to the right people who have the access and the need for that data in a time efficient manner with that high quality. So important. And it's part of the evolution of where we're seeing from data governance into AI governance as well. And those principles carried forward. Excuse me. The idea of ensuring compliance is never been more important to our business, not just because of the monetary fines and I think GDPR from data privacy is the one that is. I think on top of mind for most people, GDPR certainly brought that to the forefront for a lot of folks. CCPA in California and for all of those of us that do business or have clients in California certainly apply and California is just on the leading edge of many other states and countries who are setting forth data privacy standards. But this is the one that is really on top of mind and that we have to in a lot of ways use it as a, as a way to get funding for some of these initiatives from the business. But it's, we have to consider regulatory compliance is just a core, a cornerstone of everything we do so that we have it and that we know we're compliant and we can sleep at night knowing that we're compliant. And then move into what is really the fun stuff, which I think it gets into leveraging AI to help differentiate ourselves from competition, improve efficiencies and things like that. So these are a few of the things that are going on and I will claim to be an expert on all these different policies that are driving data privacy as it applies to AI and these are really AI level regulations. I think the most complete one right now is SR 117, which gets into model risk management and describes the validation process that has to be done within financial services. That is the one that is probably the most implemented across the, the, or most complete and being implemented I should say across enterprises today, but there are a lot of evolving standards around AI, a lot of regulations, certainly data privacy is a big part of it. Again, it gets back to that. Is it, is it ethical? Is it legal? Is it transparent so that I can have that that trust from our customers or constituents who that may be in that this model is helping and is done in the right way. And we've seen examples where companies have have leveraged AI governance to help them understand that they, they have an exposure. Apple came out with their new credit card and quickly realized that through some of the process of the application process that there was a gender bias and we're able to correct it quickly. Amazon has the same thing around recruitment software on Facebook had it with with personal data. And by understanding the, the models that were in place by being able to govern them, they were able to quickly react and correct course. IBM is another example. Going back to the spring when there was a lot of talk about facial recognition software and IBM pulled its support from facial recognition development because of there was inherent bias and facial recognition software so that we could find better ways for security and things like that. But, but also standing firm for what we believed in and leveraging AI to help us do that in a better and more effective way. So, going back to this evolution here. If we think about this business ready data, we've talked about this a lot over the last few years and talked about it some today know your data, trust your data, use your data. Now we are moving into this world of AI ready data. Know your model, trust your model, use your model. But AI ready data is an evolution of business ready data. It doesn't replace it. It doesn't replace the need to know your data or trust your data or use your data. It builds upon that and starts extending that to to the models that we are running with AI. It extended to how we are using that model, how they are validated. Who is validating them? Are those, is that in, is that validation process in line with our morals and our ethics as a company and as an organization and then also as an in line with with the legal requirements that we have. So, AI ready data and AI governance really start extending and expanding upon the terms and the concepts and the practices that we have around generating business ready data and start applying those to models and the other components of AI that we talked about. This is a rather involved and complicated chart that would take a lot of time if we wanted to walk through it, the flow chart here. But I think what I wanted, the reason I included it, the reason I wanted to share it with you today is AI governance covers a ton of different roles within an organization. And these are all people that are that are critical to AI governance and then are also impacted by and consume AI. So they're impacted by the output of AI governance. It's not just one or two folks. It's not just the chief chief risk officer, the chief data officer. It's across the enterprise. And so if you think about your own organization and you think about evolving what you've done with data governance today, you start to think about the roles and a lot of these are similar roles, but it's even more expensive. And as AI gets a lot of airtime, if you will, in an organization as being the next great thing, remember that these fundamentals and these that we have around data governance and that we're expanding those still apply. Going back to how I started this with, you know, with my hockey analogy with my kids, those fundamentals that you learn at 10, 12 years old of learning to go out and work hard and study hard that you have to do more than just what you do in school. You have to learn more than just or do more than what you just do in practice every day that you've got to take the initiative and build those fundamentals, those work habits as a hockey player. If you will, as a student, you still have to have those same fundamentals and principles around data governance and that a lot of the same roles that we include or that we touch as we govern data as we cleanse days. We generate that are also involved in the governance life cycle. They are constituents of it. They are players and integral parts of it as well. So leverage that as a way to get buy-in to the importance of this and help generating more efficient, more trusted, transparent and efficient organization around how you manage your data and then also how you manage your AI initiatives as you move forward into the next stage. So with that, I think we have a few minutes left for questions. I'm going to try and see if I can see them by expanding my window here, but I think we're monitoring the chat as well. Yeah, we've got you covered. So yeah, thank you so much for the presentation and I'm glad again we could get you logged in and go in and we do have some questions coming in and just to answer the most commonly asked questions. Just a reminder, I will be sending a follow-up email to all registrants by end of day Thursday for this webinar with links to the slides and the recording along with anything else requested throughout. So diving in here, Scott, how do you differentiate a Chief Data Officer and a Chief Information Officer as they relate to AI? Well, I think it's an organizational thing and it really comes down to... So when the CDO became popular a few years ago, I want to say that the statistic at the time was that a third of those CDOs reported into the Chief Information Officer a third of them reported into a CEO and a third of them reported into some sort of Chief Operating Officer or Transformation Officer like a CFO, somebody like that. And so it had a variety of different reporting structures and so I think whoever is charged with the AI initiatives is who is probably going to be most accountable at least at the forefront for ensuring that you have that AI governance. But it is a consideration to think about when customers set up their business or their teams to focus on AI, how does that apply to what a CIO's roles and responsibilities are within your organization versus what a CDO's roles and responsibilities are. And then also balance that with like a Chief Privacy Officer and a Chief Risk Officer. So I don't think it's a real neat and tidy answer, but it really comes down to considering who has the responsibility for it and how does that relate to the responsibility around ensuring data compliance and ensuring that you have high quality business ready data as well. And Scott, I think you covered some of this already, but maybe formulated in a different way and the question provides a bit different answer. What is the main difference between data governance for AI and the operational transactional data? Does the same governance approach, can it be taken for all data or not? Again, what's the main difference? Well, I think the main... So to me, the fundamentals still apply. Whether you're governing data for operational or you're doing something for AI, a lot of the fundamentals still apply. You have to have the policies you want to be able to discover it. I think where AI extends or expands beyond what we're doing with some of the just the core data governance stuff, is it gets into where the models came from, who created the models, are the models done in a moral and ethical way? Or do they comply with legal standards? Because if we're using AI to help automate a lot of decisions, we have to ensure that those morality and ethics are applied to those models and are consistent and that we've gone through that validation process. So it really takes those concepts and starts expanding beyond just some core data governance concepts or fundamentals and brings those into the fold as well. Are we saying that AI is an enabler for data governance strategy and its implementation? Are we saying AI is an enabler for data... Say that again? Yeah, are we saying that AI is an enabler for data governance strategy and its implementation? I think so. I think absolutely. I started out early in the conversation saying that I think the attitude towards data governance has changed drastically over the last couple of years. But I still think people are like, yeah, why do I got to do that? I mean, it's like, well, why do I have to eat vegetables? So if we are looking for ways to get more buy-in, I certainly think AI governance is a way to put a refreshing spin on the conversation that maybe... People may organizationally have some sort of bias against the word data governance as being old and lethargic and whatever. And AI governance might be this whole thing that's new because it's around AI. So it can absolutely be an enabler. And even if it's more in the way of getting budget, getting funding, resources, skills. But at the same time, you have to be cautious because it expands upon the principles of data governance and you still have to do those fundamentals as well as the AI governance fundamentals. And we get this question in relation to AI a lot. How is an audit done in AI? How do you manage the data quality? Yeah, that's a great question. And I think that the... I would leave that to somebody that is even more versed in that than I am. To me, to answer the question or take a stab at it, I think that your data quality, you know, audits that you have and how those policies are reviewed and adherence is insured. Again, it extends what you're doing there into the AI world and into the models. But I think that that's also something that you have to be very specific about with whether that's a statute, a regulation, something within your own organization. There's a lot of corporate mandates. There's a lot of folks that are coming out with how corporations are viewing AI and how they're using AI to ensure that moral and ethical and legal standards are upheld and harmonious with that company or that entity's beliefs. And so you have to apply those as well. And by set by applied, you have to review those as well as well as any sort of data quality standard that you have. I love the vegetable reference. Can the same approach for non AI data be applied to AI? And we kind of talked about this a little bit already. But considering if an organization has a well-governed non AI data already, the AI data therefore will be trusted and more manageable. Is that correct? Yeah, I think so. It certainly gives you a head start. I would argue that there's a lot of us that saw the long way to go in ensuring that we are called an expert level in data governance. But the hard work that you do there pays off not just in overall data quality but in business ready data. But it certainly sets a stronger foundation for you as you move into the AI world. How should the data governance and AI governance organizationally be integrated? One unit, mixed team, separation of duties? Well, it's a great question. And I don't think there's really one answer. Again, I think it comes down to how the roles and responsibilities within a given organization or entity are divided. Certainly, in my opinion, I'm of the belief that it's an extension of the work that a CDO would do. But, you know, there's a lot of variables that we all have that are specific from one organization to another that help drive that. And the important part is to me is understanding what those boundaries are between the different roles, understanding what the roles and responsibilities are. And if you do that, then you're able to figure out, you know, where best resides within your organization based on a multitude of factors. But those boundaries, I mean, boundaries, there's an old saying, right? Conferences make for good neighbors. It really comes down to having those roles and responsibilities clearly defined in your organization and whose responsible ultimately for that will be decided by how your organization approaches its business. There's a lot of, I mean, that's a very, I mean, I don't think that was very articulate, but in a lot of cases, I'll say it this way, people get caught up in the title of a CIO versus a CDO. And there are organizations that don't distinguish between the two and the CIO is the CDO or the CDO is the CIO. So that's really what I mean by how you, what's best for your organization. Don't be so caught up in the titles, but the roles and responsibilities. I love it. So Scott, it's not a management about governing algorithmic models. And can you provide examples? I couldn't hear you. I'm sorry, Shannon. Oh, sorry. It's model management about governing algorithmic models. And can you provide examples? It is. I'm trying to give it of an example. So I'll say, so one example I'll give is either within some of our products, we have some machine learning models that are out there that are constantly specifically around data quality and they're continuously learning as they are running and processing data, but also based as, you know, a, in our case, our product engineer is interacting with that or a data steward is acting with that from a customer and giving them inputs into it. So that model in that case, managing that algorithm, who made changes who made inputs to it. So if you had a data steward that is saying, well, you know, this, the model gave me this, but it didn't take these things into consideration. So I adjusted the model. It gets into a lot of that. Those, I mean, I hate to say it, but it's even basic things like version of who touched that model. When did they touch it? Did they change the data sources? Where did those data sources come from that were hitting that model? So those are some of the examples that come to mind. Kind of back to organizational functionality here, you know, as more corporate functions will be affected by governance as risk management, legal compliance, et cetera. What's the best approach to high number of the stakeholders with differing interest when too many cooks spoil the meal, so to speak? That's a tough question. I don't think I get paid enough to answer that question. No, I'm teasing. I think that that's a really hard thing for, to me, I look at it as where's your lowest hanging fruit. And I get that everybody, whether it's the legal department or the finance department, operation sales, manufacturing, whatever it is, are all going to want to jump in line. But it's about what's the lowest hanging fruit and managing the risk of it as well. That's a big part of it. So where can, which department or business unit is going to get the biggest bang for the buck by investing in this so that you can show ROI? I think the thing that we've learned, if you go back to the way data governance used to be done is an even application development. I mean, going back 20 years, you think about application, the way we would develop them with waterfall approaches is you would spend all this time and money. Think about all these ERP implementations. And then you wait until the end and you're like, oh my gosh, what did I get out of that? Or it wasn't the right application. That's not what they wanted. So the advent of agile and dev ops and data ops and a bunch of other things have helped us be more effective in that and determining that. But you have to look at it organizationally of who's going to benefit the most. How do we manage that risk? And then develop that along with an overall AI strategy across your organization. And that's the difficult decisions that the business leaders have to make with that strategy. And that's the importance of collaboration with whoever that is. It owns that whether that's a CEO, CEO, chief research officer, whoever it may be within your organization. Perfect. And I think we have time to flip in at least one more question here. So, where do you think the line should be drawn on what AI deployment should be governed? For example, recent executive orders sort of excludes commercial common applications. Would you similarly carve out exploratory data analytics? Any lines you think exist? Yeah, I think that that's, I think it's a really tough question. And I'm not sure I have a great answer because I think that. You have to, each organization has to set their own standards. And they have to, there's so many factors that go into that, that it's really hard to give a succinct answer. Because it's where are you in your data journey? How mature are you? And then where are you in your AI journey? And are you trying to put the cart before the horse as the old saying goes? So, I think that those are, those are really specific questions to a specific organization and entity that have to be probably taken a little bit more intimately and no more details about what's going on to answer that effectively. I think that's all we've got here, Scott. Thank you so much for a great presentation and so sorry for the technical difficulties logging in there. Glad we were able to find you get you in and on here. It's been great. Thanks to all of our attendees for being so engaged in everything we do. We very much appreciate it. And just again, reminder, I will send a follow-up email within the next two business days containing links to the slides, links to the recording and additional information requested throughout including additional information to contact IBM and information there. So Scott, thank you again so much. Thanks everybody. I hope you all have a great and safe day. Awesome. Thank you, Shannon. Thank you everybody for joining us today. Greatly appreciate it.