 My name is Shannon Kemp and I am the Executive Editor of DataVersity. We would like to thank you for joining our October's Installation of the Relatively New Monthly DataVersity Webinar Series, Enterprise Data World. This series is designed to give our Enterprise Data World conference attendees education year-round, a conference we've been producing in partnership with Damon International now for nearly 20 years. Enterprise Data World will be held this year in Austin, Texas, April 27th through May 4th, 2014. We'll close the call for presentations and we'll start reviewing those submissions and putting together the agenda before we open registration next month. Today we'll be discussing a strategic approach to data quality with Laura Sebastian Coleman, who has been both a speaker and attendee at past events. Just a quick list to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we will be collecting them via the Q&A section in the bottom right-hand corner of your screen. Or if you like to tweet, we encourage you to share highlights or questions via Twitter using hashtag DataVersity. As always, we will send a follow-up email within two business days containing links to the slides, the recording of this session, and additional information requested throughout the webinar. We will now introduce to you our speaker today, Laura Sebastian Coleman. Laura is a data quality architect at Optum and has worked on data quality in large healthcare data warehouses since 2003. Laura has implemented data quality metrics and reporting, launched and facilitated Optum's data quality community, contributed to data consumer training programs, and has led efforts to establish data standards to manage metadata. In 2009, she led a group of analysts from Optum and United Health Group in developing the original data quality assessment framework, which is the basis for her book, Measuring Data Quality for Ongoing Improvements by Morgan Kaufman. I should say published by Morgan Kaufman. And if you don't know, DataVersity has a partnership with Morgan Kaufman and I'll send a link on the promotional code in the follow-up email for this webinar to receive 20% off of Laura's book. So with that, we are very lucky to have her with us today and I will turn the presentation in the floor over to Laura. Hello and welcome. Thank you so much, Shannon. And thanks to everyone who's in attendance. I know our time at our jobs is scarce and I appreciate people choosing to participate in this webinar. It's valuable for you. I have two goals today. One goal is to get participants to think strategically about data quality improvement. And my other goal is to try to provide you with concrete suggestions about actions you can take to enable you to assess your organization's ability to move forward strategically. To do this, I will give a very brief introduction to my company and to myself. I'll make an important disclaimer. Then I'm going to review some definitions and then some assumptions about the concepts I'll be discussing. And many of you, I know work and data governance and know the importance of setting the table with accurate and complete definitions. So we'll take some time on that. I'll give you what I've called the 12-Directive Data Quality Strategy. And with each of these, I'll describe the goal of an assessment of readiness for that directive and then, as I said, some concrete things that you can do to start working on that goal. I will try to finish up in a timely manner and leave 15 or so minutes so that we can have some discussion at the end. As you can see, I work for Optum. Optum is part of the United Health Group and we are a health information and technology company. So we help people within the healthcare system use the technology to make that system work better for everyone involved. That is our company's mission. And when I talk a bit later about the importance of mission for a company who wants to move ahead strategically, please remember that the company that I work for, I think, has a very clear mission and we try to live it. We have a large 35,000 people and Optum solutions interact with people through the healthcare system from planning protocols through treatments and the like. So we are hoping that through technology we can improve this overall and deliver healthcare of higher quality at lower costs. As Shannon said, I have over 10 years of experience as a data quality practitioner. I've worked in the healthcare industry during that time and I'm on the IT side of things in the industry. So I put that forward because I know that sometimes our questions about who's responsible for data quality is at IT, is at the business side. I really think that technology and data are so important to everything we do in healthcare that you can't really separate out how the business works, from how the data works and how the technology needs to support that. My experience has been largely in data warehousing and therefore that's influenced how I think about quality. But I also have a background in banking and manufacturing. I've worked for a distributor and I've also worked in the workers' comp side of commercial and insurance. So I bring to my understanding of what IT is, what high quality data is, and how we can measure quality. One thing that I conclude from what I've seen is even though we are living in the information age, data is still treated more as a byproduct than it is as a product. Organizations continue to make incorrect or sometimes naive assumptions about what data is and how it should work and they don't always recognize its complexity and that is one of the things that tends to get them into trouble with quality because they have expectations that cannot be met. So that's about myself and my company and I did say I have a disclaimer. My disclaimer is that the 12 directives that I'll discuss are not new. I hope that none of them are really surprising. I've since then from some of the thought leaders in data quality and in product quality, I'm saying here's a brand new idea that will change your life. I want you to think about how to apply these. So I've included a biography in this slide so you can get a listing of some of the sources that I rely on when I try to grapple with some of the challenges that we have with data quality. What I think is new is that some of the ideas about how to move forward with directives. So that's what I'm going to be getting at. These directives are based on is a set of assumptions of some of the common problems that exist in organizations that rely on data to get their work done. There's also an assumption about the complexity of systems. Systems tend to be complex anyway because they're moving a lot of data through them, but most organizations have systems that have evolved over time and oftentimes the folks that are really built the system are long gone by the time changes are made to the systems. So they really have gone through evolution and very few organizations are rigorous about documentation. So there's a lot of knowledge in the system that is really hard to access. We don't always know why our systems work the way they work and they appear more complex because of that. Our organizations are dispersed and so data is produced in multiple places and it doesn't always fit together in the way that we think that it should. Most people, most organizations are not rigorous about documentation. We often know the system, but that knowledge is embedded in people's heads. And so getting it out and being able to share it is a difficult challenge. I'm assuming that many of us are in that condition and if anybody's not, contact me after the webinar because I'd like to understand how you got that and beyond it. So let's start at the beginning with definitions. Organizations talk about strategy, but they're able to actually define what they mean by it. I'd like to go back to the origin of the word, which is a military word. It's actually the Greek word for general ship. So strategy is about planning for a war for a set of battles or a set of engagements and what you can accomplish and then it includes a set of steps to accomplish it or a set of options really for accomplishing it. And those are usually referred to as tactics. And there's a lot of logic of tactics. You have a set of decisions you can make and depending on the outcome at the tactical level, you can move, you know, other options may open up. We talk about the difference between strategy and tactics if they're positioned, but they're really in part and parcel of the same thing. So when we talk about tactical success, we think tactical success can only be equal to success if it actually enables a strategy to progress. If it doesn't, then you are, you know, winning the battle and losing the war. The origins of strategy tell us about what it means to be strategic. What it means to be intentional, what it means to think out and plan for success in terms of both time and space, and you really always need to be asking yourself, if you want to act strategically, you always have to be asking yourself, what do I want to accomplish? When do I get it accomplished? Why? And how do I do it? How do I move from point A to point B or X or whatever? Questions will keep coming up. With business, we're another organization, you know, a modern entity. When we talk about strategy, we're usually talking about aligning work efforts with the mission or the long-term goals of any organization. So it's about aligning the mission and making the right decisions to achieve the goals of the mission. So in order to do that, you have to have a clear vision and mission. You want to understand what your current state is. You need to know where you're starting. You have a set of plans that move you from that current state to that future state, and we talk about those in terms of tactics. So strategy can provide an organization with criteria through which to make decisions about tactical efforts. And of course, people need to know what it is, and they need to actually refer to it when they make those decisions. So this just tries to put into a picture what I've been saying about the relation between strategy and tactics, current state and future state. And that is you always have a strategic direction. That direction always has to be forward. And you need to have plans, and depending on the outcome of any one of those sets of plans, you can make a decision about additional plans as well as with the future state in mind. So that's what I think. And that is that people get all excited about it and they kind of think it will magically happen. So, and it is in fact very operating to have a clear picture of where you're going to want to go. If you don't actually make plans, if you don't understand where you're starting, and if you don't set up the plans that you want to take to get to that future state, then you're very disappointed and you feel like you have a failed strategy. So it does happen all at once. You really do need to plan for it. The planning part is central. So it's important to apply the idea of strategy to data. Then I'll step back for a moment and define my terms. I think one of the reasons why some organizations have trouble with data quality is that they don't recognize what they mean by data. They don't know each other about it. So I've got several definitions here. I won't read them all, but I will start with the New Oxford American Dictionary definition, which defines data as facts and statistics collected together for reference or analysis. There's a very scientific definition of data, and I think it is a good starting point. When I was trying to come up with a definition to be able to articulate some of the ideas in my book, I wasn't making a longer one because I wanted to bring it into the idea that data is a fraction. So data is part of something to represent a whole of things. It's important that we have every single characteristic of real-world entities with data. We have to choose which ones we want to represent, so data is formulated and created through a set of choices. We have to represent objects, events, concepts. In order to understand the system of representation, they need to be, in play, explicitly defined conventions about what data means, how it is collected, and then in our electronic world, how it is stored and accessed. So when we think about data, we know that data is trying to tell us some truth. It's trying to represent the world in a factual way, but it's very formal. To understand how its form is created is to understand what it is trying to represent and how well it represents that thing. So we need to understand data from its organs in order to really judge its quality. I think one of the things that gets in the way of our understanding of data is that much data that we don't really understand. When we talk about our first name as a data attribute, we understand what that means and we even understand how complex it might be when we know that in parts of the world, the concept of a first name is created differently. So in different cultures, the first name has a different meaning from what it means in other cultures. So we can understand those concepts, but we take a lot of data for granted and we don't actually lay down how did this data get created, why is it in the shape that it's in and the like. So my definitions of data quality are also fairly standard. Most discussions about data quality, especially those like my own that are rooted in the product model of quality, say that data is fit for the purposes that people use it for. But data is also representational and so the quality is also in part, data all well represents those things that it purports to represent. And again, in order to understand the quality, you do need to understand its representational components, how it's created, how it's used, those kinds of questions. I'll be talking about how to assess the current state of your organization. So it's worth spending a moment on the concept of assessment. Assessing is a kind of measurement, but it goes a step beyond the most basic measurement. It can make things. You have to have two things in order to measure. But the purpose of assessment is actually to draw a conclusion. When we talk about a current state assessment, what you're going to be trying to do is to look as objectively as you can at your organization and make observations and to the extent possible quantify those observations. But you're going to draw a conclusion. And the conclusion that you want to draw is framed by the relation between your current state as objectively described and your future state. So you want to find out how far away am I from my desired future state and how do I move toward it most effectively. So we've talked about data. We've talked about strategy. And we're going to say what we mean by a data strategy. And I'm going to step back. In complex organizations, even in smaller organizations, there are a lot of talk about strategies for different components of the organization. So there may be a human resources strategy. There may be a product strategy, a sales strategy. All of these things need to align with the overall mission of the company. And they need to not conflict with each other. So if your product strategy is in conflict with your HR strategy, then you're going to decide what kind of organization rather than moving forward. So a different strategy. One of the goals is to recognize that data supports your overall mission and then apply techniques of quality improvement to data in order to get better data that can support that mission. So we're trying to pull this together to be able to identify actions that will move the organization towards producing higher quality data. And the most important thing is to get alignment between the various groups and the various activities that are supporting improved quality. So some organizations have different names for the strategic initiatives associated with technology as well as with other parts of their plan. And whatever names your organization chooses for these and however they may overlap, again, the most important thing is that you align and don't predict each other as you're trying to move the organization forward. So the mission, again, becomes the central driver. One important thing. Again, coming back to this concept of strategy, we want to intentionally plan for success. You can sometimes succeed in intentional planning, but you may even realize you've succeeded. Whereas if you do, if you do intentional planning for organizational success, you have better ways of making decisions and you just increase your chance of success. So there are 12 directives. These are not any process. I'm not telling you do this thing first, second, third. They are component pieces of what your organization will need to do overall in order to be strategic about quality. Which of these are going to serve your organization best in the short term and in the long term? Which ones you should focus on really depend upon which organization is now and what your role is in it. And so there are certain things that you may be able to do as an individual to get your organization moving in this way. And there are other things where you may need, you know, the direct support of other people. And so, you know, you need to talk to which of these and how to leverage each of these. These are also not very interconnected. So, you know, if you want, for example, to get management commitment to data quality, to data quality efforts, you also work on data as an asset. You will work very closely together because if management sees data as an asset and you can get some numbers around that, then they're much more likely to provide the support you need for data quality improvements, the two you hand and hand. And then you just break up into three sets. The first set is that you recognize the importance of data to the organization's mission. So the second one says to obtain management commitment to data quality. And the goal of assessing the current state of management commitment to data quality is to identify any obstacles to moving the management in that direction and also to identify opportunities through which you could develop that commitment. So part of what you need to assess is how your company's management actually works. And you can do that through things like renewing business plans and either formally or informally surveying the management team with the same questions that I want you to understand what you need to get to commit to anything and then specifically to the data quality efforts. The next thing that is really important when you're trying to obtain management commitment to data quality is to be able to take the language of strategy in relation to data. So your company's mission should drive the business uses of data and you should be able to relate the business uses of data directly to your organization's mission. If your organization does not have a clear mission, if they do not currently have a clear strategic direction, then your job here is much, much harder than it will be if you're working in an organization where they do have a clear overall mission and you can relate data to it. It's not, it's never impossible. And in fact, if you already have strategic thinkers. But, if you're starting, if your organization isn't thinking strategically, then you have an opportunity to contribute to that effort as well. And you can use improvement of data quality as a way of illustrating the benefits of a strategic plan. The next thing to do is to treat data as an asset. So we're going to do this kind of two times in this fork. One is to understand how your company currently talks about the value of data. They're going to be talking about the value of data in comprehensible terms, which usually means, you know, we have much monetary value. Do we get out of our data? What monetary value could we potentially get out of our data? If your company is not talking in these terms at all, you can look at other assets that your company values and determine how do they put value on other assets that aren't directly on the general ledger. And that is a model for talking about data. You can get some hard numbers by looking at the amount of investment that the company makes in data, whether through projects or through the cost of remediating issues or the investment in technology. You can use some scenario planning so you can understand some of the ideas about how data directly relates to your company's mission and you can work out what is the best case of data supporting the company mission and what is the worst case of this data. You know, what if something were to go terribly wrong with this data? What impact would that have on our company mission? And you can then present the data in terms of the benefits of improving data quality and the risks of allowing poor quality data to exist within the organization. And then if you are in a position to do a pilot project, you can use one of these opportunities and define a pretty thorough cost-benefit analysis and tie it directly to the company's mission. So these two perspectives related to recognizing the importance of data to the company's mission are to apply resources to focus on quality. If in the long run your organization is not willing to apply resources to improve quality, then the strategic thinking will not really get out of the gate. So how do you move the company in this direction? So you want to understand the willingness that the company has to formally engage a team. So one more approach to this is to assess how the organization responds to data issues. And you can find out a lot of information about this through things like help desk tickets or incident reports or break fix projects. And then you can give you information on the cost involved but also can give you information on how decisions are made about where and when to apply resources. So conduct a survey of teams that may have the name data quality but who are in one way or another working on the data and trying to make the data better. And if you do know what the problems are, that will allow you to build up the value of data and the like. But you can also identify areas of redundancy. You can identify areas where people may be working across purposes. So you can be looking for potential efficiencies if a team were formed specifically to be responsible for monitoring and measuring data quality. And as was described under Directive 2, looking at budgets, understanding investment in data management and in technology, those things will allow you to put the value of data in monetary terms. The first directive here under recognizing the importance of data is to build explicit knowledge of the data. And as I said earlier, most organizations are not disciplined about how they build this knowledge. A lot of knowledge is lost and people struggle to answer questions about data. So if you move your company toward better knowledge management practices, you need to understand how people currently share knowledge and understand how well that knowledge sharing is working. So if they are working well, then by all means, sell them and use them as models for potential improvements. But if things are not working well then find out how people work around those things. So if there's a knowledge gap, how do people fill it? And you can provide some concrete analysis of the condition of your systems documentation, of the condition of your metadata to illustrate to your management team the number of gaps and why these gaps create risks. The second one is concern with how to apply concepts that have been defined within data quality. How to apply those to data quality. So the first of these directives is probably the one that is most often used in discussions of data quality improvement and that is to treat data as a product of processes within a measure and an improved. How do we begin to do this? The first is to understand the degree of organization already treats data as a product and then to assess the properties that create data in order to hide those opportunities within that production process to improve the quality of data produced in the organization. So one of the most important things in this kind of activity is to understand whether there isn't any documentation of current processes. How is your data chain informed if there is documentation, then identify gaps. Again, celebrate the fact that there is documentation if it exists. The other thing is to survey business and IT leaders and other stakeholders about the condition of the data and put an emphasis on the knowledge of the interaction between systems. So you might find that people are confident about the data in one system, but they lose confidence outside of that system. They lose confidence for a range of reasons. Sometimes they know one system much better than another. Other times they realize that the data is different and they are more familiar with the data in one system rather than another. So you need to be focusing, as you think about data as a product, you need to be focusing on the ways through the organization and where people understand it and where they don't. If you do find examples where processes are well-defined and data quality is measured, then find out how they got to be the way that they are and use those as a model for other improvement opportunities. Directive here, again, is fair and clear that the way of data is defined by data consumers. We often talk about product quality in terms of the customer defines what is right. If you understand how to move your utilization towards enabling consumers to really define what they mean by quality, you need to define data consumers and you need to ask them directly about the condition of the data and you need to find out from them which data is most important to them so that you can understand where they are coming from in terms of quality. So you can survey them directly and you can also find objective information about data access and use through access logs and again, review of things like help desk tickets or issue tickets so that you can find which problems cause the most pain for your data consumers. The set of directives in this set are that you should address the root causes of data problems and that you should measure and monitor critical data. So when you talk about addressing the root cause of data problems, again, this is a question of your company's cultural orientation to solve problems in general. So how does your organization solve problems of any sort, not necessarily related to data. Every time you work together, what kinds of behaviors get in the way of solving problems and which ones enable the organization to solve problems. And you, again, can find a table where you can produce a cost-benefit analysis on the remediation of a problem at its root cause. Then you'll be able to show that example to others in the company and begin to build a case for remediation of root causes rather than remediation of just symptoms. But if your organization does tend to treat the symptoms and not the root cause, then there is a bit of work in trying to reorient them. And then finally within this is to measure the quality of data and then monitor critical data. So you can understand whether the company currently does do a data quality measurement and if so, what's been successful and why. You're not currently measuring the quality of data then I would suggest you look at how your company measures other quality characteristics of their products or of their processes and again try to understand from the perspective of cultural behaviors that you might be able to apply successful measurement approaches to data. The data quality assessment framework which I outlined in my book provides a range of ways that you can measure data and that you could produce a cost-benefit analysis for a project or a proposal to move forward with tackle data quality activities. The third set of directives are at building a culture based on quality and to strategically manage its data. The fourth set of hold producers accountable for the quality of data tied to your usage of the data chain and you need to understand your company's general approach to accountability. You need to understand the data chain in order to identify ways that you can improve the relationship between pieces of the data chain. So you may be able to identify gaps in communication. You may be able to identify opportunities if your company responds well to performance review goals and you can move towards incorporating goals into performance evaluations then you can give people a direct incentive to be accountable for the quality of their data. Directive 10 is to provide data consumers with knowledge that they require for data use. And again, this ties back to moving forward a knowledge-based approach to data quality. So you need to understand how data consumers are currently informed about the meaning of data and the production of data. You can use the same kinds of assessments that are described in directive 4 and directive 6 to understand how much time is spent if this knowledge is not available in a direct way. And then you can ask them what would be the best way to improve your ability to understand the data. The final two objectives bring us back to strengthening and also to the overall emphasis on cultural change necessary to actually implement a data quality strategy. The two go hand in hand. So as we're talking about the value of data in your company, you need to be understanding the potential in your industry. And you need to understand or your organization needs to understand how to prepare for the future, not only in terms of data, but also in terms of their systems and in terms of the activities that they want to engage in to fulfill their mission. So identify major trends in identifying the risks with the existing apparatus, your technology and your data can help you identify potential future risks. And when you're identified a set of risks, you don't have a basis for planning to mitigate them or take steps to eliminate those risks altogether. So in directive 11, the many are probably familiar with the book The Art of the Long View, and this is probably the best there for looking at best and worst case such as in an industry applied not only to at the industry level, but also at the data level. And so all of these suggestions bring a significant amount to trying to move your organization culturally toward a focus on producing better quality data in order to meet your company's mission. And at every opportunity throughout your engagement with any of these directives, consistently you can make that connection between your company's mission and how you produce the stronger argument you will have for many kinds of decisions that will improve the quality of your data over time. In these current state assessments, you are going to want, you are going to discover a number of things. If you do even one of these, you're going to find out something about what data is most important and what's less important. You're going to find out which processes are working well and which aren't. And get insight on the behaviors in your company that support your movement and the behaviors that you get in the way of strategic movement and of improved quality. You need to sit down and synthesize them into a set of prioritized recommendations and a proposal for how to implement them. And in most cases, most of us do not work alone. And to work like this, I think a collaborative approach is really beneficial because you can get multiple perspectives. And if you've got a small team who were in a position to review the results of even one of these assessments, you go through all of your help desk tickets and identify patterns in the problems that you did are present in your data. Have multiple perspectives will give you a deeper level of insight about ways of solving the problems themselves and also offering your company forward with the context of its mission. So I'll go into this discussion about strategy just as a reminder that you need to start where you're at. You need to understand where you want to be. And then you need to make decisions based on where you want to move you toward where you want to be. So that is the end of my formal slides. And as I said, I wanted to make sure that we have 50 minutes or so to have conversation. So Shannon, do you want to facilitate the Q&A? I already have a couple of questions come in. Just a reminder to everyone to submit the questions in the Q&A section in the bottom right-hand corner of your screen. And one of the most popular questions that always comes up is will people be getting a copy of the slides? And just a reminder, I'll be sending out a follow-up email within two business days containing links to the slides and the recording of the webinar along with anything requested throughout the day. Shannon, Mara, so the first question that came in is could you expound on how data has quote-unquote shape? Yes, Mara. We talk about data structure and use that term structure. We are really referring quickly to the table in which data is stored. But I think that data has shape even before we put it into tables. So let's take if data is a representation of characteristics of real-world objects and you as an individual person are a real-world object, then you have different choices about how you can represent yourself. You can represent yourself using just your name, your first name, or your first name and last name. You can include a title. You can describe physical characteristics, how tall you are, how much you weigh, what color your hair is, those kinds of things. You can use all those things to represent yourself. In some contexts, you may use a small or larger subset of those things to represent yourself. If you are going to the library to get a book out, you can hand your library card over and get that book, and no one is going to ask any questions. But if you want to fly on an international flight, you need to have your passport and other data about you. You're the same person, but the context is different and you need to represent yourself differently in those different situations. The decisions about how to represent ourselves or how to represent any object give the book of data a shape. As I said, the selection of which attributes to represent is in part what I mean by the shape of data. Does that make sense to me? I can ask the question, or if you want to make a little note to me on how we can ask Laura to expand that. Now we've got a couple more questions coming in. Laura, what are the size and roles within your data quality organization? Yes, so I have a team of people recently, three direct reports, and we have responsibilities for implementing many projects, monitoring the results of ongoing measurements, and providing guidance for implementation projects and providing guidance for the development of metadata. We have a team of the four of us that are different, and we realize that data quality efforts can be organized. We have IT roles, and so we have traditional IT roles, like business analysts who are responsible for defining requirements and then our developers who actually write the code and our QA people who do the same thing. So our team is really very much focused on the development part of data quality. Great. Next question. Can you expand on how you were successful in getting the organization excited about recognizing data as an asset? You know, through all the 12 directives, I've noted that if you can come up with an example, or if you can show a cost benefit, then you are going to get a better response. And I'll tell a short story here. When I first started, my boss, you know, took me in the first day and said, your job is to figure out how to measure data quality and report on it. So I got into the warehouse and I talked to people about what the challenges were and what they thought of quality and the like. And with a very small set of measurements, there were four of these measurements. And I requested that we would get these measurements automated and some people saw value in automating the measurements. So I never thought like that. I never thought ahead. And we were sorts associated with the measurements. So the first time that I got an alert on the measurement, we went to my boss and then we went to the manager in charge of our ETL and immediately assessed that we did have a big problem that he had action on immediately and he started the ETL process. And we looked at the projects that had been put in place to do the same, you know, to fix similar problems. And we saw that even the instance of such a project would be, you know, a $350,000 fix. So I actually did that analysis and said, wow, the last time we fixed this problem, you know, it cost us $350,000, it didn't fit me in a different way. It was a very hard number, a concrete benefit of having that measurement in place. And so there's a sense that, okay, we have, we know that people needed to do their jobs and we have just saved the price number of dollars. We could apply that to other parts of our warehouse and we did. And people began to turn to see, we had more control over the quality of data than we had previously thought we had. And that made them see it as not something that just got shoved into the database, but they saw the real thing that people really wanted, what people needed was the data. Okay. And then just kind of looking at those lines and nice progression. Are you using any third-party tools to measure data quality? We're not using third-party tools. The measurements that we have in place for the, I should say for in-line measurement, we're not using third-party tools. We've got processes to collect data in line and to do a set of comparisons so that we can determine whether the data is consistent with past results or consistent with defined thresholds. We're not aware of a third-party tool that does measure data quality in line in the way that we wanted to do it. We do use a profiling engine to do initial assessment of data and we're not using in-line stuff to build. The structure with committee and other forms separated from data governance committee to discuss data quality issues in dashboards? I really think that data quality and data governance are so intertwined. But there's not a lot of work to do in both areas. So I would say you can look at your organization and you can understand where you're starting in terms of either data quality or data governance or both. The original vocabulary and how it advises data governance and people are committed to that and that means that they're committed to the means of getting movement on improving the quality of data. If you're talking in terms of data quality and you can use that commitment, you won't be more successful if you have the components that are often associated with data governance. So when I say build a culture committed to data quality, one thing that is through having a strong data governance organization where you have stewards who are paying attention to the quality of the data and they know the data really well and they know what's most important and what the repercussions are if something goes wrong. So I see the two as very much connected. Sometimes people try to separate them, sometimes people try to put one above another. It's not theoretically worth it. But within your organization, how can you use the concepts associated with data governance and the concepts associated with data quality to make your organization more successful? And I'm just looking, sorry, I'm looking at the other part of the question there. And with formal structures and how to set up committees and such, I tend to think this way from the perspective of your culture and how your organization works best. There's not a magical committee structure for data governance. You work with what you have and then bring the best practices that are most suitable for your organization culturally. Right along those lines Laura, does data validation fall under the umbrella of data quality management or is that a function under quality assurance? It sort of depends where that's happening within the software development lifecycle. If you're developing, you know, the software and you have a QA team that can, then they, I think they should, as part of it should be validating the data as well because how else can you tell if the software is producing expected results? We have processes with assets, you know, assets that have already been signed off by quality assurance and are in place where we do validity checks on the data on an ongoing basis. I'm afraid that brings us right to the top of the hour. There's some fantastic questions and I'm sorry we didn't get a chance to get to them all, but I want to thank everyone. And Laura, thank you for this great presentation and thanks everyone for attending today. Just a reminder, I will send out a link to the slides and the recording from this session within Tuesday's business day, so by end of day Thursday. Laura, thank you so much. This was really fantastic. I appreciate it. And thank you for all the great interaction and all the great questions and that just makes it all a bit. So again, I'll get that email out to everyone and I hope everyone has a great day. Bye.