 Hello and welcome and happy new year. My name is Shannon Kemp and I'm the Chief Digital Manager of Data Diversity. We are proud to produce this webinar series of Data Governance Case Studies for the Data Governance Professionals Organization. And we'd like to thank you for joining today's DGPL webinar, Lessons Learned from a Mature Data Governance Program. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we will be collecting them by the Q&A section in the bottom right hand corner of your screen. If you like to tweet, we encourage you to share highlights or questions via Twitter using hashtag DGPO. Now let me turn the webinar over to Michael Delgado from the Data Governance Professionals Organization to introduce today's webinar and speaker. Michael, hello and welcome. Hey, thanks, Shannon, and happy new year to you. And thanks to you and the Data Diversity Team for always making these wonderful webinar productions for the DGPO. We really appreciate it. So thank you everybody for joining and happy new year. Glad to see so much excitement and interest for data governance as we kick off the year. So just wanted to give quick highlights. So we are the Data Governance Professionals Organization. We can find us at the dgpo.org. We do offer memberships with some benefits that I wanted to share with you real quickly. Definitely the ability to network, share, and interact like we're doing here on this webinar with each other. One of the main reasons I joined the opportunity to collaborate and interact with experts such as Susan that we'll be hearing from today. Also being able to participate in joining industry-specific working groups that are set up through the DGPO. We do host a lot of teleconferences beyond these webinars where we focus on specific topics. We have member-only access to certain content including today's presentation and recording. So just proactively letting you all know that you will have access to Susan's presentation and information on the dgpo.org members-only website. The DGPO also puts out a courier newsletter and we also offer discounts on training classes, seminars, conferences, and other products. So please do check out the dgpo.org website. We also are excited to announce that we're currently collecting submissions for the 2018 DGPO Data Governance Best Practices Awards. So if you go to dgpo.org, the link will be right there on the front page. Those submissions are due by January 31st. So just a few weeks away, so we're looking forward to hearing a lot about each of your governance programs and lots of excitement to announce the 2018 Best Practices Award this year. So speaking of best practices, we wanted to introduce our speaker, Susan Gaiman. Let me pull up here. So Susan, thank you for joining. Susan, the CPA and risk management director for the Enterprise Data Governance Program at Ally Bank. Susan brings over 25 years of data management and governance experience. She's currently focusing on integrating data governance successfully with big data analytics and analytics, a very hot topic of bringing those two worlds together. Previously, Susan's built out a complete enterprise data governance structure for Ally Bank, creating a team of professionals to support the cultural shift to data management and ownership. Data governance in her world includes the development of a business glossary and to end lineage and data quality testing on a reoccurring basis. Prior to her role at Ally, Susan was an internal audit executive with a Fortune 500 company and within the healthcare industry focused on converting corporate culture through value-added integrated audit approaches, instituted data mining, constructed audit methodologies, and infrastructure reporting mechanisms to improve that internal audit function. Susan, I'm personally very excited to hear about this governance program and how you are maintaining this and we've all heard about starting a data governance program. You have one going, it's mature and learning about those lessons is something I'm personally looking forward to and know our attendees are as well. So I'll turn it over to you as you set up your show and welcome, Susan. Thank you so much for your time sharing with us today. Thank you. What a wonderful introduction. If I can just get this screen display to just give you one slide to look at, it could be more effective. My clicking is not helping. If I go back to my original, there we go. Bring that down. All righty. Yep. Yeah, we're good to go. You're all set, Susan. All right, so today we're going to cover some really hot topics. I'm with Ally Bank, which is a digital online banking system and one of the number one auto lenders in the United States. So what we want to share today is a lot of lessons learned as we have grown this program starting way back in 2011, 2012. So these are the key concepts we're going to talk about, the importance of a solid framework, responsibilities around decision-making bodies, because there's a lot of them in the space of data governance, consistent communication for lasting commitment, and key steps to achieving data quality, shifting from results to value with managing data quality exceptions. So this presentation was actually built to provide you with a lot of content to take away with you. We're not going to cover every single point on every single slide, but that content is for those of you that are building and maybe you've already progressed into a structure of data governance, but things you can take away and hold on to and use in your day-to-day world in your own companies. So obviously I'm from the banking industry, so this just gives you a quick overview of the various product lines we have at Ally. If you're interested, don't hesitate to go out to ally.com to learn about our various product lines. But a very innovative company we're really into digital financial services. We have zero branches. We make money when you make money. We don't make money off of you. So that's one of our mottoes, along with do it right the first time. Saves you all a whole lot of money. So hopefully some of our lessons learned can help you save some money in your development. So real quickly here, I'm going to go through Ally's EDG program evolution. And as I said earlier, we started way back in 2010 and 2012 to stand up a form of data governance to achieve Basel II regulation compliance. Fortunately for us, we sold our international business about six months before that target date to go live with that structure for Basel II. And we became a North American company exclusively, which took us out of the picture for Basel II. But our executive branch had learned so much about data governance in that space that they wanted to pursue building that within Ally in the go forward position. So although we learned a lot during those Basel years about what the federal regulators were looking for in the data governance and data management structure, we didn't want to just take it from there. So we deployed a data management maturity assessment that was first put out by the EDM Council, the Enterprise Data Management Council. And we ran it across the entire corporation, both from an IT perspective and a business perspective. And we gained a horrendous amount of knowledge about where was data management within this company and where were all the pockets of success and where were all of the gaps that we didn't really have any policy or procedures around. And we took all of that information along with what we saw in regulations, along with what we saw from an internal control framework, looking at a lot of internal audit materials, and created the data governance policy that we have today. So really, after those scores came out, we actually compared our scores to an industry benchmark. And we did all of that in 2012. So we really got our feet underneath us in 2013 where we deployed the policy, the framework, the tool set, the methodology, and rolled out all the roles and responsibilities and really got our first critical data element initiative off the ground. Over the last three years, we've accumulated about over 2,000 data elements that we consider critical, that we apply all of our methodology steps to. And our tool set is corporate-wide at this point. So every member of the corporation, whether they're a contractor or employee, has full visibility into the business glossary they can also go into the lineage piece along with data quality. So we're trying to really force that culture shift across the company by having the tool set corporate-wide. A lot of the things that we're looking at now in our move forward position is really merging our tool set and suite of capabilities into our big data lake. So as we've evolved to create that big data lake, we're now kind of doing an interaction between the two structures where we can say in the big data lake there's a tag that identifies it as a critical element, which informs anyone in the corporation that it's within our tool set. And you can play between the two tool sets and identify what is in the big data lake from our business glossary and vice versa. So obviously, this is cutting-edge stuff that continues to change as technology changes. You know, we're still working on defining a really strong master data management strategy. We're also standing up a customer information analytics as a part of that lake and how are we going to interact and apply our data governance structure to that? And how can we be successful really from that master data management position around data security, data in movement, ETL structures, architecture, kind of keeps going and going. But let's move into some more interesting stuff. So our enterprise data governance policy as we first established way back in 2012 was very focused around a framework. And when I say framework, that's really roles and responsibilities and groups of people that help to make those decisions and drive the momentum for the culture shift. Consistency and alignment is really around the efficiency across LA to support the data governance framework. So that's kind of the nice words to say you have to be compliant and what is the criteria for you to be compliant as a member of the company. Then data quality really gets into our theory of data quality, which is testing data at 100% of the volume on a reoccurring basis. In our world today, we're doing it on a monthly basis. And we're taking it all the way through remediation and monitoring that. So in standard section, we really focused in our policy around those key things that we didn't see in other enterprise policies. So things like proper documentation, data flows with controls, data quality remediation efforts. If you've got bad data out there, there are reasons you should fix it and risk acceptance if you can't. Down here at the very bottom, we're showing related topics that in our discovery through that data management assessment, we identified enterprise policies that were more than sufficient to cover these key aspects of what you should have in your data management bucket. So rather than recreate a world, we just tagged these enterprise policies to somewhat be linked with us to give that overall picture. As I stated, you know, our framework I think was the key to the entire policy, was really laying down the rules of who's who, what are their responsibilities and accountability, and what role do they play in the data governance framework. So we stood up what we call a distributed model versus a federated model, and we'll talk about that in another slide. But again, I'm not going to go into all these details, but if you get the presentation, this really kind of gives you a high level look of each area's responsibility and role. And if you look down at the bottom, those are our key roles, a data governance business lead who's the data owner from the business side of the house, a data governance tech lead, which is our technical expert around that system of origination or system of record, and then our data consumer. So that's the wonderful person out there plucking that data and fulfilling some critical report out there in the world. And they play a key role that is a part of the overall process. All three of those roles can roll up into a line of business data working group that's specific to a product line or a line of business. But all of them across the entire corporation create the data stewardship working group. And obviously, you see data governance, and they're kind of almost to the middle but near the top. So we're really the strategic arm of the whole structure. So we drive the strategy. We create the policies and the standards and the procedures and the tools, and we go out there and educate and hold hands and help them all get off the ground. But we have Enterprise Data Governance Council, and that's, you know, a group of top executives that we are responsible to. So they're really more or less our decision-making body. But in our early years, we had an executive sponsorship team, and I highly recommend that. I think the tone at the top is really the only way to get this off the ground. Without the executive C-suite behind you, no one's going to prioritize the work to actually get it accomplished. So that's a critical lesson learned. So this slide depicts our current overall approach and methodology. But I have to admit, you know, again, this is a key lesson learned. When we first started our first critical data element initiative, we took a critical external report that was a regulatory report, and we attempted to reverse engineer it. And we worked with the data consumer, and they provided us with their source data elements, and they defined each one and where they took it from our technology structure, and we thought it was gold. And that's kind of the initial methodology that we created. However, as we started, we found, number one, that most of our report prepared a source from a warehouse, not from a system of origination. So if that data element is corrupt or bad in that warehouse, it cannot be fixed in the warehouse. It has to be fixed way down that chain at the system of origination, and then passed through any other applications to that warehouse to be fully corrected across the multiple systems that we have. So that was a huge lesson learned for us. We worked the lineage down to the system of origination. We contacted an owner perspective and gave them the role of business lead data owner at the system of origination. He read that list. He found that they were plucking the wrong elements, and their name and definition was exactly opposite of what the data element really was. So our methodology today is we do lineage first and foremost. We start with that data consumer, and we work our way all the way down through our technology sections to that system of origination. In that process, we also provide them with the data flow with internal control. As a kind of a give back, it's also one of the most commonly requested documents by external auditors and federal regulators. Then we've identified that system of origination, so then we apply a data governance business lead steward to own those data elements, define them, and bring them up in our glossary. Our last piece that we do is develop those data quality rules that go through multiple tasks before they're finalized, but then they go into our production arm and they are run on a monthly basis and must be worked and remediated by that data owner or his delegated staff on a continuous basis. So this is a quick view of our tool. One thing I think is a key lesson learned is there's no tool out there in this world that's going to give you everything that you need. We found that as the years progressed, we continued to add tools that some of them we actually created internal using our technology expertise. So we do have a lot of lineage tools in our bucket today because we tried to do our lineage in an automated fashion, but we also have manual lineage that's in the source-to-target Excel spreadsheet and then loaded into our tool set. Our core tool includes the business glossary and the lineage views. And our third tool is our key data quality testing tool. Everything's feeding into a centralized Oracle database. And we custom-built a reference data hub to hold our reference data codes and their meanings along with a data quality exception tracking tool. So this helps us monitor and manage and work with our data owners around those data quality exceptions. The last piece that we built was actually a bridge process to our corporate issue management system. So if we have a high severity data quality exception, we have a platform to bring it into issue management and get strong leadership attention. So some of the key parts of decision-making in a data governance structure is there's multiple pockets. You can go from the top down, you can go from the bottom up, but really to be successful, we did it from both angles and pushing from both ends. So we kind of started at the top with our executive sponsors and internal leadership. We sold our strategy policy standards and our methodology to them. And then we went down to the bottom of our framework. And we worked with those data owners, business leads and tech leads, to really drive from the bottom up. And we meet in the middle that's really at our council level. And our council level consists of the key executives that really own these business leads and tech leads. So once we got their staff committed and we've had the top of the house committed, the council was easy to please because they were getting it from both angles. So they were kind of cooked at the beginning to accept the change in culture and the change in priorities and our needs for their resources to get the work done. So this is what I was talking about between a federated and a distributed model. Here at Ally, we stood up a distributed model where we are relying on experts within the line of business. We are not their leaders. We have no span of control over their day-to-day work. But they've signed up for the role that we put in our enterprise policy and they fulfill those accountabilities. But there's pros and cons to either way you do this. Obviously, we have no increase in FTEs. We keep the cost down from a corporate perspective. However, we're challenged to keep the priority up for these people. We are also challenged as they change roles and positions. We're continuously training people to take on their work. They're extremely knowledgeable people. They really know their data. They own it. They feel responsible. So it's been very successful from that perspective. In a federated model, you could have dedicated resources in each line of business or each product line that does this work from a data governance perspective 100% of the time. That is high cost to your corporation. And it's very difficult in that type of a position in the federated model that you find a resource that actually has authorization to work on data quality remediation in the system of origination. Because they'd like to say that data governance person isn't really in the operations of a various line of business or product line. So that can cause some pickups right there. So one of our, I think, greatest lessons learned is we realize as we grew that we have all these various channels of communication and we have to fulfill every single one in order to be effective and keep the gauge moving down the road. But we quickly realized that we were going to consume our entire Central Enterprise Data Governance Team and just fulfilling these meeting requirements if we didn't get stronger organization around it. So, you know, here's Enterprise Data Governance at the top and we are creating and driving these key meetings on a reoccurring basis. In our early years, we are meeting the Stewardship Working Group on a monthly basis in the Council every other month. Here we are years outside of that. We meet with the Council Quarterly. We meet with the Stewardship Working Group every other month. So, you know, your communication is really tight in the early years and then you start to slow it down as you actually achieve that culture shift. But the key to success is having standardized metrics that you can use across these multiple groups and communities. You know, in the early years, it's really hard to show that you're making progress because it can appear at the snail's pace. However, you know, participation is a metric. If they're attending the meetings that they're required to attend. If your framework rules are taking the training that's required, that's a check mark you can promote as a metric. How many, you know, critical data elements have you collected and recorded in your glossary? How much lineage capture counts, data quality rules, all those kind of metrics, you can utilize in any one of these community meetings. And so that's the way you want to structure yourself so that you have consistency in your staff time in building all these various decks. This is an example of our communication plan. And so we try to use the same metrics in multiple places. However, the key messages change based on who your audience is. So we created a very detailed plan so that we could not only, you know, keep everything together, but also to plan for when does that deck have to be written, how much time do we have to review it, who are the review people, when's the finalization, so that we're not crisis managing these very important communication channels. So I highly recommend if you're in those early years to build yourself a strong communication plan, put in your target dates and your responsible parties across your team, and look for ways that you can really synergize on those core metrics to utilize them at a some level, at a detail level, and fulfill the needs of the various partners you have throughout your community. And of course, there's many other ways to communicate outside of meetings. We've done video recorded, training sessions, file sharing team rooms, data governance newsletters, and obviously every company has an intranet site. Here at Ally, we have a very large one and a large presence on the intranet available to the company. So data quality. Now that's your true value add for the whole data governance structure. Yes, great to know what a data element really means, what's its definition, where does it live in the warehouse, what's its lineage look like. But in order to add value, because we don't provide a return on investment as a data governance employee, we provide value through data quality and being able to show the corporation and any external auditors or regulators that our data has integrity and accuracy at a very, very high level. And if they want to dig into the details, they can see the documented evidence of that level of quality of our data. That's priceless to a bank and to many industries throughout the world. But the structure has to be, you know, doable and workable. And so that's, again, one of our key lessons learned is getting back to that data owner at the system of origination and saying you own this data. Because he's got the authority, he's got the staff to really manage the data that's within that system of origination. So our methodology covers lineage and glossary development, and then we take them into data quality development. This does take a bit of hand-holding. We profile all of our data first. And then we work directly with the data owner, the data consumer and the tech leads to formulate these business rules that really give us a relational test that tells us this data is valid. Every rule we develop, we classify in one of these data validation categories, because, again, as we're spinning this mass amount of data quality results, we want to be able to slice and dice it and articulate what's dropping, what's doing well, what's not doing well and why. And so that's when we learned we had to build something for data quality exception management. We got this all off the ground. We had the reports spinning. We had dashboards and all those great little things, summary reports, detail reports. But we didn't know if anything was getting fixed, except that we kept seeing the trends of quality going up, up, up, up, up, and the invalid trends going up, up, up, and we couldn't see anyone fixing anything. So we created our own tool called the data quality exception tracker that forced our data owners to go in and on a monthly basis tell us how are they fixing this data. Did they do a root cause analysis? They document it. Can they fix it? Do they need risk acceptance? You know, there's lots of packets in this data quality space that really become critical. You want to ensure that your data consumers have the visibility of this data quality because they're the ones that are going to throw this data in a critical report. And if it's bad and they do it blindly, then everyone's in a pick off. So our data consumers have access to all of this information. They now know who the data owner is and can pick up the phone. They can pull that information down themselves and see it at the account level. But your data quality reporting has to be built for, again, just like your communication plan, multiple audiences. We try to do everything at the line of business product level. So our commercial auto has their own dashboard and report. Our retail auto has their own set. But they can look at each other at their leisure. Only the data owner has the ability to delegate remediation to other staff. So our tool allows multiple people to be responsible for the cleanup of the bad data. And we have the ability to really manage that remediation process. We can monitor the aging. We can monitor... We classify our exceptions today by high, medium, and low with a calculation. So we're actually defining that for the business. And if you're a high, you have 15 days to resolve this data problem, or we're going to escalate it into our issue management structure and ensure leadership is fully aware of the problem. And in some instances that the value adds, because now you've got the executive leadership, understanding the importance of the exception, and willing to fund a technology repair, which is often the root cause of the problem, right? Risk acceptance is something that we've had to design into this. You're going to come across bad data that may not impact your critical reporting, may not impact your financial statements, maybe just a minor nuisance. And you have to have a venue in which the right leaders are accepting that risk and attesting to that acceptance and very well documented from, you know, an audit perspective. We'd love to see all that documentation. We don't let them close their exceptions in our tool until the next run of the test in which we see a zero exception volume. So we do have many exceptions that age for multiple months until they can resolve the root cause and bring that into complete closure. We have some that never close and are in the risk acceptance mode. But again, that is then communicated all the way up the leadership chain so that executive then is signing off on it and it can go all the way up to our Enterprise Risk Management Committee. So if one leader thinks it's okay and another leader doesn't, at least that conversation is occurring. And sometimes we've seen them spin all the way back to risk from risk acceptance to we are going to fix this, we're going to remediate it, and here's the new target date. This is an accumulation of lessons learned that we have experienced over probably the last five years. So from a governance perspective, support from the top of the house is absolutely essential. And obviously, that's essential in your early years, but even still today in 2017, I'm always looking up to that executive branch for assurance that this is still a priority this is meaningful to the company and they're deriving value from this work. Obviously, we're now moving into the big data lake and the customer analytics space and the company really feels the value now because they want that top quality data in those spaces. Data quality will require triage teams to join technology and business to really figure out the root cause of a problem. So every company is different. Here at Ally, our information technology is somewhat siloed and separated from the business. So we have created these triage teams to work through data quality exceptions and it's really seen a great improvement. So don't hesitate to pull people that don't belong together together because I think there's a lot of benefit to rise from that sharing of knowledge. Communication channels are critical for keeping all groups informed on track and on the same page. Data quality testing is the true value add which I already said. Testing results should be obtainable by all interested parties. So you can't fill this in a silo. You can't make things exclusive. At first people are very hesitant. They feel like you're airing their dirty laundry publicly. They can be very resentful. So when we first started our dashboard reports we wrote them in the positive and what I mean by that is our dashboards reflected the percentages of valid data elements. Well here we are. Four years in we're solid. The culture shift has happened. Now we report the invalid percentage on every single graph. So sometimes you have to start with more of the positive view to get the acceptance out there and then you spin it back around. So instead of looking at 99% on all their charts now they're seeing 1.5% of bad data. 2% bad data. 0.5% bad data. So it doesn't shock them anymore and they're accepting of it. But in the early days if we had done that they would have thrown us out the door. Integration of exception reporting in the enterprise issue management process. If you're in the banking industry or any other heavily regulated industry like healthcare, there's several out there. This is where you really need to build that bridge into your enterprise issue management process. Try to keep things within your own tool set for as long as you can. But if they are a high risk to the business use the venue to get it up to the executive branch. We found that we could not rely on our data owners. Even our council members would rather hide the report under their desk than to take it to the C-suite and disclose where they were at a data quality level. So that's kind of a hard thing to sell but if you follow whatever your corporation's enterprise issue management process is you're following the standard that they created for the whole company to use it. And that way it's no longer within your span you didn't do it so to speak. You just followed the proper policy. That's what we do it. Lineage will always be a challenge. Let's be honest. We started off at a manual lineage level and now we're at an automated level. We have several tools. We're continuously purchasing new modules to our tool as the ETLs that we use throughout the company are ever changing. Definitely attempt to institute data flows with internal controls as a service to your customers. You can provide them in the early part of a critical data element initiative that adds value to their operation. Something they can hold on to and take ownership of after you've completed it for them. So it kind of builds that trust relationship. And partner with your architecture and your integration services teams. Here at Ally again our information technology is somewhat siloed and we are really focused around understanding the full inventory of ETLs and a full inventory of our applications across the company so that we can make sure we're not missing any hoot pops that our data is taking. Capacity, critical data element initiatives end up drawing upon the same time constrained resources over and over again. So as you saw our framework has designated roles. Those poor business leads down there in our auto space are hit all the time. So the minute you start to do an initiative with your compliance unit or your regulatory reporting or your combined corporate reporting areas you're going to cross every product line you have in your company because all that data is being rolled up to fulfill the report. So that's something that is somewhat difficult to manage. Be aware of it going in and proactively manage it so you're not killing any one resource so that they're going to shy away. Data quality remediation may require additional resources. The data owner can't always do it all himself and never will be. So design your tools and processes to accommodate others and allow that person to do the delegation. Tools and technology again. Increasing demand is requiring improvements of tool sets and capacity on a reoccurring basis. We just did a major upgrade here to a higher version of our tool set and again as I mentioned earlier in the presentation we've created our own software internally to fulfill the gaps we couldn't find in the open market and our data quality exception tracker and our reference data hub are two prime examples of that. Currently we are working on the design of a data consumer tool to add to our suite and to help to ensure that they are getting the information they need to make smart decisions. Really struggle the balance between self-service and full-service. You know there's always got to be some member of EDG in the mix. Obviously we want accuracy and consistency across our tools so we are constantly reviewing materials before they're loaded. Try to launch your tool set corporate ride once you have a base of content to share because that's what really helps the culture shift and builds the momentum for people wanting to do more and more work in the data governance space because they want to see their data out there and viewable for the rest of the company. Now we can open it up to questions. Thank you so much for sharing Susan. We've got quite a bit of questions. I'll try to put a little bit of context around that I personally love the ability that how you guys started with compliance and then the company really bought into the value of data governance standalone not just from a compliance perspective so that's fantastic to hear. All right. So a couple of the questions. I'll start with one. Since we talked about you starting with compliance what role of any does compliance still play and then related to that Allied Bank, U.S.-based and GDPR something that you have to start worrying about in your function and where the company stations. Right. So actually we our first ACE engagement was in 2008 here because we're allies so Allied Critical Element we've marketed the brand name of ACE here which is real snappy but we actually started on a regulatory risk report known as the CCAR report and we reverse engineered it across every single product line because it's a giant report that you've fulfilled federal requirements with and as a result of that we got that built and we stood up data quality and the compliance department was paying very close attention and they were the first at our door asking us to support them because they were mass consumers of data in regulated area so in banking you have your anti-money laundering area and so they were one of our key focuses. Again, when you work with compliance you're going to cross every product line in line of business quickly. Our fraud unit was the second to the door. Obviously they had a lot of data that they were consuming to look for fraud and also investigate fraud accusations so obviously we wanted them to have the most accurate data of both parties and we actually tweaked our definition of what we deemed as critical to include any regulated processes. So any data consumed for a regulated process was just as critical as data that would be consumed in a critical report. So it seems like even today you continue to let compliance drive criticality of the data elements and what your focus areas are compliance is still a key driver. Yes, yes. And a key partner, right, because they're a data consumer area. They don't own data. They're consuming the data across every line of our company and so it gives you the ability to spread your data governance structure fast across the company because you're going to hit every packet because they're taking every packet. So actually a good angle. Great. And then related to your organization thanks for sharing some of the structure of questions specifically around you know, ally banks having a chief data officer specifically, CDO can mean various things but questions specifically around a chief data officer if you have one or not. Ah, wonderful question. No, we do not. Obviously being a data management expert, I would love to have a CDO position here. We are currently structured underneath the chief risk officer at Ally. We do not have a chief data officer position. We have talked to the executive branch on several occasions and they don't seem to have an appetite for it. But we did just this past year in late 2017. They stood up a chief strategy officer position that was filled and we've been meeting with that woman and she's all excited about getting a chief data officer. So we're hoping in the future if the executive branch approved that position we would, you know, move out of risk our entire structure and move it right under the CDO which we believe is the appropriate place. Okay, and within your organization, your function then aligned with the chief risk officer that's considered more of a business function than an IT function then. Correct. And I do have majority of my staff is what we call our business staff and then I have an entire tech department too to support my tools that yes, they're dedicated. Just sticking with organization and, you know, maybe not just your function because I know you said that you also have, you know, colleague or peer data functions as well. How many full-time employees are dedicated on just data governance versus having other day jobs? Right. So I would say, what are we at right now? 15. I'd say about 15 including my tech side and my business side. Go ahead. So we have our data quality team and we have half of that team onshore in the United States and we also have a core team offshore so that we're running 24-7. And that's how we really keep up. So we're running at night and we're running at the day. True 24-by-7 data governance. So sticking with roles and responsibilities of the roles that you shared there which role specifically is the one accountable for documenting the data lineage to some of those critical reports and which one of those roles is actually doing that work. Right. So the data governance tech lead role in our world is responsible for that lineage. And they also, you know, once we have it in the tool set whether it was collected in an automated fashion or manual source to target mapping, that tech lead is responsible for testing to the accuracy of that lineage in a go forward position. And ultimately our manual lineage is attested to every quarter in our automated lineage is reviewed twice a year in June and then in December. All right, great. And you spoke a little bit about the data quality exception process and that being one of the key value ads. Can you talk through and maybe share some example of some of the data quality remediation that you're doing and maybe share some examples depending around what specifically you're providing remediation for? Right. So as we see a high volume data quality exception, you know, I'll use an auto example. So for instance, a customer's home phone number. We have a data quality rule testing that field. It should be 10 digits. An area code plus the seven digits, right? Well, the test is is it 10 digits and is it all that, I should say thousands, that do not meet those conditions. So we took it to our auto triage team because our retail auto business is huge here at Ally. So we have many lines of businesses involved in running that product line. So we bring them all on a monthly basis to a meeting on data quality exception. So we've got those tech leads in the room that belong to IT. We've got all the business owners from all the packets of auto at the table and we can talk through them one by one. So as we hit this phone number problem, the tech leads at the warehouse level started looking at it to see are they dropping digits as they're pulling this data in from the system of record. Check that, no, all's fine. So then we went to the system of record. Is there something going on in the system of record that is causing these to get short changed by a digit because a lot of them are coming in as nine digits? We would have accepted seven, so then they started looking at their logic and their intake. Obviously, we get that data at our application layer. So the system of record team said it's not us. If they send us 10 digits, we've got 10 digits. If they send us seven, we've got seven. There's no truncation or good. So we worked our way all the way down to our true system of origination, which is our application layer. And if we work our way all the way out of that, we're in vendor operated systems that are not owned by Ally but owned outside in the world used by dealers all over the country. And they're entering these individual applications for a loan and these fields are not controlled. So the phone number field, they can put in a zero and the system will accept it and that application will process and come to Ally as electronic data. So there was no preventive control in those vendor operated systems so we had to move that particular exception because of the horrendous high volume of it into a risk acceptance mode. But because of its materiality, it's associated to the volume of loans and its association to the outstanding balance of those loans because we look at it from a monetary and a record count perspective. We bring that all together that risk acceptance had to go right up through the leadership of the auto business as it approached the top hierarchy of the auto business. One key executive said, oh yeah, we know that, that's always been a problem. We're going to have to accept it. We can't control the vendor systems. When the top executive had to sign off, he said I'm not signing it. Get a hold of those vendor systems and put in a preventive control. Enough is enough. So here we are spinning all the way back after like nine months of this negotiation and now we've got some C-sweets going out and negotiating an improvement to vendor operated systems. So we're continuing. Definitely governance and action when you can take it even to your third party and get that buy-in. You just described a lot of the systems involved all the way up to the sources work which are external. You also shared an architecture that showed a combination of buy versus build things that you put together. So lots of questions around whatever you feel comfortable with Susan sharing on some of the thought process on going build versus buy and for those things that you decided to buy if you feel comfortable sharing some of the names of the tools that you're using. And also if you can revisit, I think I saw in the middle there was some centralization of the federated world so if you can talk a little bit about the centralized engine that you have and the tools that surround it. Right. So I know Ally never likes us to talk about our tools but we do use the IBM products. We're a big IBM shop here at Ally so we acquired the InfoSphere business glossary and metadata workbench years and had to take those tools as we built data governance because the company had spent the money. As we've evolved we decided when we looked across the world again just a year and a half ago there was still nothing in the market that was really fulfilling our needs from a business glossary and lineage toolset effectively that would be better than what we already had with IBM. So we chose to upgrade the IBM governance catalog structure which is combining two pieces of software into one which is your lineage tool and your glossary into one web-based application. But as we continued to hunt across all these different software we couldn't find anything that had a value issue management structure that we really wanted to leverage so we started designing our own. Early, early on back in 2012 we realized the struggles the company was having with reference data. We all have tons of reference data. We use codes to keep our space in our system smaller to make things simpler. But a lot of people would get all these fancy reports with a ton of reference data in it and never know what it meant. So we decided to build the reference data hub again really to allow anyone in the company from any aspect to our tool set and see what that code is and what its meaning really is. So one is a GM, a two is a Ford, a three is a Mazda. They look at the numbers but they never knew what did it really mean. So those were the two key tools we bought and because we were in an IBM product line we accidentally found out all these other things worked with the IBM tool set. So basically with the IGC tool data stage again is another IBM product that is an ETL structure that we can automate our lineage automatically if we use a data stage job. But then we went out and found a vendor named Medidex that had a fabulous way to create automated lineage based on the language and the technology you use for your ETL structure. So we looked across our inventories and it was a big user. SQL is a big one. What else do we have? I think Oracle was a huge one. So we started picking up modules from them to do automated lineage and we're continuing to expand that. I think this year we're bringing in Cladera so we can capture that type of ETL structure. But the IBM product comes with an X meta database in the back end and you cannot do anything with it. It's proprietary to IBM drove us crazy. We wanted stronger reporting that the tool did not provide. So we actually created a mirror image of X meta into an Oracle database which is our reporting database and we were able to apply Cognos and also Microsoft Power BI on top of that. So that really improved our reporting capabilities. Great. I think we can all empathize with various tool sets. I don't think there's one size fits all so I think your approach as a mature data governance organization is to really look at your requirements and find the right technology fit. We're talking about the tools being used for data quality lineage, business glossaries, any tips for the audience on how you've been managing the ongoing creation and we have these third parties involved too. How are you managing keeping these definitions up to date and keeping everybody aligned? Any tips for us on continuing to do that? Yes. So you've got to make sure first and foremost never tell them this is a project because we call it the never-ending cycle. And so once they've stood up their business glossary and they've got their lineage, we updated our policy to require these data owners and the data consumers have to come back into our tools at a minimum on an annual basis and attest to the accuracy of the content. Because, you know, once you open these things up corporate-wise, the need for the accuracy is just explode. You don't want anyone going into your tool set and reading about a data element if it's not accurate. Because we merge business information and technical information in our business glossary. And not only can you gain an understanding of what is this data element really, we have a data example, we have comments that gives you nuances about that data, and then we have technical metadata. We tell you where it is in a warehouse, what table is it on, what's the column name. So a lot of the information just in the business glossary without even flipping into lineage can help you build a great new report. So we want to make sure that's accurate. The lineage is being attested to on a quarterly basis, and I think we've moved our glossary up that it's twice a year they're attesting to the accuracy. And we monitor them to ensure they complete it. And with us just almost being at time, so everybody, thanks for all the great questions. Any ones that we didn't get to will definitely use as inputs for next webinars. But maybe you can share with us to close us out on the value and measurement and the metrics that you're using. We talked about data quality being one of the critical things. If you can share how you're tying that business value and the metrics to measure the success of your mature program with the team as a closing statement for us. Sure. I think the biggest value add is obviously being able to give documented evidence of the level of your data's quality, especially that data that is critical to the success of your business. So our dashboard can give you the top 10 data elements with problems. So you see it clearly on the page very quick. But if anyone wants to understand the depth of detail, so you've got 10,000 bad data points in one data element bucket, be able to quickly reproduce the detail so they can actually go in and look at those accounts with those data specific problems. And we started off being kind of kind that they only had to define a root cause if it was at a certain volume. Now we've got a calculated ability to rate every exception a high, medium, or low. If you're not coming in at zero, you're in low, medium, or high buckets. And we're driving these people to do remediation faster, more effectively, and identify their root cause. If they're medium or low, they've got a document to root cause of the problem, even if it takes a month and a half to figure it out. Yeah. Go ahead. Always have an escalation path. Yes, that's true. Gotta make sure we get that air cover that we need to keep us going. Susan, thank you so much for sharing and congratulations on the journey so far, and good luck on the journey coming up. Susan, please give a virtual hand to Susan Yaman. These are the practitioners that come and share their best practices and we all learn from. Susan, thanks again for your time and for sharing with us. Shannon, back to you. Thanks. Thank you both so much, Susan. Thank you so much for this great presentation. And Michael, as always, thank you for keeping the Q&A going here with the only section of the DGPO. You can go to www.dgpo.org for that. And I hope everyone has a great day and happy New Year, and we're off to a great start. Thank you so much and have a great day. Thank you.