 Hello, and welcome. My name is Shannon Kemp and I'm the Chief Digital Manager of Dataversity. We would like to thank you for joining this Dataversity webinar toolkit for Building an Effective Governance Program sponsored today by Metric Insights. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we will be collecting them via the Q&A or if you'd like to tweet, we encourage you to share our questions via Twitter using hashtag Dataversity. And if you'd like to chat with us or with each other, we certainly encourage you to do so. And just to note the Zoom defaults the chat to send to just the panelists, but you may absolutely change it to network with everyone. And to find the Q&A or the chat panels, you may click on those icons found in the bottom middle of your screen. And as always, we will send a follow-up email within two business days containing links to the slides, the recording of the session, and any additional information requested throughout the webinar. As we introduce to you our speakers for today, Mike Smithman and Marius Moscovici. Mike is the VP of Sales at Marketing at Metric Insights and has over 15 years of product and marketing experience in the business intelligence industry. He helped bring analytic products to market with senior roles at Seagate Software, AIM Technology, Teleaf, Xero, and Good Data. Marius has over 20 years of experience in analytics and data warehousing. Marius is the CEO of Metric Insights, the leading provider of a BI portal that helps organizations organize their BI environments and ensure users are getting the actual data they need. And with that, I will give the floor to Marius and Mike to get today's webinar started. Hello and welcome. Thanks, Shannon. I will just share my screen here and then we'll get going. I will hand it over to Marius to start. Thank you, Mike, and welcome everyone. So today we're going to talk about what makes up an effective BI governance program. And the presentation is really tailored towards you if you are either taking this function on and trying to create BI governance within your organization, or perhaps you're inheriting some existing governance framework that's there and you're trying to sort of evolve it and make it more successful. And I should say kind of before we get started that, you know, when we talk about BI governance, we're not just talking about data governance, but we're talking about data governance together with analytics governance. It's kind of a holistic way to manage all of and govern all of the BI ecosystem that you have in your organization. And I should also mention that, you know, we're going to show you some specific examples. Some are kind of examples of documentation type things and surveys and things of that nature. Also show you some ways in which you can apply governance using metric insights as an example, but really the important thing here is not the specific details. It's not about sort of using the perfect framework or taking a very pedantic approach, but rather it's about being very just practical in your approach, you know, being pragmatic in your approach. And what I would hope that you take away from here are the guiding principles, more so than any specific examples. So with that said, let's go and sort of talk about the building blocks are the key pieces that need to be in place in order to make BI governance effective right so what are the steps at a high level as we see it. So the first step if you're if you're starting maybe from clean slate or if you're doing a reset is to build a business case for the governance. And really this is all about, you know, beginning with the end in mind, and establishing for all the stakeholders within the organization, you know, what is it, what are the goals, you know what are you trying to accomplish. And this is obviously a key first step. Then the next step is with those goals defined and and we'll go into kind of how those goals should be as specific as possible and measurable as possible. Then it's all about getting consensus within the organization it's getting everybody on the same page, so that all the key stakeholders are in agreement that this is something that needs to be done and that they're going to apply resources towards making it happen because, you know, governance is not something that can be wholly accomplished by any centralized governance team it's always going to require collaboration with both the BI team and the business within the organization. Once you've got consensus. It's very important to define both long and short term objectives. So you want to know where you're going the overall goals have targets that you're headed for, but you don't want to get lost in the overall journey and not have sort of short tangible goals that you can use to build upon and and accomplish a progress and we'll show you some examples later, you know how, how does that work how can you kind of, you know, segment your goals into into phases and and have and build on the successes as you move forward with that. Then, you know, to make this work obviously you need to be very clear about roles and responsibilities. So it's really important to define who is going to participate in each aspect of the governance process what create that security matrix that that maps the key roles to their responsibilities to go functions that have to be accomplished, and thereby you know that you've actually got to buy in not just at an abstract level, but at a very specific level when it comes to actually doing the work that makes governance possible. And then, finally, and really just as important as everything else is to manage and measure your success. So as your results you compare against the baseline that you've established when you when in your business case, and you and you see how you're doing and then you iterate right because it's an iterative process so you continue kind of in the cycle to make sure you're successful, until you are fully successful towards your goals. Let's look at each of these sort of key steps in more detail. Right so let's talk about building the business case right basically justifying you know why do I need to have a governance program when it's going to accomplish. And, and I think here, you really want to look at each areas within your organization, where you're going to generate measurable results and focus on those areas and identify goals in each of those areas but in order to have a goal. You must first have a baseline. So you must first know well how am I doing these areas where the key measures and how am I performing how we performing as an organization. So I'm going to put it in this presentation sort of for sort of typical measures around standards compliance analytics usage license utilization and user feedback and we'll go and give you some specific examples around those. But you would want to think in your case, you know, maybe all these apply maybe some and maybe there's a number of others that also apply, depending upon what the objectives are of governance within your organization. So let's take a look at an example so standards compliance so clearly this is one that would certainly be the case in every governance initiative is that there is a need. There are some standards that have been established within the organization that are critical and a lot of times things have to do with data security, a protection of PII data, ensuring that data sensitivity and classification. Things are addressed but there may also be things around business terms and glossaries and definitions and other things as well. So my point being is that what you want to do here is you want to say well we've established some goals. To what extent are these goals actually supported end to end in my an organization. And that's a very key distinction. You're not just looking at the data. It's great if you've identified PII information in in each of your data sources at the table level and you've cataloged that somewhere. Unless a consumer of that information is aware of the fact they're working with PII data, or they know what kind of sensitivity or classification applies to given report and what that means in terms of how to handle that report, then that criteria of is not really satisfied. Right, so you can look at what all the assets that you have out there, and then you know make an estimation or do an analysis to figure out you to what extent are we compliance with that particular criteria. And so, as you see examples here, PI data data sensitivity, maybe constraints around how information should be used whether that's policy or enforced within the particular technology, to what extent are those being satisfied. You know, things like metadata to station and certification you know are those are those happening on a regular basis so that once metadata has been established. You're able to go back and say oh you know this was this had a PI data, six months ago does this still contain PI data or vice versa, or what is as the data classification change for this object. So those things being able to handle that and with some kind of frequency throughout the station process on a regular basis, all examples of these kinds of standards compliance type of uses and you and you just base mark that baseline that you're well compliance for that. Another area that's very common is around analytics usage. So here, if you think about governance. It's clearly must be much more than about security and classification and maintenance of metadata. Right. Effective governance is is essentially stewardship of resources in such a way that that your maximizing the value from the investment you put in those resources. And so, one way to measure that is to look at the analytics that you have and say well okay, how many reports are available to users, what have I made available to users and at a high level what percentage of those are actually actively used. So you know we have 500 reports, sales reports between Tableau and Power BI. How many within the last 60 days have had any usage at all right so if there's if you know and that number indicates to what extent your environment is a curated, you know manicured beautiful garden that is well and it allows people to consume content easily and to what extent, it's not right, and it's so it's a very useful baseline, and oftentimes these numbers can help justify the business case for a movie for doing investing in analytics if the percentage is not very high. Another kind of slice at this is to look at cost or ROI from a more of a licensed utilization perspective. So, you know, obviously from a from a cost perspective investment in BI is can be very high in an organization, and that investment spans both the tools themselves and of course the expensive resources that are working with those tools building reports. And so if you measure and you say well I've got, you know, for tableau I've licensed 500 users. But if I look at actually who's use the product at all during the last 60 days, you know what is that number and what is that number is the percentage of my licensed users. Then, and I look at overall costs that I have from my licensing, then I can identify what is my underutilization of that hard cost that licensing costs. And suddenly that percentage is an indication of also underutilization of the BI analysts that are building content right because if I'm building content for a community, and only 40% of that community is actually using that content at all. So probably there is a lack of efficiency or underutilization of that significant investment that's going into both the licensing and into the resources that are using those tools to build out the content. So this is another sort of powerful justification from business case perspective to identify why having proper governance is very important. And, you know, the last three we've looked at were all examples of, you know, very objective measures right you just go you measure what's actually happening and you can report it. Equally important in any kind of business case justification is having a subjective measure as well and understanding, you know your user community, how do they perceive the value of the analytics they're consuming. And how did but how easy is it for them to consume the analytics, do they understand they have necessarily did a literacy coming into it, and what is the overall satisfaction level. So here's an example of just a few types of questions that you can ask, maybe ask people to provide from one to five scale to indicate, you know, are they are they very satisfied and they're not very satisfied. How easy is it to use the current analytics. Well how much trust do they have in the data are the are the do they understand the analytics that they're looking at or things clearly defined, you know what what are things streamlined so they can find their content. How productive is the time they spend in the reporting system are they are they just searching for things continuously, or are they easy for them to find the things that they need. So in addition to any sort of number of those kinds of very specific questions. We recommend that you always ask at one kind of high level question of their overall level of satisfaction. And I should say that this kind of information collection is very easy to perform right in a few minutes you can create a survey monkey survey or use any number of surveying tools and, and then you know send it out to people and collect it. The other part here is really having the courage to generate this and send it out right you oftentimes, you know the scores that come back are not particularly flattering. And sometimes they are markedly different from what maybe people within the bi organization, or within, you know, because there's a hole on the bi side might think is represents performance so you know there's always the risk that your scores are going to come back a lot I might think, but it is very important to take the step because again you cannot improve what you do not measure and understanding what it is where things are today with the good or not is critical so that you know what you're building from. So, we've talked about, we've done that let's say we create the business case so now comes the, the next very important phase of the process which is to say, let's take that business case and we need to get consensus from the organization. And this is another area where sadly, oftentimes, governance folks fall short. Right and the reason for that is, oftentimes, those of us who are working in the governance area are very passionate about data, we're passionate about visualization. And sometimes less passionate about getting a whole bunch of people into room, argue the case, getting consensus. I thought that's a, that's a messy process. So it's much easier to go in, you know, the fine definition business definitions and glossaries and, and come up with processes for tracking PI data than it is to necessarily go in and get consensus but this is really important. And if you do this, you're not going to be successful is that it's not any kind of large scale organization. It's never possible for just the governance team alone to be successful in deploying a solution. The other thing that is often missing from these conversations and the kind of a common pitfall that you want to be thinking about is that you want to make sure that when you're engaging these various constituencies. You don't make the mistake of having not enough carrot and too much stick in your argument to them. Right, what I mean by that is it's very easy to use the stick of compliance as a way to justify governance say well we have to do this because we must protect PI data. And we must, you know, our team has told us that we must have these key requirements met. You know, obviously it's very important and that should be part of the dialogue. But if you, if you have that conversation where it's too heavily weighted towards the take the stick and there's not enough carrots is none of goodies in the, in the process for these people to get value themselves. But what you'll see is they'll say yes, we agree. It's very important. Now you governance team you go do this because we have a whole bunch of other things so we can't really do too much to support this we agree conceptually that's required would you go do it. And so they won't be the level of buying that's required for them to actually, you know, allocate resources to actually to help you. And so that's that's very important. So, for example, executives, you'll be looking at my compliance risks but you know you'll also be looking at maybe ROI. And, and how does the what what kind of BI systems are why I'll go back to those metrics that we baseline and identify ways in which you can generate better BI ROI from that. From the bi team perspective, you'll be looking at, you know, content, things like hey you're going to be boosting content engagement. So if you're building us bi team are building all this content out there, if we put proper governance in place, a much larger number of people are going to be using that on a regular basis. So that's going to generate more value from you, you're going to be generating higher utilization from your license so when you go to talk to you at the CIO about about your budget it's going to be much easier to justify what you have because you'll be able to show that you know that you've got high utilization numbers there. You're going to be saving our time from your bi team, a well governed environment is a prerequisite for self service. And if everyone is able to go in, consume their own content they're going to get fewer questions where you know analysts are getting called up all the time to answer questions that can be addressed easily through existing reports that are there. Well, the value add to participate is you're going to have a lot more trust in your information and well governed environment allows you to know which visualizations are certified, what what contains where the data comes from. What are the key glossary terms that are the enterprise metrics that are referenced there. It gives you all that information to know that I can trust this visualization to make the business decision that I'm using it for. The general data literacy increases. So you're, you're from a business perspective I as a, I know that my team is going to go in, and they're going to understand the data they're going to make and probably interpret the information. And because I can self service, I'm going to have time savings I'm not going to be hunting around looking for content. I'm not going to be stumbling across content that is not useful. I'm going to be connected easily with the content that is relevant to me so that I can make the right decision so all of these are things that come out of the governance process that are the carrots that you want to focus to in addition to the to obviously the fact that hey we got to do this in order to meet compliance. So, the, the next step is you've gotten the buyer, then the next thing that to think about is the fact that you know you want to define both long and short term objectives. So, if you think about how, you know, baby at home or something and you watch them kind of go through that crawl walk run stage. Right. It's, it's clear that they are really good at, at getting incremental victories and building on those incremental victories right they iterate. They, they learn something eventually they figure out how to do it and then you know once they've gotten good at crawling the next thing you know they're on their legs and then once they start walking it's pretty quickly that they start running. They don't, they don't go from crawling to running. Right. And so, it's the same thing you want to do here you want to treat this as a journey. And, and say, and think about this as a you know what are the goals that we want to accomplish right off the bat. And when and pick those goals in such a way that they're achievable there are things that you can do and during early stages of the process. And they allow you to act as a building block for the next set of goals, and so on. And in doing so then you can build momentum because if you try to do too much. All at once, typically you just can't get to the point where you generate value quickly enough, and then you can easily lose momentum. For this purpose, we're going to, we're going to take in and give you some examples of this of specific things you can do in phases and we'll use the metric insights product as an, you know, to in order to illustrate these examples. But as you look at this, I want you to be thinking about more, rather than specifics it's more important to be thinking about, you know what would be the right incremental steps in your organization, in order to create value in this stepwise fashion. So, with that said, I'm going to switch over to Mike to show you the in some examples. The first step will be consolidated BI portal, which is a governance platform is idea being that before you can govern everything, it must all exist under a single pane of glass, where it can all be applied, same rules can be applied to every Yeah, thanks Marius. So, so as Marius said, we'll take you through some examples of kind of what this practically might look like and what you might work towards in your organization and I just switched across to telemetric insights platform which you should be seeing in my browser now and really, you know as Marius mentioned kind of in order to instrument governance both BI and data within your organization. In a lot of organizations you have this very sort of disparate environment where, you know, oftentimes there's multiple BI tools as oftentimes documents and spreadsheets out there you know there is no one place to go for BI. And these tools have differing capabilities from a governance perspective if they have any at all. And so it's important to think about creating sort of this consolidated space where both analysts or publishers of content can publish content in a governed way and we'll talk about what that might look like. And also, as Marius said, where your sort of business user stakeholder and the executive stakeholder can get value out of the governance steps that you're taking. And so an example might be like you're looking at here, this is the metric insights catalog, where we're connecting to all our BI sources and again that's everything from sort of the BI tools like Tableau or Power BI whatever you might be using. But also, it's important to bring in anything that needs governing like spreadsheets or PDFs or documents or presentations that you may create on an ongoing basis. Have a place where you can pull all that in and each of these tiles in the middle is something that we've published into this catalog and we'll look at how they get in there in a minute. But you know the benefit of gathering in here is that you can govern all these assets in a consistent way and so what does that look like if we take something like a Tableau workbook here. It's the ability to publish it in a way that again benefits both the analyst and the end user. So here when I preview this tile, you know, obviously we're picking up sort of what the dashboard is called, any descriptions. We're also picking up things like, you know, when was this data last refreshed so the user has some context about the sort of relevancy of the data. We're adding in things like ownership of the content so we know who's accountable for it both from a business and technical perspective. It may be that you're thinking about that classification or your content down the bottom here so if you've gone through in your business case or saying okay we need to identify which assets contain PII data or what the classification of assets is. Well let's use that to also help educate the end user who might be accessing this particular report so they have some context around how to use it so you know it's internal to the organization but this one doesn't actually contain any PII data. And then obviously things like glossary terms so if we're going through the process of defining our metrics and defining our KPIs and how they're calculated and what they mean. Well again making sure that that doesn't exist in a vacuum within the governance team but it's also being presented alongside the content to the end users so that they have the context of it. And so yeah by pulling this all together and again we'll look at how this gets in here in a minute. Yeah when a user ultimately comes to access a piece of content like this is tabloid dashboard here. They're getting this context around the asset that they're looking at, they're getting a level of data literacy around the asset and they have all the information they need not just to analyze the data but also understand the context behind it. And again whether they're looking at a dashboard from a BI tool or whether they're accessing maybe a spreadsheet that's sitting out on a file store out there or a version of a spreadsheet, the experience should be the same to them and they should have that context. It should also be an environment where they can search, they can use that context that you've added and that governance information that you've added to either again as a BI analyst or an end user be able to search through that catalog of content so you know this this catalog should have search capabilities where if I search for something. It's looking through all that context that we've gathered through the publishing process to make it easy for end users to find content made make it easy for business analysts to be able to see what's out there already being created by different teams and make it easier to consume and find analytics rather than typically what happens today where you know if as a business user I'm interested in some data but I don't know there's a report out there I pick up the phone to Marius and ask him to create me another version of it and you end up with this cutter and duplication and lack of trust and lack of clarity into what reports are actually being used. So being able to search through the catalog find things easily either as a publisher or as an end user is critical. And then finally on this step is we just go back to to the sort of spreadsheet example I was looking at here is by having sort of this centralized catalog this centralized space. You can also start to enforce some of those governance policies that we spoke about right at the beginning of the business case. So if you are requiring, you know, the, perhaps, you know, certain assets require users to understand certain usage policies and compliance policies around it, then let's make sure we're kind of tracking whether that is being whether that's happening at the user level so being able to communicate through the catalog when certain policies need to be enforced and understanding, you know, or requiring users to accept that they've read those policies and complying with them gives you a way of as we'll see in a minute, measuring compliance and improving that baseline that Marius spoke about. So, you know, the crawl step is to figure out how you can pull together a governance space where you know you can have consistency around some of these processes that you're putting in place. Yeah, thank you Mike and I think that you know that's that's very important right so you have to have some mechanism to do that and, and once you've established a baseline way in which that that governance can be applied across all the assets, whether you have a perspective of where how you source that information in. You're now ready to kind of take a next step and that next step will be say well now I've got the right content up then how do we, how do we kind of create workflows or mechanisms whereby, you know, maybe you've created that an example of a smaller use case where you've got everything fully governed and well established but now you ready to roll it out to the enterprise. So you want to be able to have a mechanism by which users can be I analysts can participate in the governance process, they can do their own publishing, make sure that only the right content gets up there. You know you don't want to bring in every piece of clutter that might be out there and you're in your bi tool environment, especially if your bi tool environments been around for a while there'll be a lot of clutter there. You want to have a well defined process by which only the right content gets brought in by which it gets certified the right metadata gets associated with it. And then you want to have a mechanism by which you know that when users are consuming it that data quality is preserved. So, perhaps you might have already created some checks in your ETL processes that that validate that that there's some fundamental quality of the data identify when there's data quality issues. You want to make sure that there's a way to communicate that and automated fashion to all the constituencies that are using that and that, and that, so that if there's an issue with the data, there's transparency around that. And, you know, users know that today they can't look at that particular information. So Michael show some examples of how that. Yeah, so what we're, what we're really talking about here is having this ability to instrument some sort of workflow process as Marius said within within your organization to onboard new content or, you know, enable distributed teams to to onboard content complying with with the sort of governance processes that you put in place and I think what's important with this pieces. Yeah, it's not governance workflow should be sort of a one size fits all for your content. If you think about the content that you're onboarding. You should think about what what is the necessary level of governance that needs to happen for it to be in compliance without it being overkill but also with with that being enough there to to meet the standards that you put in place so so any sort of workflow that you put in place to publish content should be flexible enough to apply to the type of content that you're you're publishing. That's what we mean by workflow. But it's really sort of the checks and balances and the addition of any of that metadata that we were talking about and ultimately the certification of the content. So that when it ends up in the centralized space that you've created for your content. When the user looks at it. They have that context that we're looking at before and they know that they can trust the data so a workflow is a number of steps that you're going to push the content through where an owner sort of has ownership of that step. It could be it goes to a developer first to review the content make sure the the data is accurate and quality. It could go to a business review where they're responsible for sort of tagging it with your your glossary terms and your metadata terms it could go through the governance team to make sure that you know the compliance standards and announcements are being added but ultimately again depending on the content and the level that it requires it is going to go through one or more steps before ultimately you can say this content is certified and something that we have complied with and something that we're going to put in the catalog. And so you will set up these workflows and just move the zoom window here. And you will then basically assign content to those workflows and the individuals who are responsible for each of those stages. And so, you know what you're looking at here is sort of a board where, you know, content as it gets created will get synchronized into a particular workflow. It will be assigned as part of the stage to a particular user or users who's responsible for it. It could be as simple as this where it's kind of a one step process where maybe the analyst is responsible for coming in, reviewing the content, maybe adding in tags or glossary terms classifying the content like we looked at before, before ultimately sort of saving and certifying it into into that final stage. So that could be again, you know, maybe it's some financial reporting that we're doing that's going more externally to the board or out to our investors and it's, you know, something that we want a higher level of scrutiny on. So that may go through multiple stages involve multiple stakeholders, adding the necessary context or reviewing the data before again, it ultimately gets published into into the catalog and into into that centralized space. Ultimately, though, again, the necessary levels so that as a user coming into that space easily identify what content has been certified when that happened and who was ultimately accountable for that process. So think about sort of the workflows that might might have to be in place for the different types of content that ultimately you have in your organization and apply content to those workflows with the necessary level of scrutiny. The second piece Marius mentioned was the idea of sort of data quality. And again, you might be doing some data quality within your ETL tools that you know is obviously probably notifying the analysts or the data pipeline engineers when when issues occur. A lot of times companies will do some level of sort of data quality alerting on the analytics as well so you know for loading a Tableau dashboard with with data every day, what's the typical level of what's the typical volume of data that gets added into that that dashboard. You know track that if we see any sort of anomalies in that volume. It can be an indication that something has happened with with the sort of final step of populating the dashboard. So you might be doing it tracking so the data quality issues is important but equally important is communicating those issues, not just to the builders but also to the end users of that content. By having a government space, you can also include announcements around that and things aren't going to be 100% accurate 100% of the time I think end users expect data quality issues, where they get frustrated is when they don't know they're occurring and I'm looking at a dashboard. There's issues with the data and I don't know that but I'm making decisions off of it. And so those data quality checks should automatically sort of drive announcements like we have here at the top associated with any results where those issues might be occurring because then as a user I come into this. Yes I see it's a certified piece of content and I should be able to trust it, but I also see that there were issues this morning and therefore you know I should step back and not waste my time trying to figure out what was going on with what I think is business results but ultimately it was data quality issues. Publishing workflows to get content in now that you've got your baseline and maintain that level of scrutiny and then communication around the quality of the content on an ongoing basis is critical. So if you've gotten that established and now you've got your content in place, you've got some data quality measures, you've got a process for publishing and decentralizing that publishing process so that the right folks are involved in that. You've implemented that now you're really ready for that kind of run phase and here we're looking at things like discoverability. So you want to make sure that you know sure you applied a security model to identify who should access the content. You know, oftentimes you'd inherit that from the BI tools, but what about making content discoverable for users so that they can find and request access to it you know that's a key aspect of a government environment, because you know what you don't want is somebody going in looking for something. They can't find it because they've not been given access to that report and they incorrectly conclude that well that asset doesn't exist and they go build it. And they go request that it be built and so that that creates a situation where duplicate contents is created by design and that's obviously the opposite of a government environment. So discoverability is a key capability that you need to have in your in your governance delivery. And then, of course, if you've gone through the trouble of bring putting all that metadata in through the publishing process that's all well and good, but that will decay over time. It was it was perfect as of the time that you released it. But then some time afterwards maybe that metadata is no longer accurate maybe that report is modified and and a new column has been added and now it has a different kinds of data classification different level of sensitivity. So having that attestation process whereby you revisit and and recertify and identify that yes this report is still relevant. And in fact, from a business logic perspective and from a metadata perspective all the key metadata attributes are still accurate that that's very important. And then the third item that we've chosen kind of for this run phase is the feedback process so you know you're getting that those surveys that we talked about at the beginning for the macro level feedback, but it's very important for a government environment to also get micro level feedback to know that of these reports that are out there which are the ones that users really find useful. Which are the ones they don't find useful. And if they don't find them useful why not right and having that feedback and create a loop that goes back to the folks that are creating that content, and are able to to then you know improve the content. So, let's look at how some of that would work. You know, discoverability so as Marius said it's it's key that, you know, once you you have government content going into the environment people are able to easily find it and avoid things like duplicate content getting created so what what does that really mean. Well, in this govern space is catalog that you're creating, you know, obviously you should enforce security and permissions and ensure that anyone accessing the catalog is is, you know, only seeing data that they have the permissions to see. But we can utilize that metadata that we've captured as part of the publishing process that we put in place to enable sort of a discoverability scenario so if for example, I was coming into our catalog and searching around, you know something like procurement data, maybe there's some information I'm interested in there or I'm looking to create an asset around procurement. As an analyst, I want to know what's out there already regardless of whether I have access to it. And so if I search for content, there should be this ability to have discoverability enabled where I can see here that actually there is a procurement dashboard available out there. I don't have access so I can't see the data it's blurred out there's a padlock over it, but I can get that context that we were talking about you know so okay who owns this particular report, you know what metrics does it contain. Any of the, you know what classification is it does it contain PII data or all that information that we might have captured when this was published, I can see this asset is out there. It seems to be similar to what I was going to create. So maybe this is something I could request access to from from that owner, you know, ultimately obtain access to it. So discoverability should help you a save time from a business user and an analyst perspective so I'm not wasting my time, you're trying to create stuff that's out there already but it should also help sort of ramp up engagement across the organization. And I guess or an educated guess when you set commissions and security of who would be interested in these particular assets, but you know letting people find them and make their own decisions help strive engagement with that as well. So discoverability. So what Marius mentioned is is a concept of attestation and review. So all be I sort of has a has a shelf life both in terms of the data but also in terms probably of the, the metadata that you're adding to it. So, we think back to sort of the publishing workflows that we mentioned before. It was this example of sort of multiple stages and asset might go through. These workflows should really also contain this sort of review stage where based on rules that you put in place again it's going to be dependent on the type of content that you're creating. You may want to automatically bring certain assets and reports back into the fold to get reviewed on a regular basis. So an example might be here where you know every month this workflow is going to look at the published and notified content that we have. That's more than 90 days old say, and if it finds any reports it's going to put them automatically back into that review stage and assign it to a user or a group of users and any content that gets put back into that review. It should automatically notify that user that they have content for review in their queue. Yeah, maybe this is content that needs, you know, putting back into the workflow to to update categorization or any of the metadata maybe it just needs review in terms of the content, maybe this this attestation was triggered because something changed within the report and therefore we need to relook at it but whatever the rules are within your organization. It should be part of that automation and that workflow so that we can, we can, you know, keep track of it and have it trigger automatically. And then the final piece is being able to capture feedback so we, we talked about sort of feedback at the at the beginning in terms of setting a baseline for your customer satisfaction for your user satisfaction. And an advantage of sort of bringing everything into this government space is on an ongoing basis then you can start to capture that at the asset level and so, you know your your space should or your catalog should allow you to basically trigger feedback from your users and you know it should trigger these two to pop up similar to what you might see on a website on on an ongoing basis without spamming your users, you should also enable users to offer up feedback if they have anything. Like that rating that we spoke about in the business case, and potentially any subjective feedback that users might have. And, you know, why do we do that. Well, we can then look at that when content comes into into our attestation process and our review process. So if I just edit this, say Tableau tile here, I should be able to look at sort of the engagement that that content is getting so the usage which we're trying to improve. But alongside any feedback or ratings that my users has given me, because then I can combine these two things together. Okay, is this a bit of content that perhaps needs to be retired because it's kind of meta shelf life. Is it something we need to update and re review, or is it something that you know still relevant that should be remain remain in the catalog in a certified state. Thank you Mike. So we've talked about the some of the phases here that could happen in the qual walk run. I think the fourth step is making sure that we drill down you've actually gotten that high level approval and support from business overall and from IT and from executives. It's very important to drill that down into the roles and responsibilities that are going to be required to support your process. So, those of you then not familiar with the racy matrix, the idea is that you basically identify either deliverable task area of responsibility, and then you get the roles across the top. And then for each item, you basically identify in the matrix you say you know who is responsible to do that particular item who is accountable that should only ever be a single party accountable they may be multiple people responsible that are doing it, who gets consulted as the process is performed and then who are is informed just to keep them in the loop. So you can really think about all the things that you've defined as key to your process here obviously this work you're working together based on what you have from what you've enabled from a tools perspective, what you've defined from a process perspective, and you identify that in that list. And then for each of those, you map the roles that you have in your organization to that process and if it's a centralized function where you know obviously where the development is all done in central fashion, then there are probably fewer roles. But in this is an example where you might have a business unit that does its own projects and has a content publishers and leads and they're working on it. There's a bi team, one or more bi teams that are working on onboarding content and supporting those business units. There's the ETL teams that are working on moving data, defining data schemas, moving data through ETL. There's leadership on both the business and the IT side, and you've really got to kind of figure out how all that works together. So, you know, for example, governance standards, the practice manager for all governance is accountable, the leads for data and all this governance are responsible and then you know, some of the leads are consulted and everyone else is informed as an example. And obviously I'm not going to read you all this whole thing and the details are not so important here, except to note that you want to go through this process. And you want to make sure that there's buy-in across the organization as to who's going to be contributing, whether it's on a consultant basis, it's a responsibility basis, and so forth. And then to have buy-in on that, so that you don't have a situation where you have a great process, but the people to execute it are not aligned with that process. So then the final step in the process, and this is really a recurring, ongoing step, is to say now that I've introduced this governance practice, I've defined specific steps, I've rolled those steps out. The entire process of rolling things out on a quarterly basis, if not more frequently, but certainly on a quarterly basis, I want to be looking at how are we doing, right? And so really analyzing and saying across those four areas that I used for my business case, you know, what's the current state, what's the goal, where do we start from? And that will give us a great sense of things. So you benchmarked in each of those areas, the high level compliance with governance standards, you're in this example, the usage and engagement, the BI tool licensing, and you use the satisfaction level. You probably have established as part of your business plan some goals to say, well, you know, our goal is over the next year, over the next two years, we really want to move this from a 40% to a 90%, you know, as an example. Now, how are we doing six months in? How are we doing 12 months in? Right? So setting up that cadence for checking in on progress, you know, some of these things, obviously, you know, you can't check in all the time. You're not going to want to survey people continuously. So some things might be less frequently than others, but usage, you can certainly look at on a quarterly basis, engagement, you can certainly look at a quarterly basis. And based on these results, you know, you hold yourself accountable and the organization should hold itself accountable to improving. So if you're not moving the needle, then, you know, going back to look to say, well, why not? Right, and digging deep and making adjustments because the reality is, there is no sort of silver bullet here of like, here's the, you know, the methodology down to the Nats eyebrow and the process that you can follow and be assured that you're going to have success. You know, each organization is different. There are going to be different challenges that you encounter. So the capability to be successful relies on having a baseline, having a set of tools you're going to apply, and then having an iterative process where you examine your progress towards those goals, and then making more adjustments until you hit that. All right, thanks, Marius. So I think that kind of brings us to the end to Shannon. I don't know whether there's any questions out there that we might want to look at in the last few minutes here, but I'll hand it back to you. Thank you both for another fantastic presentation. As always, always love the presentations you guys give. If you have questions for Marius and Mike, feel free to submit them in the Q&A portion of the of your zoom there and just answer the most commonly asked questions. Just a reminder. I will send a follow up email to all registrants by end of day Monday with links to the slides and links to the recording of the session and anything else requested throughout here. So, do you have a use case you can provide for ROI? Well, I mean, I think some of the examples that we provided there are, I'm sort of thinking about how to frame it. I mean, I think if you take the business case numbers there you can plug them into an ROI just by looking at what your costs are. Right. So, so if you take a look at that scenario, for example, I say, okay, I'm underutilizing my BI tools by x% and that and that means that the BI tools and some percentage of my of my existing BI resources spend is underutilized in the ROI. So, if I can improve that utilization from 40% to 60% that will represent an increased efficiency of a certain amount so, so that you can absolutely do the ROI that way it is not generally possible to do ROI as in like in the classic sense of like we're going to increase, you know, revenues by x% because it's not direct line, right. Obviously, there's increased data literacy contributes in an indirect fashion to reducing costs for the organization of the business users that are using it or boosting revenue. But the way you can tangibly connected to it is the is by looking at those other measures of usage, like both licensing and utilization level, and then identifying based on that what what is moving those those those items what does that represent from an increased efficiency perspective. Yeah, I'd agree. I think it's, we're probably all the same on this call right it's the age or problem of, you know, bi ROI is challenging right inherently we all know that data should, you know, increase sales if we're looking at it right but it's hard to make those links so I think, as Maria said it's looking at it from a, what are we spending on bi and how much, you know, engagement and utilization are we getting out of that is a good place to start. Now there was a question that came in early on into the presentation it says, can you define asset are those systems or docs. Yeah, so when we talk about assets it's really the sort of bi assets so reports dashboard spreadsheets documents, anything that sort of contains, you know, information if you like the end users are consuming and you should think about it statistically right I mean you know obviously it's our bi tools but you know, organize you know Excel is not going away spreadsheets are out there. So how do you get control of those in a in a governed environment. You know we create the sort of, you know maybe quarterly board deck which contains data right well we want to make sure that's governed in a similar way so asset equals sort of any document or report that you know ultimately requires some governance around it. It's not government's because it's shared. Right, so obviously there's lots of docs and PowerPoints and things that are just kind of a, you know, just people are just using for their own purpose or if the sharing has done it's completely ad hoc. So assets are all things that are shared on a regular basis like example, a board deck, a financial report that performance relative to to to budget, you know that every month or every quarter every some frequency that thing is being issued that should be governed because it, you know, obviously good. Do you think it's critical to have the data architect creating and maintaining that or maintaining the subject to mean conceptual logical model with business metadata, the building blocks of one enterprise data model and involving domain data stewards in the process, and getting a sign off in order to quote unquote certify the metadata. Otherwise it seems the big the big data catalog may be subject to a lot of ambiguity. Yeah, for sure I would totally agree with that especially you know in a central and a enterprise scale environment, you don't want the the conceptual logical model being defined by alone by a particular subject area domain expert because you know that that's in a particular line of business because then you're going to miss key areas where there's overlap around that subject area with other areas of the business that absolutely needs to be one of those collaborative exercises, and it should be the kind of the data architect, you know, who's was a central centralized function part of the COE, who should be leading the charge and working on that. And I think from a, you know, from a technology perspective if you're implementing technology to support this process then, you know, you should leverage wherever that's happening right so yeah you don't want ambiguity between your bi and your data catalog so if you're defining stuff in the data catalog that's relevant to the bi catalog it should flow through right I shouldn't have to replicate the capture of those definitions or whatever in a second area. So all this ultimately needs to work together in an enterprise environment. All right, I think we have time for one more question here. So how do you get the balance right for publishing workflow. If it's to bureaucratic then people might just not might just publish in their silo and instead of centrally how do you get by and for having or not having too many compliance stages of publication. That's really the sort of critical piece on this right that it shouldn't be as I said before, a one size fits all, because I think that's a trap people fall into either they define some of this single governance process that is to sort of heavy handed. And therefore no one ever actually uses it or it's you know not enough and therefore it doesn't really add any value to the process so it's kind of a big discussion in its own right but I think you need to look at the other particular piece of content and understand the impact that that is having or could have on the organization so think about things like the, you know, the, the, the importance of the data that that particular as a report contains think about so the scope how many users are going to be using that particular report, you know how critical is the data in terms of you know is it again a financial report that is going to be seen widely within an outside of the organization or is it just a, you know, a report that Marius and I are using internally within our team here. So that should ultimately dictate the level of workflow that's required. And, you know, if you get that right, then I think there's this. Again, if you think about the value and the messaging to each of the stakeholders if you get that right, then, you know, the appropriate level of governance. Users should want to comply with that because as a BI analyst I'm going to know doing that will mean that my asset is getting used and trusted. You know as a consumer, I'm willing to chip in my review in terms of the business process because I know I'm not going to get in trouble if I'm not beyond the organization the data is going to be correct all those sort of things so yeah again tying the, the report to the level of scrutiny is important and not trying to do a one size fits all. Yeah, and I would add to that also think about that process look at how whatever technology you're using for this, make sure it is a light touch for everybody who's involved like Mike said it could be many parties involved because of the requirement, but for each of the parties should be really easy. So that means bring in metadata if it exists in other places automatically from source systems, if you know if it's a spreadsheet or if it's in a if it's in the BI tool, and then make sure the process is minimal number of clicks. You know the users are notified when there's something they need to do, and then and then it's, it appears feel super lightweight to them, even if it's comprehensive behind the scenes. And thank you both so much for another fantastic webinar really appreciate it and thanks to all of our attendees for being so engaged in everything we do always great to see all online. And just a reminder again I will send a follow up email by end of day Monday for this webinar with links to the slides and links to the recording. Thank you all hope you'll have a great day. Thank you, Mike. Thank you, Marius. Thanks everyone. Thanks everyone.