 Hello everyone. Welcome to my webinar about transformative digital project product management. In this webinar today I will present to you the modern product management paradigm and how it differs from the more traditional models. More specifically why today it's actually paramount for every product manager to adopt this modern approach and how to apply it in their daily life would be the key to this presentation. We're going to go from a high level kind of methodology discussion to a real life example. So I hope you enjoy and that's it. So as we get started maybe a few words about myself. I'm Muli. I am the VP of product management in Glassbox and previous to this role I have been a VP of product management in other companies. I've been in other managerial positions. I actually started off from a QA position very technical very detail oriented and as a QA person I got to travel to a lot of customers and I fell in love with solving actual customer needs. Throughout the different product management roles that I held I was actually in different domains different industries and even more so important different types of methodology of execution. So I've been in waterfall companies and I've been in extremely agile companies and I've done functional product management so to speak but also very much technical product management data oriented APIs etc. So with regards to this session I can share with you a perspective that's fairly holistic throughout my career and how I actually made use of the paradigm that I'm going to show you to advance and basically to become a better product manager over time. So with that said we'll get started. First of all before we deep dive into the details I want to share a little bit about Glassbox and what we do. So Glassbox is a company that offers a digital experience platform product and what we do is we make your customers digital experience better meaning if you've got a product that's a digital product on mobile and web we will make it better with our tools and we service different types of personas who service UX people we serve as product people but we also service more technical personas and the reason I'm extremely excited about Glassbox as a product and also about this session is because I'm a product manager and I get to dog food up our product which is really exciting and fun for me. So what does it actually mean? What Glassbox does is it starts by capturing we are capturing in a secure way 100% of data of your sessions whether they're on web or on mobile and all the events that happen within that session what that enables Glassbox to do is to visualize in different ways what happens with the journeys and experiences of users on top of a specific product may it be a web app or a mobile app. So examples would be we've got a journey map that visualizes the different journeys that users take we've got an interaction map that highlights the areas in a screen that are most clickable most viewable and where interactions happen and we have a lot of different visualizations in the product throughout and then using all that data 100% of the sessions that we record we are able to analyze and service recommendations so Glassbox generates a lot of signals for our direct users which are in this case product managers that optimize their products to have more insight into what to optimize in the experience of their users. Example would be struggles that end users experience of our website and we prioritize them we tell what the impact of those struggles etc and eventually obviously that information is used to elevate the business results to increase customer satisfaction's engagement and revenue for our clients basically their main KPIs as they cascade into actual activity within the product and how to optimize it to influence them that's what we would support in this case. Glassbox is a recognized leader across multiple categories I won't deep dive into each category but just to shout out a few you can see here that we got customer journey analytics, mobile analytics, digital experience platforms and I think the key and what's really important to take away from this is that for a product manager or UX person a developer to be able to execute effectively on top of their data from users that use their products they need a holistic perspective and that holistic perspective means that we want to be able to incorporate both information from mobile information from web we want to be able to screen record but we also want to be able to to visualize feedback that was given correlated with CS. So Glassbox is playing in all those different areas of our space and as you can see on the right we have been recognized by multiple awards over time. Okay so what do we have today we're basically going to go through four different sections in this presentation we're going to start by me explaining what is the new paradigm for digital product management and why I think it's so important and then after we actually understand what it means we will discuss the importance of data as the bloodline of executing that paradigm without data it's really not possible and once we do that I'll actually walk you through a practical product strategy framework that will enable every product manager to impact the bottom line of their business by applying a customer-centric approach that utilizes data and obviously we're going to close off we kind of recapping on our key takeaways I hope that by the end of this session a lot of the key takeaways that potentially can seem like big statements that we all say but it's hard to connect to reality we'll feel really much more down-to-earth much more comprehensible and the importance of following those would be would resonate much deeper following the strategy that I'm going to present. So what is the new paradigm for digital product management? Well as product people especially for those that have been around for quite a while it used to be that it was all about shipping product it was about delivering and today it's actually more about shipping user value and what does it really mean shipping product versus shipping user value so in the past the emphasis was on delivery on time the project management element of product management the deep dive into very detailed requirement and building extremely robust product specifications and it was focused on deliverables and slightly less on outcomes whereas today we are basically focusing on the outcomes to the user and it actually becomes much more important than it was in the past which I'm going to cover why and I think one more point of information that's really important is that we used to act mostly on gut feeling and limited data as we made product decisions in the past and I don't feel that this is enough today anymore today we have to utilize other approaches to be effective in what we do I know it all sounds a little bit vague at this point but as we go through this presentation you're going to understand what I mean when I say that so why do we want to focus on value and not delivery well first of all we're living in a slightly different world than we used to right not all products used to be fully digital in the past and then we had a lot of desktop products and then we had web products but also the business models of vendors in our in the tech ecosystem has changed meaning today a lot of vendors offer a software as a service business model and what happens in a software as a service business model is that you pay a subscription as an end user and you pay only as long as you get value it's not a one-time purchase so users will pay as long as they are satisfied but they will stop when they're not this changes things meaning as a company that sells a product I'm not only optimizing to sell it I'm optimizing to retain my client and keep reselling it so to speak to the same client and it really has become as easy as as it gets today to switch products for our users so retaining them becomes actually more challenging I have to say that it's really important that to stress that this is not only a B2C issue this is also a B2B issue there are a lot of B2B products today that are offered as a SaaS model their penetration approach to the market is actually to start from the users and you know you may have heard product led growth premiums etc and then to scale up and to sell to larger organizations but there there B2B products and they still occur heavily around about providing user value in order to push their business and I think finally as you you may be aware competition is really tough meaning in the main category of tech verticals there's a lot of competition a lot of different vendors that look very much alike without understanding your users and being able to deliver to the specific pain points or needs of that specific user that that characterizes your ideal client profile we will have extreme challenges in being able to retain them and sell to those clients so with that said what am I pushing for the framework that I'm talking about is actually designed to make a product manager a strategic person it's about thinking strategically and thinking what impact can you as a product manager make by focusing on the user value that you provide now what does it mean to be strategic to be strategic is to think big and we have to connect it back to what is our role as product managers it's not about only building user stories writing the right specs or executing effectively within sprints if you do that it's then your agile it doesn't mean that you're a good product manager what being a good product manager means is about making an impact and the question is who am I impacting so actually a product manager is supposed to impact on two fronts one is their users and internally on the company and the business outcomes for the company that they work in and impact for users is exactly what we call user value whereas impact for the company is business results that could be measured by business KPIs and once a product manager is able to influence business results it means that that product manager is a strategical one and that's what I hope that you will get out of this session meaning how do you basically amp up the impact of what you do to be able to relate it to the bottom line of your business so let's talk about the strategic value of user value and I know it's a funny title but I saw it fit the way I see it it's all about consistently providing value inconsistency is really important because we talked about the fact that products became subscription based and that users can walk away whenever they want and their cost of switch so to speak is fairly low so if you do deliver user value and you do it consistently you will be able to impact the bottom line and what I'm going to do is I'm going to be presenting you three specific business KPIs and some of you may seem that those aren't directly product related and it is true that they aren't only related to product and they are owned by other departments in the company as well potentially but I think product plays a very significant part to drive those KPIs and especially in product led companies B2C ones honestly product is probably the biggest player that influences those KPIs so we're going to start off with the first one and that's called cost of acquisition what does that mean so cost of acquisition is the amount of money a company has to spend in order to attract new users or new clients and to convert them to be paying clients now why is user value in the product related to cost of acquisition so first of all if part of cost of acquisition is marketing when your product is delightful for your users and they're addicted to it they will talk about it they will tell their peers in the same company that will enable an upsell that's organic but they would also tell potentially colleagues that they have outside of the company and that would push a word of mouth marketing strategy other than that when we as a company invest in a freemium strategy what happens is think about the marketing funnel you pay for advertising and then people drop off between the point of being exposed to your product on your website not the product itself to the point of actually paying for it but when you give a freemium offering then you actually reduce the abandon rate between those two steps and you get people to get to play around with the product assess it and only then make the decision of purchase so you actually can potentially reduce the amount of money spent on advertising because you are converting more people to try the product and if it's delightful they will convert and become paying customers um and obviously other than that just think about the entire lifecycle you know support on boarding getting a client using the product and pay for it over time it all has associated cost and when the product gives it out of the box then you reduce that cost then when we look at the next kpi we're talking about customer attention right so it's all about in in SaaS it's all about retaining your customers having a low churn rate and then every growth that you experience as a business is affecting the bottom line ARR revenue we all know that companies that have high churn rates uh are actually really really struggling in the SaaS model so customer attention is built on happiness of our customers and happiness of our customers is built on the fact that they can achieve the value that they want out of the product with is um and that's core so once we do a good job in providing value to our users they will not leave us would uh which would translate into a higher retention rate and a higher client lifetime value which impacts the bottom line of our business and as I mentioned eventually the customer lifetime value it's all about optimizing to that if you look like on the uh cost graph of acquiring a new client you actually invest quite a lot before they become a client a paying client you invest at the start when they're onboarding they have a lot of requests uh they're still not fully adopted and then it becomes what's called potentially a milking cow after a certain point in time where you mostly see revenue from that client you keep delivering value for them but they're not as high touch and when they're at that point those are their most lucrative years uh in the lifestyle below the client and you want to increase the span the lifetime uh span of a specific client so the customer lifetime value increases and obviously with products that actually charge by action like um could be payment products where uh the vendor makes some money off a commission so the more actions that you enable the more money you would make in actual dollar value meaning the customer lifetime value in dollars would increase not only in terms of just being able to retain the customer for more time and then they pay the subscription for a um for a larger duration of time okay what does that really mean it means ROI as a company um we invest we invest in R&D we invest in support and operations and we invest in marketing and we want to see revenue coming in to support that investment so basically if you do all the things right and you push the KPIs that are presented before your overall ROI would increase as a company it would make you more competitive it would make you more positioned for your next challenge as a company at your growth opportunities um it's all about balancing the amount of investment that you're making and the amount of revenue that um that you generate from that and we have the ability to influence that as product managers and a lot of product people don't necessarily feel that on a day-to-day basis but that's the actual reality when we do a good job when we're killing it and that's what I want you to come out of uh from the session I want you to have the realization that you can influence the overall ROI for the company okay um so we talked a lot about value and I think it's slightly a vague uh term to an extent so I want to show you how data fits into this paradigm of promoting user value and why it's really important to talk and use data in order to actually practice this this paradigm okay we started off before from the fact that it was all about shipping product right and now it's about shipping user value but then what is user value so the question arises what does data have to do with user value and especially if we want to measure the value consistently it even becomes more important value is quite elusive I mean if you try to look for value um and if you even go to the dictionary and you look at the definition of value it's not really easy to answer I mean the dictionary definition of value is is uh importance worth or usefulness of something all those terms are terms that are um slightly vague and the question is if we want to make sense of them and we want to be able to measure them what do we do right that's where data comes into play data is basically an indication of value and if you have data you're able to derive the value that you create for your users so first of all I think it's really important to touch on the point of that there are different types of data there is qualitative data in the examples you see over here feedback and user interviews are qualitative meaning specific users provide specific uh feedback the feedbacks or interviews there's aggregated data such as analytics and trends analysis adoption trends uh stuff like that there is self-reported uh data such as feedbacks and there is uh inferred data such as watching a session replay of a specific user this is a type of data point that's actually both inferred as well as qualitative and not aggregated and out of the data we can actually derive two things we can derive both um issues meaning things we need to improve pain points for our users but we can also derive opportunities and both of these are really important if you think about the product development life cycle of um of a specific set of functionality or a specific feature it start off with an id an id is usually based either on an opportunity or a pain and it follows to um to execution and eventually optimizing over time so we might start from an opportunity identification from the data come up with an id and then as we start executing optimize the experience and find the smaller issues that prevent the solution from being tailored to the value it's supposed to deliver over time and we use the data for these two things in parallel and if we want to have a good answer to value we actually need to use all the different types of data together doing so will enable us to better interpret the data and to validate it as well as focus on ways to solve the things that we see um i can give examples we might see a certain trend and a drop-off and the drop-off may have multiple reasons it might be because of an issue with the product but it might be because of uh something seasonal you won't know that without actually trying to validate and seeing what happens when the user use the product so session replay in that case will give us an ability to see a recorded session of the user and validate if the drop-off that we saw is because of a certain experience or a struggle that the user had and if we don't find that we're going to try and find different reasons and they might take us to a different direction for example a seasonal effect on the trend that's not related directly to the product um other interesting types of data points that we use as product managers include funnel analysis the ability to see conversion between steps journey mapping such as you see on the top here on the right and also technical signals for example the fact that there was a slow loading time for web page we want to use all those together to come up with a holistic understanding of what's going on with our users how do we um how can we impact the bottom line with a customer-centric approach so this part of our session focuses on actually actually applying a pragmatic way of impacting the bottom line of our business with the customer-centric approach and doing so by utilizing data meaning we're now going to review a specific framework that has specific steps of how to execute this in other words what is the the framework that enables us to execute our product roadmap and make sure that the added value that we created for our users is translated to better business results okay so i'm going to present now a three steps approach that will outline how to execute this strategy using data and we're going to start off with the first step the first step is to identify the key business kpis that we're trying to push basically what's included in this step is both to define a problem or opportunity that we want to tackle from a purely business perspective this should come from a company's leadership discussion it can be also influenced from a bottom-up approach but a lot of time this is something that the leadership of the company is struggling with to understand what are specific business kpis underperforming and want to be able to define that business kpi in a way that we can measure it that's first step of this process the second step is to derive the related product kpis that influence the business kpi that we're talking about so it's about identifying the metrics and it's about analyzing the data that we already have to find out what are the current values of those metrics and which ones we think are at the right at the right place so to speak are at par with benchmarks and which ones we think could be improved improved and then once we do that we want to walk into a cycle of optimizing the product experience to influence the product kpis that in turn will influence the business kpis and what that means in reality is to identify the usage data of our users that are relevant to those product kpis to find correlations and identify root causes with the evidence and then based on those data points to suggest a solution a an MVP one a minimum viable product and to execute that solution in a build measure iterate cycle meaning we want to build something we want to then measure its impact and then we want to iterate and potentially change our plans now sometimes we actually have to apply a bottom up approach or at least a sickly approach meaning we don't necessarily always know the exact product kpis that we want to track we know the area so to speak and we have to look at more detailed information to be able to refine our definition of the product kpis that we want to track so we kind of combine the top to bottom and to bottom up approach and we do that by looking at users behavior in our product and understanding how they actually use our product preferably also by separating it to different personas and then when we understand that we actually understand also what are the product kpis we want to track in order to make the overall experience for those users good in a way that would influence the business kpis meaning that would push the retention of those users as an example and one note that I think is really important is this framework is relevant in general but for companies that utilize the okrs framework objective and key result I think that's really really connects into the practical steps of how to hook into an okr framework as a product manager meaning the business kpis would usually be the okrs set at a company level the company objectives it would be a fairly small set of objectives and then the product kpis would be the cascading objectives to the departments in this case the product department and then we derive from that the right usage metrics so if you think about a yearly planning cycle or quarterly planning cycle you start off by the company defining or tweaking if it's on a quarterly cycle their objective and then cascading this to the department level and then taking these two actual features and usage data etc so this entire process they're presenting actually happens all the time and it kicks off on the strategic level at the start of a planning cycle if it's here at your quarterly and it trickles down to actual product metrics that are consistently used in track but modified from time to time as objectives change now I think that we talked about theory mostly until now and I have to make it a little bit more tangible for you so what we're going to do now is we're actually going to examine this framework with a case study to demonstrate how to apply the framework and in this specific case study we're actually talking about a fast food delivery company that sells most of their orders so to speak from digital channels so it means that the product manager's role here is really really important with regards to the business results of the company and the business problem that they identified they have quite good metrics at the top level business kpi is the business problem that they identified is that their customer lifetime value is fairly low and they do know which is interesting they do know that their ratings of food are great meaning they know that they deliver a good physical product the food that they deliver and they know that they have a delivery service that's fairly that's operating fairly well and still they see that the customer lifetime value is lower than they expect when they compare to benchmarks so if we're looking in this I want to define the problem then our gut feeling is that it's about customer lifetime value and it's also so focusing this on how is this impacted by the digital experience of users because we know that the food is good and we know that the delivery service is good and customer lifetime value is a business kpi that we'll see in a second that is measurable okay so now we start drilling down we're walking into step number two which is to derive their related product kpi for that business kpi right um to do that we actually have to first understand what is a customer lifetime value calculation and what are the building blocks that compose it okay so customer lifetime value in the case of a food delivery shop is basically the customer value times profit margin times the average customer lifespan now if we look at the average purchase value in this specific line that would be 25 on average per purchase and it is in line with the industry average for businesses that are similar to this business and when we look at the average purchase frequency on average a user would order once every two weeks from this place that means 26 times a year and that also seems reasonable when we look at the market analysis studies so we think that's fairly in line which means in turn that the customer value is 25 times 26 that's 650 per year now there is only one question mark which is what is the average customer lifespan the average customer lifespan in this case is 1.5 years usually for companies similar to this it's longer than that it's not an extremely pricey product and usually customers go and buy from that same place for a larger duration so we want to understand what's actually hindering the customer lifespan and driving it down from the expected value we want to focus on this but we have to remember this is still a very much business-level KPI and we want to get to the product level KPI so let's see how we're going to deduct the measurable product KPI from this what is the customer average lifetime average lifespan sorry so basically this is where we start doing the back and forth between data exploration looking at our data and top down cascading from the top level metrics we want to understand deeply what would be the product API that influence the average lifespan and to do that we have to do some research and play around with the data so we started off by looking at our different delivery channels of this service and we found out that within Moberg we have a mobile application and we have a mobile web experience meaning users that order on the web through their mobile device not through a mobile application both in both cases it's the same delivery service meaning once you make the order the same delivery guy and the same service send it over to you and still we found that there is a discrepancy between those two channels meaning one channel was actually generating more orders or actually converting more orders from the point of starting a flow of ordering to the point of actually making the order so understanding that the delivery is the same and understanding that it's pretty much the same cohort of users you know users that use the mobile to order the to order their food we contemplated that this is possibly a digital experience issue meaning something is different in the experience of those two of those two types of users or in the different experience in the two channels so once we did that we thought well where do we start meaning if we see that we have a different average lifespan for those two different channels but we assumed that it should be fairly similar what what actually drives the lifespan and what drives a prolonged lifespan for a user is the fact that they keep coming back to the product and that's actually based on the fact that they need to have a first good experience and what does a first good experience mean a first good experience is the ability to actually achieve the outcome that you desire from this product from this experience successfully with ease so you would come back and do that now imagine the an experience where you try to order food it says great food was ordered it never gets to you because there is a broken linkage between the systems and it never actually gets to the the restaurant to prepare the food right you'd never order again so yes a good first experience is mandatory looking into that good first experience we wanted to look at conversions to successful purchases because this is the main job to be done by the user they want to order food and if we believe that that increasing their average lifespan is built on the fact that their first good experience would be good then we have to look on the conversion of specific flow that represents that job to be done the ability to purchase so we looked at the conversion to successful purchases and we compare that between the mobile application and the mobile web users and once we did that we actually saw that the conversion rate for mobile web was lower than the conversion rate for the mobile application and we already know that they have different experiences in the mobile web and the mobile application the mobile application is much more personalized it remembers some user information so it's streamlined some of the steps so this made it quite evident that what we need to dive deeper is into is the conversion rate for mobile web users and that is the product API that was decided to focus on so that's pretty much kind of summarizing step two step two it's about defining the product API this is the product API that was defined to focus on and again if we take it back to the okay our framework imagine that the company level objective was about customer lifetime value and then the product API was actually increased conversion rate for mobile web users and on their first purchase okay now that we know what we want to measure we can go to the execution piece about how to use that information and actually improve the experience right and solve the gap so what are the steps that we're going to take in this case we still need to know what is the exact pain point that the user experiencing meaning we know that there's a different conversion rate but nothing more we can come up with a solution at this point in time right so we have to dive in even deeper than we then we didn't turn out and what that means is that we have to look at usage data that's relevant to the product API so we started off by investigating the funnel steps right a purchase flow has multiple steps it starts off from filling your cart and then you go to providing contact information payment information address information only at the very end you basically submit get confirmation and the food is you know heading your way so there are multiple steps where there could be friction points and there could be abandonment of that flow and we want to analyze that flow specifically for the mobile web users what we saw from the data is that one out of five visitors were running into a missing information error meaning they were trying to make a purchase and at the last step they got an error that that's basically that you're missing information so after that we see that there is 20 percent of users that never tried to complete the flow exactly at this point they're already gone they don't try and fix the forms fix their payment method fix their address they just they walk away that's fairly significant meaning if you kind of calculate it so one out of five that's 20 percent and 20 percent of that that's quite a significant rate of overall users out of the mobile web that never finished their their initial purchase probably not likely to come back and and execute on an additional purchase which means that it influences the overall average lifetime of that cohort of users of mobile web so at that point we have to ask ourselves why why don't they try again why one out of five are running into this error now if you remember we discussed the importance of utilizing multiple types of data this is exactly the point where a product manager needs to dive even deeper and not only look at the aggregated data on the funnel's data but actually look at evidence to really understand what's the pain point for that user so what happened in this step is basically looking at recording sessions for evidence this is a digital product all sessions are recorded last box provides this capability so you start looking at sessions and when you start looking at sessions it was observed really quickly that users failed in two places one was the address details and one was the contact information submission but the error message was only at the very end not specifying what was missing meaning the users themselves they didn't know if the issue was in the address details or in the contact info we inferred that that was the case because for the 80% that didn't abandon we saw that those were the things that they were fixing so the problem of not having a clear indication for the users made it really tough for them to change the information they already submitted which led to that 20% of abandonment and that's that now we have to send the pain point and we can start coming up with ideas of how to solve this issue right now we all we all know agile and we all know the importance of lean in theory but I'm going to show you now how you use data to actually execute lean and build true MVP's okay so when you when you see when you see an issue and you now know what you need to solve you start coming up with ideas on how to solve it right so now in this case prioritize the different ideas were fairly simple it's a fairly small issue to resolve so the first idea was to add real-time validation for each field in the flow so you don't have to get to the last step to basically find out if something went wrong now the next idea was to improve the UI of how the important information is presented imagine you know credit card details to separate the different sections of the credit card information to validate according to the last four digits that it's a valid credit card number or date fields etc etc etc step number three was about using auto detect of the address by using the device location I mean we're using we're talking here about the mobile flow right it's the mobile web but you still can use the device location and then step four and five were about motivating users to actually register so their contact address and payment details could be saved and suggesting and promoting users through the experience on the web to convert to be mobile app users where the experience is already personalized only information is saved etc we as product managers we don't want to execute on all these steps in parallel right so what we do is we want to execute in a build measure iterate cycle what does that mean it means first to implement one solution at a time not all of them in parallel um then when implementing a solution right after releasing it to measure if the in this specific case if the conversion rate actually increased and I'm reminding that we're talking here about the conversion rate of the purchase flow so on their first experience which leads them to not come back to the product if it actually increases then we want to validate if it actually increases the user return rate meaning it's actually connected to the fact that we return and and purchase again and then in turn will increase the customer lifespan and then if this is effective and we still want to push that that conversion to a higher rate then we can decide on implementing the next solution right it's an iterative approach and continuing to measure and validate and over time to track the customer lifespan which is which is already the metric that's more related to the business kpi now what's important here is to also understand that we can use data to decide when to stop or if a solution that we thought would be effective is effective or not in this case think about the mobile application flow if that flow is converting quite well and it's already providing a personalized experience we could use the conversion rate that we have on that flow as a benchmark to what we want to get at at the mobile web and once we gather we might not choose to execute the different steps so potentially steps four and five here would never get executed so overall this is how to apply the framework that I suggested and if we if we try to kind of recap we went through from the start of defining a business kpi right down to how to build a solution for a product problem but we can see the threaded line in between so at this point what I would love is to capture a few key takeaways and stress their importance in context of what I've described and I hope that it will not resonate much more than as if I was to present those at the start of the of the session so key takeaway number one focus on user value will impact the business kpi basically it's all about investing the time to methodologically define the relationship between the user value and the business kpi and if you do that well and execute in alignment meaning choosing to invest in the things that support that relation and choosing not to invest in the things that don't support it then you will impact the business kpi as a product manager secondly the relationship between data and value those two concepts they fuel each other if data cannot be measured then you basically cannot know sorry if user value cannot be measured then it can be optimized and that's why we need the data right but also the data a lot of time will help us to define better what we think are the value pillars for our users so it's a cyclic kind of process you have to look at data in order in all different types of resolutions aggregate and qualitative and you let you have to look at your what you would call the value pillars of your users that you can come up with by defining persona type user interviews and you have to correlate those together all the time and cross-reference them to come up with the holistic approach of how to drive value and then the third takeaway is about connect product kpi's and usage data to business kpi's and strategic goals and what it means is that the key to effective execution is based on the fact that you're able to systematically connect and explain the dotted line between those different hierarchies of the data that you have meaning the business kpi the product kpi and the actual usage data and if we're able to do that then we have an operating framework that enables us to understand how doing some change in the product affects the top level business kpi's and it actually assists us with removing the guesswork meaning if we don't have that dotted line we do stuff in the product and we hope that it does something in the business on the that it influences the business kpi but we don't understand the direct relation the next key takeaway is about using multiple data sources to validate your assumptions so as we reviewed there are multiple types of data right there is feedback data customer experience data tracked by systems like glassbox technical issues like slow responding web pages user interview data there is direct data and indirect data meaning that you infer it every type of data point gives a different perspective and a different type of hint to a problem or an opportunity that a product manager can exploit so using all those different types of data together and combining them and reaches the product manager's ability to understand the needs of their users in a holistic manner and following that it's really really important to incorporate both qualitative as well as quantitative data in your analysis if you try to understand what's the difference I think in general quantitative data enables you to find a problem and quantify it for example seeing a drop-off in a funnel but qualitative data enables a product manager to come up with a suitable solution because they already see what the user actually does or they hear from the user what's the pain point as well as refine the impact of the problem because sometimes you understand just from watching or talking to a user that without solving a specific pain it's detrimental to them being able to achieve value out of your product and obviously post release of a functionality to be able to use session recordings and to the user interviews is a really good way to validate using qualitative data are you choosing the right solution and quite similar to the previous one but it's really important to collect evidence and what I mean here is that sometimes a product manager or product analyst they will not know for sure what is acceptable when you look at aggregate level data meaning if you've got a drop-off in a funnel or if you've got a drop-off in adoption over time graph to know for sure if it's reasonable or not is really really hard without looking at the evidence so in order to get to root cause incorporating evidence is critical and that type those types of evidence again could be user interviews could be recorded sessions could be feedback that's collected from within your product by an NPS survey right that has free text and it's really important to use that to validate what you think are potential problems and opportunities from the aggregated data and finally working iterations deliver fast and minimize waste so this goes back to being lean and practicing this out of an agile framework but if you want to actually be able to define what's in the MVP do I go to the next step of enhancing this functionality what should be included in the next step of enhancing this functionality then you have to use data that will pinpoint you to how much value you're creating for the user it's not only about the ceremonies of edge on it's actually about utilizing the cycle to inform your next steps of development and the outcome of that would be faster delivery of value and minimize waste meaning you'll choose what not to invest in and you'll choose what to invest in and overall you'll invest less to achieve the same level of value for your users and this is it I hope you enjoyed I do want to thank you all for joining me and if you're interested to hear more about the digital experience in general and specifically about how AI is impacting the digital experience world and connect with leaders from all around the world that really care about this topic it would be a great opportunity for you to join us in digital world you can see all the details over here we got a virtual event on September 13 and you can sign up in our website so please do and I will meet you there thank you all have a good day