 This patient will be having now is a product usage metrics practical trainers from Kiran kasha, so he's on stage now. Yeah, thanks for the Let me share my screen So, thank you all. Thanks for joining good. Good evening and good morning to everybody from wherever you are and Before we start the topic let me Introduce myself. So I am Kiran. I work as a Strummaster at Lowes and And I'm also an open source contributor and and I'm a blogger Also, I'm a conference speaker. I wherever possible. I try to share what I have learned and And I love playing badminton and and I'm a traveler So, yeah, this is my intro So before before we start the topic the the topic name is product usage metrics, right? So what exactly do we mean by product usage metrics? Let's try to understand that and then we can then I'll share you few stories among this topic So here let me let me try to divide this Term into three. So product usage and metrics. So what is a product? product is something which solves a problem or Or which gives you something of value that that's a product and usages. How do you use the product and The final term metrics. How do we measure? to use the product So this is what product usage metrics means. So what we are going to talk for another like 15 20 minutes is how You know, in my experience, how we made use of How we measured the product is being used by the customer so So we I think we are very well familiar how to I know how we build product There are multiple process like Scrum Kanban. We have multiple things But this topic is mainly around what happens after we give the product to customer How the customer is using our product? How can we? I know what are the insights we get based on how the customer is using the product? So this is the talk So what I'll do I'll share you three stories. How we made How we measured the product being used, right? So I'll share you in the form of three stories So So let's to to share this story. Let's let's assume we have a team, right? The team contains the product owner Development team and Scrum Master Let's let's discuss the the first story So the one with the prioritization problems so so we often have this problem like The product owner often ends up saying, you know We have Priority, but but he finds it difficult to you know, prioritize a single item if two or three High priority items are there. We often end up product owners, you know, all these three items are equally important and And it's difficult for me to prioritize And and I'm sure you all know there are other other techniques where you can prioritize the stories Let let me this this this story. I'll try to share another technique which we can use to prioritize your story So let's say we have a product and it has two features feature one feature two and We have two stories here story a and story B so story a modifies feature one and story B modifies feature two and Both have equal business well, right and both have equal agency So we often end up in this scenario where It's difficult for product owner to choose which one is important here So so what did we do here? So we started, you know, we heard we heard about this Concept called product usage metric. So what did we do? We created a data table and We started collecting metrics here. So for example, let's say feature one We had multiple User subscription. We had a diamond kind of a customer and silver subscription customer So we understood that diamond customer work was using the feature one on an average 350 hits per day when I say hit he was using the feature around 350 times per day and And the same feature was also used by the silver customer And that was around 450 hits each day So like this we started collecting data for all the features Right and and and it also based on the user category whether he's a diamond customer or a silver customer So as we started and another point to add here is we didn't find much change in performance while we were collecting data So as we as we started collecting data We generated a report which was something like this. So we understood that more number of Users are using feature one more than feature two So it was it was way more the feature one was used way more than feature two irrespective of Whether the customer was you know diamond subscription or a silver subscription So this held the product owner to sort of help his prioritization problems Now now product owner has two options. So, he knows feature one is being used more Let me try to improve the feature one so that user can have better experience because he's using the feature one more No, and another way the product owner can think is I see the feature two is not used much at all. Is it because Is it because that you know the the feature itself is like that or it's not used much or is it because it's not so Friendlies that is why the you know the user is not using it So so here the product owner can get more insights and then he can decide which particular story He can pick it up. I mean can he pick up can he prioritize the story which changes the feature one or can he Pick up the story which modifies the feature And it did not stop there What did product owner do he went on? You know, he went to use it even better. So what did he do? He understood there is a customer eggs Who is very important to us? Right and his renewal is ending this month Let us prioritize feature one because he uses this feature more. Let us try to make it better and You know and it gives us more probability of retaining the customer And it is another way this is the extension of the bus story where product owner leverage leverage that data we got So there ends our feature one and and let's let's discuss the the story to So This story is around how it the cultural aspect of how this matrix helped us to to know to to do better So it helped us to motivate developers. How did we do it? There was a time where We received an email from the customer saying Thanks a lot. This features says at least 10 minutes of manual work time. I was doing and While the time it reached developer developers were saying we never knew it We never knew we are adding this much value to the customer So what did we know what happened here? You know the product owner gave vision and help the importance all these things were good enough Good enough for the developer but adding to that This can add even more motivation to developers when they see value of their work actual value Where it is helping the customer, right? If if so, what did we do? We took it to we took it One level more and we started generating reports Something like this. So what did we do? We we displayed this We started collecting how much time for each feature the customer is saving for example, because of this particular story or this particular Product of feature we were able to help customers save like, you know, 10 minutes every time he uses it Or five minutes every time he uses it. So we started adding The the time each time the user uses so this was the displayed out in the open Where with the developers it in the office so they can see it every time and they And and it makes them Visible of how they are actually adding value to the business right this this help the developers to Contribute even more and it motivated them they they got to see their actual value what they were adding and And all these graphs which you are seeing right right now It's the graph which you're seeing this helped we we took it from the Grafana So it helps us to generate all this kind of graph So So let me see how much time we have Okay Yeah, so let's discuss one more story before we end so so another story What we did was So the product owner was a little confused. So how did we help? To clear the confusion of the product for you here the the confused word I'm using not in a negative sense So the product owner wanted to make sure the best value comes you can he can he can get the best value of the Work done by the developers. So in that term he was sort of confused what can give the best value So there was a particular feature What it did was it it gives the cost we give the user to configure an email and And then send out you might have heard this feature like called as email campaigns So where user can perform the email campaigns. So what he does is he'll he'll just put a lot of You know email recipients in the two column and then he goes to the the email body Pills in the relevant information and this is this is how the flow goes Now now the product owner was stuck which approach to follow Why because we had two solutions Right in front of us and both solutions seem good. So the solution a was let's say Screen one enter Email recipients and screen to draft the email body. This is how the screen goes in the solution a and the solution be was First, let us show him that no to draft the email body in the first screen and later We last the user to enter the email recipients So here both both solutions seemed good, but we were not sure which one to take So here we had to consider either one of them You know, we had two options. We can either consider either one based on You know just gut feel or experience or the other The other way around is We can decide this solution based on the actual data we get So what we did was we we requested some beta users to use both these solutions For example beta users. I mean, they are actual users, but they were willing to help us, you know in testing or Or to check some features if they are okay with it or to get their feedback So these are set of beta users where we had We requested them to Use the both the solutions. How we did was we had one set of beta users We requested them to use only the solution a and we had another set of beta users We requested them to use only solution be Now the next step, how do we know which solution was better what we did was if the solution is good What we did is we asked both the set of beta users to raise a support ticket whenever they have a concern Right, so if the concern can be anything whether they are not understanding the new feature or or there is slowness or No, we just can't use it. This is not user friendly. Whatever they feel There is a support ticket So what we did was Which ever solution gets more number of support tickets, we will ignore it We will take the solution which was quite natural for the I know for the beta users to use which was which was more user friendly and What we understood the solution be Beta users were raising more number of Tickets Right, so we understood that let's go with the solution a because people are it looks like The beta users are getting along with it pretty well without raising any support tickets and Also, whatever the support tickets were being raised for solution a right We we also went through the tickets and we understood what problems they are facing even in that and and even that Even that changes we made in our Feature and then we were able to give it to the actual customers So, yeah Here it is. So thank you all for listening. Let's Let's hear if you have any questions there Okay, Kiran. Thanks a lot. That was awesome We have some questions. I guess you can there's one under Q&A A bunch of discussion going on in the audience Yeah, you can probably go through them and answer them as Yeah, I'm looking at them in case of Difficult or legacy customers. We don't get any feedback from them They just want to get their work done What do you suggest? So is this question for the third scenario Yes, okay so so here we can so one is We will have somebody from the product Business side right like the product owner will be representing from the business we can We can either create a relationship with the product owner and explain him how to get in the beta users How to get in the actual users to help you along or If not, you know, we can we can you talk to even the leaders and explain them this problem. There is no I Would say there is no clear cut solution here. We'll have to We'll have to create a relationship It all depends on how good relationship we have with the management with the leaders and if you're a Strum master, maybe you can you can post the product owner to to help the actual users to help us or Or if you're in management, you can directly talk to the business and see get sort things out So did I answer your question? I'm Or was it any helpful? But I didn't get your name there So she yeah shock she Yeah, there is no clear cut answer there. We'll have to just trial and error. It's this is all That's a be testing rotation. Yes, that's a be testing. We can call it Me know how such metrics can be built on a product where we have multiple customer and they require their customized version Maybe we can Create multiple tables For creating graph That's a good question. I even even I need to explore their menu Yeah What tools did you use to collect the analytical so the tools was Grafana, right? The the table we added the once we got the data on the data table We used Grafana to create all the graph Any any other questions from anyone? Alright Okay. Thank you so much for Giving a platform to to talk. Thank you. We come and thank you