 Hello everyone. Welcome to today's session on how to effectively measure product success. As product managers, it is important for us to have metrics informed strategy for a product. However, precursor to that is to understand the product performance and what metrics are you measuring and how are you evaluating. In today's session, or I would say a presentation, I'll help you avoid some pitfalls in defining those right metrics to measure product success. Before we dive into it, a brief introduction about me. I'm a product manager at Squarespace part of the growth team responsible for the conversion world where I'm responsible to get users to start a trial. It's been almost close to a year of working at Squarespace before Squarespace. I worked as a product manager at on deck. It was a small business learning company, and I got an opportunity to work on different products and different related to different customer journey. And before that, I have an engineering background and leverage of business degree to transition to product management. So, going back to what we're going to learn. So, the first learning, like throughout my career, as we were defining the product success, what I learned was we need to look beyond vanity metrics by adding context. And I know what was going, what would be going in your mind is like, what is exactly vanity metrics. So, vanity metrics are those metrics that look good on the like that look good or impressive on the surface, but do not provide any understanding or context into what have what's happening. It doesn't help you to find what your future strategy should be. And like some of the examples like number of app downloads. It doesn't tell us anything. It just tells us that how many app downloads have happened, number of visitors on your website. Again, it doesn't tell us anything. Why, why do we know what are some other other things around those things like it, it can often be misleading, and it will not help you improve your product success in any form of it. Now, let's look at an example of an Instagram influencer. As an influencer, you have 100k followers. On surface, it will make you think that this influencer is really successful. It's like, oh my God, this influencer has 100k followers. But what if I said that it took five years for that influencer to get to this number, or only 500 followers engaged within any posts. Will you still think that having 100k followers is a success. Maybe not, right. So, sometimes you need to make vanity metrics actionable like these are these are important metrics to give you a little bit of indication how your product is doing, where it is at, but to inform you around the strategy. There are some other aspects that you need to bring in. So some of those things are timeframes, like, okay, how much time did it take to get to that number. Data traffic sources is that traffic on mobile desktop, is it from US region, North America region, is it from US specific, where is the traffic coming from. The other one is the rates, conversion rates, how are those numbers converting. You can also look at those numbers on a per user basis per visit rates basis. And lastly, you can do a ratio between those metrics. And I think once you do that, that's when it starts telling you some story. But you might be wondering, but why do some companies still use it. Vanity metrics can be helpful. I can imagine imagine a product which you're just launching for the first time it's an early stage as a product manager when you're doing and when you're launching a feature. How would you know what's the success you have to have some benchmark. And I think that's when the vanity metrics is a good way to go about it. Okay. So if you want to set a milestone on a success metric is like reaching 100 k website visitors is your goal, or getting to your first 100 customers is your goal. And I think that that's when those metrics are useful, but still making them actionable metrics can go longer way. Now, coming to a second learning percentages can be misleading always have a secondary metric to give a full picture. This is, I know percentages. Yes, this is a very obvious learning and like, yeah, we know it. But sometimes a higher percentage. It's helpful to like remind ourselves that what's happening. Many times we get so busy in our work that just percentages are more than enough so it's more for us to remind that how percentages can be misleading. So sometimes our higher percentage mislead us to show a higher improvement or lower percentage can mislead us to show a lower improvement. So adding another color or another metric as a supporting metric can give us the real picture. Using confusing are misleading numbers like personal changes puts you at risk of over or underestimating your impact. Let's look at another, let's look at a website example. So as a product manager, you want to increase on your traffic, you ran an ad marketing campaign, and you got the report. What does the report says report said traffic increased by 200% This is exciting right this is a very impressive improvement. But what it doesn't say is whether it was an increase from five to 10 traffic, or from 10,000 to 20,000 traffic. In this case, metric secondary metric could be an absolute number of landings. If I say the absolute number of planning is 50 K. What would your answer would be, yes, it would be that you improved the landing by 200% from 50, from 50 K to another number, which is which is impressive. So this gives you a little bit of a view into. Yes, this is a very good improvement. Let's talk about another example of using a percentage as a relative percentage change. If you launched two features to see whether it's solved a given problem. Feature a was for desktop it had a relative improvement of 20% feature be was for mobile and had a relative improvement of 10% product manager if you just looked at these numbers. You would make you would make a decision to roll out feature a but that would be wrong, because what you didn't see is these numbers that the 20% improvement was that an increase of conversion rate from 0.05% So an absolute percentage change is just point one person, but the percentage change is different. Looking at these numbers, both improvements are same. And what would you do as a product manager on this. Yes, you would make a decision. And at that point of time, you need to bring in your product sense hat and decide that which one would you prioritize. Maybe you launch both of those features, maybe you launch one of them and you launch the second feature later, but until unless you looked one level deeper, you wouldn't know that both of those features had a similar impact. Hence, percentages are important to show emphasis, but can can give incomplete picture. Lastly, what is the last learning time series are important to tease out the seasonality effect or trends. Throughout the product management career, what I've seen is. Yes, you can look at absolute numbers you can look at relative numbers but until unless you have the time series you wouldn't understand what's happening with your product is it is it overall growing declining. So you need to take a step back and look at a completely time series sometimes you look at one year back sometimes you look at two months back or sometimes you look for our five years back. So let's talk through an example. Imagine your business will sell plants. As a product manager you are responsible for a product to drive plan says your manager reaches out to you and asks, why are we seeing a sudden increase in this in sale. And you go back and you remember that you launched a new feature just a week back. If you if you look at this. Yeah, like you would go back to your manager and say that. Yep, we launched a feature. That's why I'm seeing a such big jump, and you can see it's a very very steep improvement. But what if you look back the full year. The answer change. Yes, when you look back full 12 months, the improvement that you saw is diminished a little bit there was a seasonality effect into play. This is a month when plant sales are usually high. Therefore, to get a full understanding of an impact of your new feature, you have to look at full time series. You have to look at your your ear over your increase to understand the real improvement driven by the features release. So if you look at this, it's not saying that the future released and see an improvement in the sales. But what is saying is that you might have accounted for maybe 115% improvement in sales, but that's, that's not the real impact. When you look at the year you over a year, it would be just a 10% increase or 15% increase. So it's important for it as a PM for you to take a step back and look that is there a time is this a time period when you see some kind of seasonality impact. Final wrap that I would have for you all metrics are core to PM's job. And it's important to bring context and time to tell a full story. Thank you.