 Hello, everybody. Thank you for joining me today. My name is Ajay Swamy and I am a Product Leader with Amazon Web Services. At AWS specifically, I am part of the AWS Solutions Group, where I manage various solutions and products that we build towards our customer segments. Within our worldwide customer segments team, my focus is on small and medium businesses, greenfield enterprises, DNBs, ISVs, digital native businesses, independent software vendors, and startups. So I build and deliver various products and solutions for all of these segments across the world. I'm glad to have you with me today because today we're going to be discussing something that's really important for you as a Product Leader or a Product Manager. It's about measuring what matters in your product. Without measuring, you can improve your product. Without measuring, you cannot iterate. And without measuring, you don't necessarily understand where to actually focus your investments. So today we'll be chatting a lot about how to actually set up metrics for your product and then really sort of become outcome-driven and metrics-driven so you can have a more productive and efficient working mechanism to actually impact the lives of your customers. So today's agenda is focused on these five topics. The first topic is around being more outcome-driven versus output-driven. We'll dig a little bit more into why that is important. The second is about determining the right product metrics for your product and your organization and sort of how to recognize which ones to use versus others. Here specifically, we'll talk about vanity versus clarity metrics and where they are useful. Next, we'll talk about setting up the right product metrics from the get-go. We'll take a top-down approach and then apply that to across the different product features or different product areas where you'll be measuring different product areas and their outcomes and how they sort of roll up to your one North Star metric. And then lastly, we'll be talking about OKRs and KPIs and how you sort of measure those. And then we'll wrap up with communication. By the way, if you don't communicate all of the work that you do in terms of measuring your product metrics, it's not really that useful. You've got to measure your metrics and you've got to celebrate your successes. All right, so with that said, let's get started. Okay, so let's talk about being outcome-driven. So in order to be outcome-driven, you need to sort of get into the mindset of what is really impactful for your customer. So Marty Kagan, the founder of Silicon Valley Product Group, he championed the term Empower Product Team versus a feature product team. Across many organizations today, product teams are broken up into feature product teams where PMs manage one to many different features for a product suite. However, the work that goes there is around being able to talk with the field, being able to talk with your customers, and then understanding their pain points and opportunities for improvement and being able to design features, build those features, and releasing those features to solve for that pain point. But is that really outcome-driven? No, that's more output-driven. So let's sort of walk through a very simple example. So let's imagine you have work, you're exhausted, it's afternoon after lunch, and you're feeling tired, right? It's the afternoon blues. So you go to the cafe next door and you buy yourself coffee. At the coffee shop itself, you know, to some people, the outcome of the event is the coffee, and the output was you walking to the coffee store, picking out the coffee that you want, paying for it, and then getting your coffee. Also, the output could also be all of the work that went in at the coffee shop to actually produce the coffee. And you know, some people just walk away with that as the measure of output and the measure of a product. But that's not really, really complete. The outcome is not really the coffee. The outcome is that you feel more energized and you're able to feel more productive in the late afternoon and you can sort of get to all of the tasks that you need to get done. The outcome is the way in which your problem was solved by the output. Now, if you sort of plug that mindset, the mental model into a product, it really impacts how you sort of define product metrics. So, for example, if you focus on product measurement from an output perspective, the number of features delivered, the number of bugs fixed, that's not really outcome-driven, that's more output-driven. By focusing on outcomes, for example, let's assume that you own an e-commerce product and you have various features within that e-commerce product. And one of the outcomes that you want is, hey, I want to reduce my card abandonment rate. So, you, as a PM, can actually dictate that specific objective to your product team that's responsible for looking at the shopping experience all the way through checkout, right? And using that as an objective or a goal, you can create that autonomy and allow the team to find the best design, the best builds, the best solution to actually accomplish that goal. And that might take one or more iterations, right? You're not going to get this right the first time. But nonetheless, you're able to at least look at that objectively and say, hey, is my card abandonment rate going up or is it going down and how do we get to improve on it? That is being outcome-driven. By being outcome-focused, I also help you become more customer-focused because you're really sort of focusing on what problem you're solving for the customer, right? What are the jobs to be done by the customer and how well they're accomplishing those jobs to be done? Is it making them more efficient? Is it really solving their pain point? Or are they just sort of going through the motions because there's no other alternative? If you look at it from back perspective, you're really sort of laying the foundation to become more customer obsessed and building the right set of features that result in the best outcomes for your customer. So moving on, let's talk about determining the right product metrics. You have to recognize which metrics are more applicable to a particular situation. To me, the way I've always looked at metrics is across these two broad buckets. On the left-hand side is something called vanity metrics. And on the right-hand side is something called actual metrics. You can also hear people talk about actual metrics as clarity metrics. So let's go through what the differences are between both of these metrics and where it's applicable. So when you look at vanity metrics, they are generally good for understanding the size of the business, right? They're sort of understanding. It's good for you to understand how many people are aware of your product. But they're not really good for you to understand whether your product has a good product-market fit or whether how many customers have really adopted your product. However, if you look at actual metrics or clarity metrics, they are really good indicators that you have a strong product-market fit. And they can also tell you very quickly that your product-market fit is improving or you're losing it as a result of releasing bad features or other new entrants into the market. So when you look at vanity metrics, like I said before, they sort of help you determine the size of the business, right? These are sort of the metrics that talk about gross quantities. You can track them pretty easily, right? So if you look at usage analytics, for example, you can look at the page visits, you can look at followers, you can also look at if you have an app, you can look at number of times that apps have been downloaded, you can look at the total customers acquired, total revenues, etc. This is really good when you want to understand how your product is sort of moving in a certain direction and how you want to scale it. So for example, like if you're not getting a whole bunch of users to your website, then obviously, you know, that's not really good because you can't scale number one because you can't convert. However, when you look at actual metrics, you know, they are really representative of how a particular customer segment is behaving. You know, you can sort of really sort of drill down and understand the personas that are impacted, the segments that are impacted, and it helps you sort of understand that through the lens of ratios and unit economics. So for example, you can look at conversion rate and you can see how that's trending. You can look at the customer acquisition cost for CAC and see what's happening there, right? Because ideally, you want that CAC to reduce over a period of time either through referrals or word of mouth or strong network effects. If that's not happening, then you know that you're not probably getting a stronger product market fit as you mature. There's other things such as churn rate, customer lifetime value, NPS, et cetera. So clarity metrics sort of really, really help you understand whether your product is doing the right thing for your customer. And vanity metrics sort of help you understand, okay, like, what is the size of my product that's being impacted across the space where it operates. So really sort of understand which ones to use when. And sometimes, by the way, what I like to do is use vanity metrics as a precursor to clarity metrics. So for example, I like to understand the number of visitors or the number of downloads and see how that trends. And I like to sort of tie that into the conversion rate and see if there's a correlation there. So think about it in terms of vanity and clarity metrics and make sure that you apply the right set of criteria for your product to better understand the problems that you solve for your customers. So let's talk a little bit about within your product organization. If you haven't done this before, that's quite okay. If your practices are ad hoc and you're measuring some parts of your product, but not others, that's okay as well. It's never too late to actually sort of set up something more structured with your product organization. And the way to do that is sort of what I have here on the slide. So you want to create a holistic top-down view of what you think success constitutes. So starting off, you want to look at your North Star metric. Your North Star metric or NSM for short, it should really be your focus to track your company and your product's growth. It's singularly the most important leading metric that allows you to quickly align and your team and your company around your product and it sort of brings your focus, clarity and sort of customer focus around what problems you're solving for them. So if you look at some of the examples in the market today, if you look at Spotify, their North Star metric could be something as simple as time spent listening. Why does that matter? Because the more time you spend on Spotify, the more revenues they generate for their artists, the more ads that are shown in case you are a premium user and the more value that you get as a customer. So it's not just music. They're interested across podcasts, shows, et cetera. So that's a pretty good North Star metric. Facebook could be something as daily active users or monthly active users. Daily active users I'm a little skeptical of because it varies a lot. You want to be able to set up a North Star metric that's not sort of thrown off by periods. So a monthly active user metric is a good North Star metric to measure. Airbnb experiences, for example, that could be the number of book experiences. So this various way to sort of corral your team around the right North Star metric and then being able to say, hey, you know what? This actually makes sense for us to understand if we're moving the right direction. Second, you should be very aware of your feature metrics. Feature metrics generally act as input metrics and they sort of roll up into your North Star metric. They should materially impact your North Star metric. If it does not move your North Star metric, start to rethink whether this particular feature metric is really the one that you should be tracking. So let's go back to our previous example for Spotify. For Spotify, time spent listening can be impacted by various other sub metrics or input metrics. So for example, you could track how frequently users come back to the app and start listening. Can we do something there to build and enforce that behavior to use Spotify more? That could potentially be an outcome focused objective for one or three product teams that's focused on customer conversion and customer retention. So that's one area where you could influence that. Second, it could be increasing per session listen times. As you sort of understand your customer behaviors, are there areas where you can say, hey, I think I understand the customer's behavior. I think I understand where this person uses Spotify during what times. And I think I'll be able to either recommend or enhance the experience of that customer during this time so they can actually listen more. So again, this kind of outcome driven objectives will actually give your product teams autonomy over how they actually try and achieve that objective. So that's a couple of different examples about how your feature metrics actually sort of roll up into your Nordstrom metric and that sort of helped you become more outcome driven. So again, empower your feature product managers. Make sure they understand what the Nordstrom metric is and make sure you understand and work with them to make sure that you are defining the right metrics that roll up into the Nordstrom metric. So this is great, right? You've done your work. You've identified your Nordstrom metric. You've identified your feature metrics. You have a good pyramid that you have the underlying metrics that roll up to your Nordstrom metric. However, that's only half the battle, right? The second half is actually sort of being able to put that into work continuously. At Amazon, for example, right? We measure everything, right? We measure deployments. We measure page visits. We measure vanity metrics. We measure parity metrics. But we also sort of are very thoughtful about when to actually act. So we have monthly product sessions or monthly product meetings. We have quarterly product meetings. And then we sort of look at all of these identifying metrics. And we actually set goals at the beginning of the year to hit certain objectives. And then we get to sort of measure month over month, quarter over quarter, how we're doing and what those metrics are telling us. So we can be very thoughtful about assessing where we need to improve and where that investment needs to go either from a quarterly perspective or a monthly perspective. And then we can figure out what to change in terms of our product or our feature. So in a nutshell, when you sort of identify the metrics, make sure you capture those metrics. Make sure you analyze that at a frequency that matches your particular use case. Act on those metrics, monitor again, and then repeat. If you do those things, you'll start to see a gradient where you'll get to see a product improve over time and customer experience also improve over time. Another key area is around utilizing OKRs and KPIs. Now, you as a product manager within your organization, you might probably not have the capability or the remit to actually set up these OKRs and KPIs. But if you do, it really makes sense from a product organization perspective to align with your senior leadership teams to establish some of these OKRs and KPIs. So what are OKRs? OKRs, they serve for objective and key results and they really sort of think about it as providing the missing link between ambition and reality. They sort of help you break out of the status quo and then help you achieve these objectives in terms of how you want to achieve a particular goal. So OKRs are short qualitative descriptions of what you're trying to achieve. And the key results are accompanying each objective and they quantify the metrics that help you measure that progress. Now, let's talk about KPIs, right? OKRs encompass key performance indicators. They are inclusive of KPIs. They measure how you're performing to achieve those key results defined in your objective above. So really sort of the main differences between OKRs and KPIs lies in their purpose, their scope, their duration and how you sort of measure them, right? So let's take a simple example here. For example, your objective could be, hey, I want to get back in shape after the holidays. This is your destination. This is your goal, right? It answers the question, where do I want to go? Where do I want to be? The key result is I want to decrease my body fat by 5%. This is a mechanism that tells you whether you're actually getting closer to your goal, right? Because if you cannot measure, you cannot succeed, right? How do you know you're actually getting there? So the key reason your body fat is within your circle of influence, but you don't have full control over it. But how do you know that you are actually getting to achieve that key result? Well, you can measure it, right? And this is what KPIs are for. So KPIs can track, you know, the number of miles you walk per week, the weights you're lifting, how many times you go to the gym, the time per mile, is that increasing or decreasing? But they might not directly correlate to your key results because you need to measure those key results, but it at least tells you exactly whether you are headed in the right direction or in the wrong direction. So, you know, think through these from a perspective of an organization top down, right? So if you are able to define an objective for your organization or for your product org, think about it from a perspective of what the goal should be and what those key results are that actually helps you think about whether you're achieved that goal. And then you can start to define specific performance indicators that help you achieve those key results. If you can do that, I think you'll be in a better place because all of these things are the tools and the mechanisms that helps you achieve greater product market fit or greater customer satisfaction. All right, so this brings us to another part of the metrics discussion. So there's other types of metrics that you utilize, right? And they sort of broadly fall into three different areas, proxy metrics, counter metrics, and then leading and lagging indicators. So what are proxy metrics? The short answer is that proxy metrics, they help you, they exist to help you determine whether your particular hypothesis will improve a high level engagement metric within your product. Proxy metrics are generally more easy to measure and optimize and they change more frequently. So for example, if you look at conversion rate or if you look at registration rates, these are probably proxy metrics that says, hey, your product actually has a good value proposition and people like registering or using your product because they're converting. However, it's not really a stand-in to say, hey, is the product actually doing a good job from an outcome perspective, right? Is it really sort of helping the customer achieve their jobs to be done? Is it really sort of achieving the customer's objectives? We don't know that, but at least it tells you directionally that you're on the right track. The second metric, which is actually very useful to measure and I alluded to this earlier is counter metrics. And the key to this is that you want to pair metrics with counter metrics for balancing. So for example, you can pair the growth of your product with quality metrics. What does that mean? So you can probably see that your conversion rates have increased, a lot of people have been registering for your products and that shows interest on your product. However, let's assume you survey those people via NPS and you realize the NPS is really poor. So these are counter metrics and then you get to start to understand deeply why that is the case that's happening, where on the surface, the product's value proposition might be very, very strong and that's why people convert, but as it's hard to use your product, that quickly fades away. And also another good example is about web trafficking conversion rate. A higher web traffic to your website or your product page or your landing page does not necessarily relate to a higher conversion rate. So you can start to see how those two work in tandem together and you can start to optimize for the right shortcomings. The third metric is basically they're called leading or lagging indicators and these are super useful. Lagging indicators already tell you what's happened and the leading indicators can point you to future success or future friction in your product. So within Amazon or within AWS, I should say, we track both, right? When we have our monthly business reviews or quarterly business reviews, what we are actually tracking are lagging indicators. We get to look at revenues, we get to look at how many customers have used the product. We get to look at other kinds of metrics that tell us what's already happened. However, leading indicators can also sort of help us understand whether we are on the path to growth or whether on the path to decline. So some of the key examples here are profit, revenue, et cetera, there are all lagging indicators because that's already happened. However, if you start to look at a trend in growth and the number of signups or the number of registration, that can potentially sort of help you tell a different story which is, hey, you know what? People are interested and that might lead to greater sales. That might not always be the case, but it tells you that he's sort of moving in the right direction. So be thoughtful about what kind of metrics you want to use and how you wanna measure them. Proxies can help you sort of uncover and optimize. So for some of the deeper level attributes, behavioral attributes that you might not be able to directly measure, but it helps sort of get you in the right direction. Countermeasuring sort of helps you tell a different story and help you sort of double check yourself whether you're moving in the right direction. The example that I used earlier was around conversion rates and NPS, right? A growth in conversion rates doesn't necessarily mean that the customer is fully satisfied with using your product and then leading a line of game indicators sort of tell you where you've already been, what the product has already achieved versus what is the potential for your product to achieve. So all of these things are very, very useful and be thoughtful about how you use them. So this brings us to the end. So it's not just enough about, you know, with measuring your product. You have to communicate the value of the product it has had on your organization and better yet, it's if you can sort of tell the story from the perspective of the impact on the customer. Metrics are a tool for you to celebrate your wins, right? It's not enough just measuring and improving but you have to sort of, you know, take some time to reflect and give credit where credit is due to the PMs, to the product team, to your engineering team, to your design team, to the sales teams and the product ecosystem because you're all in this together, right? So make sure you understand that and then make sure you make some time to celebrate those wins. And then lastly, you know, highlight you as a PM. You have to highlight your positive impact on the organization and the impact of your product team, right? Make sure you have sessions such as state of the product. During the state of the product, you can get to talk about, you know, the wins and the successes you've had with your product but also acknowledge that there are other areas to improve. This tells the bar organization what's happening with your product, how your, what your key successes are, where are you the areas to improve and what's upcoming, right? So make sure you always take some time to measure, to analyze, to celebrate and always improve. So get building, make sure you work with your teams to figure out what the best metrics to measure are and then, you know, always improve. If you have any questions, feel free to reach out to me for advice. I'm on LinkedIn and on Twitter. I will, the product school will also be sharing this deck after this webinar and I hope this was useful. So happy building and have a great day, everybody.