 Hello everyone and welcome to this products cool webinar today. We'll be talking about how to measure product manager impact. My name is Josh Falma and I'm a principal PM lead at Microsoft. I work in the Microsoft Azure division, more specifically I'm a product lead in Azure Kubernetes Service, one of our most popular cloud based products. You might notice the image in the slide is not a strange shot from the Golden Gate Bridge. It's actually a famous bridge in Lisbon, Portugal. I am originally from Portugal as you can also see in the picture behind me. I have a background in computer engineering. I've done different roles from software development, system administration, then technical evangelism. I've done support consulting services, then technical sales, and technical leadership before actually joining the product world where I've been from junior PM, mid-level PM, senior PM and principal PM, and more recently a PM lead in AKS. We'll be talking a little bit about the experience that I've had. I have had the opportunity to lead teams prior to Microsoft, in Microsoft, product teams, non-product teams in official capacity, non-official capacity. That has given me a lot of opportunities to observe impact, to evaluate impact, and to really gain a lot of different perspectives about the differences from measuring and evaluating impact outside of a PM role and inside of a PM role. These are the perspectives that I hope to expect with you. I don't see this as a lecture, this is just really me having a conversation with you, and I hope that you can use some of these perspectives in your own teams, in your own career. I will be using knowledge from a lot of folks that are way smarter than me, and that I've also used and adapted to make sense of my teams, and also the result of a lot of trial and error from teams that I've been in or with for things that have worked and not worked that well for us. So without further ado, through our agenda, we're going to talk a little bit about why is it challenging to measure PM impact, how you can actually evaluate it and measure it, how you should possibly be communicating and rewarding it, and also how you can improve it. Now the short answer there really is become a great PM, and be the best PM you can be, and you'll typically will be having more impact. So we're not going to dive into that area, this is product school, it will help you break into PM and also make you a great PM. So you can just browse product school content and many, many talks and webinars and presentations and blogs out there on how to become a better PM in here. I just want to do a brief overview from a lens specific of impact and how your impact is perceived and measured and how you can improve and optimize for that. So we're going to be focusing really much on that, and not just the broader how to become better at PM. Now, starting with why is it a challenge to measure the impact of a PM? So the first thing is really the role. I mean, if I go to say my mom right now and say I'm going to rejoin a band and be a drummer in that band, I'm not sure how she would feel about that, but she'd probably have a very good idea of what I'll be doing next. And I'm still trying to explain to her today what is it that I do as a product manager? So it's a role that is fascinatingly unique. And although this meme of being hard to explain what the role is, is becoming less and less prevalent, it is still true that it's hard to explain what this is. There's many presentations about just this specific thing and what the job entails and what it should entail and the tasks that comprise of this role. So instead of going, again, too deep into it, I'll probably use an analogy that I borrowed from a friend of mine, which is the analogy of the underpants gnomes from South Park, where they had this kind of three-phase approach master plan, which is like collecting all the underpants in the world, phase one, phase two, question mark, and phase three profit world domination. And I think even though there's different schools of thought about what the PM really entails, I think most of them would agree with me if I just kind of split it out into, there is some problem space, some ambiguity somewhere. A PM comes in, Raleigh's resources from many different teams, typically, from many different backgrounds, and together collectively will somehow transform that problem space and that ambiguity into business value. So typically that is, so it's that phase two where kind of the magic happens. So, okay, so if this is fairly non-controversial, then did we just go to it, is then business value, the way to measure the PM impact? Well, yes, however, there are few issues with business value alone as the only thing that you look at when you are measuring the PM impact. The first thing to come to mind is the time horizon for that evaluation. If you were measuring your impact from like the value that you deliver to shareholders and that takes years, are you just going to evaluate that in years, basically? That's a little bit tricky. And perhaps even more importantly, it is the attribution issue, which is how much of that value added to shareholders was because of you or that person individually, when really the kind of job description that I put before is really collective with many, many individuals, again, driving from that problem space into business value. So how much of the PM, you know, how much would you attribute the PM to have changed that and to have contributed to that business value? So I found that to be a little bit flawed and challenging to just use business value alone. And the role itself with all its different hats that sometimes it has to use makes it a little bit challenging to do. Moreover, the discipline itself and the value of the PM discipline is still fairly up in the air. Once again, different schools of thought, I think more and more now we're in a golden age of PM or the teenage years of PM, however you prefer, where more and more folks are breaking into PM, more and more PM roles are being open. And that's because more and more it's clear what happens and not so good outcomes that happen when PMs are not present. You can see kind of that the whole when the PM is not there, but it's still somewhat hard to quantify the actual value of the discipline, the actual value of PM. As a PM, now we brought this to the company. It's still very, very hard to do that. And so once again, if we could do that very easily, well then it would be much easier to also quantify the impact and to pinpoint what impact individuals in that discipline would do, but that's not always easy. And finally, the latter. This is not specific to PM, I would say, but once again, it's exponentially harder in PM because of the factors that I mentioned before. In any role, it will be a little bit more challenging to measure someone's impact based on the career stage that they're at. Someone at the higher level, how do you actually measure that impact versus someone at a lower level? But in PM, again, because all these things are so sometimes subjective and depend so much on so many factors, when you add up the career ladder on top, it becomes even harder to do that. I mean, if you were, once again, that musician, maybe you know how to do a scale, maybe you know how to play a song or full sets or now you know how to read music or compose or play by ear, you can kind of see almost a split that you could say, okay, these are the levels and this is a person that is in that level. And so to measure impact would be for that person to do that and to do that and do A to B. And if you're doing A and B, then you're having impact. On the PM, that's not as easy as such. So what strategies can we do? And so my favorite framework that I have been using over the last few years and that have been working great for me is one that I heard the first from Shreya Doshi. I think this image is from 2020, but I think the first time I heard it was in 2019, which is kind of the framework of input, output and outcome. And the way that I kind of interpreted is, I see the PM role and me as a PM as kind of a system and where there are inputs, where it's basically your investigations, your market insights, your PRDs, your product requirement docs, your one-pagers, the notes that you take and send out, the communications that you send out, the decisions that you take collectively, all of those are inputs that you're putting into that system that will typically generate outputs, which could be features or the feature velocity or improving quality of existing parts of the product that maybe were causing issues and pain to customers and you just improve that quality and that's a great output as well. Or it can just maybe a feature existed, maybe even has quality, but it's a little bit rough on the edges and so the users kind of struggle with that part of the product and you might just go and smooth that out, iron out the edges there and make the experience much easier. That could, for example, try a much better outcome, which could, let's say, be an improved C-SAT. So again, you improve that experience that was a little bit rough. That drove a improved C-SAT of that part of the product, for example, so that's the outcome that you had. And I see outcomes as not necessarily direct business value, as you can see, I mean, improved C-SAT of a part of a product hardly is business value directly, but it can be seen a little bit as a proxy metric for business value because if you have customers that are now happier with that part of the product, they'll probably retain more, turn less. They might be happier, they might spend more with your product, they might stay with your product longer, they might do good PR for you, which brings all their customers. And now you start to see the increased revenue, the increased customer base, which drives even more revenue. And now you can see how that was actually a leading indicator into the business value itself. And the great part about this is that you can actually measure them in much better periods than just taking years to manifest on that business value. And you can still tie them to both on one hand, the business value, and on the other hand, the actions that PMs are taking day in and day out. The second thing that why I think this is great is that it's really works across different stages and across very different factors. But you might ask now, okay, so it's the case here now that we're gonna just do inputs plus outputs plus outcomes and that's our impact, not as simple. So you need to strike some balance depending on things like the PM career stage. If it's a junior PM or senior PM, for example, the product stage, is it a early stage product or a very mature product? And things like organizations, the types of industry, et cetera, they can all influence this equation. And it's gonna be typically not as linear as just adding them up. You might need to have different weights for each one of them depending on these factors. For example, in a product that is already very mature and you're focused on extracting value. So it's already past its growth phase and it's already a very mature product, you absolutely want to do a little bit more weight on the outcomes themselves than just the outputs and the inputs, not zero, but you probably, in either of them, but you probably want to put a little bit more weight on the outcomes themselves. You want to be more outcome driven and you want to put more impact and measure better the outcomes than the outputs and inputs. Similarly, or conversely in this case, for an entry level PM, the impact that an entry level PM can have on outcomes is going to typically be much lower than what a senior PM can have on outcomes. And so typically for an entry level PM, I focus more on measuring the PM's inputs. Again, the PRDs, the quality of the PRDs, quality of investigations, quality of the reports, quality of note-taking. And then as the PMs stay more time in role and becomes more senior, the outputs, the feature velocity, the number of features, the quality that they're delivering with each feature and with their area of the product, the improvements of key quality pieces of the product. And then obviously as they continue that journey, give more to outcomes, but then for an entry level PM, inputs typically carry a little bit more weight than then outputs and outcomes. And I might say, okay, so now we have this kind of equation and that might be easier, but what about if someone is like mid-level but then has a very big scope, how do you know exactly when you start to move these weights, right? When this, I told you as a PM goes, for example, in their career stage, you might move from giving a lot more weight to inputs to then moving into outputs and then maybe then later on to more into outcomes, but how do you decide that? When do you say this is the right time to change the weights, to change the equation, to measure impact slightly different for this person? And so for that, I normally like to use kind of this ash framework where if you take it from the top, it stands for kind of area of focus, solution, how and execution. So an area of focus, you have someone that is able to actually define an area of focus, a problem area, a strategy, a vision, if you will, someone that is defining a vision. Just before that, there's someone that can just pick up a vision, a problem area, an area to focus on and start to drive solutions. Right after that, you have the ability naturally that from a solution to understand how it can be implemented, how it can be done, and then you have actually executing and driving that to completion. So you can have as a vision, we're going to be focusing on dinners, as then the solution for that, okay, so we're going to need to have dinner tables, dinner chairs, cutlery, all these things. All right, the first thing we're going to be doing is a dinner table. You move on to the next stage, so okay, so how do we assemble a dinner table? Don't know how, let me go and figure out how can we assemble a dinner table? And then finally, Ikea style, you get the instructions to assemble the dinner table and you get to execute on that until completion and getting the dinner table there. And as you go up this ladder, you move from having instructions included into actually creating the instructions yourself and to actually defining what is the thing that you're gonna do and what is the thing you're gonna do first and then to actually see what should be the priority, what should be the vision for this area, for this product, et cetera. And as you might see from this plot, you can actually plot yourself in any of these points and you'll notice that it is actually a little bit easier to go to move laterally than it is to move vertically. And in particular, there's a big, kind of seemingly very hard and very hard to have a mountain to climb there at a particular stage. And that's when you're really starting to move from just execution and then when you are a bit more comfortable to even doing so without instructions or with less instructions into really going into defining what to do and what's important and what's the vision and that part is really, really hard to climb. So I got a lot of questions on like, how do the PM impact stages kind of map into that? Typically, once again, this is not linear but typically they map a little bit like that. So you'll notice that as you typically grow from an associate PM to PM role, you are now just instructions included, being able to execute. As you move into PM 2, you now start to have to be the one that knows and defines the instructions. So you don't get instructions, detailed instructions anymore and you start to define those. And then as you move into the senior band and now that you see the gap and you see the challenge there is you need to move into what are we going to do? What are the solutions or the problems that we need to do? What are even sometimes the problems for this area of focus, for this vision that we're trying to do? And then as a PM or group PM or director, you're going to be doing actually that vision definition, that area of focus definition for the rest of the team. And obviously you'll notice that as we did that, we were actually growing in scope as well. So we were growing from just a feature to a feature group to a vertical or product area to a product or even a set of products afterwards. And so it will typically be almost easier to do that but the challenge then happens when you start to grow on the Y axis a little bit more. So if you plot yourself in any of these positions or somewhere close to this line, even if it's not exactly in this line, you'll be able to know exactly how your equation should look like. So obviously if you're more in that execution, feature perspective, your inputs are going to be the most prominent and the highest weight factor. If you're conversely more in that solution and product area kind of region, then not normally you're going to start to have a lot more outcome-based weights. So things are going to be more, the outcomes are going to weight more than then the outputs. And if you're a little bit between maybe more than the how vertical area, for example, then you probably are really focused about having the outputs be your main thing. You're focused on feature velocity on features, on quality of features, et cetera. And so this is a way to both see how you can measure folks differently based on their stages, based on their place on this ash framework and also what you might need to do in order to get to the next stage a little bit. And this enters a bit into the next phase that I had in this presentation, which is the communication phase. So you want to be, as I was doing here now, have a very transparent communication. You want to know where someone is, communicate that to them, communicate the goals that you expect from them and that you're setting. And you want to make sure that those are adequate to the career stage of that PM. And if you're a PM, you want to have a very, very crisp understanding of these. And if not, you'll want to ask. Thirdly, you want to have predictable cycles of evaluation. So you don't want to, once again, just casually one day come up and say, hey, we should talk about evaluating your impact. No, it should be fairly predictable. The goal should be aligned to that cycle. Again, many companies have yearly, some companies have semester bays, other companies have quarterly evaluations. Whatever works for you, for your organization, for your team is good, but make it predictable, make it clear, communicate it well. And then on those times, also to take the time to reflect, reflect not just measure the impact, but reflect where you are in that framework, reflect where you are in terms of career stage, see what coaching opportunities you have for that person or that you as a PM feel that you need and want to ask. And then for ways to grow that impact and to further reward impact, make sure to be providing opportunities for that growth. Be it, again, a way to grow laterally or a way to grow vertically on that framework so that folks can increase their impact and be rewarded accordingly for that impact increase. All right, so the last piece that we have really is all about how can we improve RPM impact? Now, once again, the key thing here is improve your PM senses. Be have a better execution sense, have a better analytical sense and have a better product sense. And if you keep improving your product senses, you typically will be improving your PM impact. Now a common misconception or myth that I hear a lot is focus solely on your strengths and let your weaknesses be. Just be the best you can be on the things that you're good at. As a PM, I don't necessarily agree with this. I think that's a generic good advice to say you should boost your strengths as much as you can, but as a PM, you really don't want to have your weaknesses pulled you down. So you want to at least get your weaknesses to a baseline that is acceptable. Think of the framework input, output, outcome or into any of those areas of the ash framework and think if any of those areas is a zero. So if you're giving yourself a one to three, right? Like you are really good at maybe defining solutions, but you're terrible at executing. You're a zero. You're like a three in defining solutions, but you're a zero in executing. So that's your ceiling. So now you can't really get anything done because even though you might come up with great solutions, you actually are not able to execute them and have them follow through. So you don't want to be at that stage because that zero will cancel out everything. It's like a multiplication, right? If you multiply anything by zero, you're gonna get zero. So you want to have at least a baseline that doesn't hurt you and doesn't hurt your career and doesn't hurt your impact. And so that's why normally the screen on just focusing on your strengths, make sure that your weaknesses are at an acceptable level and then absolutely make sure to maximize your strengths. Be the best you can be. Be a three in, for example, solution definition or vision definition, whatever it is that you might be good, but then be at least a one or a two or whatever it might be in execution and analysis and creating those instructions, et cetera. Otherwise, what can often happen is something like your quality of execution could be awesome but because you're so focused on execution, removing blockers and getting things on and pushing that thing to release and going and compromising and reducing scope and changing scope and aligning stakeholders and finally you get it done. And in fact, that idea doesn't even solve the problem anymore. So once again, you want to make sure that you sometimes step back and use all your product senses. Like if you're creative execution, you need to make sure that you're deliberately stepping back sometimes and using our product sense to see am I even solving the problem anymore? Does this idea even solving these? And as you see, there's a lot of pitfalls. There's only one right quadrant in this chart. The rest are all unsuccessful. Now, the next thing that you'll want to do is make sure that you prioritize. And you're like, okay, prioritize the, I mean, I'm a PM, but you'll notice an interesting thing, which is very often you will have, you'll be good on that Y and X axis, but so you'll have to be doing a good quality of execution and you'll be doing an idea that does solve the problem, but sometimes you'll be missing is that even the problem that you should be solving. Now as a PM, you might be saying, okay, but I'm a PM, I know how to prioritize, but really it's not just about prioritizing the features and the problems that you need to solve. It's about then following through. For example, I is countless, the amount of times that I found PMs that do a great job at saying, these are the two most important features that we have. And then the other two are really P2s, are really secondary. These two are definitely the most important ones, the ones that I prioritize the highest. And then they get at the end of a period and they got maybe one of those and then the other two less important ones or maybe none of the important ones and then they got the other two less important ones. And then there's all kinds of reasons for that. Oh, because the other two, I mean, I tackle for the same, but then those two were easier. So the engineering team got to them and did them and these two were so much harder. So they required a lot more effort from me and I was so busy and I had so many meetings and so many other things that I just couldn't get to them. So yeah, they're still more important, but that just didn't happen. What that tells me is that you prioritize, potentially the right features, but prioritize your time and your actions wrong. So when I say prioritize here, it's not just about those charts that I showed you and about knowing what to do and how to do it and that is it solving the problem and is it the right problems to solve, but it's also are you yourself prioritizing your time, your actions, your calendar accordingly to make sure that the two things you said are important are the two things that you are effectively working because if you said they're important, ideally they're gonna be the ones that are gonna have the biggest influence on your impact. If you miss them, then the reverse happens, right? And you're gonna have the least impact possible because you didn't meet them. And then more often than not, the case is a little bit more nuanced where you might have two very important features and then maybe eight others that are less important and at the end of the semester or the quarter or whatever you might have, I don't know, maybe I have done three. One very important and two or three less important. So yeah, four, it's really bad. It was really good, right? Or maybe I did even six. I did six, five less important, one very important. And I just, yeah, I couldn't get to that one super other important one. But hey, six, it's more than half of all the features out there. It has to be a good impact, right? Well, potentially, but what if you had done only four but you had included the two most important ones? Now that probably with those four are probably more valuable than the six that you did, right? So it's really all about making sure that you have quality versus quantity and you have that really, really ingrained your brain that you're optimizing for impact and not for quantity. And similarly that you are prioritizing doing things with quality. Once again, you don't want any zeros. So you did all 10 features, great, but you did them at 20% quality. Then you actually did zero features because a feature without quality is almost the same as no feature at all. So this is really important to kind of understand this dichotomy and prioritize accordingly in this dichotomy. The next thing is horizontals. And this is great for both getting opportunities across different things to expand your scope as well as opportunities to work on things that you might traditionally not have a lot of exposure to. It's great to give you visibility. It's great to get you to work with different parts of the team because horizontals are typically those cross team processes that you can help improve or create from scratch. Those cross team initiatives that you can lead. So you have all kinds of opportunities even if you're like a junior PM leading a horizontal will actually allow you to work across the whole ash framework because a lot of times you'll be, okay, for this horizontal, what exactly is going to be our strategy here? What exactly are we gonna do and how are we gonna do it? So you'll get in charge of that in a case where it's not directly necessarily linked to the product right away, but it will allow you to actually have an impact that is across the team. And it will also allow you to grow on that framework and to grow on that and to plot yourself probably a little bit higher in that chart, which again will move the needle of your equation and move your impact as well. The next one might be a little bit controversial, but it's about acknowledging optics and visibility, which is sometimes again, maybe you did those two features that were super important that we discussed before, but you didn't really communicate them. And so from everybody's perspective, you did zero features because two features that you didn't communicate, that you didn't brought out, are they even theirs? Like a tree fall into the forest, did any tree actually fall? So it's about the same thing. It's not necessarily bad, sometimes optics are seen as a very, very, very bad thing and sometimes they are, that's true, but there's a baseline of communication and visibility that you need to ensure so that your impact is perceived in the same way that you are perceiving it. So always ensure that other folks are perceiving your impact the same way and be open to the fact that the way that you are perceiving your impact is not correct because again, that your leader or leaders of the chain might be focusing more on the output or the outcome and you might be very focused on the input and then there's a miscommunication on actually how much impact you're having. So make sure once again that you communicate well with both your reports as well as your leaders into how exactly your impact is being measured so that there's no miscommunication and then make sure to communicate and give visibility to what you're doing accurately. Again, don't over communicate, don't spam. So work on an agreement with your leaders and your reports and what's the best way to communicate. Now coaching again, I put coaching twice on both how to reward and to increase someone else's impact as well as how to improve your own PM impact. If you think you need coaching on something, ask. Ask your leaders, ask your teammates, ask your mentors and on the flip side as a leader, make sure you're coaching folks on how to increase their impact. If they're doing a great job, but once again, their eye per focus on one of those areas and they're really leaving the other ones down and then basically at the end of the day the impact is being completely thwarted, coach them, make sure that they know about that and that they can act upon it and then make sure to leave as a PM and as across my teams, I want to always to make the point that all the product managers are leaders. Don't mean that they're the bosses of anyone, the managers of anyone, they don't necessarily own something, but they are to me the truest definition of a leader, which is they set the example and do the right things regardless of if anyone is watching or seeing and if they do that and continue to do that eventually people will follow them. And so that is super, super important because if you want to be doing always the right things and you want to be setting a good example and that is the best way as well to improve your impact and to improve your different PM senses and to also move both laterally and vertically in that plot. And finally, understand the expectations of your current stage and next stages. How can you move to the next stage? How can you move to the next vertical ladder or to the next horizontal ladder in that chart? What is required for the next stage? Am I meeting all the expectations for the current stage? Understand those, know those very, very, very deeply because once again the equation to measure and evaluate your impact will depend on it. All right, that is all from me. I hope you've enjoyed this time or thank you very much for your attention today and for spending this time with me. I hope this was helpful and it got you to reflect on PM impact a little bit more. It may be even give you some tips and strategies on how to measure it for your teams or how to look at your own PM impact and how to maybe even improve it into the future. Thank you very much, rest of a good day.