 Hey, everyone, Andrew here again. We are on Episode 3 of How to Interview at Top Tech Companies, and today we'll be going over execution, metric change. Just as another free food cap on myself, if you haven't been listening in in the past, my name is Andrew Oh. I'm a former VP of Products and Chief Design Officer. I previously served at companies such as Grab, which is the Uber of Southeast Asia, TikTok, everyone's favorite social media app, and PuLong, a Series B stocking crypto trading app based in Indonesia. I've also founded several different startups, and I'm also a coach on exponent for product managers. Today, we'll be going over a few things. What is execution? What is metric change specifically? What is the framework that we'll be using to break this down? As mentioned, today we'll be going over execution. So, we'll be going over a few things today. What exactly is execution? What exactly is going to be obsessed in metric change? What is the framework that we'll be using to break this down? And how would we be going about tackling follow-up questions? So, just as a reminder, if you haven't tuned in for the last episode where we've been over metric definition, execution is a categorical interview type which product managers will face, of which they'll be going over metrics. And this particular definition will be going over metrics. And so, there's two broad categories of this, metric definition, which is about how would you go about defining metric or success metrics for a specific product or company. Really, it's about establishing goals and goal setting for your team. Metric change is slightly different. It's going about how do you diagnose a metric that could have changed within the company? And we'll be going over examples of that in the next slide. Why is this being assessed? And what exactly is being assessed here? It's going to be assessing your analytical abilities. And you'll see in the follow-up questions later on, it will also be assessing your stakeholder management abilities as well. This is, of course, crucial for any product manager in the field to be able to quickly diagnose any particular issues or irregularities that you're going to be seeing in your product dashboard. And also, how would you navigate the complexities of your organizational environment to address those issues as well? The people that will be interviewing you for this specific kind of interview would most likely be other product leaders within the organization. Very rarely, although I have seen it, you may also be interviewing a data scientist lead who may also be assessing you on this attribute. So there are some examples of metric change questions that you will be asked. Here are a few of them. Uber rides are down 10% determine what happened. Another example would be TikTok engagement is down 20%. How would you diagnose this problem? And another, a little bit more fun would be McDonald's GMV is down 10%. How would you diagnose this or what happened? So as you see here from these questions, it's pretty straightforward, similar to other questions that we've had in the past. The TRA and tier one companies will tend to give you a very clear yet ambiguous question with not a lot of context and it's up to you to define that context upfront so you can set yourself up for success. And so that's going to be leading into our next segment, which is us breaking down that question and providing context. So let's take the case question here. Uber rides are down 10% determine what have happened. As usual, a key part of strategies for most frameworks here is to first set up that context by asking clarifying questions. So there are a few here that you should be asking. So first clarify the product with Uber. So there's not so much a question, but more so just a walkthrough of describing what that product is. So you can just say, for example, hey, I just want to clarify my understanding of Uber with you and what parts we're covering specifically. So just to recap this, Uber is a ride hailing app of which everyday people can become drivers and get allocated a ride from a rider who is requesting to be taken from point A to point B within the app. And these drivers are incentivized based on a, I think it's a 20 or 30% commission on each ride. And likewise, if they reach a specific target such as like doing 20 rides a week, they may also get a bonus payout at the end of that week as well. So I just want to clarify that understanding with you. Interviewer might say yes or no or make any iterations to that understanding based on that premise that you have presented. So it's always good to just clarify the understanding of the product very clearly. So you guys are on the same page. This is really, really key to do upfront. Don't do it later. When things get messy and you've already kind of synced in 10 to 20 minutes over time. Second piece here will be to define the metric. So how do we define rides internally? This is another key question to ask. So you see here, the question most more specifically, urban rides are down 10%. So let's define rides. How do we define rides? Is it from a time of which that passenger is finished going from point A to point B? Is it actually determined by just the points of which the customer makes the transaction and the processing of that transaction is completed? There could be different interpretations of how rides are determined inside this company. And you want to just double check and make sure that you and the interviewer are on the same page of this definition because that could be crucial later on for you to try to troubleshoot this problem. The third question you want to ask here is when did it start happening? You know, did it start happening two weeks ago, two months ago, last week? You just want to double check and clarify again when did it start happening? Cause that's going to be essential and crucial to the next question, which is was this a spike or gradual drop? Now this is probably the most important question that you're going to ask in the clarifying questions bucket because if this is a spike, you can start to assume that this is likely due to some internal issue. If it's a gradual drop, you might be able to immediately start thinking on whether this is an external issue, something outside of the company, something outside of the product. So more so a market or global force that's causing this drop in rights specifically. And then lastly, where did this start happening? Do we have a rough idea on what kind of users this has been affecting? So, you know, did it start happening in the US or is it like a global problem? Are we seeing this with new users more specifically or all users across the platform? These are again, really central questions to ask as it will help to provide more clarity and depth on your structured way of thinking on where the problem could be starting out. Now, one thing, sorry, I forgot to mention is that you do want to be speaking out loud. You want to talk out loud on where your thought process is going as you're going down each of these questions because that is essentially what's going to be assessed here is your way of thinking and how you're putting these clues together will change your hypotheses of where things could be going amok. So, you know, if I have gotten up to the point where I've determined with the integer that this is more of a internal problem due to a spike. You know, I might say, well, given that it's a spike that started happening just within the last two weeks I would start to imagine that this could be due to a more of an internal issue, whether that's a bug related issue or maybe it has something to do with the UX change. You know, again, just given that it's a spike and not a gradual drop, I would expect that something abrupt had happened of which had started to really affect the metric. So you can just explain it off like that. Again, you're just thinking out loud as you're going about each of these questions on where your hypotheses is started to formulate. So what that thought process, let's first go over the buckets. So there are two main buckets that you're going to be going over to determine and assess where the problems might be facing. So first bucket is internal issues, the second bucket will be external issues. So what cover both? Let's first go over the internal buckets. So mind you, what I'm going to be presenting is just only several of many that you could be brainstorming. Because you want to get this interview down to about 20 to 25 minutes, don't go too crazy. You're not trying to just cover all bases. You're more so really just generally trying to solve this problem with your interviewer. So there are a few buckets here that you can go over. The first is product. So any kind of product related categories, you could be asking your interviewer such as, you know, did we have a new version release recently? Did we, had we done any AB testing that might have also affected the majority of users? Could the AB testing have actually been accidentally pulled over to like half the user base or more? Or is there a new change in the UX or maybe even some product integration that ended up causing these problems? So that's more on the product side. Technical side, you could be asking questions like, you know, again, do we have any third party integrations that we did that, you know, might have actually gone down? Maybe that's not an iron, that's actually on our third party provider's end. If you've experienced any downtime for a significant period of time that might have also caused this downtime to occur, I mean, the drop in rights to occur. And, you know, you could also be asking, you know, were there any bugs that we had seen through the software or through customers that might have actually been blocking them recently? Then we go on to marketing. So marketing is another bucket you could be talking to here. So another example, this could be, you know, to get a marketing strategy change. Like as we know, Uber's customers are quite part sensitive. It is more like a journal commodity. So, you know, have we dropped anything like new user signup promo codes or referral promo codes or city-based promo codes? All of this could have, of course, been attributed towards the drop that we had seen in rights. Now, this is a little, now for this, I would then also caveat that, you know, if it's a drop in promo codes, I wouldn't expect to see a spike in drop unless there was like a truly massive, like, global campaign. So that would have been how we defined it earlier on, but again, this is maybe not one of those where I would expect to drop, more so a gradual drop, if anything. This one's pretty important, metric. So by the metric itself internally, you could be asking questions like, did we change our metric platform, our analytics platform in the last two weeks? Maybe that was why we ended up having this veiled drop in our metrics. Did we change the way that we define our ride in the last two weeks? You know, that end of itself could, of course, drop it 20% at the spike and being represented as a spike itself. Another thing you could ask is customer service. So, you know, have we, you know, is there anything in terms of a spike that we have been seeing internally among the customer service reports that have indicated? We've been seeing, you know, maybe for example, customers getting blocked or maybe there's some safety issues that have been going on that caused this 20% drop quite suddenly. So these are just, again, just some examples of many different buckets that you could be assessing and using to determine whether or not there were any internal issues and many points towards the drop in rides. What I would recommend is, of course, any time you go on to the next part of this framework, just ask for a minute to start writing down your thoughts. Write down all the buckets that you think could be leading to this specific drop in rides for this specific product. Let's move on to the next part, which is external. So let's just assume, hey, you know, a lot of the seconds that we're getting from the interview is that we're not really seeing a big problem here that's leading to the spike in terms of a drop in rides. Then let's move over to the external forces that might have been actually covering this. So I have here three listed out that we'll be representing for this problem. First would be competitors. So was there a new startup that might have came out recently with a huge PR release, something truly disruptive that maybe just sucked out 20% of our rides or 10% of our rides, et cetera? Was there maybe just one of our incumbent big competitors that launched a new feature or a new promo campaign and went really hard on that? So these could be some examples of reasons from the competitor side that could have been coming out that you could also just address. A second, one is world events, which is very, very relevant to modern days. I'm actually reading up on your news, but one of those could be COVID. If you were interviewing back a couple of years ago, maybe within that specific country or city that you've identified based on the clarifying questions, when COVID is active, was that the time when maybe the city went into a lockdown or an early curfew? Maybe that's why rides went down 20%. Or in another case, Ukraine war, could there have been some sort of conflict or a strife within that region or that specific country that had also caused this to happen? That's a very specific question that if you're a little bit more up to date with world events, you could ask if you noticed that where the users have been dropping off in terms of rides was in a particular region that's a little bit more sensitive right now. So again, that's why clarifying questions are just so important. And then lastly, it could be political or legal. So for example, were there any new regulations that had been released or amended in the country of which we're operating in that we saw these rides start to drop? Or if it's a global issue that's affecting all the users, maybe this was a new regulation that hit the US, which we can assume is probably the biggest single market for Uber. And maybe that impact was so great that we actually saw the rides drop 10%. So these are just some of the external issues that you could be going over. Again, make them specific to that specific products of which you are assessing that metric drop or metric spike upwards starting to occur to try to determine where this problem is. Now, again, you would have gone through a pretty healthy list of internal external forces. And maybe you're at a point where you still can't figure this out and your interviewer is not really giving you any any good hints on it. What's most important to demonstrate here is that you've is again, your way of thinking that you've covered a lot of the basis here. And as a last resort, if you really can't find this out after 15, 20 minutes, the last resort I would say is just say that you'll go talk to customers straight out. Show that users in Tracy decided of you that you're willing to go in person, do a survey, talk to users through one-on-ones to try to get down to the ground truth of why this might have been happening. And if the interviewer had early on hinted that maybe, hey, yeah, we have been seeing something a little bit more on the technical side. That's at a point where you can just dive in and hone in on that specific issue and just ask more questions and dive a little bit deeper as to where on the technical side of things, that's the drop in the spike might have gone wrong. So again, this whole interview is meant to be a little bit more collaborative. You're fishing for information, trying to put the clues together to try to determine where the problem is. If you can't find the end answer, that's actually okay. I've seen a lot of people pass this interview while not being able to find the end answer. Cause that's not the exercise or the point that's being assessed here. The point again is just assessing your way of thinking and how you're able to collaborate and analyze holistically where things might be going wrong. Now, at the end of that case, you will very likely be asked some follow-up questions with the remaining time left. You'll likely have five or 10 minutes left in your interview to solve for these questions. So very common question types will be both trade-off questions and stakeholder related questions. So let's first talk about the trade-off questions. Here's one. Rides are down 10% for Uber, as mentioned before, but we've actually seen the GMV up 20%. So as the PM for Uber rides, what would you do in this situation? So this is an interesting follow-up question that you could get asked because what this question is essentially saying is that you represent the rides portion of this metric that's down, but the overall business is up. This is likely not hitting your OKR. This is not likely not hitting your KPI. And of course, this could also be a performance hit to you in your performance review cycle. And so what would you do in this situation? So what they're looking for here, and again, this is gonna be more dependent on the company itself, but usually what they're looking for is selflessness. Selflessness and striving towards coming towards a decision or a series of decisions that is ultimately going to benefit the mission and the company itself. So, it's fair to ask some more questions about this upfront to just try to determine, for example, why is the GMV really up and why the rides are down 10%. What you wanna do is just always go back to the mission of the company. Go back to the mission of the company, read up on it, or just make sure you know what it is. If you don't know about it, just ask, you can ask your interviewer what that mission is, or you can also just make an assumption of what that mission could be. And so for me, let me just maybe state that quick assumption of what this mission for Uber or Uber rides team could possibly be. Given that Uber is a ride handling company, I would imagine that its mission could be something akin to enabling anyone in the world to seamlessly and affordably go from point A to point B. That's it. Now, I'm making an assumption here because I actually don't know what Uber's mission statement is, but given that mission statement, the principles of the company follow through with that as well. Because what that mission statement, if you read between the lines, what it's actually saying is, we want to democratize and enable anyone to be able to take a ride to go from point A to point B. And what we're valuing here is more mass consumer level transportation rather than just making that's a premium service that only wealthier people could afford and write on. And so based on that principle, you go back to this initial question and write it down 10%, but the GMB is up 20%. So you could ask more questions or you can make assumptions up to you at this point, but you can make an assumption that if we had done the analysis and we've seen that the GMB is up 20%, but they're rising down by 10% because the growth team decided to did an analysis and realized that they could be charging users more money. But that came at the cost of 23% of riders no longer being able to ride with Uber, that doesn't align with the mission. And because that doesn't align with the mission, this is something that I may want to work with the growth team PM on to try to come to a conclusion on number one, one of the principles that we're standing by based on the mission of the company. Number two, how can we actually make this a win-win? Not, this is irrespective of my department and your department, but how can we actually make this a win for our consumers who are ultimately both trying to serve? That's the kind of response that they're really looking for here. So again, the way I kind of described that was it's selfless, it's mission oriented and irrespective of whatever team you're on and as a product manager, you should be serving ultimately your users in the best way that you can. Yes, business metrics do matter, but hey, we're seeing a 20% GMV uptake, but if that's coming at the cost of 20 or 30% of our riders being able to ride with Uber and afford that ride, I think that could have longer term implications and it also gives a competitive edge to our competitors to potentially overtake us and take our market share later on. So try to make that connection not just towards the mission, the users and the products, but also towards excellent factors like the competitors to try to make that justification. Now, that said, you could have a very simple follow up question to that itself, which is the stakeholder from the growth team doesn't align with what you're saying and your justification is like, how would you handle this? So you're getting pushback. Very fair question to have and I'm sure a lot of you have encountered this before. So you've done the approach where you try to be reasonable with by doing the right thing and you also try to go by the principles of the company which is not working out. That's fine. The growth stakeholder is not aligning with you on this. The next thing you would want to do then is look at data. Look at data to see how you can possibly reposition this to make it a win-win for both of you. So maybe you can do some sort of like analysis to determine what would be a great metal ground or ridership to go up whereas the GMB target could possibly still be hit and try to make that work. But even if that approach doesn't work out but the data that you have for your case checks out. At that point, what have you done so far? Reflect on this. You have tried to do what's right by the customer, by the mission of the company and you've also presented a really good business case for how you could at least try to make this a win-win for both sides. If that doesn't work, you have really, really good case to escalate this to your manager or possibly the scale level to try to align or force a decision. And so that's the other piece that some companies or some interviews will be looking for is at what point would you actually escalate this to your manager because not all problems can and should be solved by you alone. You should still show that you've took initiative and that you've done everything you could within your power to try to swing this into a positive direction but if it's not working out, at some point you will have to escalate and that is the role of your superiors and your leadership team to help out with. And so, yeah, at that point, you have a great case escalated to your head of product or your skip level, your manager and let them work it out with the other counter parties leadership team to try to come to a conclusion. And that's it. So that is execution, episode three of this five-part series on how to interview for top tier companies. I hope you found this really helpful and I am looking forward to seeing you guys next week on episode four when we'll be going over behavioral interviews for the IC track of, interviews that you'll be facing for top tier companies. Thank you so much. I will see you all later.