 Hey everyone, my name is Andrew Oh. I'm a VP of product and head of design and I'm doing episode two of how to interview top tech companies. Today we'll be going over our topic, execution. More specifically, we're going to be going over metric definition. This is the second part out of our five part series where I am covering how to interview at top tech companies like Facebook, Google and more. Covering the basis of the different types of interview categories you're going to be getting asked. And so without further ado, let's dive right into it. All right, so just as another quick background about myself, I am again a VP of product and head of design that most recently served at Coulomb which is a series based on crypto trading app based in Indonesia. Before that I was at TikTok as a lead product manager working as a founding product manager on a zero to one product. And before that I was at Grav, it's a senior PM working on a product product. Aside from this my corporate career, I've also been a founder. I've found a startup strategy from e-commerce and sports products accessories to manufacturing to social. I've also been part of the Aquamis Accelerator Program based in San Francisco and I am an exponent product management interview coach. So today we'll be going over execution and more specifically that will be metric definition. Afterwards we'll be going through the cold framework and breaking down the most important piece that I registered strategy which you'll use to then define your key metrics and then we'll be going through a walkthrough on a specific metric selection. So I've done this presentation a little bit differently. This time I have a little bit more detailed slides and more specific examples that we could go through. And so let's get right to it. So what is execution? What is metric definition? Execution is a term that may be used differently across different companies but the one that we're most familiar with or the way that we're defining it today is all about metrics. So you have two types of metric interviews that you're going to be doing. One is metric definition. So that's defining and setting goals for your team and the last would be metric change. So you'll be given a prompt on a metric that changed for a specific product company and you'd have to diagnose why that metric changed. And so why is this being assessed? We are assessing more specifically around your way of thinking with regards to setting with regards to two parts. So I'll go over the definition part first and only this because we'll have a separate episode for the metric change. So for metric definition, what's really being assessed here is your ability to set and establish North Star Metrics and goals to help orient the team's efforts and also your roadmap for accomplishing impact which is ultimately what we product managers have to do. So driving impact in the right areas that is essentially what is being assessed here and more specifically you're going to be assessed in terms of your framework and way of thinking on how to structure the pathway towards coming up with really great goals for your team. The person that will be interviewing you for these kinds of interviews will still likely be product leads and up. So could be managerial based product managers or it could be VP's directors, et cetera. Some examples of metric definitions questions you're going to be asked could be for example, the fine goals and success for Instagram, how would you define the North Star metric or key metric or TikTok? And lastly, how would you structure the team's goals? These are all alluding to essentially the same question, same assessment criteria and we'll be going over one of these questions more in depth. So for the case question we'll be reviewing today, how would you define the North Star or key metric for TikTok? So there's a few terrifying questions you're going to want to ask here. And as you can see from the slide, I've run down a little bit more detailed footnotes that we can go over together. So the first question is, and again, I think this is a great tactic that more advanced interviewees should be using. Just ask if it's first okay to make the data assumptions. Is this going to help you make data assumptions on the fly to better make prioritization and trade-off decisions or to help magnify opportunities? The second question is not really a question, but more so just reviewing the product and walking through with your interviewer, the typical TikTok UX in this case. So walk through from A to Z, what is the typical TikTok UX? Talk me through what do people go to this app? What is it really used for? Who are the people that we think are kind of using it? Just like touches, but really I'm just trying to get a better sense to make sure that we're on the same page for our understanding of the baseline products. Third, we'll then be asking if there's a regional focus. So like the product sense, this is a really important part of your strategy to help define goals. So TikTok today in the U.S, which I believe has captured well over a hundred million Americans that are 300 Americans out of there, 300 million Americans. It's done really well. And you can argue for its target market, which could be Gen Zs and millennials that's probably captured over like well over 50%, maybe even 80% or more of people today. So the regional focus in that case, if you were to place that regional focus and attribute a stage of product to the U.S, you might say, well, it's already captured most of the users. We are probably not thinking of it being more towards the mature stage as we start shifting goals towards monetization. And so that's what's important to really determine and align on what is the regional focus because if you are building this for say Southeast Asia, that could be a very different story, this stage of the product could be in its growth stage, not the mature stage, and therefore the goals would also be different. So for this case, we're going to say maybe the interviewer said, hey, let's focus in Southeast Asia. If the interviewer doesn't provide a specific market, then you can go ahead and make an assumption on which market you think makes the most sense for us to focus on. So in this case, we'll choose Southeast Asia. And here's why. And so this is when you can start using some of the knowledge you have to make some data assumptions. It may or may not be right, that's not the point, it's just about seeing how you think. So Southeast Asia, I can see why they make a lot of sense. We have four billion people, which is 60% of the world's population just living in this market alone. With the GDP growth of over 4%, bullish 20 year outlook in that case, it's a 10 billion opportunity today going 10% year a year, which means it's likely to be over 10 billion, over a hundred billion dollars in terms of digital ad spend opportunity markets in the next 10 years. So yeah, so we don't focus, Southeast Asia makes a lot of sense. Given the Southeast Asia, and let's just make another assumption, we only have 20% penetration in terms of the target users that we're targeting today, which is millennials and Gen Zs. Our goal is going to be centered around music growth. So now we go to the second part of this framework, which is about context and users. And so first start with the mission. What's the next mission is to inspire creativity and bring joy to its users. Next, you can talk about the strategy and how that ties back into the mission. The strategy here is really the key component that you're gonna need to provide that direction on how to structure your goals or in other case, your metrics when we get down to it. So I wrote a little bit of a prompt here. The strategy is in order to grow the user base in Southeast Asia to hopefully one day encompass our four billion people in APAC, which is 60% of the world's population, and therefore fulfilling our ability to inspire creativity and bring joy to the world. Our strategy will be this. If 0.1% of creators of users can create 90% of the most highly engaged videos on the platform, we'll have a really strong growth loop activated. The more highly engaged content we can create, the more people are going to refer the friends to come on board, the more likely they're going to share really cool TikTok videos, like download it, share it on Instagram stores, et cetera, or just WhatsApp it or I message it to the friends. And so if you can even get 0.1% of users to create really great videos, we're gonna have a really strong growth loop there. Mind you, let's also just assume that we've seen this kind of behavior already, this referral based behavior already validated in other markets that we've been in in the past. Now the short-term goal would be to bring global entertainment to a pack region, the Asia Pacific region. The long-term goal would be winning this region which was to cure the $100 billion market opportunity in the next 10 years. Now let's think about the user segmentation here. I've identified three specific types of users. The first is creators. So these are people who are just creating videos. They could also double up as viewers which is the second user segment viewers. So these are people who typically come to the platform. Their goal is to view the videos, again, sustained. And then last we have advertisers. Advertisers are people that with the small businesses or big enterprise that come to the platform to curate ads and hopefully end down more traffic and sales. Now what I've done here is I've further segmented each of these user segments. What I like to do is I like to call the first layer, layer one. So layer one is the creators, viewers and advertisers. And then I call the subsections layer two. So for creators, I've defined the layer two as friends. So these are creators who are just creating more for the front-end family, long-tail creators. So long-tail creators are creators that maybe have in between a couple hundred to maybe like 10,000 or even 50,000 followers. So these are the long-tailed creators of which are hoping to then transition to the last layer two segment here which is the full-time slash powerhouse creators. So these can be celebrities. These could just be huge, massive creators that have, you know, and are beyond 50,000 followers and are then able to monetize and do this full-time. But I would say, you know, these people like we have hundreds of thousands at the very least. So these are the layer two creators segments that I've identified. We then have viewers. So for the viewer side, we have what I call casual viewers. So there's the people who just, you know, momentarily throughout the day, open the app, entertain, maybe there's even lunch, et cetera. Just casually browsing, no real goal in mind. And then you have the addicts. So the addict viewers are people who are addicted to the app. So they're constantly going on the app to get into the chain, get that dopamine hit and just contribute a lot of their free time and day-to-day on the app and scroll through. The last is advertisers. So for the advertisers layer two, I'm identifying that as SMBs, mid-market businesses. So mid-tier businesses and enterprise. Now, in terms of the user segment, which I think it's going to be really key and crucial for us to prioritize when it comes to really activating this growth of strategy and making it work, I think it's going to be the creator side. As mentioned, if you can get even just 0.1% of these users to start creating videos and really good videos, we're going to have a pretty impressive growth group activated which is going to increase our user growth ratio by a factor of a very high level. So more specifically than, this means I'm not looking at the full-time or powerhouse creators. I'm more specifically considering brands or long-tail creators. And so this is where I need to maybe just take a step back and think more logically as to which one makes the most sense. Does it make more sense to try to convince people who may also again be a little bit more private to create videos with the goal of trying to attract lots of attention to themselves but also to increase the follower base or does it make more sense to go after the long tail of users to constantly then create more videos and hopefully we can help achieve their user goal of obtaining more followers and hopefully help them monetize in the future. Now, we don't have any data here. So I might have to make an assumption to help make this right or at least in a better direction. So for the long-tail, if these long-tail users are 0.5% of the entire user base and our strategy here, as you can see is to get 0.1% of users to create highly engaged videos. It means we need 20% of these long-tail creators to be successful. That might be a bit of a stretch because let's be honest, not every single long-tail creator is going to be able to create really great content. However, maybe let's just say that's 2% of creators that are actually more long-tail creators who are trying to again achieve the user goal of getting a better following. And taking this re-creation process a lot more seriously, then that may make more sense. If, and then maybe the interviewer goes ahead and says, hey look, actually we did look into the data and we saw that long-tail creators are actually in fact only 0.5% of all the creators or all of the users on the platform, then you would need to make a logical decision if it still makes sense to go after that specific segment or if you should be going after the friends. There is no right or wrong answer here. You just wanna try to lean yourself into a justifiable argument for it's one way or the other. Now in this case, I'm just gonna stick with long-tail for simplicity sake. We'll just assume that it's actually 2% of the users. And also I just think that more realistically the long-tail users are people who already have that intention to create really high quality videos and then to try to go viral. There's probably gonna be a lot less friction and a lot more opportunity there to solve for and then to achieve the goals and help us obtain this growth loop. Then you can go through the user journey. So I covered this also in the product sense video last episode, but basically you just wanna walk through what someone and the long-tail creator segment would actually go through when they're creating videos. What does that entire process like? Even before they pick up the app and start creating videos, you can just tell me even what their day is like. Captured content, where would they go? With the mindset of I'm going to be creating something at the end of the day and collating all of its content to create a viral tech video. So you can talk about the journey of why the person wants to be a long-tail creator or an influencer. How do they plan their day in terms of capturing content? What are some of the problems that they're facing along the way? Talk me through the process of actually going on the app and stitching together video and then what is the process like even after the launch video? Setting up the tags, being responsive to comments and likes, et cetera. So you wanna just kind of walk through the entire user journey. That's gonna be really insightful to not only showing your empathy and depth and empathy, but more importantly, you're going to also be able to potentially see new opportunities. And in this case, more specifically, it could help you really identify some of the key metrics that you might actually wanna consider when it comes to that final piece. And so after you walk through the user journey, take a step back, ask for maybe two to three minutes so you can start writing down what are the key metrics that you want to prioritize. And so the way that I structure the metrics here is in four different parts. Number one is the OCR. So the OCR is going to be that department level metric that has been assigned to the department in order to achieve. Second is the North Star metric. So what is that key P0 North Star metric that's going to help accomplish the OCR? Three to five KPIs or key performance indicators. And then lastly, one to three guardrails. And if you don't know, guardrails are more like defensive metrics. No examples of this could be fraud. Could be fake content, could be, let's see, well, we'll leave it to that for now. But those are just examples of such a guardrail topics of which can derive metrics from. So I've already prepared some specific metrics that we could be reviewing for this case. Now, I'll walk you through the different columns that I have set up here. Column one is just simply one of the metric types. So OCR North Star KPI and guardrail. Second column that we have here on this table is metric name. So just the name of the metric itself. And the last column here, and I'm sorry, I didn't spoil that definition, right? But the last column here is specifically about how we're defining that metric. So I'll go over each of the metrics in the roles that I have here so far. The OCR is directly tied to the stage of the product. And we mentioned that early on that it's since we're focusing on APAC, our goal is centered around user growth. So the OCR for this department is to increase the number of users on the month-to-month basis. The North Star I will get to later because I like to actually go through the KPIs first and then think about what the North Star could possibly be. So for some of the North Star, sorry, for some of the KPIs, I'll watch on the following. One is the engagement rate per user. And we're just assessing that by time spent per daily active user and weekly active user. So this is great. This just gives us kind of like a baseline understanding, you know, as a PM, you should just be a little bit more aware of what your rates like for these that you have on board. Because this can actually feed into many other metrics that could be relevant or strategic in terms of insight that it could be providing you. Also it's a social app, I think, engagement rates are kind of a given. The second KPI that we have here is the percentage of users who are creators. So the way that we're defining this is just creators over the total number of users on the platform. And you would still have to even define what we mean by creators here. This creator, someone that created one video, five video, was it like one video a week? So you can also just decide to define a place on a timeline. So one video a week versus one video a month. It's up to you. Just make sure it makes sense the way that you do it. Maybe the creator also has to have their TikTok account turned to public and not private. That would make a lot more sense as well, given that the way that we define the long tail creators is that they're trying to gain amounts of following. So set up the criteria parameters for how you're going to define that specific user segment. So just to recap, KPI number two is the percentage of users who are creators. KPI three is the average time spent to create a video. And we're defining that from start to finish of the video creation process. Sorry, I made another type of there. KPI four is the average number of users who were referred per user. So that could be either tracked by the referral link or copy and paste. Could be from the address book as well. But yeah, that's how we can define that. KPI number five, and this is the last KPI is the D one, so day one to D seven retention rate. And we're tracking that by say, it could be the user opening the app simply by opening the app because they're going straight to the TikTok homepage where it's more like a video feed, or it could be say watching one whole video per session up to you how you want to define it. So these are the five KPIs. And so what I like to do is I like to just talk about the OCR first, think about the KPI second, and then I'd like to actually then come up with a North Star because sometimes almost KPIs could in of itself be a North Star metric or it could just help really again, allow you to get the time to take a step back and think more objectively from the whole ecosystem that we just discussed from the user journey, the user goals, like what really is that North Star metric that's going to really feel that OCR. And for us, we can just look back at the strategy. Our strategy was 0.01% of users will create highly engaged videos which would feed the growth loop. So therefore, I'm going to be placing the North Star metric as the increased percentage of highly engaged videos. And how we can define highly engaged videos could be some kind of factor ratio of likes, comments, and views. That could be a way to just indicate that these are high potential, hybrid videos. Sorry, just to clarify the definition again, it's some percentage factor of likes and comments or reviews. So that's how we can maybe define highly engaged videos. And then I also just have one guardrail metric that I just put here as well to check a box. So for us, that's going to be the percentage of fraud slash abusive users. Fraudulent users could be just fake accounts that came here for a specific negative reason. These kinds of accounts or abusive users could be people that came onto this platform and started maybe leaving harassment, comments, or messages to other people. How we can define this is maybe by some factor of either user got reported X number of times or it could even be say, mental high tier fraud score. And so the guardrail in this case could help us to filter out bad users and clean our data for how we're looking at the rest of KPIs, North Star, and OKR. And that is it. So, hope you really enjoyed this session. This is just again, a quick walkthrough of how to break down a question for metric definition. And you can replicate and rehearse this framework with your friends or with your marketing partners for yourself. And once you're ready, feel free to hit apply to all of those companies you want to spread and pray to. And I really wish you guys all the best in the hunt. That's all for today. I'm looking forward to seeing you in the next session. Take care.