 This is Dr. NCD, a director of product. I help engineers and international professionals transition from worker B to a product manager and business leader. Today we had the pleasure to invite Big Joe in city company to share with you guys how Google built product. What are the roles and responsibilities as a product manager in Google? And Big Joe is a product manager at Google with eight years experience in Google and then he has a lot of experience to share with you guys. So Big Joe, can you introduce yourself? Hello guys, my name is Joe. I am a product manager at Google, been at Google for a few years. I've been working with a few product Google including ads and we try software and video-related products. So today I'm going to share with you a few tips as a product manager at Google. But one small disclaimer, I'm going to talk about my own experience. I'm speaking from my own experience about how I feel about how I define product. It doesn't necessarily represent Google. It doesn't necessarily mean that all Google products managers do the same. I'm just speaking from my own experience. Hope that will help. Awesome, thank you Big Joe. Yeah, let's do this. Given you have been Google for eight years, let's speak for your own experience. Can you share with us how would you build product inside of Google? What kind of specific framework? The reason I ask this question is that maybe lots of people outside of Google are like, wow, Google is the best tech company in the world. They must use the best framework and methodology we'll offer you to learn from you today. Can you start to share with us how would you build a product in Google? All right, so my own experience of being product are with this framework. So there are multiple steps from beginning to the end. So very beginning is I always focus on the users. Who are our users? What do they need? And how do you know what people need? Most importantly, first, user research. Interview people or send out surveys as well as looking at data on our own product that you can track your behavior on existing product and find what their patterns of behavior on the product. Starting from there, and then you can build what we call a persona or what kind of user we're actually targeting for and what exactly their need and particularly on that need. I want to emphasize that. On that need. So you're already using a product. There's a gap between what they want and what you actually offer. That's the one you want to tackle. This is the first step to really understand the user. Any questions here? Great, Big Joe. So actually something very interesting about user research because at Verizon we do the same thing, customer first. Now the sensitive question I have for you is that how do you do user research? How many is enough? For example, any time you reach and like launch any new product, you can launch in many different verticals, five different verticals. Each vertical there are lots of users you can interview. So, and also it's also to survey. What's your prioritization in terms of interviewing customers? So in the user research, there are two major methods. One is qualitative research. One is quantitative research. Qualitative research includes things like user research, including things like user interview and focus group. And quantitative, I think like survey. So you got things from data and the metric and tells your story. So quantitative are the ones actually give you the end result. So it need to be very large scale. But qualitative are the ones you have very handful, a small number of participants, but they give you a qualitative result which can help you define your quantitative research. For example, you can interview as a six people, of course, they're very small than people, but based on your conversation with them, they can tell you what kind of options you want to put into your question there. And you send out to as a thousand people, that scale will give you the end result of understanding user review. That's how you combine both qualitative and quantitative researches. Awesome. So Bichou, I do have one more question. I like how you put in a qualitative and quantitative. So which one goes first? Do you send out massive survey and then you pick a few people or you talk to you first, then you send out massive survey. So which one comes first? Usually qualitative comes first. So usually qualitative comes first. Because you talk to people and based on your conversation, then you come up with a question there. Makes sense. Awesome. Great. Let's move on to the second part of your framework. Okay, the second part is to actually define the product you need on that user need. So based on user understanding, you build up a persona and then you know, okay, they want this and then you offer this. So there's a gap in between. There are a lot of things you can do. One very important thing that a PM needs to do is to prioritize all those needs and then build a roadmap. What do you do first and what do you do next? For example, what would be the MVP? The same thing you build very first and then what's the next iteration? What's the next iteration? This one involves what we call market sizing or opportunity sizing. You need to size the whole large in that opportunity. For example, if you do that one feature that is only Nancy can use and Nancy needed badly, right? Which is awesome. Nancy would like it. However, if you build a feature for Nancy alone, it doesn't help most of the users, right? That opportunity for that feature is only one, which is Nancy herself. So you have to look at all the needs out there. We have different needs and find out the largest need, the largest amount of need, largest opportunity out there. And then you have a sequence of feature on the view. Always start with the largest opportunity with the least cost. Basically you spend the least of resources but at the highest impact. Then you start actually writing the requirement. Great. I like that you actually emphasize on like this is a trade-off, right? You actually focus on the maximum people you can serve and more money you can drive for the companies, which is also relevant to the program management interviews that we talk about in our book and the three webinars a lot. You need to emphasize on the achievement. You can emphasize on what the achievement and the benefit impact you bring to the companies. Also something you need to bring into the program management interviews. Awesome. Thank you, Fijio. Let's continue to the next session. Yeah, next session is to come up with PRD and then work with the core functional team to actually define and finalize the results of the PRD, such as you work with engineers, you work with designers, you work with other teams. Because sometimes your features depend on other teams' features. So then you talk to all of them and put everything onto your documents, every sign-off, say, okay, I supported this on this television and then supported this timeline. For example, I'm going to fill this in as a Q1. Everybody agree with Q1. That's basically kind of like the definition of what we want to do. Put it into a document. So, Fijio, let me ask you a quick question. Lots of students ask me this. Can you give me an example regarding good requirement and bad requirements? The good requirement is basically have done all the things that I talked in the past. For example, user research, data analysis, as well as partization of the opportunity so that we can read this requirement, understand the background and the motivation. And to them, it makes sense to do this because there's a lot of opportunity out there to need it. The bad requirement basically do this because I say so. That's kind of the bad requirement because yeah, you are looking at U-defined features. Yeah, people will do what you want them to do, but you want to motivate other people to do the right thing. At the end of the day, if a product succeeds, everybody succeeds. So everybody need to have a faith in the feature that you define. So all the upfront work, as I mentioned earlier, are very important. You need to do them. Second thing is you need to listen to other people which I particularly emphasize on. Oftentimes, I have an opinion on my own so I visit the best feature for the user, biggest opportunity, but I do not always insist on my own opinion. I listen to designers and I listen to engineers. I personally find it very important because oftentimes you do not have the big picture or the overall picture. And then it will tell you, well, from a system design perspective, there's actually a certain constraint. Designer will tell you well from the overall design perspective, there's certain pattern that's better to follow because there's some good practice from the design perspective. So I'm very open to listen to them and I'm actually very open to multiply my definition to add the different variations to my definition, which I think is going to helpful. First, from a kind of definition perspective that you get more input. Second, also, the team will see you as an open minded person instead of someone who is very dedicated. I tell you, you have to follow what I do. I think that is not a good PN. I think PN should be open minded to listen to other people's input. Awesome. I like your example regarding how you engage and listen to engineers. I also heard that Google specifically is more engineering-focused company. You do need to bring everybody together to make decisions together. I do like this. Awesome. Thank you, Victor. All right. So should we move to the third part? So moving forward. So we're going to do the last part, which is the execution part. So now you have user need. You have product requirement, which includes engineering includes such as designs and mocks, et cetera. Then you will need to actually execute it. So a few things that are important in the execution phase. First of all, it's important to figure out what is the metric that you're going to measure to define a success launch. After you exact everything, you will need to see whether this product, the launch actually meet your initial thoughts. For example, you define the success metric as let's say a number of people use this feature every day. Or the success metric is the revenue of your product. Anything, you can define that. And of course, you need to agree upon yourself, your engineering team, your designer, leaders, everybody, everybody define a same metric. And then as you implement it before your launch, you need to run an experiment, which is also called AB testing. The experiment will be run against those metrics you define. That is very important. You have to define that first. And they run an experiment and they see whether that experiment meets the initial metric that you define. And you say, okay, we run an experiment, half of people will have old treatment, half of them will have the new treatment. And new treatment does increase the revenue by X percent, which I defined earlier. And that's the successful execution. And then you can launch it and go back to your first step. Say, okay, analyze the user behavior in the product and also new ideas to meet our net need from the user. So this is kind of a general framework how I define that. Awesome. So Victor, I do have a follow-up question because during the product management interview, majority of the students will ask this question, how would you define the metrics of your product? And in your example, you mentioned metrics, AB testing, you know, the typical metrics in general, daily active users are also pleased at many different ways to define the metrics. Can you give us some examples regarding how to define the product metrics and what type of metrics do you usually use? So this is a very broad question to answer because it really depends on the situation, depends on individual project. Because even like all the features that I have built in the past, they have been in a very different bucket. So they have different metrics. For example, if a feature is a stage of a new feature, it's something really, really new, then it's very difficult to measure the actual impact that we have nothing yet. So in that case, things like the user happiness could be a measure to measure or you can have some initial estimation of the overall market size based on the MBG launch that you launched earlier. If the product is in a mature stage, meaning it's already out there and then you are just launching new things to improve it, then you have a lot of old user and old data to compare with. So in that case, you can just define something for them, either revenue or either users, then you compare with the old, see whether they actually improve or not. So it actually depends on the stage of the product. I would say it's gonna be a really case by case thing. And also from an interview perspective, there is no framework in my opinion to address this question because every question is very different. You have to think outside the box to address that particular case, that particular product. Awesome, yeah, you're right. So in summary, I feel the same way about the metric. The metric is so flexible. It has to be case by case as an example. I also like that you mentioned you build MVP, the entire launch force apps, all the way you launch MVP, right? And then you continue to improve it. That's the key of the tech industry as I described in other interviews, that one part of product management is in tech industry. It's very fast and people need to build new product using MVP about how to launch a product on concept execution. You can check out those videos here. Thank you so much for being able to share with us. Do you have any summary or final thoughts you wanna share with us and how can we get in touch with you? Thanks Nancy for your invitation. So I'm really glad that I can share the framework I use in my work. So if you're interested in learning more about career development in tech industry, feel free to come to my channel on YouTube and the BDBD for in China. So unlike Nancy, my channel is a Mandarin speaking channel. So I speak Mandarin in my channel, but I'm adding English subtitles gradually. So if you look at some old videos in my channel, roughly like there's like a one or two weeks delay, you can see I added English subtitle in my older videos. So I talk about all kind of career development in Silicon Valley. And Nancy is actually one of my guests in my channel as well. So yeah, please come to my channel, search for Big Joe in Silicon Valley at YouTube. Thank you very much, Nancy. Thank you very much Big Joe. Yeah, make sure to check out both channels. I've done a link to Big Joe's channel in my description so you guys can check it out. But make sure to join the private face group I have for product managers and change makers. I go live in the product manager groups every Sunday to share with interview tips to become a product manager. All right, I'm gonna see you next time. See you guys. All right, see you next time, see you everybody. Bye-bye.