 Hi guys, my name is Rakesh and I work as a product manager at Google and today I'm going to talk about foundational product insights. I was told that a lot of you here are aspirational product managers so can I get a show of hands on how many of you want to become product managers in the near future? Okay and how many of you are already working as product managers? Cool and how many of you are just curious and want to explore product management? Cool that's a fair mix. Cool so I'll quickly start with what I want to achieve today with this talk. My goal is to inspire curiosity amongst you guys around how to think about foundational product insights and I'll be using a lot of practical day-to-day product examples to do that. So first of all, since most of you are either aspiring or are curious about product management, I would like to start with two most important attributes that I think a product manager should have. The first one is leadership. I'll just briefly touch upon this attribute and most of my talk will be focused on product intuition. So coming to leadership, so PM is responsible for the overall experience received by the user. So imagine if you are a hardware product manager, let's say you are a product manager for Apple and you are responsible for iPhone, then it's not just the product that, it's not just the iPhone hardware that you are responsible for. You are responsible for even the customer support, you are responsible for the packaging, you are responsible for even the iOS software that goes into it. So as a PM, you are responsible for the end-to-end user experience that the user receives from your product. And note that you are responsible but you do not have any direct authority over people who are responsible for building the product. So that's very important. So you would not have direct authority over the engineers or the program managers or the UX designers who are working on the product but you have to influence them so that the right product gets delivered. As product managers, you need to be bold. You need to be making bold decisions. So there will be instances where you'll have nuanced options in front of you and then instead of just using an average option, instead of just using an average option that makes your team happy, you need to be bold about the decision which should be around what's your ultimate product vision that you want to achieve. You need to be able to show people what you see. So you need to be able to convince them about what's important to work on and help them see the important priorities for your product. And the last thing that I think a product manager should do is the product manager should be able to steer the ship. When I say ship, it's mostly your team and if it's a startup, then your company do in the right direction and this can be done by a ruthless prioritization. So if you have 10 things to work on, then make sure that everything is stacked in the right order, in the order in which you, in the order of your product priorities and you have to cut a lot of corners and make sure you're just making your team focus on the most important product features. So I guess this is very cliched most of what I've spoken about. So that's I'm not spending too much time on this. I'll focus mostly today on product intuition. So I believe that all successful products are built around foundational product insights. And the way you can discover foundational insights is through understanding jobs. How many of you have heard about jobs to be done theory? A few of you. So a jobs to be done theory is conceptualized by Professor Clayton Christensen from Harvard University. And it's a very powerful tool that a lot of emerging companies are using today, like Airbnb. And it's very powerful for understanding your customers. So I'm sure in all your talks and in all the courses at product school, you might have heard about this phrase that you need to understand your users well. So I think if you want to really understand your users, jobs, understanding what jobs they're trying to solve would really help to, would really lead you to the foundational insights. So I'll explain what jobs is because this will, you'll see this term throughout my presentation today. So jobs is nothing but a bundle of needs with a very rich contextual information. So to give an example, here you see, if I just express a need in traditional sense, let's say if I need a hotel or a place to stay in New York, if I just think about this problem with respect to how traditional needs are defined, I would just say that I need a place to stay in New York. But if I were to think it from the perspective of a job, which has richer contextual information, I would have more information around it, which is I need a place to stay in New York. I want to live like a local in a homely space and explore health kitchen area, which has the best food in New York. And I'm a little tight on budget and wish I could crash at a friend's place. So this tells you a completely different story. So if I were to just tell you that I need a place to stay in New York, then you might just suggest me, okay, you can go for hotels. In pre Airbnb days, hotels or like if you had a friend, that would have been your only options. But now, after Airbnb came to picture, now, if I tell you the rest of what I want, then you'll probably suggest me like probably the best option for you right now is Airbnb instead of just taking a hotel because Airbnb provides you more than just a place to stay. It provides you all the local experiences. And it's also slightly inexpensive than your traditional hotels. So I'll get more into Airbnb examples in a bit. So this is just like in summary, a job is a combination of needs with more richer contextual information. And it's in a way defining some kind of progress that the user is trying to make in their lives. And our job as product managers is to build products that people want to hire to get their jobs done. Any questions so far? Like what jobs are cool? So start with my first example, which is around Snapchat. How many of you use Snapchat? Can anyone of you tell me what do you like about Snapchat? Anyone? Sorry? Cool. Yeah, so spot on. So a lot of people love Snapchat because the messages disappear in a while. And if you guys know the history of Snapchat, the way it started was it was a sexing app for teens. And what Snapchat identified was that teens needed a way to communicate with each other, escaping the prying eyes of the adults. And they wanted a more personal platform than what existed in the market. So if you look at Facebook, at that time when Snapchat was beginning to be developed and launched, Facebook was the most prominent social networking player in the market. And still Snapchat was able to carve out a niche for itself. And the reason was Facebook had a fundamental gap in its core product, which is Facebook is based on permanence and perfection. You go on Facebook and you want to present the best of yourself out there. So when you post a profile pic or any photograph or you post any statement, you want to make sure that it's the best sentence or it's the best photograph that your friends and other colleagues would appreciate. So Facebook in a way is it actually implicitly encourages you to do that through a field path loop which is driven by likes. So most people when they post on Facebook, they keep track of how many likes did I get. And now if you look at Snapchat, what they did was they based their product on ephemeral social experiences, which means that it's like fundamentally different from what Facebook is about. In Snapchat, when you send a message or a picture, it's there for a few seconds and then it disappears. And then later on Snapchat released a transformational product feature called Stories. So if you think about Stories, it's so transformational that Facebook has copied it outrightly across all its products like WhatsApp, Instagram, Facebook Messenger and Facebook itself. And the cool part about Stories is that you can post whatever you like and it especially caters to this particular niche which is you don't want to care about what people feel or what people think about what you're posting. So Snapchat created this product where you can post whatever you feel like, you can capture the moments of your life and you don't care about whether anyone agrees with you or disagrees. And there's no feedback loop in the form of likes and it's intentional. So you can't like a story because the fundamental nature of the product is that you don't want a like feedback loop because it's all about like being yourself rather than presenting the best self, rather than presenting your best self out there. And in a way, if you think about it, Snapchat is very analogous to the way we speak. So whatever you speak, it just like goes away after you've spoken. And Facebook is more like recording yourself. So if you think about, if you're recording yourself, if you're recording a birthday message or something for your friends, then you would want to rehearse it a couple of times and then you want to make sure that your voice and whatever message you're trying to give is the best one and it's perfect. So that's one way to think about it. A difference between Snapchat and Facebook and why Snapchat got so successful. I can't say like if they'll survive in the long term, but as of now, they were able to find a great like niche in the market and were able to build a great product. Any questions? Cool. My next example is Airbnb. So the insight, the foundational insight that Airbnb had, which was contrary to the general belief at that time, was that people are comfortable sleeping under strangers' roof and hosting strangers. When Airbnb pitched this idea to VCs, people laughed at them. They said that like, it's crazy. Like people, why would someone's, why would a stranger stay in someone, why would someone stay under a stranger's roof? Or why would someone host a stranger in their house? So people just could not get their heads wrapped around it. But Airbnb founders saw this need in their personal lives and believed in that. And they also found the insight that people valued homestay experience rather than the cookie cutter hotel experience. So these were two key fundamental insights that Airbnb found from their founders' personal lives and that's how they built the whole product experience around it. Now, if you think about pre-Aerbian B days, people, people care, travel still care about trust a lot. But pre-Aerbian B, hotels had a huge advantage over any other form of, any other form of stays, which is hotels in their concept and as part of their brands, they promised trust, trust in the sense of safety and security and also quality. And what Airbnb did was they created a, they digitized this trust model. So if you think about it, as host, you are incentivized to provide the best place to your, to your travelers. And as travelers, you are incentivized to keep your host's place as pristine as possible because there's a feedback loop there, which is your host can leave reviews for you and you can leave reviews for your host. So that's the trust model that Airbnb built. And what this did was really powerful. If you think about it, pre-Aerbian B days, as I just said, trust was most important for, for the travelers. And what Airbnb did was they neutralized this trust advantage that hotels had. So earlier, if say, had this, had this trust model not been built, if I, if I were to ask you, would you stay in this stranger's place versus a hotel, you would have always chosen a hotel over a stranger's place. But now, because, because Airbnb has neutralized the trust, like hotels and Airbnb have equal amount of trust these days. So now, Airbnb is competing against other parameters. It's not trust. It's other parameters like experience, cost. So if you think about it now, as I said, Airbnb found that people value home stay experience. So now Airbnb is completely focused on that experience. They've, they've added new features in Airbnb, which is around providing local experience, experiences to users. So they are doubling down on that insight now that they have this trust model. And they are able to compete very effectively against the hotel industry. So what the home stay experience that Airbnb is providing is like, what is it like to live a day to day in a different country, in a different culture versus a typical manufactured experience that hotels provide. And if you think about it, Airbnb's tend to be slightly inexpensive than hotels, than most typical hotels. So it's now competing against two parameters with the hotel industry. And before Airbnb, it was just trust that hotels used to capitalize on. Any questions? My third example is around television or cable industry. So the boxes on the left, which are colored in yellow, are the jobs that cable industry used to fulfill. Let's select cable. I'm using cable and TV interchangeably here. So the boxes on the left are, which are colored in yellow, are the jobs that the TV industry used to solve for the users. So we used to watch television and pre, pre internet services era to get informed in the form of news channels. We used to watch television to get educated by watching channels like Discovery National Geographic and so on and so forth. We wanted television for sports and storytelling, storytelling in the form of movies, TV shows, and then escapism. This is a really important one. It's like the way I describe escapism is like when after work or after you go back home, maybe like after like after your school or whatever, people had this habit of just mindlessly browsing through channels. So that's what I call escapism. Like you just wanted to kill your time. So now after internet services got dominant and more and more people started getting access to mobile phones and internet, these jobs were taken away one by one from the cable industry or the TV industry. So information was being taken away by likes of Facebook, Google, education, again, likes of YouTube and other online services. But interestingly, sports is still dominant with cable industry because they haven't made streaming available for sports yet. It's going to change soon given that ESPN is launching its OTT service pretty soon. But sports is still something that people use traditional television for. Then storytelling has been taken over by the likes of Netflix and escapism by the likes of Snapchat, Facebook. You can ignore the columns on the right. But any questions? Sure. Yes. Yeah. So what I'm trying to explain here is, yes, sports has not been able to move towards, let's say, another service provider because of like rights and economic reasons. But the jobs part here is these things like if you think about television as a product, it was solving these jobs for day to day users. It was solving jobs like getting informed, getting educated, watching sports. So these were the jobs that users had and television was fulfilling that. Now, like as and as in when services became dominant, these jobs were taken away. Now if you think about television, most people watch television like the traditional cable network for watching live sports, not for these other things. For these other things, you have other alternatives today which are becoming more and more dominant. Cool. Cool. So now coming back to job, like I want to now touch upon how can you identify these jobs? When you would go out as product managers, this jobs theory would really help you to discover these insights and understand your users better so that you can create transformational products like in your own companies or like you can start your own startup using these. So I have I didn't I will talk about three ways you can identify your customer's job. It's helpful for both life. You have your own startup or if you are in a tech company or just in a if you're working for someone or you have your own startup, this this is a really good way to identify jobs in people's lives. The first one is you can look for jobs in your own life, which is like look for needs and problems in your own life. And you you will be able to figure out something that still has not been solved for. And you will be if you like, if you want to create a product around it, you can be really successful on that. So one cool example is Khan Academy. So the way Khan Academy started was Sal Khan, who is the founder of Khan Academy. He wanted to create videos to teach his knees difficult maths concepts. So his knees always had problems with maths. And she used to like lessons from Sal Khan. And because he he could not be in touch with her on a regular basis. So he started creating some scrappy videos for her. And over time he saw that a lot of people started sharing those YouTube videos and that insight like made him think that what if he can help more and more students to understand maths and other educational subjects better. So that's that's that's where Khan Academy got started. So you should you should look for the problems or jobs as as we call it in your own life. Next is non consumption. So if you think about Airbnb pre Airbnb days, there was not a singles dominant service where hosts could rent out their apartment for a short term to random strangers. So so if Airbnb wasn't there, then even today, it would have been difficult for hosts to provide to sublease their apartments to random strangers. So it's it's like they would not have used this opportunity. So it's like non consuming. This particular aspect that that they could do with the help of Airbnb. And when Airbnb rolled out a survey, they found that 40% of their users would not have either traveled or would not or would have preferred to stay with a friend. If Airbnb did not existed. So so that shows that these 40% people would would have chosen either to not just just not do anything. Would they would either not have traveled on would not have like taken a hotel. So that's what I call non consumption. So you should look out for things where people would not do something because a solution does not a perfect solution does not exist for them today. So there will be people who just choose not to do anything. They would love to just stay without a solution rather than like do something about it. So that's another key area where you can discover jobs. The third one is workarounds. So this is for users who see that today a perfect solution doesn't exist. But they go ahead because they care about that problem so much. They go ahead and they create workarounds to solve the problem. So one example is stories. So earlier, I told you some aspect about stories which is ephemeral experiences. Another aspect is I don't know if you guys have seen it in your friends or families that people like to take 10 to 20 seconds videos when they go to new places or when they experience something new outside. They just want to capture those 10 to 20 seconds of videos so that they can relive those moments later on. I've seen this with my parents when they traveled somewhere new. They want to capture like the area and for 10, 20 seconds so that they can look at it again and again later on. So stories is a really powerful product which is solving that need. If you think about it, I don't know if you guys have used stories. So stories allows you to capture these 10, 20 second videos and it just caps you at that and it allows you to even save those videos, add filters and make really cool videos out of it. So the founders or whoever created stories really understood that need that existed. People were using workarounds like using existing camera and video recorders to record those clips and stories, the creators of stories just banked on that insight and created a wonderful product around it. My second example is splitwise. How many of you know about splitwise? Before splitwise existed. So for those of you who don't know it, splitwise is a way where you can manage your expenses with friends. You can split bills or you can keep track of what you have paid versus what your friends or family have paid and then you can settle expenses later on. So please splitwise. People used to use tricks or pen and paper and other kinds of stuff. But what the founder found was he could build a really cool product around it to automate most of these processes. So that's another example of where people use workarounds. If you observe people using workarounds in their day to day lives, there's a great opportunity to build a transformational product around it. So once you have identified a job and a key insight, there are these three things that you should address in your product to make sure people adopt your product. Because there's a lot of resistance and inertia that why users don't want to use a new product. And I believe overcoming these three things that I'm going to talk about would help adoption of your products better. The first one is current frictions and struggles. So all products try to overcome some kind of friction and struggles in users' life. And if you want users to adopt a new solution, you want your solution to be at least 10x better than what's already out there so that they can leave their existing habits or their existing products and they can switch to your product. Because there's always a switching cost associated when you try a new product. Let's say if I tell you, if you're used to using Facebook, if I tell you today, use Google Plus for all your social networking needs, I'm pretty sure you will try to avoid it. You may try it once, but that usage will not survive. Because all your friends are there, you're already used to using a particular product. So if a new product is not at least 10 times better than the existing solution out there, people will not adopt your product. So to give an example, when WhatsApp started, SMS was already there. And a key friction for SMS was that it was paid. Not so much in US, but outside US in most of the countries, SMS was very expensive. And if you look at WhatsApp, WhatsApp started becoming a dominant messaging player outside US. And then later on, people started adopting WhatsApp in US as well. And the key reason why WhatsApp became successful and why people adopted WhatsApp was because it was fundamentally just a free SMS app. It was like people attribute a lot of things to WhatsApp success like user interface and the simplicity around the product. But the core reason why WhatsApp made that carved out that market for itself was because it was essentially a free SMS product. And here, if you compare to like paid SMS, WhatsApp was exponentially better because it was free. You can't get any better on that price parameter. So WhatsApp was able to overcome that switching cost for the users because it was exponentially better in terms of the cost that users had to pay to send messages to their friends. The second one is anxiety of the unknown. So people always have, they're twice as much loss averse than if you tell them there's an opportunity of gain. So let's take an example of phone trade-ins. Let's say if someone tells you that, if there's an offer from let's say HTC and they tell you that return us your old device and we'll give you some discount on the purchase of new device. So what most people do is they want to retain their old phones with them because in the back of their mind they think what if my new phone breaks sometimes and like in case of emergency, I might use this old phone when things break or I lose my phone. So this loss aversion is always at the back of people's mind because they don't know what lies ahead. So this anxiety of the unknown is a great barrier to product adoption. So in case of Airbnb, I just as I just talked extensively about was people had this anxiety about trust. The house that I'm going to stay in, will that be of a quality house? Will there be like quality mattresses or washrooms are clean? So that trust and that anxiety was there. And also, am I really safe in a stranger's house? So Airbnb took that anxiety away by building that digital trust model in their app. So next one is current habits. This is slightly similar to what I spoke about in the first point. So here, as I said like any, if you want people to use a new exam, a new solution, it always has some switching cost. It may not be monetary. It may be just like learning something new or like the act of just understanding a new app or even inviting your friends to a new app is really painful for the users. So what WhatsApp did was really cool when it was competing against SMS. It retained your phone book as a network and it kept phone numbers as your login or as your IDs. So it's still a very powerful tool. So at that time, all messaging or messaging apps like Yahoo Messenger or G Talk, everything used email as primary logging mechanism, which required email password and everything else. But WhatsApp had this inside that because it's just a free SMS, why should you add another layer on top of it, which is like creating user accounts and all of that. So they leveraged your phone book. You already had a phone book on your phone. You already had your friend's phone numbers. So it used that to connect people and just digitized those connections further. So in a way, they did not make people change their current habits. They retained that. But this is just one example where people just retain current habits. You need to be conscious that people don't want to change their current habits no matter like who they are. So when you're designing your products, think about what are their current habits? Either make sure you are retaining those current habits in your product experience or creating something transformational like what WhatsApp did, like overcoming the friction in a 10x form so that people don't care about learning new habits. Cool. So that was most of what I wanted to talk about today. There are three books and one blog that I would highly recommend to all of you who are specially setting on this path of product management. The first one is competing against luck. It's all about discovering foundational insights and discovering customers' jobs and coming up to an insight around which you can build products. That's a really cool book and I would highly recommend reading that. The second one is the lean startup. You might have heard about this one as well. So this is about the mechanics of building the product after you have discovered the insight. So let's say like you come up with some transformational insight. So how do you go about building a product? So this just talks about that and it's a brilliant book. The next one is design of everyday things. It's about developing a design intuition. So everything around you like from the stable chair, even that can of coke, everything that you see around you is a result of someone's decision. Someone actually made a design decision to make this product happen. After you read this book, trust me, you'll look at the world very differently. You'll get critical about everything that you look at. So it's a very powerful book to read. The last one is a blog called Stratechry.com. So that TV jobs image that I showed that was from this blog and it's a brilliant blog to understand these strategies that the leading tech companies come up with. And the author is Ben Thompson and he has really key insights on understanding go-to-market strategies of these leading companies and he breaks it down into really easy to understand form. And you can read this on a regular basis to keep yourself updated on the new strategies that tech companies use. So yeah, that was it. Any question and answers? How is the concept of a job different from a user's story, particularly in a strong sense? Can you describe a user's story as an example? The typical format is that you stay as a person. I would like something to happen because of, so you say who you are, what you want, and why you want it. So how is that, is that conceptualized the same as a job or is this a different concept? I think these concepts are somewhat similar. In job what you also do is you think about a particular circumstance. So in a user story the way you described it, it's about who the user is, what they want generally and why they want it generally. And the job is focused around what do they want in this particular circumstance and why do they want it in this particular circumstance. Like there's an example on this book which really describes it well, which is around milkshakes. So a customer who goes to buy milkshake in the morning while he's going to work has a different use case in mind. He wants something to keep himself busy for a long drive. He wants to not feel hungry after the drive. He wants like the milkshake to be slightly entertaining like it should have some nuggets of fruits in it so that his boring journey that's like one-hour commute becomes somewhat interesting. He should not feel hungry after the end of the journey. So it has a different job when he's buying that milkshake in the morning just right before he goes out for work. And the same milkshake can solve a different purpose. Let's say when that same person comes with his child at that store so he might want to buy that milkshake to just be a nice stat to his child. Like he wants his child to enjoy that milkshake. So the point here is that the same product or the same user might have different jobs for a similar product at different instances. So if user stories can add that contextual information as well then that would become a job. However you want to say it is going to use your product. So the way I answer that question is the user story to a job versus a user story to a job. To me I think of user stories as a task. So if you look at a lot of user stories as a blame that's generally the same type of user or a subset of users that's the blame. Is that how you would think of it? So I think that's one way to see there's no right answers here. So that's one way to think about it. The way I think about jobs is adding personas and user stories together. So the way I think about personas is it tells you user characteristics the demographic data, the behavioral data about the user. So that's giving a background around the user. And then if you add more information which is around user stories like the tasks that user is doing in a particular circumstance if I add both this information then that makes a job for me. And for that persona there might be number of jobs but that persona remains the same. That this is a particular user and that user has this, this and this job which is slightly similar to the user story. So get too restrictive on this because it depends on the product in a very simplistic, oversimplified rule that I've heard often with personas and the question I have is like do you find jobs being the same? Usually you don't want to have more than six personas with a product. Do you find that you want to be really, I would think you would but do you find like there's some form of constraint on jobs or does that not really matter as long as your product is servicing a large number of jobs that do it versus like a persona you probably don't want to have 40 different types of jobs. Yeah, so let me rephrase your question so that I understand it well. So you're saying that a persona generally people try to restrict it to five or six so that the product is not all around the place and a job can be like 20 jobs, 30 jobs so how do you maybe like how do you prioritize or how do you restrict the jobs? So I think different product teams work differently even at Google a lot of teams have personas and then they define these tasks or these jobs and then they prioritize so it's not like you have to work on all 30 jobs so the user persona gives you some information about the user like understanding their current habits or what kind of anxieties they would have and the jobs like a user might have 20 different jobs related to your product so you then figure out what are the core jobs that you need to solve for and then prioritize. So there's a prioritization exercise that goes like how severe is this pain and how widespread is this to like filter that list down from like 30 to like maybe top three, top two. Go in the context of user story versus the jobs. What you're saying is that user story is the customer's perspective and the general objective and the job is the company's or the product's perspective of a specific objective kind of combination in relation? I have actually not done a good job here to explain what jobs because I wanted to keep it very high level but if you read the book over there you'll see some examples and it's actually from customer's perspective. The way I've listed the insights here it appears to be from company's perspective because I wanted to really show you the key insights but jobs is really from customer's perspective. Their stories on what's going in their mind while they're taking decisions and how they are solving their needs and where they are, who are they with. So all this contextual information gets added and we try to create a story from the user's perspective not as we see it. And the best way to find jobs is by talking to people. So recently the project that I'm involved in I talked to a lot of customers and going to really in depth questions and answers and I just keep on capturing those things. And then I try to build a user persona then what are the different tasks that they do and then try to really paint a picture around the user. Yes, very good question. So for people who are watching online so the question was how do you determine which insights are powerful or which insights you should prioritize. You may get like 10 different insights initially. So the way you can converge to the foundational insight or the core insight is by doing a series of experimentation. So let's say if you develop a hypothesis that like these are the top three things, top three insights that we want to solve for then you should do some ab testing for each of these hypothesis and get collect data and see which ones are really working which ones are not. So I think no one can if you ask different people, everybody will have opinions but those don't really count unless you experiment and validate your hypothesis, your insights and like the book that I mentioned the lean startup it has got really cool models on how you can set up these experiments and that can really lead you towards the core insight around which you would like to build your product. That's a very good question and that's a very tough one. So as a company like Google or Facebook data is given a lot of importance. So as I just said through your intuition you can develop hypothesis through observations and intuitions and now you need to validate those hypothesis. So until you have validated something, we don't really proceed with scaling a product on the basis of a hypothesis or an insight that we have. So we generally like figure out these are the top insights, let's build hypothesis around it. These are assumptions and hypothesis and set up experiments to prove that and through these experiments you'll always get to the right answers and you might be surprised sometimes that you and your team might have certain hypothesis that like we should have chosen option a for a product but through this experiment you would get the right answers that probably option a was never the right option, you should have gone with option two. So the key part here is like when you're launching experiments make it as small and as simple as possible. Don't invest too much resources into building a complete well-polished experiment or product because you would if at the end of the experiment you realize that whatever you're testing doesn't make sense you would have wasted all your resources all your engineers and all the other people who are working on it. So one key thing that we practice is going lean, design a man viable product as lean as possible so that it really addresses these hypothesis whether these are true or not. I found aap does a good for incremental changes if you're doing something radical or very different then more likely than not you just take time to get used to it aap does doesn't do well and so I'm not sure if you get very good insight so you draw the wrong thing. Yeah I think it depends on what kind of experiments or what the scope of your experiments are. So if say, do you have an example in mind? Example let's say if you're trying to change search ranking. Yeah. Maybe that might be good for something like that like you just get it right away. Yeah. Other results relevant to the search where you're not but if you're changing the line in some part of a different way you just take time to get used to it or engage with it. There's some learning that may or may not reflect something that might be good in the long term and the short term it doesn't show the results. Yeah so we are very careful about these long term experiments versus short term experiments. So for example the YouTube UI change that I don't know if you guys have noticed the UI changed so that took a lot of time. It's not that we thought about the UI three months back and boom the new UI is there. So we experimented with the UI for a long time and then based on like what really worked well with the user behaviors and habits based on that when we were really confident that yes this is worth rolling out further we start like we started with 1% experiment and 5% and 10% and if by 5% we see that the metrics that we are tracking with respect to user behavior and habits they do not align with our goals then we roll back the experiment then we try something else. So we don't really roll it out to like 2% or 5% until we get more confident with the metrics that we are measuring. So as you find that the metrics are aligned with the goals that you had in mind the success criteria you had in mind you keep expanding the experiment to more and more users and the experiment might last a long time. So there's no fixed timeline for any experiment. It really depends on the product and the features and the scope of the product. So what I enjoy is working on these like on the problems that I work on these are really critical problems that I care about and the freedom that I get to think about solutions and to take decisions and to work on things that I really care about is something that keeps me motivated and that's something I really love about working at Google and with respect to sorry what was the next question? I think the struggle is alignment there's so many cross functional teams at Google and it's really sometimes it becomes a challenge to get everybody on the same page like one of the attributes that I talk about was leadership and convincing people on the most important things to work on so that takes a lot of time everybody is smart and everybody has strong opinions so in that environment how do you move fast and are able to account for all the voices and then take those bold decisions so that's something I find challenging and again exciting as well sorry? Yes so data and like whatever is good for users that is our filtering criteria so let's say if both of us are working on a project and I have a really strong opinion than you as well then what wins is like we look at the data we look at what's really best for our users and then we make the decision and we always start with the assumption that everybody has good intentions and everybody has users intentions first and everybody has a rational theory behind their opinions so we always start with that assumption rather than thinking that oh the other person is trying to sabotage my project yes so the question is how important is technical knowledge for doing a PM job at Google I would say it's fairly important you need to be technical enough to be able to have detailed technical conversation with the engineers it helps if you have a technical background but even if you don't and you can prove that like you had the way you can prove it like by doing some side projects or showing some experience where you had some technical contribution to a project that really helps so the sole purpose why we want people to be technical is so that they can really have meaningful discussions and take the right decisions when they're thinking about technical solutions for the products so you'll be in a room where you might be designing the system with an engineer like you may have arguments like the system should be designed in this way you need to be technically savvy to understand how things work to be able to make decisions and to be able to put your point across so engineers would not respect you if you don't have technical expertise so that's why it's very important and there's nothing to discourage people who don't have a technical background I myself don't have a technical background and I know a lot of successful pms at google who don't have technical background but they were able to demonstrate technical capabilities either through side projects or through some contribution to tech projects