 So let's give it up. Thank you very much. All right. Hi, everyone. My name is Sung. I'm the product manager of youtube.com on the desktop platform. Previously, I also worked on the live streaming services, so I'm sad this is only streamed on Facebook Live. So a little bit about myself. First of all, word of warning. I like to dance, so my hands can get a bit aggressive. So apologies if it's dramatic. I work at YouTube, so things are a little bit less serious as you'll see in this presentation. My background. So before I used to, before I worked at YouTube, I used to work at a company called BCGTV. Essentially, it's an incubator for startups. There I led a product team of about five people from its inception to launching a startup called Tact.com. That powers a lot of the personalization services behind some large companies. You probably haven't heard of us, but for example, if you've ever used the Starbucks mobile app and it's suggested that you buy a breakfast sandwich that was my fault. Those are pretty good money for Starbucks. So before then, I also worked at Microsoft and that was for the Windows platform apps. So any pre-installed applications that you get when you first install Windows, I worked on those as well. So why am I explaining all this? It's because over the past few years of my career I've worked at extremely small companies where we've started the founding team to extremely large companies and ironically not much in the middle. But I think it's interesting. We talk a lot about startups. We talk a lot about what it means to be a PM at startups. But we don't really talk enough about how that differs when you talk about a product that's million users or hundreds of millions or even in the case of YouTube, billions of users large. So that's what I'd like to talk about today is, how do we, what are the challenges that we face when we are working on a product at scale and especially when we want to do something big with it? So our agenda for today, I'm gonna give you a little bit more background about what it is and why I'm standing here, why this timing specifically about what I'm doing. And then secondly, we're gonna walk through the steps that every product manager does, but potentially might be some interesting lessons for you. Whether you work at a large company today or whether you just as a startup PM today, I think a lot of these lessons transcend all that. So a little bit of background. YouTube. So I'm pretty sure hopefully everyone is here has used YouTube before. We have four major clients that we call the main app. I'm responsible for the desktop website. We've got the Android client, the iOS client and the mobile web client. So across the world, we have 1.5 billion monthly logged in users. You can imagine that gets a bit larger when you have logged out users as well. Over a billion hours watched per day around the world. And as for the website, we're the second largest website in the world. I think you can probably guess the first. There's another random tidbit. If you're in the TV industry, I would recommend get out now. YouTube, it takes up more hours of prime time watch hours for the 18 year old to 49 year old demographic than TV does today. In fact, our mobile clients alone already outpace TV. So a lot of things are changing. So with that, the website, which I'm in charge of also changes quite rapidly. You'll see that every year so far, or almost every year, we've changed the layout of our site. People say, if it's not broken, why fix it? There are very good reasons why. And most recently, we've come to this version of the website. This is the new youtube.com as per two months ago. This is a project that I worked on for about roughly two years. It's not just a facelift. We've rebuilt the site entirely from scratch. So every single line of code on that entire site that serves billions of users has been recreated. The entire period was two years, but actually for one year, we just spent it experimenting, constantly optimizing it. And that's the reality when you have a billion users that are very passionate about your product. Small size story. A few years ago, there's an urban legend that somebody at YouTube PM made a change that was not very popular. Someone showed up to the YouTube headquarters with a weapon looking for this person. So passionate users, you gotta be careful about what you do. All right, so today, this process everyone should be familiar with, right? As a product manager, you create a vision, then you establish a process about how you're going to execute over the next few months, you execute it, and then you finally launch it. So let's go through each one of these steps. Vision. All right, so less serious part of the presentation starts now. Vision helps other people walk off clips. And actually, it includes yourself. So let's start off with an analogy. That's you right there. And you've got a bunch of friends with you and you're all starving to death. Now, you're looking for food and it just so happens to be on this other side of the cliff. But you don't know that. Without knowing that, there would be no reason for you to walk down this cliff, risk your life, and climb up the other side. However, even if you did know that, you'd have to somehow convince all your other friends who are starving to also take that dive with you. This in a nutshell is product management at scale. Someone's job is always on the line and if you make a mistake or if you make a big enough change, someone's job is going to be negatively impacted, almost 100% of the time. So no one is willing to follow you off that cliff unless there's something that they can believe in and that you can convince exists on the other side. It'll never be, of course, concrete until you actually do it, but you have to find a way to convince them. So realistically, what does this mean? Now, this should be hopefully familiar to everyone who's been in product management. But if not, let me explain this to you. This is the question of local maxima versus global maxima. The current spot is where you are currently as a product manager with your product and there are some things that you can do to make it better. You can probably, for example, in the case of YouTube, drive a little bit more watch time, get a little bit more engagement, get people to open the app a few more times. And if you keep doing that, you will get to this level where you think, wow, I've really hit the end of what I can do. Everything I do now is in the single percentage digits of impact that I can make. And this is again a problem of every large company out there. When it's somebody's job on the line, it's really hard to walk downhill, especially if it's your job that's being measured in that metric, than it is to walk uphill. And it's so easy to get stuck in that trap of being at the local maxima. But why does vision matter? Vision allows you to see the global maxima, or at least a faint outline of it, and so that you can convince people to walk down that slope and then walk up in a few months' time. It takes a leap of faith. All right, so how do you actually execute this vision? You have this idea, you think you can see something in the distance, but you really don't know how to start this process. That is an analogy for my engineer, my lead engineer. He actually kind of looks like that. So the point is, when you start building your story, you can't build it alone. As product managers, there's this quote I, no offense to anyone who believes this, but there's a quote I hate called, you are the CEO of your product. In some cases, that's true, but in a lot of cases, you are not the one person that makes all of the decisions. You will need a team of people to support you. And that starts with your core team, whether it's your lead engineer, whether it's your lead UX designer, or whether even if it's your lead UX researcher. There's gotta be some people in your camp that you work with on a daily basis that will support you. And so build your team with them, have them sign off on it. So how do you create that story? So when you create a story, especially for a large product, I think it's easy for a smaller product potentially, but for a larger product, let's just say I want to redesign youtube.com. That's not something that you just float out there. You have to build a whole story. But ironically, the larger and more complex your story is, the less effective it is. So lesson number one here, build the simplest story possible. The larger your product choice, the larger your product strategy, the simpler the story's got to be. Otherwise, people are just gonna ignore it. So quick anecdote, for the redesign of youtube.com, I put together a story. It was very well thought out, had tons of data. I gave it to everyone in the company. Executives would come up to me and be like, why are we doing this again? Could you explain this again? And I thought I had spent years, not years, but a few months building these beautiful stories, but of course it wasn't effective. I talked to my user researcher. They said that the higher up the chain you go, no offense to any executives here, the shorter your memory gets. So stick it to like one or two lines, all right? Especially the more complex and the higher up it goes. All right, so you've got a cool story. It's fairly simple, but it's not done yet. To be an effective product manager, you have to use data. And I hope I don't have to convince you of that. And there is data everywhere out there. A lot of people get stuck in this idea of saying like, oh, I have this really cool vision, but the data just doesn't exist. That's false. And I encourage you to continue to believe that is false. The data is out there, but it is extremely hard to get, but you can find it in certain ways. And if I could implore you as well, stop ignoring qualitative data. A lot of people think of data in terms of what I can measure in my analytics dashboard, in my statistics dashboard, but there is more than just hard numbers. There's also qualitative data, which is of course numbers as well. Do your user research. Talk to your US designer. Do a lot of testing. It's out there. Okay, so you've got this cool vision. You've got this story. It's in the form of some sort of presentation and ideally a prototype because people like to look at things and they can feel it themselves. What's next? Well, you're not done. You need to specify the goal posts because even if everyone loves your idea, if everyone is convinced with your idea, no one is gonna sign off on it, especially at a large company, unless they know exactly where it ends. And this is for every team. You know that product manager I talked about who is gonna risk their life to walk down that cliff. That's reflected right here in this metric. Metric V, negative Y percent. That's something we don't talk about a lot in terms of what our goals are. We usually talk about our goals about the good things. I'm gonna raise users by like 10%. I'm gonna create deeper engagement by like 20%. But nothing large comes without a cost and you have to spell out those costs very clearly or that will come to bite you and probably crash your product. So make sure you don't sweep those under the rug. And then finally, share it out. Let's just say that everything is great. It looks amazing. It's just not the right time. And this happens so often in larger companies. We've got processes galore like OKRs, quarter planning, annual planning, who knows. But if you just put out a good story out there, you never know who will pick it up. Quick anecdote, I did that for myself. I created a story about a year ago outside of the YouTube redesign project and it just went sitting there for about a year. No one touched it until this year's annual planning came up and we needed a change in strategy. And voila, it was right there. So now it's one of our main strategies heading into 2018. Yeah, let me twist my actual example. So with the YouTube redesign, a lot of things changed. And let's just say that, what metric should I make up? All right. Let's just say that the number of shares that we have per video is something that we expected to take an impact because we were going to emphasize something else. For example, I'm making this up. Let's say I decided to increase the video player size so that I wanted our users to get a more in-depth experience. But at the same time, that would push out other items from the screen or below the fold. And so we know that would have an impact, but we were ready to make that trade-off because we explicitly said the number of shares would go down, let's just say, by 5% that we estimate. But the bundle that it comes with is that the number of, let's say, watch hours or playbacks are going to increase by, let's say, 2% to 3%. And at that point, as a shared PM, you're like, yeah, that really sucks for me. But at the same time, I can understand where you're going. And for leadership, it's a fairly simple decision for them to make. And by the way, those hard decisions that you want to make, make them early, not later. Because that later, they will turn against you because once again, someone's job is on the line. But if it's early, then everyone signed off from the beginning. They can't really push back at the end. All right, so you've got an awesome vision. And now it's time to actually figure out how are you going to spend the next two months, the next six months, maybe the next two years. So first tip I have is define stages to get to your MVP. If you're a startup PM, you're probably like, why? I don't need this. And you're right, you don't. This is for once again, a very large project at a large company. Oftentimes, we think of our MVP as something that we need to strive for first. And there are a lot of definitions for what an MVP is. My definition is a product that is usable by an end user, that they can get the overall objective, even if there's a lot of handwork, even if you're delivering mail yourself. However, at a large company, that itself also takes a while. So you actually need to break it down into further stages. And what you're delivering at those stages is not a product, necessarily, but it is proof. It is proof that your product is worth going after and is worth investing into. So that's what I've tried to break out here. In stage one, which is extremely small and extremely fast, you want to just prove some things. Prove A and B are true, that your product is interesting, that your product has promise. And that might be a prototype, that might be a simple test, who knows. Stage two, build on that. Prove the rest of the things that you need in order to say that my MVP is ready, that my MVP is actually a viable thing. And then finally, go and build it. Go and build that MVP. So on that theme I said earlier as well, simplicity, I'm sure a lot of people here, if you've been a product manager, you've seen the Gantt charts from hell. Processes on processes, various product management tools, roadmaps, endless backlogs, et cetera, et cetera, et cetera. And most people, I think, think that that gets more complex as you scale up your product. And that temptation is so tempting. Gosh, it is so easy to get into that trap. But what I found to be the most effective is you need to do the opposite. Kind of like how I said that as your scope gets bigger, your message needs to simplify for why your product is important, why your vision matters. So does your process. You need to extensively simplify your project. YouTube.com had thousands and thousands and thousands of features that probably no one here can really wrap around because there is so many people that we serve for various weird use cases. But ultimately, we could not create some behemoth of a planning algorithm. We had to simplify it extensively for everyone to be on the same page. And this goes back to once again, product basics. Just have, for each stage of what you're trying to do, rely on sprint. Have a prioritized list of what you're going to do, draw a cut line, and keep prioritizing that one list, I'm serious, that one list for that entire section of work that you're doing, and move that dotted line around. Whatever falls below the dotted line goes into the next stage and keep iterating. I can't emphasize how important this step is and how it saved our lives during this redesign. That you're sharing across. Yeah, I think it's whatever system, I know everyone uses their own, system that you'd use to track actual engineering work. So for example, at Google, we use a system called Bugganizer. It's an internal tool. You can see it, think of it as a JIRA for Google. It basically lists out features and tasks in a linear format, right? There's a lot of services out there. They all do the same core thing. Every week prioritize and make sure that you've got roughly the right order. You don't have to prioritize this list as well. Another mistake people do is they're like, oh, God, we gotta reprioritize everything. That's false. Just reprioritize what you're doing next week. Make sure it's the right big things. And oftentimes you can rely on your own memory. If you can't remember that it's not in this top section, it's not that important to you right now. And I know that sounds really sketchy, but it works and it saves a lot of time. All right. So you've created a process, you've created your rough stages, you've created your excessively simple ways of working, which I don't know if anyone believes in me actually works, but trust me it does. All right. How do you execute? Be very principled. I think my designer, if he were here, if he would love this, we established very clear principles of how we would work from day, not day one, but let's just say day 10. Fairly earned early on in the process. And in principles, I'm usually talking about two things. One, in terms of your process. Once again, larger company, people think, oh, I gotta get less agile. My sprints are gonna be now one month. The temptation is huge. Don't give into that temptation. Believe in the core principles of agile. Honestly, you don't have to follow it to the letter, but the principles of communication, of accountability, all these things are essential. And so for on youtube.com, a two-year project, we had sprints one week at a time. We had daily stand-ups, and that saved our lives. Because at some point, our project had been in danger of being canceled, but that process saved us. So yeah, believe in agile. But also the second thing, establish your design principles. If you're working on a large project, project scope creep is the scariest thing that you can ever imagine. If you already have 2,000 features, somebody's gonna be like, what's another one feature to your list? Like, is it that big a deal? And the answer is yes. Every one feature is a big deal because you are setting a precedence. And even in your design decisions, you don't know how many times I've been asked, can I just add a button here? And that is like my least favorite thing I've ever heard. Can I just add a single button here? And the answer is honestly, no. Like as best you can. And obviously you can't say that to everyone. I see it would kill me. But at the same time, you have to try and stick to your principles. And if enough problems arise, then your principles are wrong. Change your principles. But don't stop being principled in how you're building your product. Because once you let that go, your product goes downhill in the long run. So yeah, learn to say no, which a lot of product managers have troubles with. And then finally, an execution. I'm not gonna go too too depth because everyone's hopefully aware a little bit. Keep up team morale. And in a large two-year project, honestly I can't explain how important this is. Those stages as well really help. Give definable like goals every two months at the very least. Make sure that something's achievable, something's worth celebrating and you'll get through it. And then finally, it's time for a launch. Yeah, I've used the analogy of launching youtube.com felt like trying to, I don't know, like steer a massive cruise ship to turn a right angle or something like that. And this is the closest image I could find. But this is basically what happened to our project as well. At the last minute, everything happens because suddenly it's a reality. You're replacing the website for over a billion people on this earth. There's always gonna be like, oh there's a legal concern here, there's a last concern here. Just hold firm to your principles. Don't give in, you know, just stick it out to the end as best you can. However, set the expectation for turbulence. And this is not just for yourself, but for your team. If your team is thrown off by last minute requests, which happen all the time, you may not get there. And there are a lot of products honestly at youtube that have died towards the end because of just that. Because the team got demoralized, progress slowed, they never hit that moving target, and it was over. So don't let that happen to you. All right, and that's about it. So let me recap kind of like my final thoughts about what I've been talking about today. One, that vision. Build it from your core team. Get them on board. Back it up with data and share it out. Number two, make sure that you prove your way to your MVP, especially if your MVP is pretty large. And simplify, simplify, simplify your process. Number three, be very principled in your execution. This is really hard. And keep your team happy. And finally, be ready for that rough landing. And that's it. Thank you very much. I'll take questions. Yeah? So you mentioned the deep backlog. I'm sure there's like thousands of ideas from the organization, but there's like probably millions of ideas from users given the size of YouTube. Do you ever use tools to organize some bottom up initiative from all that feedback? Or is it more like a top down approach of, hey, here are the themes and initiatives of the product for the year. And I'm going to dig into the feedback looking for input. So do you ever drive initiatives bottom up, or is it always top down looking at it? It's absolutely mixed. And for that, I actually prefer people to not be too strict about being top down or bottom up, because you never know what will happen. At YouTube specifically, we have tools to help us consolidate all user feedback. By the way, if anyone writes anything bad about the website on Reddit, I read personally every single one of those things. So yeah, feel free to bash me in the comments. But yeah, we consolidate a lot of our feedback, whether it's from social media or whether it's from our actual feedback tools. And it bubbles up to us in terms of like some hierarchy of what are the top issues that we're hearing about this week. And we get this every day or every week. Honestly, that's a bit more of estimation. And what we estimate against is our key metrics. So I can't say our key metrics, but let's just say we have a specific set of them. And so if there's something that we think is hurting one of our top line metrics, we have user happiness metrics, we have usage metrics, we have consumption metrics, all that jazz. Depending on how much we think it's impacting those, we take a rough estimate. And that's where we shuffle the list around. And honestly, there's no exact science to it. It's literally myself, my tech lead, my UX lead, sitting in one room and deciding, is this big enough to be more important than the other? And at first it is a bit of a crapshoot, but after a while, as you start working with your team, it becomes really natural, everyone's on the same page. And we knock those things out every week in like 20 minutes. So, yeah. Oh, sorry, I forgot to repeat your question. Your question was, how do you define the priority list on that, the one with the dotted green line? Sorry, for the recording. You said your project took two years. Yeah. No. I think everyone's really optimistic when they start off. And it started off as kind of an idea of one engineer. He was like, I think that the YouTube site is getting really heavy, it's getting really old. I just want to refactor the code, so let me just create this on Polymer, which is our web stack now that we use, to test it out. And he built it within like two months by himself. The base framework of it. So originally, we thought it was going to be like a year or so. But then as we got closer, as we started building it out, we started realizing that our estimates were off, as any project in history does. However, what didn't change is our overall goals, like our metrics. So you know how we talked about those key metrics, your goal posts? Those stayed the same. And because those stayed the same, those were like our anchor. So even if we were slipping six months, three months, by our projections, we could show, we're still getting to these goals. And so we kept the project on track. Oh yeah. We spent one year experimenting with the new site in the wild. So a privilege of working at a large company is you have a lot of users. You don't need that large of a percentage base to get actionable insights and data. So we would launch like 0.5% experiments, 1% experiments consistently. So screenshots of our new site have been up around the internet for over a year, actually. So yeah, a lot of iteration. Sorry, keep forgetting to repeat the questions. Yeah, go ahead. Yeah, fair question. Okay, so the question was, we talked about gathering data in order to launch products or to achieve what you're trying to do. But if you're talking about a brand new product, how do you gather that data? So for me, I'm in a little bit of a different situation because of course I am running a large site that already exists and we're just bringing it to the next stage of that. So for the new product though, I think what you're trying to achieve is you're trying to first test if there's a market for what you're trying to achieve. And if there is no competitor in that space, which rarely happens, but if there is that possibility, then I would try and get data to suggest that that market opportunity exists. Even if it's as simple as just quickly going around doing coffee chats, gathering user research about, are people interested in this? Is there a niche for this in this specific market? And if you can prove that, that's already step one. You've got an idea that in the coffee shops of San Francisco, people have a niche of this. And then you can start expanding that. You can start seeing if that's global. You can start seeing if that's nationwide. So once you've defined that market, then you've already got your first step. And then I think you keep on trying to prove incremental steps like that. Like that for me is still data. Even if it's not hard numbers, even if there is a lot of bias and you have to take it with a grain of salt, that's still data. Okay, so the question was, comments on the size and composition of the company, especially transitioning from a small startup to a massive YouTube structure. Okay, and then the accountability for all the various roles within the team. So I could probably rant on forever. I'll try to keep it short. My startup, you feel like you, actually I'll take a step back. As a product manager, I think a lot of people have different definitions of what a PM does. For me, the PM is the person who oversees and helps manage all these experts, whether it's a design engineer or whatever, and bring them together so that you can make the best decisions possible for your product. So at a startup, of course, that's much more loosely defined. And so you have a lot of bringing together to do. So I was responsible for a lot of stuff there. The composition of the team was, I think we had a PM team of about four people that was reporting up to me, as well as an engineering team of around 10 people. And we relied on very simple agile methodologies. Like we had a whiteboard where if someone erased that whiteboard, we would have lost our product plan for the next like year. So it was very simple, it was very clean, and it constantly changed. That's why it was a whiteboard because we never could make up our ideas. I'm gonna make up our minds. Transitioning into a larger company, the most important thing I realized was, there's no hierarchy. And I think this depends on which company you're going into, but there's no clean hierarchy, at least in my case. At Microsoft, there was a bit more of a clean hierarchy. People reported to this person owning this section. At Google, you own what you really love. So if you're really passionate about it, you take ownership of it, which makes that really funny XKCD comic, which has all the lines crossing over for Google. That's our org structure. So the second part of your question, or actually to finish the first part, it was crazy moving into Google with this incredibly complex structure. At YouTube, we have what's called a matrix structure in which almost every major feature on any YouTube platform is owned by two PMs. So I'm wholly responsible for the desktop website, but there are YouTube comments PM, for example, who stretches across all the different platforms and they make decisions. So we partnered together to create a big feature for the comments on desktop, for example. So that became a lot more complicated figuring out how to navigate. Essentially, the politics became more complicated. The second part of your question, what are our roles? As the product manager for the desktop website, essentially my role is very simple. I create the plan for what we're building on desktop in the next three, six months, one year. And Google, I think, is a little bit special and allows me a lot of flexibility to do that. In Microsoft, for example, it was very top down. They defined what I should be doing and I was more executing within a smaller area, but I have more reign in terms of what I build next year. My TL, my tech lead, he's responsible for the engineering decisions over the next year and my design lead, we work together to define that roadmap. Yeah. Okay, so the question was, I described some moments where some products dropped off either due to team morale or it's stretching too long and what are the differences between those experiences at a small startup versus a large company? Is that right? Okay. So honestly, at my startup, I've been lucky in that a lot of our ideas have panned out so I can't really speak with a lot of experience to that but in my current role, and also I've also been lucky in my current role, but I've seen a lot more in my current role. The criteria for why we end up dropping those projects is honestly very simple. It's just cost benefit. It's, does this extra three months or extra six months that we now know require to lead this project from today, will that six months with this project eventually give us something that is worth it versus another project? And oftentimes the sad thing is that, you know those goals I posted, a lot of them are of course unrealistic sometimes and you get that sense of maybe it's not gonna increase my users by 10%, maybe unfortunately, it's only another one or two percent and that's gonna take six more months. That changes your story. So ignoring the past, ignoring what's behind you, just constantly looking in front of us and saying is the incremental cost that we need to put in worth the effort versus another project? And if that question is ever no, you're basically looking at potentially canceling that project. And I would imagine those same principles kind of apply to a startup scenario as well. Okay, I'm gonna try and remember both your questions. A bad memory, so remind me. So the first question, oh sorry, I'm gonna go backwards. The second question was, could you describe the goal setting process of my company? Let me try and quickly get that out the way. I probably can't, but I don't think it's a stretch to say it's roughly the same in terms of timeframe of what we try and do with every other large company. Like we have a quarterly system, we have a yearly system. And one thing I can say is that those aren't like very fixed for us. Those are more like guidelines of what we try and achieve. Google is a very bottom-up company. So if we just define like the rough directions and strategies we wanna do, then that's about what leadership tries to do. It's just roughly point us in the right direction. And then the PMs and the various teams try and figure out on a more granular basis. And the first question, sorry, could you repeat it one more time? Yeah, actually, so the question was, how do you weigh the short-term benefits from like A.B. testing, like the incremental wins versus the long-term, let's just say harder to measure things like your user retention, your user happiness and all that. And of course, as with anything Google, the answer is, do you worry about it? The answer is of course, we worry about everything. We have a lot of people worrying about a lot of stuff. But more specifically, you do touch on a pain point that I think the entire industry has, which is it's easier to measure one thing over the other. And oftentimes that leads us to incrementality versus creating products that people love because love is a hard thing to measure. So it's really, I think, one of the understated roles of being a product manager is you have to somehow champion that ladder bucket. And because it is harder to measure, but it is not impossible to measure, it is just harder to measure. So at Google or at YouTube specifically, we do have ways to try and roughly approximate some of those harder to quantify things. And that's one of the focuses we always have is how do we quantify that? How do we make it measurable so it's as easy as looking at a dashboard for that as it is for the incremental AV wins? In practice, what ends up happening is that we dedicate some time to try and do what we think is absolutely right that is good for retention users. We set aside a bucket because we know we can't measure it, but we know some of the things we have to do. And the others, of course, are towards either the strategic initiatives or incremental wins that we want to improve. So the question is, how do you sync roadmaps between my product and the other platforms so that we make sure we roll out the same product at the same time? It's a very good question. At Google, because of the crazy work structure, let's just say it's not the most efficient process that we've done. But at the same time, ultimately, because we are defining our roadmaps and plans at the same time, effectively, if we have an idea that will likely impact another product or that we think should impact another product, it's part of our culture that we just go and talk to that team. So for example, if the Android team wants to create this really cool feature, but they think that it has merit across all the different devices, they will come up to us and let us know that this is happening. And what's really nice about Google is that we have internal tools for our launch processes. Once something starts getting into experimentation and launch, it becomes fully known within the entire company. So what I mean by that is let's just say I have a feature I've been cooking up in secret, and then I want to start getting into launch approvals to get it out to the market. As soon as I do that, an email is sent out to everyone who's potentially involved. And that's a really nice thing, is transparency helps a lot. Like forced transparency within your company helps accelerate a lot of things. And that's one of the things I'm really grateful for. Basically, we started rebuilding. We decided to see the old YouTube.com was too slow or whatever the performance issues. I'm guessing technical debt. Can you talk about that process of making that decision and what the product involvement was, the product management involvement was, to get that buy-in? And I think that's like a common issue that a lot of software companies face. Yeah, that's a good question. So the question was, what was the process like in terms of product of getting buy-in for this project that essentially scrapping the old site and moving over to this new site on this new tech stack? So I mentioned that this project started as an engineering effort, as a one guy engineering effort. But actually, it became a lot more than that. Originally, the idea was, I'm just going to rewrite the code in the back end without people really noticing the difference up front. But we actually made a big strategic decision. And I think this is a lot where the product comes into play is the question was, how much do we invest in the future of the web platform? And I think a lot of people, if you work in tech, you are probably wondering the same thing. How much do I do mobile? Is it mobile only? Or is it mobile first? And I think those are the only two options unless you're also Google who says AI first. But if you're in that situation, how do you decide if you're going to invest in something that's so important for, that won't see gains immediately? And that's essentially what YouTube.com is today. It's actually not a finished product. So don't hit me up with too many bugs. The idea is that it's the foundation for our future strategy of building the best web platform. We're very biased. I'm very biased because I work on the website. But I believe in the power of the web and the future of it. I don't believe it's just going to be all purely native. So from that end, we made a strategic decision to invest in it. And to that end, we didn't just replace the code. We replaced the design as well. So we wanted to build a design framework that would scale towards the future as well. So in the next few months, you'll see a lot of interesting features coming out on the desktop website. And that was all enabled by the strategic decision.