 All right, product con. This is my first product con. Can I get a show of hands for who else is a first timer here? All right, I love this. So about a half and half. I'm really excited as was mentioned up here. There has been so much work that has been put into this conference, both from the organization standpoint and from every single speaker. So I'm extremely excited. I'm going to talk about de-risking big bets to drive impact. So we'll start with a little bit about me, but I'm not going to spend too much time here. My name is Michelle Parsons, and I am currently the executive in residence with product school, and most recently served as the chief product officer at Hinge, the dating app. But in my 15 years of my career and in product, I've worked at companies such as Netflix and Spotify and Kayak. But I think honestly, when it comes down to what I find the most passionate, I'm a product nerd by heart. I actually started my career as a high school science teacher. And so my time here at product school really brings everything full circle for me. I lost my slides. Awesome. So since I'm the first speaker of the day, I also have the obligation to get everybody amped up. I know it's early. I feed off of your energy. You feed off of mine. So help me work for you. All right, can I get that? All right, amazing. So I want to start with a question for you all. So can you raise your hand if you've ever had to come up with a product strategy from stretch and then create a plan to actually implement and execute against it? All right, so a lot of us, it makes sense. We're all product people, or most of us are. To me, great strategy is nothing without great execution. And so I want to talk to you all today about de-risking products and features that you build so that you can deliver more impact faster and more often. So let's start by orienting on what product strategy even is. And I'm not going to spend too much time here, but I do think it's important to level set. So product strategy, it's the process of defining what you want to achieve and how you plan to actually get there. And it's based on your company's vision, their mission goals, and ultimately your overarching product vision. All of this is basically informed by the data and understanding of who your users are and what they need, and then clearly define outcomes that you hope to achieve. So there are a few key steps in this process that I like to orient myself around. First, it's really about developing and creating that product strategy, leveraging data insights and input from your cross-functional partners. And then we go and get buy-in. That's the most important thing. We have to story tell. We have to actually get people excited about the things that we're going to do. But most importantly, we actually have to create a plan. We have to break that strategy down into its components and then understand how we're actually going to deliver against it and execute it. So for me, it's really in the implementation and execution piece where all the details really matter. And this is really where things can either go right or extremely wrong. Execution really is where we're harvesting all of the fruits of our labor, all the time that we spend doing the analysis, the user research, talking with all of our stakeholders. Ultimately, it's all the time that we put into creating and aligning around that well-informed product strategy. So I said earlier that great strategy means nothing without great execution. But great execution also means nothing without a really great product strategy. They go hand in hand. So let's fast forward. So now we have our product strategy. We have our roadmap and our list of ideas. And then we have the question, OK, well, where do we actually begin to effectively execute against this? So this is often time a place where I see teams struggle. Sometimes we work on too many things that are only delivering incremental impact. Sometimes we are underestimating the time or resources that it actually takes to build something. And other times, we're working on these really big, bold ideas. And we actually haven't validated or embedded them thoroughly enough. And we don't really actually know if they're going to produce impact. And so we can spend our time spinning our wheels a little bit too much. So I want to talk about a concept that I've used in my career a couple of times. And it's really helped me and my teams balance and create structure when choosing what and how to build. So this is called the Balance Portfolio Framework. And it's really used to help plan and execute effectively. So it helps you break down your roadmap, your plans, and your ideas into three key categories, quick hits, small bets, and big bets. So I want to walk through each of these really quickly just to orient ourselves. So quick hits. These are really focused in on your low-hanging fruits and learning opportunities. And I really want that to stick with you, learning opportunities. They help unlock learnings. And they typically should be really low-cost, low-cost, low-resources, low-effort, low-design. And what they're trying to do is unlock these really key insights that help feed into your other bets. Then we have our smaller bets. And this is really about taking the most out of your current investment. So the current features that you've launched, that you've already proven out, how do you enhance that, how do you continuously move the user experience forward and drive impact for your business? They're going to be a little bit lower in terms of cost, but they're still going to be costly. And they are going to produce a bit of impact. So nothing major, but again, they're those base hits. So if you're a baseball people here, I am not. But I oftentimes, when we go to big bets, this is the home run. This is the errand judge. Those are the home run hits. And what we're trying to do is get all these small base hit runs so that when we get to this moment of our big swing, we have a greater chance of success. So what are big bets? These are the things that actually push their product forward. These are super high impact. They're industry defining. They are things that actually lead the pack. But they require a ton of investment often. They come with a bunch of unknowns. And so what we're trying to do here is figure out how we can leverage this framework to actually de-risk your big bets, those ideas that are unvetted to actually ensure that there are more opportunities for you to reach success. So this is nothing we don't want to say at the bottom here. All right, so things that are easy to do are quick hits. And that's the end. What we really care about is clearly articulating the data and hypotheses that are driving the data and insights that are driving your hypotheses so that you can orient around the outcomes that you're trying to achieve. And then through this, you can start to pull back those big bets and identify your core insights that are driving your core hypotheses. Because oftentimes, we have a lot of hypotheses, right? There's a lot of data. But we need to figure out what are the key things that we want to actually focus on. So to create a balanced portfolio, I oftentimes use this 80-20 rule where you're placing enough small bets in pursuit of enhancing that user experience and harvesting the gains that you've already made from features you successfully shipped. And using those to learn towards your bigger goals. And then the rest of the investment is really balanced between your big bets and your quick hit learning tests to help you then de-risk those bigger investments. And again, I want to make it a really key point here. Quick hits should not be leveraged simply to skip the hard work of identifying key data and insights and just kind of building up your bank of like, hey, I shipped 15 features this year. That's not the point of them. The point of them is to actually learn. So you're trying to ladder these up into your big bets to ensure that you get one step closer to understanding and validating that unknown. All right, so let's talk about big bets and we'll spend the rest of the time here in the big bet space. Big bets are important because they unlock exponential business and user growth. This is really important for us. This is what we're all here to do. When we show up into our roles, we're not saying, I want to ship incremental impact. Who says that? Who says I want to ship amazing items and amazing features that actually impact the business and change our users' lives? All right, okay, I hope more people want to do that. But they can be scary because they can go wrong. And I oftentimes, even early in my own career, have experienced hesitation. Wow, that's a really amazing idea. Wow, that vision is awesome. I really think that this is gonna be an impactful thing. And then I get scared. Like, oh, that's gonna take a lot of resources. It's gonna take a lot of time. My trade-offs, if I do this, I can't do this. This thing has not been vetted. I have no understanding of this. And so oftentimes we kind of stick in a small bets lane and we're walking there and we're making some bank shots and we're good to go, getting the 2% gains time after time. And at the end of the quarter, at the end of the year, you've made some movement, but it hasn't really 10xed your business. When things go right, though, you build credibility. You gain trust with executives and your stakeholders. You start to deliver significant impacts that actually changes the business. And you are driving industry-wide change. You are becoming the market leader. And all of this together is where we want to be. And I wanna show you kind of how we put this framework into action because I'm not a huge fan of just theoretical talks. I wanna actually show you leveraging a case study from some work that I did earlier on in my career. So during my time at Netflix, and a time before every media company had their own streaming service, this was kind of like in the days right before Disney Plus Launch and Peacock and HBO Max and all the competition really revved up. We knew that was coming at Netflix. And I joined the team to lead our kids and family group. And when I got to the company, there had not been a product leader in place for over a year and there was not any clear kids and family strategy. The team had been executing events to a couple of different features and stakeholder requests. And so when I started, I said, wow, we're about to be up against some pretty gnarly competition. And in fact, the kids and family space is one of those really interesting spaces because it's basically based off of a few really big IP. So think of the Frozen's, the Moana's, the Coco Melons of the world if you have kids. Kids know and love these pieces of content but they're owned by other media companies. And at the time, Netflix was licensing all of this content. So we were about to lose everything and almost all at once. And so for us, that was a pretty big deal. So we had to go and invest a ton of resources and money on the content side to basically build up a brand new catalog and then figure out how to get kids to actually want to watch this and not Moana or Frozen with zero marketing budgets. Pretty hard. So I dove in, I consumed everything I could about our users, the data and insights and then I got to work with my team. And we started that very first part, our product vision based off of our company values and goals. And we said, we want to become the number one trusted service that empowers kids to effortlessly engage with their favorites and delightfully discover new ones. And this was really the ethos through which then we began the rest of our work. Our goal was to help kids find their favorites and engage with new ones in a delightful way in a less intimidating way. And we need to be trusted by parents because our users weren't just kids. They were kids of all ages and of all abilities and their caregivers. So we got to work and we broke this down into three key areas. First, we must build trust with parents by increasing their understanding, control and meaning. Second, we want to connect kids with the content that they know and love. And third, we want to facilitate discovery that is grounded in the familiar, in the known, to make it less intimidating. And I'm a really big fan of visuals. I think this is really important as you're elevating in your career to be able to use different mechanisms and methods of communication with your stakeholders. So we created this really amazing castle metaphor. I'm a big fan of metaphors. It has a foundation, it has your users and it has your key goals here. And this really illuminates who our users are, what our goals are, what their goals are and how we're actually gonna go about accomplishing that. All right, so now we wanna make a pretty big shift. If any of y'all watch Netflix, oftentimes we come to Netflix under the guise of, I wanna watch something. Maybe you are watching a show, but oftentimes like let's get together on board, let's put something on. I heard about these new movies, let's go. Kids are very different. So we needed actually a very big shift in the way that we are gonna approach this part of the business compared to the rest of Netflix, which is really hard, it's a very big company. So things are pretty built, right? The foundations are set. So trying to go about changing something, whether it be UI or algorithms, is really, really costly. So kids start out with this really key insight. This is what we gleaned, what kind of led the rest of it, is that when they come to Netflix, they come to Netflix saying, I want Pikachu, I want Cocoa Mellon, I want Moana. And from there, we realized that we had to approach this problem much differently. Rather than discover being the first stage, we needed it to be the last stage. And so we need to connect kids with their favorites, deeply immerse and engage them, and then finally use all that insight to connect them with things that they would love. And then rinse and repeat. We create a nice little flywheeler loop right here. And so we keep our user in our ecosystem. All right, so now I'm coming to my team, I'm coming to my executives, and I said, we have the perfect strategy. We're ready to execute. Look at this data. Look at the beautiful visuals that my designers and our researchers came up with. And they said, this all makes a ton of sense. Great, all right. So we said, step one, we want to connect kids with the stuff that they love. And this is gonna be the kind of foundation for how we move forward. And basically we got to brainstorming. We came up with a lot of ideas versus character universe where kids can interact and engage with their favorites, all the content that exists around them. We wanted to have kids land right when they opened up Netflix, their favorite characters, visualized by their character images and not boring box art. And we wanted to give kids a way to quickly input and select their favorites because it's oftentimes really hard for us to know what a favorite actually is. So we had all these questions in our mind. How do we learn what a favorite is? How many favorites do you have? And the team asked all these questions. We got to a point where we're like, okay, we're kind of ready to start. We kind of get a, we have a good understanding of our data of where we're gonna start first. So now another question to you. Based off of our original goal of connecting kids with our favorites as effortlessly and delightfully as possible, which one do you think that we started with or proposed to start with? So ever who thinks one character universe, raise your hand. Not a soul? All right, one, thank you, all right, I love it. How about two? All right, some more folks, how about three? Okay, see, as a former teacher, I know the good students in the classroom right here. All right. All right, so let's go. It's kind of a trick question. Ultimately, we wanted to do all of the things because these were all part of our ultimate strategy. But we started with the favorites on the homepage. And this was really important for us because it solves some of those immediate problems that we had identified, which was when you come to Netflix as a kid, you have no idea where the stuff you wanna watch actually exists. And so you have to go find it, it's really high friction. So in order to get here, we needed a brand new kid's algorithm. One of the key constructs on Netflix is that they actually penalize anything that's been seen before and they pull it out of your homepage. Completely gone, poof. For a kid, not so good, right? So we needed a brand new kid's algorithm. That was gonna take two months at the minimum to build. All right, we needed new kid's character art. One of the things that we found was that kids actually don't understand things like tone and box art and theme, right? They don't get it. They're like, the dark background does not mean eerie or scary to them. It just means I have no idea. So they actually resonate with surprise characters. They're familiar, they wanna be them. They dress as them as Halloween. We didn't have any of this content. We would have to create it from scratch. That was gonna take another several months to build. Then we needed to overhaul the entire homepage. One of the problems here is that the Netflix app uses kind of the same UI. So if you click into your normal adult profile, it's gonna be the same as the kid's profile. So we were actually gonna have to invest a lot and actually changing and kind of bifurcating the experiences. And there's risk with that bifurcation because it adds things like technical debt, for example. And you have to have multiple design considerations, et cetera. And then lastly, one of the things that we also uncovered in some of our early research was that we needed a new click-to-play experience. So right now on Netflix, you click, play, and what do you get to? The details page. Well, that's a next click for kids and it's really hard for them when you're thinking about how they navigate their ability to actually use remotes, et cetera. So it's a lot of friction that we entered about the entire part of this process. All right, so I brought this idea to my team, to our executive stakeholders, and they're like, there's no way. This sounds like a really big project. And actually, we don't even believe that this might work. Like, how do you know it's gonna actually work? We're like, no, no, no, here's the data. We have the data, but we had no proof. We had some research, sure. All right, so gonna take a long time to build a lot of resources. We didn't have really the knowns around the impact that it was gonna produce, though we had hypotheses. And the other key thing was that we actually got a lot of pushback from our content executive stakeholders because their main goal was what? Create all this brand new content that had never been seen before and get kids to watch it. So it was actually an exact tension with what their goals were. So I had to prove that executing the Connect Engaged Discover Plan was actually gonna get them to their desired outcomes faster. So, all right, what did we do? I said, I'm not ready to stop here. I went back to the balance portfolio approach to my big bets. I had a big bet. I had a lot of core hypotheses and data points. And I said, how can I test my core hypothesis really quickly? My core hypothesis was that connecting kids with the stuff they love faster would lead to increased streaming, greater days with a qualified play, and ultimately more retention, which leads to more revenue. All right, I wanted to create value with my stakeholders and I wanted to advocate for resources and investment. So I said, all right, what do I do? What do we do? So I went to my user researcher, my data scientist. I was like, hey, we already have this data point that 57% of the hours that are watched in kids profiles are coming from rewatch compared to 20% in adult profiles. That's huge. That is a huge number. More than half of the hours that are consumed and by 100% of our users are coming from rewatch. Really intense. So I said, okay, I'm not gonna get the buy-in right now to do this on the homepage. Where is this currently happening? Because they're doing it. All right, so I went in. It turns out that 37% of those hours were coming from the search tab. The next 26% were coming from the continue watching row on the homepage which floats up and down the page. And the last 20% from the watch it again row, it's also on the homepage which floats up and down the page and it's not always visible on screen. Then we went one layer deeper understanding that we again have two types of users here. We have parents and kids. And I said, who's actually navigating to select this content? Well, it actually changes as a kid ages, but one of the things that we're doing here is we're putting a lot of friction and pain point on a parent who might be rushing to get something done, put on something for their kid because they're crying or because they need to go cook dinner. And what do we do? Well, we introduce so much friction for them that it probably was easier to put something else on. So they bounce out of Netflix. All right, so very interesting data points. I asked myself, what do we have currently available in our toolkit? All right, well, we have an under optimized search UI. When you click into search, it's filled before you actually start to type. The cursor is on your A and then the canvas is filled with an algorithm that's by popular searches. Kids don't want discovery, but yet there it is. We also have a watch it again algorithm that's being leveraged by kids already by 20% of the hours on the homepage that it's unreliable to find. So I said, okay, aha, this is an interesting opportunity for us to test our core hypothesis quickly without any extra investment or work. So our hypothesis was that merchandising rewatch titles in the search pre-query canvas, that canvas to the side, would actually help connect users with a content they knew and loved faster and basically get them into a play mode much more rapidly. All right, so I'm gonna walk you through this quick hit test that took, again, one week to build, and that is build design QA. With the goal and intent for us to learn whether or not our big bed idea had any legs. So if you look at the control, cursor on the A, filled by popular searches. Now let's walk through and deconstruct the test variant. A couple of things we tested here because beyond our core hypothesis, we actually had a couple other hypotheses. Remember the character art. Remember the number of favorites. So we swapped out the algorithm and replaced it with watch it again, change number one. Now you're seeing content that you've already seen before there. The next thing, friction to access. We moved the cursor from A and started it on the first box art. Wow, because if it takes a millisecond to pick your title, the likelihood that you actually get there versus clicking over seven times, you might get into a play moment faster. We also moved to vertical box art versus horizontal box art because they actually, they presented characters in more full screen than the horizontal box art. So it made it a lot easier for individuals or kids to recognize their favorite character. Because you think about it, not every single parent knows what the name of the show is called, but they do know the name of the character. And so all of these things together helped us really quickly learn if whether or not our hypotheses were right without actually having to invest significant time or energy or create anything new, which is the really important thing. So not a shocker. This was actually extremely successful. We got three X increase in watch data and watch streaming, total streaming hours and qualified play days. This far down the user journey, several clicks in. As a result, we were able to get buy-in for our investment and really drum of excitement because not only did this actually impact things like streaming and days of qualified play, which are our leading metrics, we actually were able to see retention impacts. And so as a result of a highly optimized platform like Netflix, that's a very hard thing to actually move. And so we were able to then take our quick head idea and advocate and quickly test out our hypotheses for things like favorites, for things like the algorithm, for things like character-centric or character-first UI presentation. And this had a tremendous impact on our kids UI and our overarching Netflix goals. But we didn't stop here. And I think this is the goal. This is why I want to really drum home for you. Every single test is a learning opportunity and it folds back into your original strategy. And that's why it's so important to come with a firm and well-informed strategy. So discover, remember our stakeholders? They were really upset and scared. Oh my goodness, this is gonna impact discovery dramatically. And in fact, that previous test actually did. We saw a 30% decline in new discovered content. Huge, it was really, honestly, that was a conversation to be had. But I said, don't worry, not to worry, we have a plan for that. We actually are gonna now take this canvas because remember our original strategy? We wanna help facilitate discovery that's grounded in the familiar. And so what we did was we introduced this new concept called unboxing, where we actually took our favorite algorithm, we swapped out the fourth item in the row, and we delightfully uncovered your next favorite. This was so impactful for us to test into our next discovery hypothesis, leveraging an algorithm we already had, popular. So I wanna leave you to keep in mind this balanced portfolio approach to creating your execution plans. And six key takeaways. First, please develop an informed and clear strategy. Dive into your research and data. Understand what is currently available to you today. Test your core hypotheses quickly. Leverage those learnings to de-risk your big bets and gain buy-in. And then never stop. Continuously learn and apply those to your future decisions. Thank you very much. Thank you.