 Thank you. Welcome. My name is Arvind Tripathi and I'm an associate professor here in business school and I would like to welcome you all and I see that we have a lot of people from faculty of science and engineering and from business school and thank you for coming to this wonderful event. Before we start, just a few health and safety announcement. So exits are right in the back. In case of emergency, get out of there and get to the vineyard straight and I'm sure the building is very well designed so you can find bathrooms would not be a problem. So today the talk is about product design and we have wonderful speaker Andrew Law from Netflix and so I know many of you probably use mobile devices for streaming videos and all kinds of content and Andrew is the one who is behind all that. So in case your device or streaming is not working, you know who to call. So I would just invite Andrew now. Thank you. Thank you. Awesome. So just a little bit of context setting. I'm going to do a presentation that's roughly 25-30 minutes and then we want to open it up for questions. So if you have those burning questions about Netflix, how we work, how we develop our product, please get them ready and I'll be happy to answer as many of them as possible towards the end. Before we get started I want to show we are a streaming company so I'm going to play a video from YouTube. It's when we made. Hopefully the audio works and it's not too loud. So that's a video that we made for a feature that we created called smart downloads which launched not too long ago and that's the type of innovation work that my team is responsible for specifically on mobile as well as the website. So those are my two kind of areas of responsibility. So I gave this talk earlier today. I've been talking for like five hours but I'm really excited to be able to share some of the stuff with you guys and hopefully have a conversation about it. I got into design about 14 years ago. I started doing very very small projects just kind of for fun. So I did things like posters for friends bands and album covers and t-shirts and and fun things like that before I ever knew that this was going to be my career. I was sitting in seats very similar to this back at San Jose State which is where I went to school and I went to business school and I have a degree in business marketing and administration but I use very little of my degree. Don't worry you guys will. But a lot of kind of what got me into design was really wanting to understand why people make the decisions that they make and how I could influence those decisions through design. So I'm going to be talking a little bit about failure today. Specifically failure at Netflix and how that leads us to building better global products. And there's three different areas that I'm going to talk about. I'm going to talk about A B testing what that is and how it works and why we employ it. Research the different types of research that we cover. And I want to talk about empathy. Empathy is extremely important because I am from the Silicon Valley. A lot of my design team is from North America but we're building a global product for 190 countries for over 130 million members in 22 different languages. But we are not representative of the people that we're actually building our products for anymore. So how do we build empathy along the way so that we build the best products possible. Sound good? All right. This was supposed to be moving. So this last 20 seconds. So we'll just wait. We do lots of shows. How many people here use Netflix? Okay that's what I like to see. How many people here pay for Netflix? Okay so if we could just document the people that need to pay I can accept all forms of credit. That's great. So I want to talk a little bit about product design and that's kind of my field. That's where my expertise are. So the product designer's job at Netflix is to find if there's a there there. And what I mean by that is find if there's an opportunity that's going to grow the business. And that's really my team's job. So we don't just push pixels. We don't just create pretty things that are usable. We identify different areas of opportunity inside the existing business and potentially outside the existing business. And then we build, prototype, test, and do qualitative and quantitative research to better understand if that's something that we should actually roll out to our members. Smart downloads being an example of that. So let's say that we're all product designers at Netflix. Congratulations on your new job. This is a bigger team than I've ever had before. This is great. So how do you take an idea and actually validate it? So how do you take something and understand whether it's something you should pursue or something that you should kind of let fall by the way side? I kind of use this rough structure as a framework. So the first thing you want to do is try to identify an opportunity. I'm going to use smart downloads because that's a video that I showed earlier. So we already had a download feature in the app. It worked. But there was a highly manual process. You'd go in, you'd find something, you'd click download, you'd wait for it to download, and then you'd hopefully consume it later. So we thought that there was an opportunity for this when we started looking at the data of how many people are using downloads and how they're actually using it. So we had an idea and it was this idea of smart downloads. We didn't really know what it was going to be at the time. But we asked ourselves, does that align to our business objectives? And we want more people to be able to use our product in situations where they have little or no connectivity. So we said, yeah, like that actually aligns with our business goals. Does it mean a user need? Absolutely. Through a lot of the research that we did, we discovered that people oftentimes are not using the North American use case, which is what we think of where, hey, maybe I'll use downloads if I go on a flight or I'm in a subway or something like that. We studied people that are in Mumbai and in Singapore that have connectivity sometimes, but it's really spotty at best. And sometimes people are spending three hours in a car commuting to and from work. So they absolutely there was a user need. How large is that need? So it can be a need, it can be a valid thing. But if only a subset of your members are using that or would use it, is it actually meaningful for you? Probably not, it's probably not something you actually want to pursue. And then the opportunity costs. We do not have finite resources. We have a small number of people from the design side. We have a smaller engineering team. So we have to re-prioritize. So is it worth the opportunity costs? Because if you can do this, you can't do this oftentimes. And if it aligns, then you should absolutely do it and pursue it. So once you have the idea, how do you actually take that and build it? How do you take it from zero to one? The skill is not incoming up with the ideas. The skill is executing the ideas. Everybody in here has ideas. Some of you will pursue them. Some of you will throw caution to the wind and absolutely go for them. And others are just going to have good ideas that never get out the door. And everything is in execution. So we use this framework to basically, and it's very idealistic. It doesn't always happen exactly in this way. So take it with a little bit with a grain of salt and all kind of caveat when sometimes we spend backwards and come back to ideas. But we try to start with what we know. So with the data that we had, we realized for smart downloads that there was this behavior where people would watch a few episodes, usually about three, or sorry, download about three episodes. They would then consume them often when they were not connected to Wi-Fi. And then they would redo that same thing when they were connected to Wi-Fi again. So there's an opportunity there. Any time you have a repetitive process, there's an opportunity for innovation to take some of the weight off of the user and for us to do that work for them. And that's what smart downloads is. So there was foundational research that was done. So you have to start out with what you need, but you also have to ask yourself what other information do I lack? So where are the gaps in my hypothesis? Where are the gaps in my thinking? And how can I fill those? And again, ask yourself if there's a there-there. The next step is to ideate. Ideation for us looks like kind of very, very broad. We start in a bunch of different directions because we don't know exactly where we want to go. And we want to explore and not leave any stone unturned. So we will kind of take a very kind of wide, I guess, brushstroke approach at the beginning of something and then narrow that down through different qualitative research methods. The next step is to prototype and learn. It's extremely important that if you're going to show something from a personalized service like Netflix that it's as accurate as possible. So we have prototypes where we can actually sign in with your unique IDs and show you content that is familiar to you, including the last things that you've watched. But it's all in kind of a prototype, so it's all in a build that no one else has access to. But through that process, we'll try to get the most realistic understanding of how impactful that feature or idea is to you, the member. Oftentimes, what we'll see in qualitative research, we actually have to turn into actionable insights. So some of those things, when we synthesize the learnings, we'll come back and say, actually, you know what? When we explore something like smart downloads, we should really be mindful of conserving data. So we should maybe be conservative and do this as something that's only while you're connected to Wi-Fi, which is exactly what we did, and that's a learning that came from that process, from that project as well. At Netflix, we do A-B testing and a lot of people understand A-B testing as kind of if you take a user and you drop them on website A or website B, which one performed better. That is a very basic approach of A-B testing. We do multivariate testing, which means that we'll have 12, upwards of 12 to 20 different cells to identify each individual variable. So not only will I be able to understand why website B worked best, but exactly what nuance of that design outperformed the other and then why. So I'll talk a little bit about more about A-B testing in a couple slides. And then again, once you actually have something that you've learned is meaningful for our members, it's something that we will productize. I was doing a talk earlier today to project managers and product managers group, and this is where there's a little bit of friction between design and PMs. Design wants to do what's right for the user and usually PMs want to do what's right for the business, and so there oftentimes are concessions that are made around when you're going to actually productize something and ship it to your members. And then if that all works, you do it again. So to have a culture of testing, you have to support it from every single level. Reed Hastings, our CEO, started the company back in 1997 shipping DVDs in the U.S. mail. Everyone's pretty young in here. So a DVD, I'm just kidding. So that's how the company, that's how the company started. And he was really interested in understanding where that could go. So could that scale, could that go globally? They really kind of encouraged everyone in the company to explore ways that the company could continue to grow and scale. And so the level of that culture is really embedded in who we are as as Netflixers. Anybody can come up with an idea and inside of our freedom of responsibility culture, you have the opportunity to go explore if that's something that you think is actually meaningful. So I've done projects, again I have a business degree, but I've done projects that had no design, no pixels at all, and tried to understand if that was material for us as a business. But inside of that, at Netflix there are no sacred cows and every now and then you need to make a hamburger. We didn't get to where we're at today because we thought DVDs by mail were the right way to go. We got to where we're at today because we thought it was the wrong way to go and we challenged our own assumptions and so we just we disrupted initially brick-and-mortar blockbuster stores where you'd go rent a DVD and then we created streaming and made that a thing and expanded that out to North America and then we licensed more and more content to make sure that we had things that everyone in here would like to watch and then when that got really expensive we started investing in our own content and creating Netflix originals. And so we continued to disrupt ourselves and the next kind of phase of what that looks like is hopefully disrupting storytelling. So the reason that anybody watches a 30-minute show and you really only get 22 minutes of that is because the advertisers had an eight-minute ad block. We will never have ads. I'll say that right here now. You'll never see an ad on Netflix and because we don't have that we can really support the stories that people want to tell in any fashion and format that they want to tell it. So if we want to tell something that's culturally sensitive we can tell that because again we're not appealing to advertisers. We're appealing to you, our members. And if you like it then you'll tell us. You'll watch it and if not then we'll probably end up making less of that type of content in the future. So AB testing. Again this is we test everything that is in the product today so everyone in here is probably in 20 or so AB tests right now and you just don't know it. So your Netflix experience is vastly different from your Netflix experience and hopefully you guys don't compare notes. But why do we do it? So it really helps us rule out those bad ideas and focus on larger more risky product bets. The earlier you can get a read on something whether you should actually pursue it or not the better because we're talking about time we're talking about money and we're talking about resources and we want to conserve whenever possible. We apply the scientific approach. This is why we do multivariate testing because we want to isolate each of the individual variables to understand what worked and why or what didn't work and why. Because ultimately a failure is not a failure as long as you learn something. And we test large ideas we don't test like the color of a button so we would never have a test that is really kind of like nuanced into like typography or the granularity of you know iconography even. We'll get a lot of the sense of that just because of the fact that all my designers have been designing for so long that we understand what people are going to be able to gravitate to what patterns people are going to understand and be able to adapt to and learn. And one of the really important things with AB testing is you want to understand what success looks like before you start. So it's really easy when you run AB test and you get a green result or if you get a a win as we call it to just like not ask any other questions and just ship that out. But it's really difficult when something doesn't work if you haven't asked yourself why you might think it won't work before you go into it. So make sure you understand what success is before you run the tests. And failure is absolutely only the beginning. If I had a scoreboard behind me tallying my wins and my losses I would have far more in the loss column than in the win column. But all of those things all of those failures helped me to learn and helped me to get better and produce those wins. So I still look at them as failures but I was able to glean some kind of insight out of that which is why we're able to drive the business forward. So as I mentioned before we're in 190 countries 22 languages with 130 million members. We are not representative of the people that we're designing for anymore. So we really can't start with this as our basis anymore. This worked for a while when most of the netflix business was inside the united states. It's no longer. We're a fully international company. So we really have to be mindful of not just this and not just these which is where a lot of companies will tend to focus and spend their their dollars. And this is where we focus originally as well. But you have to expand into latin and understand the cultural nuances and family structures. You have to understand outside of that as well the rest of the world and understanding multi-generational households in India and how one piece of content to one family member will not be accepted by another. And so when we do when we go and do research oftentimes we will spend time in people's homes. We'll spend entire days with people shadowing them around and just seeing how they use their phone how they use their the website how they use the internet for entertainment to try to understand and identify opportunities that again will grow the business. This is by far the thing that keeps me up at night the most. It's a huge challenge. We have last year we spent six billion dollars on content. This year we're spending more. We're just about the largest studio in the world. And with all of that stuff coming to the service hopefully there's something that's great for everybody in this room. But how do you deliver the right thing at the right time to the right person in the right context? That's a huge challenge and that causes me many sleepless nights. So like I was saying some of the research that we do we're really trying to build empathy through that research. That's why we try to get on the ground as much as possible and meet folks and talk to them what works well for them what doesn't why. We try to find out hacky ways that people are using the service that we never would have thought of before. We have a share button in the app but a lot of people will take a screenshot and like SMS that to a friend. So we wanted to understand why. Why are they doing that? Why aren't they using the afford and sits right there and how can we support that behavior if we feel like it's something that we should model. So empathy is a muscle just like any other it can be built stronger and shaped over time. I have learned a lot a lot from my team in the last couple years about empathy and I've seen a lot of and heard a lot of great stories that have helped me to kind of bond and that's really what it is. We do a lot of travel we do a lot of research with participants and we do a lot of person on the street interviews to again clean some sort of insight into somebody else's perspective and life and that helps us to be better product designers. We're flying through this by the way. We'll have plenty of time for questions. So really understanding someone's frame of reference can lead to meaningful innovation. This would have been kind of scoffed at even a few years ago if you just put this up on a screen. But we can prove it through the the success that we've had in the product by understanding and cleaning insights from someone's life and then mirroring that back in the product to to create an experience that is meaningful. When you empathize you tap into emotions and you bond over those shared feelings and experiences. As a little kid in North America we always do like summer camp so we'll go out into the woods with other kids and we have tents and campfires and all those things. And what was great about those is the bonding experience that you'd have with those other kids. You'd have a shared experience and you'd often come back to your siblings you come back to your family and they wouldn't be as excited as you were. You didn't really know why. It's because you were building empathy with those kids. You were building an understanding. You had that shared emotional connection and emotional bond. I would like to replicate that as much as possible with as many people as I can so that I create a diverse perspective of understanding and then I can actually use that in material ways for the business. Everyone in here has biases. There's some amazing tests that you can do to to identify your unconscious bias. But you really need to understand these. If you're a designer this will help you immensely. If you're not a designer if you're just trying to be a good human this will also help you. We all have our biases. I was really surprised to learn what some of mine are. I won't share them with you. But the first thing is really awareness. You can do things to kind of curb them over time. And this is really important. You don't do empathy, you have empathy. It is something that you have to make a conscious effort about. You have to employ it on a daily basis. And again being aware, being conscious of it is going to help you. And you're absolutely going to going to mess this up. That's okay. I think that the more people you are able to meet and the more people you're able to connect with is going to eventually lead to you being a more well-rounded human. And again if you're if your goal is to create a business or build products for people this is a skill that you really should be developing now. Okay, research types. This is I have two slides left. So noise is extremely common in research and you have to know that going in because you want to take things with a grain of salt. People are terrible at predicting their own behavior. If I asked if I pointed to one of you and asked what would you do you know tonight after after this is done you're going to tell me something and at most at best it's going to be aspirational. Because I'm asking you in a room like this you might say something like you're going to go home and watch a documentary. When really you're going to go watch cupcake wars. So we're really bad at predicting our own behavior and more even worse at socializing what that is. So you have to understand that when you go into a research scenario that you have you have to be looking for kind of more the themes than the actual words that people are saying. So I mentioned this a little bit. The the research that we tend to employ especially at the beginning of a project is more ethnographic in nature. This is great to get a foundational basis of understanding. We did this pretty recently in Japan before that we did it in Singapore because we were seeing patterns of behavior that were kind of anomalies like it just didn't make sense. The data wasn't telling us the complete story and so we traveled there and met with folks and talked to them and spent time in their homes with their families to better understand how they were using the service. The first one can be quite expensive. The second one is you can do this on a shoestring budget. I still do this today even as a designer in a corporation. I think that this is a really really meaningful and powerful way of connecting with people and we just call them personal on the street interviews. This is you see someone on a phone you walk up to them you introduce yourself you say that you're a student that's what I always say. They still believe me like I'm going back to school or something. I say that I'm a student and I'm doing a research project and I would just like to understand how they're using their phone for entertainment and then you just have a conversation with that person and and you don't have any objectives other than to just learn and listen to them. We often give them like a $5 startling gift card or something as well. Video diaries so if you can't travel you can do different types of recruiting in different countries where you ask them to do a certain kind of step by step process. So we might send them phones or we might have them download an early build of our app and then they'll use it for a few weeks so that it's not just this lab lab scenario and that's the last one but it's not so it's a lab like an artificial home scenario. We want people to be able to use it in their daily lives and see if it's a behavior that becomes repetitive for them. And the last is lab sessions. We typically use these early on in the process and at the end of the process and a real quick note if anybody who's taking pictures like go for it totally it's totally fine I will post this whole deck to my website so that you can get everything you get the speaker notes and everything else so just know that. And we employ lab sessions at two different times early on in the process when we're when we're trying to get a good gut check of a direction of taking something and then later on in the process when we want to do a usability study or a usability qual it's what we call them. So really understand we want to see that if we create a new feature it's usable but our our user base is not is not you know 18 to 24 males. Our user base is three year olds to 93 year olds and so we have to create things that are going to work well not only for you know people in Palo Alto that are tech savvy or people here that are tech savvy but it's got to work for again that multi-generational household in Mumbai that I mentioned before. So I know all of you are students you've probably had a long day so we'll end it there. I do offer if anybody has questions if you are looking how to launch your career if you want to get into design I do offer mentorship it's free it's unlimited it might take a little bit of time for me to get back to you give them the time difference but if you ever have any questions or anything like that my emails up there my my Twitter's up there feel free to to reach out. People poured a lot of time and attention into me and my career getting getting to the place that I am today and I want to give back as much as I can. So any questions that you guys have I'm happy to field them here if it's something you don't want to ask in front of a crowd feel free to email me or DM me on Twitter and I will happy happily get back to you. Thank you very much. We have a few minutes for questions and we'll take questions from anyone. When you do do multivariate testing what kind of minimum sample size are you looking at? So that's that's a great question so if anybody didn't hear when we do multivariate testing what's the sample size that we're looking at. We typically allocate a few million members to each cell which again is not it's not feasible for everybody to do that but we're at relatively small scale compared to something like an Amazon where they can turn on an A-B test and turn it off five minutes later and and have you know millions of impressions of a page or or something like that. So we want to and it's also because of our our structure that we offer like a free 30 day trial so we need to make sure that someone comes to the service and then retains as a member past that first pay period to determine whether that's a successful product feature or idea or not. We want to make sure that they stay paying members. Yeah thanks for the question. Question there? Oh sorry. Hello I'm here. So how do you identify your biases? So there's a lot of tests that you can take if you just go online and search unconscious bias. I think Harvard has a few different tests. They're really easy it it's basically time and reaction based so they might show you a picture and then ask you a question and it's all about like how quickly you can answer these questions to try to uncover some of these unconscious biases that you have. But just do a google search you'll find there's tons of tests. So how do you find good product designers? How do you hire or call them easily? That's a great question. So my job is basically building the best team that I can giving them the right context the right resources and then getting the heck out of the way and letting them do their job. So I spend a lot of time looking for diversity of thought. I know from research has shown that the more diverse team the more likely you are to succeed. So I try to get people with diverse perspectives. It's very hard in Trump's America to hire people from outside of the U.S. but it is something that I have I currently have a centralized design team where everybody kind of sits in one location. I'm often kind of exploring the idea of hiring people in our India office hiring people from Amsterdam and our Amsterdam office and having a decentralized team to get that diversity. But a great question. Hi I'm just curious what do you test in your A.B. testing or the multivariate testing? So you mentioned that you don't of course test the color of the button or whatever but what do you test? Is it usually the type of content that they prepare to watch? No that's a great question. So for the feature that I showed called smart downloads a test cell might look like the number of episodes that we download automatically for you. So it's basically and we actually tested this. We tested whether we should automatically download like three episodes for that viewer or another test cell was we'll just we'll download as many episodes as you've consumed and then another test cell might be we'll download as many episodes as you started to download so if you downloaded one we'll delete one and swap that back in. If you downloaded three and you watch three then we'll delete three. So it's just these like different larger ideas that we're trying to understand which of those aspects of the feature are successful. Does that answer your question? It's it's not a it's not a really easy concept I wish that there was like a you know straight way of talking about it. It is and in that test we had 12 different cells that were doing those things so I'm not a professional at that stuff. I have amazing science and algorithm folks that that help us to kind of go through and identify every kind of nuance that we want to test. Isolate those variables across the cells and then we might actually product size something that has a little bit of cell eight, a little bit of cell five, and a little bit of cell two. So one is always control which is your current experience. Yeah you mentioned the challenge of becoming global a global company. One of the things of course with content is that some content in some countries will be offensive or not wanted and in other countries to be fine. So how do you manage that challenge like do you prevent some countries from watching certain things or what do you do there? So the only way that we prevent people from watching things are would be because of a licensing deal. So the the catalog is not the same in every region around the world usually because of that. A Netflix original should be available just about anywhere in the world except for some of our first few originals that were hard for us to license. But no we don't we don't gate or filter or sensor or anything like that. We want to provide as much information in the UI to allow that individual to make a good decision for themselves. So we don't want to bait and switch and tell you that it's hey this isn't graphic when it really is. We want to provide as much that information up upfront and we actually work with government agencies to kind of standardize rating systems so that we can actually tell different governments hey this is this is how you should be thinking about this type of content and then they'll kind of use that in country to figure out if that's actually right and then we'll build trust with them over time so that we're we're trying to pick up those cultural sensitivities as well. It's not a perfect system it's getting better. Hi, in your experience with product design what technical skills and soft skills do you think are important for like this sort of role? For my role? No skills at all. For your role and like the role in your team? So hard skills and soft skills. So the hard skills look like the software programs that you use. So Photoshop, Illustrator, Sketch, some sort of prototyping tool like Framer which uses coffee script or Envision which if you go on Envision's website you'll see me all over it. Any those those are the hard skills that you need. The soft skills that I find that are really really valuable are sales skills, marketing skills because it's not the best design that gets built it's the one that was socialized the best and sold the best and I don't mean sold to like other people in a in a you know used car sales way. I mean you're positioning right you're trying to position why this is impactful why this is meaningful and you want to create a case around that. So I find that people that are have had some kind of sales background even if they were a kid or something like that and you know sold cookies by the side of the road it's it's beneficial and it plays out and it kind of pays dividends as well. Great question. So how do you choose which countries have other offices because it's an international organization so yeah so we we basically created regional offices to handle kind of content in a specific region so we have offices in London and Amsterdam that handle a lot of Europe we have London's we have offices in Brazil that handles a lot of Latin Mexico City so it's it's more regional it's not about like hey this country is really important let's make sure we have an office there it's just more general than that but there are a lot of it is content plays as well we want to make sure that there are people that are representative of those markets that are buying content for those regions as well back there. So when you're designing your product do you design in sequence in terms of if you're designing for the desktop and the mobile application or do you do it simultaneously what is your process around the platform? Yeah that's a great question so we are not really rigid around having a unified product experience on all the platforms at the same time and actually because we're a very iterative culture and because we're always doing different tests oftentimes my iOS app and my Android app want to actually be the same and that's okay like we just that's something that we're just because we want to iterate because we want to kind of do these things really quickly that's just something that we have to kind of own so there are features where if you're on an iOS device today or if you're on an Android device today you're going to have kind of different things that we've rolled out and what we'll do is we'll test on one platform that has the resources for it we'll take our learnings and we'll do a hold back test on the other platform to understand if it has any negative implications on that platform and if not then then we'll have kind of unification but again we're always kind of doing the stair step yeah great question yeah um how much variation do you find between different user bases like and can you keep everyone happy uh what do you mean hold on don't give the mic back yet what do you mean by user bases like for example different age group different regions different gender or different platforms yeah so um I we we don't have gender information um that's not something that we typically I don't think we collect that in our sign-up flow or anything like that um we would only have like an email address to go off of and I don't think we're parsing those to try to figure out if that's a male or female um so I can talk a little bit more about like the platform stuff um so there's often this question of like well hey you know iOS has a platform and they have standards and they have design patterns and Android has a platform and they have standards and they have design patterns shouldn't you create two different experiences one that satisfies these and ones that satisfies these because there's different use cases and and different user needs um we've tested that many different times and many different cases and found that not to be the case at least for Netflix so our navigation pattern in both of our apps are the same we do not have an action bar for android that's at the top and a tab navigation bar for iOS that's at the bottom um we have a tab navigation bar for both um and so so a lot of times people think that there are different reasons we'll we test those to better understand it and often find that there are not that that large differences it's it's more so on the content side like people are very different with the types of stuff they like to watch which makes sense yeah um what do you guys have against movie trailers or am I in a test cell wait what do we have against them yeah they vanished from my netflix where were you seeing where are you seeing them when like navigating down and opening like the info on the website yeah you're in a test cell google google and you'll find you'll find those trailers sorry i was just wondering if i'm sorry uh do you use archetypes or personas if so how and do you iterate on them we do not use them um they kind of i know designers that that do enjoy them uh and it helps to kind of get a frame of reference to begin with um but i i think that those are often just generalizations um and it's better to to really actually like go out and meet someone and then build a kind of case study around an actual person um then then those generalizations um there's there's lots of designers like you can ask 10 different designers and they'll tell you 10 different reasons why they love those or hate those um but my team doesn't use those or employ those anyway i don't know who has the mic so if you have a mic you can ask questions you mentioned that everyone at netflix has the opportunity to innovate um and obviously it's a massive company so you've got engineering teams and analytics teams and design teams um so how do you deal with the tension between a great idea that comes from a design perspective um and the engineering side or the vice versa where you have an innovation from an engineering side but it doesn't make design sense how do how do you guys decide and deal with that tension yeah um it always comes down to opportunity size um what to one of my like first or second slides um if you think of it like a funnel um and you drew at the top of the funnel you drew 130 130 million right that's that's our member base if you do something that only is going to affect a small subset of those whether it's engineering or design driven we're probably not going to prioritize that work over something that's going to affect a larger population of folks um so it's really it's a give and take there's concessions all the time that are made um but it's it's oftentimes a conversation it's it's not like argumentative i think both as long as you come at uh as long as you come up with an idea that is in the best interest of the business and the user it's probably going to get built at some point but it'll it's just a prioritization game yeah um you mentioned testing against success criteria and defining the success criteria really clearly upfront are there any examples where you've uh defined what you believe success was and passed a test or gone gone live with the product and then realized you were actually testing against the wrong success criteria and if so what are some of the common flaws second question what are the common flaws and success metrics that you've seen and how they kind of get applied to testing so when you have success metrics you try to not make them like a snapshot in time where hey our business goals are so narrow that these are our success metrics and if we're able to accomplish those we've you know got ourselves another six months of of business before we have to reset so our our two metrics that we track are streaming which is streaming minutes and retention and they're highly correlated between the two so the more someone watches the more likely they are to be retained as a member and continue to pay for the service so those are our success metrics and they have been over quite some time but we're constantly kind of challenging the assumption that those are the right things over and over and and developing new metrics actually right now that are that are geared towards understanding a little bit more nuance that again it has to be based in the the data that we can actually gather so you could say that our our best metric is you know consumer happiness but how we measure that is is really difficult to get a concrete view of that relative to how many minutes somebody actually watched so i think the question is just always challenge the assumptions that we're making in our in our metrics but yeah does that answer your question thanks and with so many tests running at the same time how do you make sure you're getting the right insights out of them they're not impacting each other and then also along those lines how do you scale so that you can run so many tests at the same time yeah so we basically have flags in our system that determine that hey you can't be in this test and this test they're conflicting but a lot of our tests are algorithms right so we're running different algorithms to on the personalization side to make sure we're giving you better content that's more appropriate for kind of your viewing patterns so a lot of the like when i mentioned that everyone in here is in probably 20 tests half of those if not more are algorithm tests and it's easier to to get feedback on those relatively quickly because we'll buy something to you know 10 million people for a little bit of time to better understand it before kind of rolling out whether we go with that algorithm tweak or not so we we make sure that we aren't muddying the water um so that we can get those clean reads hi um i was just wondering why the content we have in new zealand is different to the ones in states or like other parts of the world yeah so that comes down to licensing um there it's it's really kind of an archaic system of uh how you buy content and distributed around the world so there's like first run rights second run rights um so when you're trying to find uh if we want to license a a marvel movie or something like that in the U.S. it's really really expensive to do that in that market it might be easier for us to do that in uh you know in New Zealand or it might be more difficult so that's why the content catalog is different depending on where you go in the world you know we're creating so many hopefully great shows and movies that people love and they'll be available everywhere Netflix is offered with no limitations and that's really the world that we're trying to get to hi there again oh hi just two questions that i have for the first one is um which other company apart from Netflix of course do you think has really great design UI design and UX design that was my first question the second one is how do you keep up with good design is that something that changes throughout years or is that through testing that you keep up with good design yeah so the first question around what other companies kind of I admire from the design perspective I think that Apple would probably be one of them just because of how how kind of clean and focused they are but another would be Airbnb around how they kind of tell stories that you can kind of create these memories through this experience when you're renting a room from a stranger right and I think the the last one would be would be Disney because of the power of storytelling that they've just developed and the brand that they create like they create worlds around their content and I think that that's something that's that's kind of really special and how they do that and staying on top of of good design I mean design in some ways is just subjective so what I want to create is I want to create something that's highly usable first and foremost and that's also hopefully provides people with joy at some level but ultimately it's a utility right you could think of us like a digital vending machine right like over down what kind of candy bar do you want what kind of show do you want the challenge is how do you kind of innovate on that canvas and how do you develop that to be something that's even more efficient and it allows people to make the right decision for them so I read tons of blogs and books and different design things to kind of inspire and another one is just listening to other people talk about designs and like kind of their passion and and things that they've tried in the past as well. Hey firstly great talk it was really awesome thanks for speaking thank you. I have a couple of questions on consistency and design systems so firstly how do you maintain consistency with such a global product in terms of user journeys if someone wants to make a change and across the UI do you have separate design systems for different products or is there one unified one? So you're ready to gasp we don't have design systems we don't have pattern libraries we don't have any of that stuff you absolutely need that if you're gonna have a junior team you absolutely need that if you're gonna have a really large team all over all of our products we have 51 product designers if you compare that to a facebook or google who have 400 500 600 designers and a lot of we don't hire I don't hire anybody directly out of school I only hire people that have like five to seven roughly years of experience so they've cut their teeth somewhere else right they've made those mistakes somewhere else so I don't have to deal with that and I don't have I have like more I'm hiring people that can make good sound judgment and that tend to be a little bit more mature with their judgment their logic and their reasoning and so I don't have to create those like hard and fast rules for them I want them to kind of go explore the edges and the fringes and find things that they think are interesting and bring them back so we don't have those systems it you know ask me in a couple years if if the team's doubled in size if we started to employ those but we don't have them today and then the consistency if you have a spectrum and you have consistency over here you have innovation over here okay like they're at odds with each other and so I'm much more and being inconsistent and innovative than I am about being consistent and not innovative okay I'm really curious to know how you factor in human ethics if at all into the design of your product so if you optimize design then someone's going to want to keep watching it all over again sort of thing is that considered or do you think that's bad if someone keeps watching it? I don't know I mean like blowing so so what somebody who asked me a question earlier today which was around you know don't you feel bad if somebody watches Netflix and gains three kilos to which I said plenty of people watch Netflix on the treadmill so so my job is not to make decisions for people my job is to help them make decisions for themselves and this might be a little bit of a somber note I'm done but my boss grew up homeless and in a very very rough neighborhood and so what video entertainment meant for him was safety he couldn't go outside because it was so dangerous outside so for him it was all about you know watching shows with his friends watching movies with his friends that's not an experience that I had at all I had the exact opposite of that and so I had similar questions when I started Netflix of like don't we have ethical more responsibilities to get people to watch you know better content right like why are we why are we jamming documentaries down people's throats but that's that's not what our job is our job is to tell great stories and distribute them around the world and let people you our members determine whether we create more of those stories by consuming them so I feel like I have an ethical responsibility to to at the very least be mindful of any negative kind of outcomes but there's a reason that I work in entertainment and not medical device manufacturing I don't want anybody's death on my head all right thank you all very much really appreciate it well thank you very much thanks well thanks for coming here and thanks for a wonderful talk and we knew Netflix a company where that is streaming all kinds of video but Andrew provided us a great insight of how understanding such a diverse content and consumers is such a challenge actually and so we have a small gift for Andrew oh thank you very much everybody's trying to get me drunk oh no it's an umbrella oh it's an umbrella okay good so keep me dry I didn't say get drunk awesome thank you guys again and please reach out on twitter or my email if you have any follow-up questions thank you