 here with theCUBE. We're at Stanford University at the Ariag Alumni Center for the first annual Women in Data Science Conference. We're really excited to be here. We got invited, came over, I think it's about 500 women are here talking data science, some keynotes, some panels, and there is no other poster child for algorithms and recommendation engines than Netflix. We're really excited to have Caitlin Smallwood, the VP Science and Algorithms for Netflix. Welcome. Thank you. It's been a great day so far. So you just got off your little talk. It was fascinating. I think we could sit down and go for like an hour to follow up on all the things, but we won't do that. I'll only hold you for a few minutes. But the first thing that just jumps off the page, I think it was your first slide, was the simple thing as the name of the labels when I look at my Netflix screen, you said there's over 50,000 row names. That's right. One of the biggest pieces of evidence of your personalized recommendations on Netflix are those row labels. And some of them are very intricate, romance dramas with intense female characters or something like that, right? And so these very specific labels, it's interesting to study the distribution of those and how much they're viewed by different members and over 50,000 combinations, which is somewhat a function of how we even do the labeling and the tagging of the titles in the first place, of course. But it's just fascinating to see how tiny the head is and how very, very long and flat the tail is. It is, and that you can extract enough value to really slice and dice it to that degree to have such a long tail, because obviously you wouldn't do it if it didn't pay. So what are some of the things that you measure that say it is worth that 49,827 label for Jeff that we know that it's working for him? Yeah, great. So obviously we can't study every one of our 69 million customers and understand your tastes deeply and see what worked for you and what didn't. Instead, what we'll do is compare different algorithms that might use different labeling techniques or might use different mechanisms for showing those labels to you, right? And then we can kind of compare and see over time who's retaining better than members who saw this set of row labels, maybe, or this set of row labels, or different algorithms that are powering which ones get surfaced to you. So it really comes out of being able to compare whole populations. And the row labels was just the first of many of those attributes that you think is just kind of a basic distribution of metadata around the movies, but there's actually, you guys are optimizing and tweaking on a whole bunch of those things. Yeah, that's right. And in fact, the algorithms really are mostly unsupervised learning, meaning we're really looking at the raw structure of the patterns that are going on in people's consumption and the patterns of that data and what's successful for people and what's not. And it's less about first labeling the rows and then figuring out which label works for you. It's not that so much as finding the clusters of content that are interesting to people and then after the fact kind of putting a label on there to help humans navigate. Now you talk quite a bit in your talk about experimentation and really being active in your experimentation process. Talk a little bit about that versus just kind of, there's this vision of big data where you throw all the data into Hadoop cluster and magically all the answers are going to come out. And would you clearly say no? There's a very active hypothesis driven experimentation that we're doing as well. That's right. And we definitely from top down throughout the company we really very dedicated to the idea of experimentation. It's the one thing that can really help you get at causality instead of correlation. So even the best of statistical models with the best of data as an input, it's still correlation at its core rather than causality, right? And really a controlled experiment is the only more reliable way to get at causality. Causal inference, that set of techniques are really fantastic and you can't always, for practical reasons, run an experiment. And so in those situations, sure, the models and algorithms that are out there are certainly much better than just human judgment alone. But still they don't hold a candle to what you can also learn if you augment the practice of experimentation on top of that. And when you say you're getting to causality because you're using basic scientific method with baselines and fixed positions that you're experimenting against, you're doing AB, you are actually driving kind of classic scientific method to get to causality and not just correlation. Yeah, that's exactly right. Just like those middle school experiments you did in the science fair, it's just like that just on a very, very different scale. Very big scale. So a big part that you talked about too was ethics. And there's a lot of ethics around experimenting. And you talked about it kind of at length, that you've got customers that are paying. You know, I'm paying, Greg is paying, but maybe we happen to be the subject of two different experiments. And really being sensitive to the fact that, A, you want to deliver a better service, you want to get the data to deliver that better service, but at the same time, these are your customers. Yeah, that's right. We feel very strongly about being great stewards of our customers' data and really using it for positive purposes that are for the customer and not for us as internally for curiosity or things that we might learn just for fun, right? And so we take great care to think carefully about our own sort of internal guidelines around what types of experiments we'd run or not run. And also giving customers control to opt out of experimentation. It's an account setting that you can go to to opt out. Probably what most customers don't understand is, if you opt out, you're still gonna get a lot of change in the product because we're always changing the product. So you wouldn't even really notice necessarily, but still we feel it's the appropriate thing to do. And then we take great care to never run an experiment where we're ripping out a core part of the service. Like, no matter who you are and what experiment you might be participating in, you're always gonna have access to all the content on all of our devices at all times, unlimited streaming. We wouldn't run an experiment where some people could only stream on weekends and other people could only stream during the weekdays or something crazy like that, right? So we're very careful to be fair about the kinds and with positive intent about the experiments. So the other part that you spoke about that I would have never guessed, right? Recommendation engine, of course. Give me a movie that I wanna see. But obviously Netflix is much more involved now in your own original content development, right? We're doing that in the old days, the DVD days, I guess there's still some DVDs out there, but you guys are developing your own shows. And I thought the story that you talked about and using your own data to try to figure out kind of the scripts that you're looking at, the shows to produce, and even the names of those shows for uptake was a fascinating story. I wonder if you can share a little about the audience that wasn't in the talk. Yeah, yeah, sure. So that's actually one of my favorite areas of trying to apply data science at Netflix is this area of originals because it's an incredibly hard problem. You don't really have any data to speak of when you get a script from a writer. You might have a little bit of a talent attached to that project, right? But you don't really have much to go on from a big data kind of perspective. So what we try to do is triangulate and leverage really the expertise that our content organization has. They've got all this deep expertise. They can read a script and they know a lot about how successful that script might be. So we try to leverage their knowledge to piece that together with other data that we do have. What are the titles that they think are similar that we have already on Netflix that might be similar to what they envision this script becoming? And then from there, we can kind of get a little deeper into the audience that might be interested in the title. Depending on how it's executed, there are still many, many unknowns, right? But it still really helps us kind of play so which portion of the audience might we be satisfying with this new project and is it worth a big investment for us or not? Because we really aim to satisfy that long, long, long tail of member tastes. And beyond the title, you must do it for cover art too. I mean, I just think back to the old days, right? Before Netflix, before recommendation engines, you would go to the video store and really alls you had was the title. Maybe there was like new releases and you had the picture on the cover. I would imagine doing very similar things with the actual photographs because as you're scanning through, of course, it's a great recommendation engine. They're going to give me something I want to watch but it is really a function of those two then to maybe go to the next step, check the reviews, go to the next step, see who the actors and the actresses are. Yeah, that's totally true. We try to get into the weeds on all that stuff. We have endless dreams of experiments that we would love to run or things we would like to try in the service. Some of, you know, we have to always, of course, be very sensitive to our partners the studio partners and others that we work with the device partners and studio partners who are massive part of our Netflix ecosystem, right? And so often they have particular requirements or preferences around say the images and things like that that we have to be very careful about. But with originals, you know, we have more flexibility there on the kinds of things we can try. It's really exciting to have that, you know, the platform that can be so flexible like that, yeah. Well, I wonder if you could talk a little bit because you guys do a lot of stuff. Some of the business decisions that you make in terms of business trade-off because there's a smorgasbord of things you can do. You've got X number of computing horsepower, which goes up and you have a lot more than you had years ago, but right at the end of the day, you got to make trade-offs. What are some of the factors you guys think about when you're making some of these business trade-off decisions about should we do this or should we do this or maybe that's got value, maybe not, or is there a way you can kind of jump in halfway? How do you make some decisions? Because you don't have ultimate, I mean, you don't have infinite computing horsepower and you don't have infinite budget to do these types of experiments. Yeah, I mean, that's obviously a really tough question. It really depends on the specific decision at hand and what are all the trade-offs at play. I would say generally we're quite a thoughtful organization from the perspective of the people who are hired tend to be very strong experts in their area. And so we have product innovation leads who are focused on different parts of the product, right, who are very, very, very strong at their ability to make strong, good judgments and they'll evaluate all the data through the data sciences or other things as well as competitive information or what have you and make a strong decision. But we have a lot of internal debate, which I think helps because it surfaces all the things that one might wanna consider when making a decision. All right, so last question before I let you go, what are you most excited about that you're working on say in the next three months? Ah, well can I answer in two ways? All right, one thing, the thing I'm most excited about Netflix lately and this is a newer one as we've moved toward going global. We're aiming to be totally global by the end of 2016. And to me this brings me a new sort of personal philosophical satisfaction in the idea that we can actually share storytelling around the world from stories around the world. So the idea that cultures can get more knowledgeable about one another, more comfortable with another, learn in subtle ways through entertainment is really super motivating to me. So I'm deeply excited about that. And then in terms of data science, I'm particularly excited about some of the stuff we're doing in the content space right now where we're really trying to marry, it's a difficult space, because we're really trying to marry that, like I said, that creative expertise with what is the information that we can empower. And so we're doing a lot in this space of originals and trying to really help empower those decisions, but not in any way get in the way of the creative piece, right? And so how can you have tools and things like that that let the creative folks play what if games about their potential content that they're thinking about in a way that really informs them in a trusted way? All right, well, it just begs the question on the recommendation engine. What, you know, you're sending me stuff that you know that I'm gonna like versus maybe I need to see something I've never seen before from South Korea or Japan or India or whatever. That's a whole different type of a challenge because I want to explore, I don't want to see what I'm used to seeing. Yeah, that's definitely true. And that's when we hear, you know, we hear that from customers that people are worried that they're getting, you know, put into a box. We do actively experiment with how much diversity we share and how that diversity unfolds over time for customers. So we generally have discovered that we're really not keeping people from any type of content that they might not otherwise see. I sometimes play around that with that, which anybody can do with Netflix, go in and create a new profile. And then it'll start without personalization and you can kind of start over and see, oh, is there like a vastly different set of stuff that you see that you wish you would have been seeing, you know, and that's kind of an interesting game you can play. You're just gonna say log in and watch a couple of movies that you usually don't watch and see how that scrambles everything. I'm not sure you can do that too. All right, Caitlin, well thanks for spending a few minutes. Great talk. Love to get you on another time. Really enjoyed your talk. Thanks so much. Absolutely. So Jeff Rick, we are at the Women in Data Science at Stanford at the Ariaga Center. You're watching theCUBE. Thanks for watching.