 Thank you so much. Super exciting to be here at Slush. My name is Gustav, as mentioned. Right now I work as a partner at Y Combinator in San Francisco. Up until early this year I worked on the growth team at Airbnb for almost five years. So this is the team that I spent those years with. Amazing. And today I'm going to share some learnings from what I learned working on growth at Airbnb. So the first disclaimer is that whatever I'll talk to you guys about today does not apply to every company. It's sometimes very tempting to start doing growth things when you're starting off with your startup. That can often go wrong. So what you want to make sure before you start accelerating your growth in your company is that you build something that people really love. And that's actually the really hard stuff. Most people that build companies don't end up building something like Airbnb that ends up being an amazing service that lots of people are using. So that's the disclaimer. This does not apply to everyone right away. So the first couple of slides are from my talk I gave to the new employees on the growth team at Airbnb. I gave the talk every couple months. And the first question I got from some people was why is it important to work on growth? If you have a great product, doesn't great products use grow by themselves? It's a good question. So I want to give an example here. So Facebook, we all know, is a massive, massive success. Two billion something users and their growth team, the Facebook growth team is famous for being kind of the original growth team that a lot of us get inspired of what we're doing today. So they have a story basically around their growth and the story goes like this. So Facebook has started in about 2004 and they of course wanted to become like a billion user company. And they had a great data science teams that made predictions about the user growth. And in about 2007, 2008, they made a prediction and they said Facebook will be about 400 million users. And this story is not, I didn't work at Facebook, but it's retold to me from people that worked on that team. So the forecast said it will be about 400 million users. Now sometime in 2007, Facebook figures something out. Anyone have any idea what they figured out in 2007 that actually accelerated their growth? Any ideas? Sorry? Like button. Any other ideas? No. So what they figured out in 2007 is that Facebook was all in English, the content was in their local languages. That wasn't enough. So when Facebook launched translation, basically they overnight translated into 200 something languages and that re-accelerated their growth. Same thing happened again. 2010, the data scientist team on the growth team made a prediction and said Facebook will be about 600 million users by 2015. So that's kind of how big it is. It's not going to be a billion. And they really figured something out again. And they had a massive spike in growth. And this is not one thing. It's a number of things. But anyone have an idea what they figured out in 2010, 2011? No ideas. They figured out mobile. So the big change in 2010, 11 was that all of us were switching our efforts to all of our users to mobile phones. And Facebook weren't very good at that. They didn't have a great app. They didn't have a great service on mobile. And when they did that, they eventually got Instagram and WhatsApp that growth massively expanded. Now the same thing happened again in 2011, 2012. The forecasted out will be one point something billion. But it turned out that it became a lot bigger. Anyone have any idea of what they were working on on the growth team at Facebook in 2013 and 2014? I barely can't hear you. They basically figured out that there are only about 2 billion people online or 3 billion people online. So we need to get more people online. Otherwise we're going to have more users on Facebook. So the growth team started something called Internet.org or FreeBasics that basically made deals with carriers to get more people to get online. Now, what can we learn from this? What can we learn from this graph? Well, the initial forecast said 300 million, roughly, not 2 billion. So how can they be so wrong? Well, the whole point here is that you made a bunch of changes and a bunch of product decisions that massively accelerated their growth. So this answered the questions, why should it work on growth? Because it matters. Like basically doing a bunch of product changes and knowing what actually works will make it enormous different to your growth rate. This is the first slide I gave to everyone at Airbnb's growth team. Now, natural adoption will always slow down over time, but you can impact your growth rate. So measuring your retention. So the first thing you want to do here is basically figure out if your product is awesome. And you do that by looking at if people are using your product again and again. And you don't want to have a graph that looks like this. You do want to have a graph that looks like this. So basically people are coming back and using a product again and again and again. And every single dot on this graph is a usage. So when you get repeat usage, you get a graph that looks like this. And that's what you want. So let's say you have a great product. You get great usage. People are using it and now you have some users. How do you make decisions at scale? So Mixpanel put out this quote. They basically said, most people in the world, most of the world would make decisions by either guessing or using the gut. And they will either be lucky or wrong. That's true. So decisions have consequences. If you go to Airbnb today and you make a decision and make a change on the website, some metric will change. You don't know what metric will change because the website is already so busy. That actually is true for almost every single product. So when we make decisions at Airbnb, we use experimentation. Let me give you an example. If I add something to the website, I have to remove something. It's already pretty busy. If I remove the star rating of each listing, the page will look like this. And you can imagine the amount of metrics that will change if you make that simple change. So small changes will have massive differences. And we cannot comprehend all those changes by just looking at the product. So we use experimentation or use, I used to work there, use experimentation to validate the decisions. And this is how it works. So how do you really know that an experiment is important? Well, let's say I make a decision. I launch a new feature on this date. And this is how my metric go going forward. So it keeps going up. So that was a good decision, right? You don't really know. You don't know if that was a good decision. You have to have something called a counterfactual, which is basically how would a world look like if I didn't make that decision? Only then will you really know if this was a good decision. And this is basically the essence of experimentation. So Airbnb, eventually, you get something like this. This is our experiment framework at UI. But when you start off, you get something like optimized or something that's more simple to use. So another important factor of this is what we call experiment review. Now, how do you get people thinking that experiments are important? I'm going to do that with you now. So this is something that we learn from Facebook. Every month, the growth team at Airbnb will get into a room and we're going to do what we're just going to do now, which is we got into a room and we talked about the experiments that we launched. So here's an example of something that we launched at Airbnb. So on the left hand side is the share sheet for iOS, the standard one. And the right hand side is our own version that we made. Same sheet, same functionality, just looks quite different. And I have the answer here. But before I give the answer, I want to figure out what you guys think. How many here think that, just to summarize, the metric that we're trying to drive was more people sharing the listing from Airbnb. So how many people here raised your hand think that the control share sheet had led to more shares raised your hand? How many people here think that the experiment share sheet raised your hand? A lot of people you guys didn't raise your hand. I'm just assuming that you guys don't think it was a difference. So it turns out that in this case, this was 40% better. You guys all disagreed. There was actually not ecosystem opinion here, which means that this is a hard decision to make. You need experimentation to make this decision. Second example, last example, we had a trip invitation email that basically went out and the subject line of that email said, you have two options. You basically you got to invite your Airbnb trip. First subject line said, join Joe's trip or join Gustav's trip. Second one said, accept Joe's trip invitation. How many people here thought if you're looking at sign ups or basically the conversion rate metric of this email, think that a subject line like join Joe's trip was better. Raise your hand. There's some people. How many people here think that accept Joe's trip invitation was better? More people. Interesting. You guys were right. So this was 14% better. Now, what can we learn from these two things? Proc decisions are really hard at scale. Your gut is not good enough. You need to use data. You need to use data that is better. Conflict of words are very important to make decisions. You need to make decisions, and you need to get ready for those basic experimentation. Otherwise, you'll make a lot of mistakes. If you rely on the average of this audience, we'll make a lot of mistakes. Three things to remember about building a sustainable growth. Measure your attention. Choosing a Northstar metric, that one metric that matters and then setting a goal. And finally, building a culture of experimentation where you do these kind of idea of experimentation as a decision framework. Cool, I'm going to take some questions. All right, how do you measure love to know if people love your product? So there's two ways to measure things. One is to talk to people and do a qualitative research. The more people you talk to, the more you get a sense of what they think. The second way is to look at numbers. And one way to look at if people love your product is to look at retention. Are people coming back and using your product? Are they engaged using your product? And the more people that come back and the more they engage you are, in general, the more they love your product. Let's see. Looking at Facebook, what does growth come from continuous innovation? Yes, so basically experimentation is all about compounding effects of experimentation. So basically, you have a metric you're trying to grow. Let's say you're trying to go new users. And the consequence of that is basically a bunch of experiments. So every time you go to Facebook.com or the website or the app, wherever me.com, you are subject to many, many experiments that they're running as we speak. And you guys will all be in different of those groups that I mentioned. And maybe on average, you'll be in 100 different experiments. You don't really know. But that will give you an idea of the amount of compounding effects that these companies are having, just testing what works. And these are small changes from pricing to UI to use experience to, you name it, anything. Let's see. How large sample do you use? This is a common question I get. So there's actually a very easy way. You can go to Google and you type in AB testing calculator. There's lots of websites that are AB testing calculators. And what you want to do in those websites, put in the amount of traffic that you have to your site, the amount of people coming into your experiment. And then you say, what's some expected change that I want to see? And then that will tell you if you have a good enough sample. Basically, sample is a function of the size of change relative to the audience. So if you have a large change to a small audience, it might be statistically significant. But you need to have confidence in the significance. Otherwise, experimentation is very tricky. OK. What's the ideal composition of a small growth team in terms of capabilities, qualifications, and experience? So when I started at Airbnb, we were three people on the growth team. It was me and two engineers. When I left, I think we were almost or above 100 people. And we had merged with the performance marketing team and the SEO team. When you start, you basically want to have someone building the experiments and building what you're doing, which means usually an engineer and a designer. You want to have someone looking at the data. Now, that can be the same people. It can also be a product manager. It can be a data scientist. Those are the most important things you need. So the team can be as small as one or two people as big as four or five people. And you can cover all those things. As you are growing, you want to have different teams working on different parts of that funnel, driving different metrics. And basically, at some point, you're going to start looking at how many experiments am I running? Like, how can I run more experiments? Because we saw that there's a direct correlation with the amount of experiments that you're running and the impact that you're having as a growth team. OK, Airbnb have revenue from day one. So you had a budget to invest in growth. What if you pre-revenue and not inherently viral? It's a great question. So if you don't have revenue, you cannot invest in paid marketing. Period. Even if, let's say you have no revenue and you invest in paid marketing and you get lots of users, have you succeeded? No, because you have to pay for all these other users. Now, you've learned that you can acquire users for money, but you don't have any money coming in. You just learn nothing that's useful. So basically, you can only really scale performance marketing or marketing if you have money coming in from your users. And you don't really know how the performance marketing is going to work unless you're charging for those users. So before you really invest in performance marketing as a growth mechanism, you want to charge your users or find a way to make money from your users. Otherwise, it doesn't really work. And most of the companies, unfortunately, if you look at the large platform today, Facebook, Google, they are getting very good at making companies pay for distribution and making it very hard for companies to get free distribution, which means you kind of have to be pretty good at performance marketing or marketing at some point. There are many of the most successful startups that have grown really fast lately are good at performance marketing and they're charging for things. So if you have a free product, you kind of have to ignore marketing and just go for some other channels. Now, virality can have many different types. You often think of online virality, things like email invites or things like that or Facebook virality. There are many other types of viralities like physical virality. Like, imagine we have this emerging field of AR, which means we're all going to keep our phone and hold the phones up doing these things in the real world. That's also virality. So there are many different things that you can do that will get other people to notice your product and make them start using it. What are the best books on growth teams? I don't know. But everything I've learned about growth, I've learned from other people that work on growth and by reading blogs and stuff online that they write about. To be really good at growth, and I think it takes a curious mind, like you basically, when you land on a product, you want to figure out like, oh, how are they doing their onboarding? How are they growing? How are they getting traffic? You kind of always have this curiosity of trying to figure out how a new product is getting those users and how far into the product they get. But I don't think that there are any good books on growth. It's also a subject that change fast enough that if you write a book today, it probably wouldn't be useful in a couple years. Let's see. Are metrics important before you have significant growth? Metrics are always important. Metrics is a way for us to at scale measuring anything about our product. So the stuff that you learn by talking to 10 users, if you have 1,000 users, you can learn similar things by just looking at measuring those users. But the very, very best growth teams have a combination of best product teams, have a combination of user research that talk to users and data, the science team or metrics that look at users and what they do. You want both of these. These are the best two tools you can have. A lot of questions. What stage should you focus on growth? So I try to answer that in my presentation. If you have built something that people love and there are many ways to kind of judge that, one of the ways is are people coming back to use your product again, then you should probably focus on growth. Like you have a few thousand people using your product and they're coming back pretty repetitively. That's a good sign that they like what you're building. Then you want to start thinking about how should I get a lot more people than those few thousand. The mistake that people often make when they are investing in growth is they invest too early. And they invest in top of the final growth channels before they actually have a product that people like or people love. And what ends up happening is you turn through a lot of users in these channels and this is very tempting because the top line growth numbers always look like a great number. But you can't look at the top line growth number. You have to look at the kind of growth accounting or the sum of all growth numbers. So how many new people are coming into my product? How many people stop using my product? How many people came back and started using my product again? What's the sum of all that? That's the number you have to look at. You can't just look at the top line number. That will fool you. How do you capture feedback from users? This might be the last question. So, at Airbnb, what we would do is we would go downstairs to our atrium and meet with strangers and show them the signs of what we're working on and just see if they got it. And if they didn't get it, then we had to go back to work. Another thing we did is we installed this JavaScript snippet on our website and when someone was in a flow, we were literally just like chat with them and say, hey, can we call you right now and give you $25 on Amazon just to hear what you're thinking about the product right now. That's a great way to capture the kind of user research in the moment when you're like, we want to get people in the right mode when you're doing that. Other ways is surveys. I recommend any growth team to have a great survey person who really understand sample size and how to accurately and properly ask the right questions that you can learn from. It's probably my best advice. We're out of time and I wanted to thank you so much for listening to this talk and there's another panel about growth at 5.15 at Fireside Chat. But thank you very much for coming. And thank you, Gustav. Thank you so much. And amazing job, everyone, with questions. Thank you.