 Hi. Hi, everybody. Welcome. Thank you all for being here and thank you for the wonderful introduction. I actually have a confession to make. This is my first presentation that is live without Zoom, so if I look a little bit lost, it's probably because I'm looking for the share screen mode, so just bear with me. But it's so great to be back at Slush. This is probably my third time, the last time I was here was in 2018, and I'm so excited to be talking to you all today about how to talk to users. This is something that's very important. I'm just going to go and introduce a little bit more about me so that you can see where these advice and these tips are coming from. So we started our company in 2006, not as a mobile gaming company, but as a corporate social network. We ended up building fan communities for TV shows and sports teams, and then we pivoted into building games. So some of the games we built were with Hollywood franchises, such as Marvel, Fast and Furious, Star Wars, Hobbit. Basically, if it was a PG-13 franchise, we built a game with them. Our most popular game, you can still download it on the app store today. It's called Marvel, Contest of Champions, and you can play it if you're a huge Marvel fan. We grew that company to over six offices around the world with over 1,000 employees, and then in 2017, we exited to Netmarble and Foxnext. Since 2017, I went to YC as visiting partner. It was there, I think many of you know who they are, but they are the first seed stage money into behemoth companies that you know today, such as Airbnb, Twitch, Stripe, and Instacart. And it's really there where a lot of things just crystallized for me. And with these two things together, you're going to hear a lot about how to talk to users. It's going to be informed by a lot of our founder experience as well as mentoring more than hundreds of YC founders. So when I went to YC, the first thing I learned was their motto. Their motto is make something that people want. And this motto is pretty funny because it's a really great guiding principle for a lot of founders. It's actually the goal for founders. So if you're building a company, this is actually your goal. And I love starting with the end in mind. So looking at this motto, there's a lot of things that can tell us a lot of how we can make something that people want. The first is a lot of people start focusing, particularly founders, on making something. That's because this is within the founders domain. I joke a lot with founders saying you can make anything. You're incredibly talented, incredibly smart. Let's just make sure you're building something that people want. And this is the whole reason why we're here is how do we figure out what people want? And the most important thing to know of what people want is that is owned by the user. It is not owned by you as the founder. And I think that's a really important distinction of who owns what. And then oftentimes, you'll hear that investors talk about problems and solutions. So even within this motto or this guiding principle is that you are responsible for making the solutions and the user is there to tell you what the problem is. So oftentimes, you'll see people interchange that here. Now, oftentimes, I've definitely seen people focus on just one side of this statement. And if you just focus on one side of the statement, you'll get some funny stuff. So I've definitely seen founders who just focus on making something. Oftentimes, we call this a SISP, a solution in search of a problem. And sometimes you end up with some of these inventions. These are inventions from the Japanese. It's called Japanese useless inventions. Some of them obviously are over engineered. I'm pretty sure there's probably easier ways to keep your shoes dry or faster ways to cool down your ramen. I actually think this butter stick is very interesting. I'm really waiting to see if it'll hit the markets. But oftentimes, if you just focus on making something, this is kind of what you end up getting. And if you just focus on what people want, then oftentimes, particularly in the crypto world, you're, you're going to end up building a rug or a scam, right? Because you're going to spending all your time talking to users. But you're never going to build anything. So the biggest thing is really focusing on trying to figure out what is it that people want. The first thing I did want to kind of set the tone for everybody is trying to figure out what is it that people want? Oftentimes, and this is probably one of the few times I'll invoke investors here because investors are not your users and not your customers. But investors are investing in teams that are making something that people want. How do I know that people want this? And they'll either put in their time or their money, right? And you could see this in terms of Facebook, right? People invested so much of their time that it was, it was seen through daily active users that over time, these daily active users were growing. Oftentimes, money is a really good measurement. And for the vast majority of you who are building a company right now, money should be pretty much your top revenue or sorry, it should be your top KPI, which is revenue, right? We see it all the time. I paid money for my streaming services. I pay money for transportation. These are things of value. It's either time or money. But what if you're at the beginning of your startup going from zero to one, and you don't even have anything, and you're wondering, am I going to build something that people want? Well, I'm here to give you some advice. Things to do, things not to do, and some rules along the way. The great thing about a lot of this advice is that even if you have a million users, you could still use a lot of this on kind of how to talk to users. The first rule in talking to users is you actually have to get out of your chair and you have to talk to users. I know this is really funny. I only say this because this is like one of the two things we repeat over and over at IG. One of the things we do to create an online platform that has a UI combinater for our founders, is you have to get out behind of your screen. Stop writing code and start talking to users, and I think it's probably because code is so much easier to manipulate than people. Not that you should be manipulating people. But to get out and start talking to users, it's a very, very vulnerable thing. The secret You're trying to figure out what is it that they want so that you could build a great solution for them. Rule number two. This should probably be the rest of the rules every single time. You are not the user. I know oftentimes investors will say, oh, you should start on a problem that you had. That's great. Oftentimes a lot of people will start talking about the Airbnb founders. The Airbnb founders, they didn't have any money. They tried to solve that by selling cereal boxes, right? Well, not cereal boxes, cereal, with Obama O's and the Captain McCain's, and they were running out of money while they were exploring this couch-surfing idea, right? And what had happened was one of the founders at the time, he decided to use their own Airbnb product and discovered how much he wasn't quite the user or some other problems along the way. So he went and he stayed at one of the Airbnb's and he got up the next day and he was hanging out with the host and he realized, I have to pay this guy, but there was no way to do it online. So he had to run all over town looking for an ATM and paying, and then he had to pay cash and he's like, this is really embarrassing. And he realized, look, there's no way this platform will get adopted unless we had a payment system, right? So he realized that it wasn't just about making money, it was about making the entire system working so that it could get adopted. And he was only able to do that because he realized, even though he is the user, he's not really the user. Now, before I jump into a little bit more of what to avoid and what to do, I really wanted to focus a little bit more on looking at these problems because the most important thing when you're talking to users is doing a lot of this problem discovery. So for you as a startup, I just wanted to go through three things that make really good problems to solve, right? The first is that this problem is very urgent. We call this a hair on fire problem, right? So you could see this in your own life when you're a customer or something's urgent, you're like, I don't care, please take my money, right? The second thing is this is incredibly valuable, right? So the most valuable thing is you'll either spend your time or money on games. We spend so much time on games, TV, when it comes to money, if I need a transportation, when it's raining, absolutely you'll pay more money, right? And finally, good problems are what we call common or pervasive. So if you follow the rule that you are not the user, you will be building for more than just yourself, right? Really good problems are problems that many other people have as well and that helps solve with your technology, right? That's what we talk in terms of scale. So I want you to keep those in mind as you're talking to users of what is some good problems to be solving as you're doing problem discovery. So as you're talking to users, there's a couple of things you should avoid. You should definitely avoid asking would questions. Why? Because everyone has a plan until they get punched into the mouth. We see this a lot in our own lives, right? Where people ask me, oh, Holly, sign up for this. Oh, yeah, sure, sure, I'll sign up for this. Oh, will you tweet about it? Oh, yeah, absolutely. And I'm pretty sure there's many more people who are way more committed than I am in the room. But for me, sometimes I either just forget or, honestly, it's to keep that social contract really good and my future self just kind of wrecks things. So it's really hard to answer truthfully, would you do something? Like, would you use our product? And I'll tell you later about how we found out the hard way. Avoid talking too much about your solution, right? So at this point, you're trying to figure out a lot of it. What is it that users want? If you talk too much about your solution, they will get distracted and they will have anchor bias. A great example of this is when I first came out of grad school, I worked at a tech company called AOL and I was a product designer. And what we would do is we'd post our mock-ups around the office so that people, we can encourage conversation, we could talk. And the thing is the mock-ups looked really well done. And nobody would talk about it, nobody would stop, nobody would engage. And what had happened was we decided to move it into much more of a sketch mode, right? And once we started doing that, people thought that they were unfinished. So they weren't as distracted by the pretty mocks and they were able, for us, what we wanted was feedback at that stage. They were able to interact a lot better because they were less distracted. And you see this in your personal life too, as you're thinking about brainstorming, oftentimes they're like, give me your idea, no idea is bad. And how many times do we censor ourselves before we even say what we need to say or give our idea? So you should try avoiding talking too much about your solution, especially if it's not created yet and you're just thinking about things. Now, rule number three, I think this is like a very important one is to know is that humans are evolutionary, not revolutionary. This is actually a picture of one of envisioning of what the first railcar would look like and it's no more than a bunch of horse-drawn carriages kind of pulled together in many ways. And I think in many ways what happens is there's a lot of familiarity with that, you start with one, and then you build from there. And oftentimes it gets really hard to hop. And so in particularly free to play game design, we would talk a lot about how we would need to have one third that was innovative. And it's kind of interesting because oftentimes you think of innovation as like super cool and super neat. But it could be something as boring as a business model. And that's what it was for us. So we kept about two things, or two-thirds of the product, very familiar. And then the one third that we played with was innovating on the business model. So at the time for anybody who is a bit unfamiliar with the gaming industry in the Western world, how we monetized was very similar to how movies monetized. So if you go buy something for your Xbox or even early games here, early mobile games, it was basically you had to pay money. And if you opened up the shrink wrap or you started the game after you paid money and you had a terrible experience, you could say goodbye to that. Right? What we did is we came along and we said, look, you know what? It's going to be free to play. If you don't, if you like the experience and you want a premium one, we're happy to upgrade, you're happy to give you more premium experiences. But if you don't like it, you can still continue playing. Which was like a very new and innovative way of thinking about games at the time. And everything else we actually kept very similar. So we kept our first flagship product, which was called Kingdoms of Camelot. Very normal IP. That IP was open IP. People in the West understood Camelot. You could build a lot of lore off of that. Our genre was very familiar. It was very similar to Civ. It was a strategy based simulation game where you build up your resources, to build up your town, to raise an army and go battle one another. So really as you're building out and thinking about the solutions, just keep in mind as people are talking, humans are very evolutionary and not revolutionary. And in fact, when you become too revolutionary, you might end up with something like this. Now, I apologize to anybody in the room who has something like this. But as a parent, I look at this and I just think, oh my goodness, this has got to be super dangerous. There's just no way. Of course they wanted a faster horse and a faster baby carriage. But I just don't know if I could put my kid into that possible danger. Now, I've talked about things to avoid. And oftentimes with the don't list, you always wonder what should I do instead. So here's a couple of things that you can do instead. You can be user led versus user driven. So in our world in games, oftentimes user driven does turn into data driven. That's because we can record every single click that you do. And that's the only way you play our game, is through clicking things, swiping things, any type of motion. And what happens often is that people become driven by the data or what the user is doing. And I think talking to users really helps paint the entire picture. If it's just data, I'd like to say data helps turn the lights on in the room, but it can't tell you where to move the furniture. And that's really what being able to look at the data in the context of talking to users, it really helps you to be user led. So also be super careful when listening to the user's solution, right? We've all heard the old adage of Henry Ford who said, if I wanted a better car, people would have said I want a faster horse, right? So he listened to the faster side but ditched the horse because he's like, I don't really, he knew that he was in charge of the solution, but what they wanted was something faster. So he had to really, really think about it. So you just have to be careful. A lot of times users will give you some great suggestions. You just kind of have to weed out some of those things. The next thing you should do is ask open-ended questions. Asking open-ended questions is really helpful. And those are really easy as who, what, where, when, why, how. It gets them talking. Remember the secret to talking to users is getting them to just start talking just enough so that you could spend most of your time listening to them and wondering what their problem is. Number five, rule number five, this one's actually really hard. It's more important what they do versus what they say. And we ran into this, particularly our first time with our corporate social network. We invited some friends and family in and they said, oh yeah, absolutely, we'll use it. And then we launched and hardly anybody used it and we're like, oh my gosh, why did they tell us these things? Our friends suck. And when you start out, all you have is people's verbal kind of commits and everything. So how do you get to the root of what is it that they would do or actually do and get a good read into it? So certain questions you can ask is ask about their past. I love asking things like, what was the last game you played? How did you find it? Did a friend tell you? Where did you play it? It's much easier to recall even if you think in your own personal life, you can recall better than saying like, okay, I would do this and actually end up doing that. So asking about the past really helps bring to light what are the things they actually did do and eventually that can translate to what do they do. The second thing is you want to ask them to show you. This has been incredibly enlightening. So if you ever get a chance to embed yourself with your users, I highly recommend this. I have three great stories of like all this context that you can see if you just ask them to show you versus tell you. My first story is about my friend. She had started a very large recruiting software company and she spent months inside another company as part of their recruiting arm. Just doing, just talking to users, talking to recruiters, seeing how they recruit, figuring out what the problems were before she even wrote one line of code. And then finally when she wrote her first line of code, she already had about three marquee customers because she knew that the something that she was going to be building was definitely something that was wanted. So it's more important if you can just follow somebody around. Now some of us don't have the luxury of that. Some of us aren't building B2B companies. In our B2C world, as I mentioned before, many of these tips and these rules can be used even when you had millions of users and we already had millions of users on our fan communities but we wanted to learn more about what they were building. And what we had done is we called up one of the users, she let us dial in and we watched her use our product. And I kid you not, there was just no way I ever would have figured this out. I watched her screen and I've never seen anybody browse the internet with a bookmark pane open. So she was only working with 66% of the screen and the resolution was really low. So that informed a lot of just context. And then finally, my friend, he was working on an insurance, a car insurance product and he was talking to a user asking her to register. And when it came time for her to put in her car registration number, she said, hold on a second. And he was just waiting there for 10 minutes and she came back and he's like, where did you go? She goes, well, I had to go to my car, I had to find it to pull out that registration number. And at that point he realized like, oh my goodness, this is a big problem that we should probably ask for the registration number later. So it's really important to get that extra context if you can ever find any bits of why the user is doing what they're doing. And that could help inform how to make the solution much better for everyone. And finally, in terms of what is important on figuring out what they're actually doing, it's really important if you can record, record, record. Now the most important thing here is after obviously you should get their permission but what's really cool about today is a lot of things are remote. So you can record Zoom sessions, you can record all these things. So you can always go back to it and you can talk with a team about it. They know kind of, you can have discussions on it and not fighting about what happened, right? And finally I'll put in this bonus rule. If you have more than one person on your team, that person should be involved. Talking to users is not only a team effort but it's hugely cultural setting. I love companies that have onboarding. I think it's Zappos. They basically force you to be one week at the call center. Your first week is you are a customer service agent because they put the customer right at the center and they built a whole culture around it and I love that because your users are the ones that are going to make up your company and make them who you are. So at this point when we were even about 30 people we made sure it was a team effort when we were going into games. We had a family and friends night before we were launching our first game away from fan communities so we needed a lot of feedback. We didn't know what we were doing. We brought our fans and family because we thought they'd be kind to us but in reality they didn't hold back any punches. But we asked the whole team to stay. We divvied up people into pairs, not engineers with non-engineers. They watched them do some tasks that we asked. We asked them to talk out loud. We even put them into focus groups, talked about what they played, what they liked, what they didn't like. And finally afterwards, after a long evening we decided to stay in debrief. That debrief lasted probably till 2 a.m. And I remember our quietest engineer, I know it's stereotypical engineer and quiet but she really was our quietest engineer, would not stop talking. She was incredibly engaged. It's because she saw things that we wouldn't have been able to see. She was coming up with solutions and there was just so much energy in the room by involving the entire team. So for as long as possible I highly recommend to never lose sight of your user and to bake a lot of this stuff in at the beginning and in ways like try to scale it across the organization. So just remember your goal is to make something that people want. And the best way to do that is to talk to your users, remind yourself and everybody else on the team that you are not the user. And know that humans are evolutionary, not revolutionary. And be user led versus user driven. And just remember it's more important what they do versus what they say. And if you do all these things this will help you talk to users by listening to what they want. And hopefully likely you will be making something that people want. Good luck and thank you.