 Cool. So, how's it going? My name is Kevin. Today I'm going to talk about Gorilla user research tactics to get mainstream adoption for your DEP. So the original version of this presentation actually focused a bit more in the beginning on some of the user research I've worked on globally at Facebook and Quartz and some lessons to be derived from that for blockchain and payments. But after doing some user research of my own, a lot of people say that the second half of the presentation, just the actual tips and tactics were the most useful part. So this presentation is going to focus entirely on a bunch of things that you could do for your DEP starting tomorrow to help validate the idea for your DEP and the concept for your DEP and all the way down to launch. So a little bit about me. First, I've touched user research and data from a few different perspectives. I cut my teeth spending summers at Google as a product data analyst running a lot of Python data jobs and analyzing user data for the Geocommerce team. I spent a summer at Microsoft managing user research for Cortana. Then spent a few years at Facebook working on growth product, product marketing and helping drive user research initiatives both on the product and perception and sentiment side, but also across a number of different continents. And currently, I'm a product manager at Quartz. We're focusing on memberships and subscriptions and doing a lot of scrappy, scrappy startup research. So I guess, you know, what is user research? Instead of a dictionary definition, I think I'll just give you my own colloquial definition, which is a bunch of ways that you could interact with users, potential or current users that help you understand how they do things. And so these ways or research methods, you could kind of cut in a variety of different ways. You could think of the methods in terms of what do they track? How do people think or how do people actually behave? You could do it by the phase of the development cycle that you're in and which tactics map best to that. You could also do it by the context in which the study is actually taking place. Don't worry, you don't have to memorize these slides. Another main split is qualitative versus quantitative. And for those of you who didn't know, a lot of user research is considered quantitative. For example, for surveys, especially when you do them at a really large scale, you need to calculate the margin of error and make sure you have statistical significance in your findings and results. So what is gorilla user research then? It's basically just a term I made up. And what it is is it's trying to do user research for your DAP in 2018. It's scrapping and clawing to get any sort of validation for your DAP that probably, if it has launched on main net, probably doesn't have a ton of users while dealing with few resources and the dozens of other barriers that you face as a blockchain founder in 2018. So this is just a quick thing I drew where this is the preferences of a real kind of person with an actual problem. To the right is the solution that they're using now. They like it. Slightly to the left is your blockchain based solution. The farther to the right it is, the more they want the solution. So you've heard the saying that you need to be 10x better than your competitor in order to get over switching costs in order for the person to actually switch over. I've always thought 10x was kind of a lot, but let's charitably say 2x. So you're already starting off at a huge disadvantage. You're already twice as far away as you were before, just by being a competitor. Then you introduce the other things that are happening. People are skeptical of crypto and blockchain because of the market crash. People may not even know that you could build apps on blockchains, all the standard exchange and metamask barriers, all the UX and copy confusion that has been very well documented and talked about, I'm sure, covered very well by a lot of presentations here. And also just, you know, if you're a platform that requires regular other people like businesses, artists, musicians, you're going to have to convince all of them to get on, and that's going to be really, really hard. And if you're an enterprise product, you have to convince at least 10 different stakeholders within that company that this is real and valuable and secure, which is going to be very, very hard. And so you're basically off the map now. That's kind of your starting point as someone who's building dApps today that are competing with actual centralized solutions. And even developers and those who invest in them don't use dApps. So this is from a great medium article I read. I thought it was pretty funny because it seemed pretty accurate that even developers who build them and the investors who invest in them, they just don't even want to use them because it's so annoying and hard to do and they have lots of other busy things to do in their life. That's what we're dealing with. Yeah, so great. So some initial questions you may have thus far. So isn't user research only a thing that big companies and fancy design consulting companies can afford? No, I mean, obviously it helps to have money, as you could see, I've been on both sides of it. So I've seen firsthand. But the problems are kind of different. For a small company, you're just trying to validate your idea at the beginning, you're just trying to find any time to do user research, whereas at big companies, you're more kind of launching new product lines, kind of protecting existing products against competitors, etc. How many of you guys work for a company that's above that maybe has more than one full time user researcher by any chance? Cool. I'm guessing most of you people are consensus or perhaps Coinbase or status, which is no, which is good. It's like, I think we should all aspire to do that to be honest. But I also know that a lot of you guys are really small and just trying to hack and get something out the door. So these tactics hopefully should be pretty helpful. All right, well, we're too small, we're short on funds, we can't hire a researcher. Well, so you don't you may not actually need to hire a full time researcher. So in terms of finding your researchers, so one Ethereum token means you don't have as many resources for means you have more resources. So when you're pretty small, five people or less, it's gonna have to be you. And that's okay, because a lot of the tactics that you'll actually be doing like interviews, surveys, usability tests, which are just, you know, kind of walking people through your DAP and seeing how they respond and giving them direction. Those are fairly accessible to pick up with some study of the methodology. Also, a lot of some of the best people in the world at user research actually started from a founder point of view. So you're set up pretty well. When you get to maybe like 10, 15, 20, 25 people, probably want your designer or PM, you hopefully have a designer to spend part of their time doing that it maps pretty well to their skill set. And if you're if you're really in a crunch, I think two other skill sets that could map well to user research are one, maybe like a marketer that's really interested in marketing research and really cares about the user, or perhaps someone on the business side that used to be a consultant that cares a lot. Usually the kind of research a lot and deliver actionable insights within a time constraint kind of method helps consultants transition over. If you have a few more people, you might be ready to make your first full time hire. So obviously it would be nice to get someone with both the academic background, usually some grad degree and one of the experimental social sciences, and also the practical background having some experience at a tech company. But that's pretty hard to find. My opinion on two kind of requirements for a full time research hire that you're making if you're on a bit of a budget are one, make sure they have some sort of UX portfolio, you know, similar to designers, a lot of aspiring UX researchers often put together a collection of articles or they walk through the ideation process, they walk through user interviews they did, they document, you know, their thought process and that's helpful just because you could actually evaluate their aptitude. And also because it's just proof of commitment as opposed to just some theoretical thing that like, hey, I'm like a product manager and I might want to do research and I'll convince you in the interview. And then also some professional work experience, I think would help just because you need this person to execute for you now or your DAP might be done. And often people who have been in a professional work setting are just more familiar with the types of constraints that they have to work under. It's not a hard rule, but those are my personal opinions. And then finally, if you have the money, for example, if you have ICO funds and hopefully they haven't been siphoned off through a smart contract by anonymous CEO, you I think it'd be good to poach an experienced user researcher from a tech company or design consultancy, especially if you're doing a lot of upfront idea validation and generative research. Because in terms of evaluating concepts from scratch, it's really hard to beat a lot of actual practical experience. All right, so okay, we could find someone, but it's expensive still. I don't have money for all of these things that you need for research. Well, you really don't need much. So this is just a quick example of some of the tools. And pretty much all of these are free. A lot of these are even just Google sweet products that you could use for free. We at Quartz use a good chunk of these. And don't you don't have to like memorize these. I'll find some way to share slides after. But I think the idea is that there's really nothing stopping you from right after this talk, setting up your own user research stack. It's just screen sharing, video conferencing, survey tools, actually scheduling is the most is one of the most important ones here. If you share your Google Calendar link, the other person can kind of ask ask you for a meeting or propose a time and scheduling actually is a big barrier. But there should really be nothing stopping you in terms of actual tools from doing any user research. And this is just an example. Like when I sometimes when I do mobile remote usability tests, I just plug in my I asked them to plug their phone into their Mac if they have a Mac, two clicks, and they're mirroring their phone instantaneously using QuickTime on their laptop. So they're just screen sharing. All right. Great. So all that's great. But I'm also just I'm just a developer. I'm just a founder. I don't really know much about what to actually do. Do I just talk to people? Well, that's what we have the rest of this presentation for a bunch of things that you could do at each stage of the journey. And I might optimize a bit more just I might go a little fast just to make sure I cover everything. But at the end, I'll make sure to provide my contact information and we could kind of talk through anything. I'm happy to answer any questions. So first, you just want to start small, not big. I want to make the distinction between protocols, which I think is good to think super, super into the future top down, how is this going to survive in 50 years governance? That's great. But if you're a DAP, which is just an app with some smart contract component, you should smart small, just like regular tech companies. Pretty much every giant tech company started with a super small slice of a pie. Airbnb literally was just air mattresses at just conferences, then went to just air mattresses, then went to, you know, renting people's apartments. Now they're doing luxury travel and experiences and concerts. Same with Uber started with just luxury limo cars, then all types of cars, then autonomous vehicles. And the reason why that is is because when you win a small slice, you get a unique vantage point into the entire kind of pie of that industry as one of the winners to see the other opportunities in that pie that no one else can. You've learned the best practices. You've learned the industry better than everyone else. You have a well oiled team with great operations. Every one of your people in their field is an expert in their particular duty. So you're just set up really well to dominate more and more of that pie. And that's kind of the mentality that I think would be beneficial. And just a quick example. So this entrepreneur I know, I don't know him very well, honestly. He actually was never really a crypto guy. I think he, he's in his mid 20s and it sold the e-commerce startup to e-bates for a modest before hand. But it's a super simple experience where you just download a Chrome extension. And if I want to buy something at one of these retailers, and there's a lot of retailers covered, a little pop-up comes up. I click it. Great. It's activated. I go through my normal checkout flow. And then two days later, I just have Bitcoin in my wallet. And there they have custodianship of the wallet. And I believe you could withdraw after a certain amount to either a bank account or you're like a like a Bitcoin wallet. But I don't know. I mean, it's this is I think a good example of starting small. And for what it's worth, even doesn't matter too much. They have a lot of strong investors on their side, a good amount of funding and they seem to have good adoption. So I feel like, you know, there's really nothing separating any of you from being able to just really tackle very real problems. Yeah, you want to do lots and lots of interviews upfront. Depth is more important than breadth, I think. And you don't have to quite go so far. But I did want to highlight a cool example that I saw of a stablecoin company, Selo, who went all the way to Kenya to do user research on the ground to understand how Kenyans use Mpeza, which is mobile money that they could send really easily over your phone to other phones. It's extremely, extremely widespread and popular in Kenya. And for example, I thought interesting insight was Mpeza kind of spread adoption wise, not by people looking it up and, you know, kind of trying to get it or download it or whatnot. But through other people in the village that people knew, teaching them physically and showing them how to use Mpeza is a very like local growth that way. So yeah, just sometimes you really need to just talk to people to understand the nuances that really define the ways that people think and act about certain products. And the tough part of this stage, especially for dApps, is that at this point, you probably have the least resources. Like even if you're funded, this is probably a pre-funding point. But research is the most important stage at this point, just because of the large amount of time it takes to create a set of secure and well tested smart contracts. So it's kind of a lot of pressure. But at the same time, a lot of the tools that you're going to use are relatively straightforward for you to pick up. And as long as you focus and try, I think that will be a big improvement in and of itself. So when we talk about design and prototyping, I actually went to a pretty great workshop yesterday by these designers named Samarin Hanny who walked through a few great prototyping strategies. But you basically want to use interactive prototypes or kind of a web mobile MVP, no blockchain at all, just use a centralized server with some dummy data, just to get something in front of people that resembles what they would actually be using. So if people didn't like your idea in the interviews, you could show this to them. And this should be a fairly representative sense of this is a real problem that they're having or not. And you want to get as many users as possible on it. So briefly walking through an example of what exactly usability test is for those of you who don't know. So at Quartz, we launched this product, which was a story series with 180 degree video called Machines with Brains. And we decided to be pretty experimental in terms of launching it. You know, you kind of scrolled down in your article amongst different cards to read your article and swiped left and right to get to other stories. We thought it was intuitive, but we also had a big time crunch. And we heard some bad anecdotes upon launch. And so we conducted some quick usability tests. We found that the large blank spaces at the end of cards made people think that the story had ended. And so they just left. We also found that scrolling card to card because they were different heights was not very intuitive and people felt like their scroll was getting hijacked. And we also found that people were accidentally swiping horizontally when they might meant to be swiping vertically, probably because they're swiping at a bit of a diagonal angle. And so we quickly kind of enacted fixes to these solutions. So usability fixes, these types of things can really help you in terms of your product. You get a lot of great information upfront. You want to design for the least blockchain savvy and technical version of your user possible. And so I actually wrote an article about this a while before. The use case, we did a hackathon. The problem was build a farm to table local supply chain DAP built it on hyper ledger. And they mentioned that one of the users was this old 70 year old farmer that doesn't know how to use email. So we kind of designed with that in mind, made it super simple. So this applies to really everyone. Like if you're building prediction markets, you probably want non blockchain experts to use it at some point. So you should probably try to optimize and design a bit more for like a regular kind of gambler. Even if you're building developer products, you probably want developers who don't know that much about blockchain now to kind of onboard onto blockchain and use your product. So you want to make it as block blockchain jargon free as possible, at least in some of the main areas, just so it's really easy for them to onboard. Since actually a lot of develop, there's a lot of really skeptical developers for blockchain and crypto out there. And even crypto products, honestly, there's a lot of people who trade crypto and own crypto don't really know how it works. So you just always want to optimize for a little less technical and a little less blockchain savvy than what you might expect. And yeah, so you, you got to, you got to test your copy, both product copy and marketing copy for product copy. Product copy is always not as clear as we think. And don't use lorem ipsum use real copy in the context of the product for marketing copy. Even for like developer products. So, you know, these are some really complex developer products with some really, really simple marketing messaging. Stripe, Watson, Twilio. I know we're all proud of the technology we built and want to show it. And you also want to impress other engineers and investors, but especially on your main homepage, you know, permissionless decentralized credit market algorithmic efficient money markets. It's just kind of not that necessary. You're just not optimizing for as many people as you could be. Even financial products, you know, fidelity, it's not like they're a fast moving startup. Credit karma, wealth front, you notice that they're focusing on value propositions like what the end result of what this product will do for you, how you will benefit as opposed to here are all the small technical things that we have. And you know, for marketing to global audiences, I'll just kind of let this slide speak for itself. But please just, you know, even just through some through the crypto network or just talk to a few people. Translations can go really wrong. So yeah, just talk to people. And I would highly recommend that you don't move past this phase unless you have users in the bank. And what I mean by that is having a bunch of users that are kind of you interview them or you do testing with them. They're so into the idea they immediately see the value this would bring to them. They're bugging you every week asking you when your products going to launch. They're talking about other people kind of in their field that meet their profile. They really, really want it. And you kind of have to be honest with yourself here. But once you get into the smart contract development phase, you start to get to some, you know, really heavy upfront costs. So you really want all of the work or most of the validation work to be in these first two steps. And then when you get to the test net phase, you know, test net's mainly just to test out your smart contract functionality. By this point, your idea and your actual product experience should have been vetted in the previous steps. You will want to make sure you just explain in a simple way what a test net is to all users, even though it has a word test in it, it's not that obvious exactly how the components work. For example, on MetaMask, on your test that it still shows a dollar value of the ether that you have in your test wallet, which is kind of confusing because when you're looking at a wallet, you usually and you see money, it was like, I was at like money that I have. Just make sure that you explain what it is. I would recommend that you have basic event tracking, like the funnel, major actions, user sessions, Google Analytics is free. You could just use that. There's a ton of documentation. And if you really care about user privacy, just don't associate sessions with any particular user identifiers. I just I'm not sure how you could realize if you've hit success or realize how close you are to success without tracking basic metrics. And one idea I have is to and that we've, you know, I've seen a lot is to use support messaging products on your DAP. We're actually in the middle of installing one at Quartz right now. You know, I'm sure most of you have seen this. This is intercom. You see a little chat window. You click it, it pops up. You could immediately start chatting with customer service or you could read a few different resources. I think this is a great way of just setting kind of a set it and forget it automated way to catch people in who go to your DAP and get some feedback from them. Because you really remember this is gorilla user research. You need to use every tactic imaginable at your fingertips to try to get any, any feedback at all. When you get to the mainnet launch, another example of an automated feedback tool that you could use are in-app surveys. So in-app surveys actually intercom has a plug-in for in-app surveys. There's some other vendors like SurveyMonkey, Instabug, et cetera. Just a quick sample that you could do, you know, hey, this is really important. How'd you find out about this DAP? Why are you interested in trying it? Provide some free form feedback options and then collect their contact information for follow-ups. Just tell them that we won't associate it with your wallet, like your product data, but you need the ability to follow up with these people to really understand some of their use cases or pain points or problems. This is probably like the most relevant set of users you could get information on. So you really need their contact information. Obviously, you could go really extreme. So it's well documented that Facebook would send in-apps, would show in-app surveys or web surveys asking people if they cared, if they thought that Facebook cared about its users. It is a bit of a blunt force technique, but I have to say, I, you know, while I was there, I looked at a lot of that data and it mapped pretty well to how people felt about Facebook. So it was valuable. Be desperate for feedback. Aggressively scour the web social for anyone who has used your products. Similar themes. Don't be too proud to pivot. At this point, if you realize that your idea isn't something that people want, you've already learned a lot about how dApps work, how blockchain works. Be honest with yourself. Don't do the, you know, everything else is great, but the throughput. So I'll just wait two years for sharding or it's just metamask. That's a problem. I'll just wait a few months for delegated transactions for more people to download while it connects. You should probably be solving a problem that, in spite of everything that's happening right now, some people at least want, really, really want to use it. So if it doesn't meet that bar, I would just really, really stop. So that's that. Here's some of my contact information. Any questions? I'm seriously happy to answer them. There's also just a lot of talented people at this conference that could answer some of these questions better than me. And yeah, I could send slides to anyone or I could find a way to post them. OK, thanks everyone for Kevin.