 So, apologies, I don't see my presenter notes. Okay. So, an app that is a nightmare to use, or spending your own time in hours and like efforts to develop something that then is not used by people, or just building something and then looking at it and it requires hours to support. So, and this is really the cost of not doing user research. So, my name is Elena Bulahina and I am a UX consultant in an agency called the Credible and I'm doing mostly user research now. I just wrapped a very intense week of interviewing, so I will be a bit low today. But the point is that I'm going to tell you a story from another time. When the project I was working on suddenly got big and so October 2011, two things happened. Coldplay released their fifth album and Google replaced the native sharing in Google Reader with that Google plus abomination. And I was furious and I was so sad because to me, it was the part of the internet that was just worse being in. Because when people shared the best bits of their RSS feeds, and then you could have a meaningful or somewhat meaningful discussion around that. And the whole population of Iran was me on that one because for them, it had even larger consequences. They were using RSS as the ways to get unfiltered and uncensored access to the news. And it was just sad. Somebody was building some whatever replacement by then, but six months in, nothing was happening. I was still sad and I had this brilliant, brilliant idea. We should build the RSS reader for ourselves. And the first person I recruited for that was my husband who happens to be a brilliant infrastructure engineer. And he said yes in hopes to never hear about this again. And then the second person I pitched this idea to was my good friend who was 19 at the time and he was very upset that he still hasn't achieved greatness yet. So I promised him an amazing project and three of us went. And then next two years, they honestly feel like a blur to me right now. But we managed to do some design work and we set up infrastructure, we get it working, we got our hundred users and then 500 users and then we got 5,000 people from Brazil. They're amazing by the way. And then we got to 10,000 of users and then something happened. Any idea what was that? Google closed the reader and suddenly we found ourselves on a very, very different scale operation. Suddenly nothing we built was supporting the amount of people who came to the product. Suddenly every minute or like 10 minutes of downtime were accompanied by hundreds of tweets, angry emails, things like that. And then a lot of our design decisions, because I was doing the design, were not supporting the scale at which we were operating. This is how our numbers looked roughly before and after. Those are the 10k users I was so proud about. And then it was so often trying to do that by ourselves for half a year, getting a very severe burnout. We, the old reader was acquired and I'm very happy to say that it's thriving as much as a reader can be thriving in 2019. But this is just for the context of the scale we were operating and while user research is even more important in the scale. So our user research approach, to be honest, there was none. And when I say that is what I mean is that if you take any of the existing design or research methodologies and map it against what we did, you will see a lot of white spots everywhere. So mistake number one, we didn't do discovery. And discovery is a stage where you learn, where you explore the problem space, where you learn everything about your users or potential users, the context they operate in. And my discovery approach was to take a look at all the existing readers, then maybe speak to 10 errors as junkies, you know the people who subscribe to like 2000 feeds and then they say they read all of them. And also to then be very, very heavily inspired, as I say, by the Google reader. On one hand, it could be worse. But on the other hand, it wasn't the right approach to take because we used the discovery findings for validation and they should never be used for validation like that. And we just took our assumptions, kind of validated them and progressed with them. But not talking to users in a field, by not figuring out what's even the value of RSS. We were never able to do this big strategic decisions that would be so beneficial for us a couple of years in when they were needed. And then we did not, mistake number two, we didn't try to understand the why behind the design that we did. So we set up a tool called User Voice, where people could submit their ideas and then vote for them and then kind of used it for prioritization. And also we used Google Analytics to track the flows, to track like the bounce rates, you know, the usual stuff, quant stuff, right? And it kind of gave us the picture. But again, it wasn't the right approach because the most essential insights and the most important decisions they do happen behind those whys. And for example, people in User Voice, there was a very vocal minority of hardcore RSS users and they all voted for a feature called Book Marklets, where you could share the bits of the internet that were not in your like RSS feed into your reader, right? And it was voted as the most wanted feature. And we took it and we spent hours doing that. Iterations are like not getting right. Lots of developmental effort as well. And it was done by like one guy. And in the end, you know what? It was used by less than 10% of actual users of our core audience. They kind of didn't care. And then imagine if we would just do it right. And if we just got what is important in the sharing. And then we would never start with a Book Marklet. And honestly, it might not even be in our roadmap at all. And finally, and this is the value of user research. So like this picture is from Twitter. And like, you know what? If you have these balls fall by the beginning of the day. And by the end of the day, you come to three empty balls. Are you sure that you understand the process that happens in between? Because most likely you not. And this is why it's essential to understand the why. It's essential also to test with the real users, which was our next mistake. We did not test with the real users. And looking back, I don't know why. So we didn't have the resources. And then we were spreading ourselves so thin that it was never in top of our priorities. But let me just talk what we missed there. We missed usability issues. And we had to fix them on the go. So imagine bad user experience. And then imagine even worse user experience because somewhere there's a team of people trying to fix that. This is just not a viable approach to take. And again, testing with users will uncover those little, small, tiny insights that in the end will be built up into this beautiful, beautiful golden nugget of research. So by now you probably are persuaded, or partly persuaded, that user research has some value. Who here knows about this whole shoot designers code thing? Anyone? No? OK. My friends know. Good. So a bit of context. Last three years, the internet has a lot of think pieces, medium posts, and rants about whether designers should be able to take on some developmental tasks or just understanding computational thinking is enough, if yes to what level, stuff like that. I don't think that this is an interesting question. This is an interesting question. Should developers research? So I think that everybody who is building something here in this room should. And because essentially if you're one person working on something, or if you're a team of three people, two people, five people, you realistically, you're just one choice. It's between not doing any research at all and doing something by yourself. And in this case, I will always, always advocate for doing something by yourself. Because essentially user research is not science, even though we kind of use scientific methods. And user research definitely is not art or design, even though both of them are involved a bit. It is risk minimizing. And if you're building something, you need to have confidence in the thing you're building. And you do it by research. So even some research is still better than no research at all. So some of the things I wish I knew six years ago. How do we usually do it? The plan is super simple. First of all, get the people, sort your research questions, figure out the method, then prepare, run, analyze, and then do it until you get the picture. I'll start with the users. So where could we get the users for research? It's a very easy answer to that. If you're building something, it's for somebody, right? Find who your users are, where they hang out, and talk to them there, record them there. We had user voice, we had Twitter comments, we have Tumblr blog posts, we had emails, we had people just reaching out for us, and we could just use them and schedule small, tiny interviews, and then have the clarity that was lacking in a lot of the decisions that we did. And what else is there? So find them, try to recruit them, and for those who are super, super new to this, we tend to compensate people for that time, and I understand that nobody has budget for anything. It's 2018, nobody has any money. But it could be Amazon vouchers, small ones. It could be charity donations, but careful with suggesting them in one sentence just like I did. People will opt for charity donations even if it's not their choice. So then how do you get to the research questions? It's very easy. Essentially, it's what you're trying to find out. So what do you want to learn? Then you transform those into research objectives and then think about how to meet those objectives. It is a very straightforward process and the more you practice it, the more you will be getting it right. So for example, what is a good research question? An example of a bad research, I'll start with a bad research question, right? So a bad research question in our case would be will our users like the old reader? That's like leading and also that's not really a research question. And what would be a good research question for discovery? Hmm, who are our users? What are the problems they're currently facing? What is the value of RSS? How do people currently read news? And again, what are the problems, obstacles and barriers they face when they do that? And then those could be translated into specific methods. So methods. The golden rule here is that methods should come from the research objectives and goals. And it's a beautiful, beautiful statement but reality always disagrees with it. And there's always time, there's always cost involved or scope that and so you can just never get it perfectly right. So we are entering the danger zone of my talk because I took all the methods and then I just left three of them that I think everybody should be able to practice in one sort or another. And those are user interviews, surveys and usability testing and also guerrilla research is an approach. I couldn't figure out whether to cross it out or not so I left it. So let's start with the interviews. Those are usually moderated discussions that may involve other methods but usually it's just you, your user, and you see it and you have a nice talk that partly feels like an interview but also in reality it should feel just like a dialogue. So if I were to describe them in one slide, I would say that. They're great for discovery, they're very useful for usability testing and if you were doing an interview right now, so first of all you have to practice listening because the other person will be doing speaking the most and so to quote Hamilton it would be like, talk less, smile more. Then write a guide, it will help you because the interview never goes right. So write a guide that will have the order of the questions, the flow of the interview but never use it verbatim because people will see when you're trying to read the questions. Then also beware of the leading and closed questions. This is what we are absolutely obsessed about in the researchers community that you should never ask something like which button will take you to a checkout because that is a leading question. But however asking something more generic, something more open, something like if you were to proceed right now, how you would do this, that is not a leading question. Also watch people's body language and this is something that is very hard to get right but people will say one thing and then they will be very tense. Most likely it means they mean something else and also diving with the right mindset to the interviews because you're not there to validate people or test them, you're there to learn everything about them and so usability testing is another, it's like a golden standard of what we do in user research. So prepare everything in advance, always do the pilot if you're insured to several pilots because this is where most obvious flows and most obvious things will just pop up. Again the same, listen much more than you will be speaking, do not prompt people and maintain impartiality. I cannot stress how important is the last one because you have to be there a bit like independent and if you're building this thing, you're so usually in love with that or you hate it so much then it's very hard to be impartial in this sense. And then the surveys, now those are very hard to get right because they look easy but in reality they don't and what I suggest to you is to use very focused, very small dedicated surveys for early product research. They do wonders because this is how you can learn about people's attitudes and behaviors and also try to figure out how prioritize things you're doing. Now you will have to do a lot of research on your own which means that Google, Google it. And there is a brilliant approach that other people are taking is Google Ventures and their approach to this. So just Google, find them on Google Ventures medium and that's all. And I'll skip a bit because it's five minutes left. So this thing, when running research again, those look like general good rules for life as well. So plan everything in advance, allow yourself more time, always plan the lunch break. I recently had a research session where I didn't plan it and I didn't have a lunch break, it was horrible. Then be flexible with your approach because you never know what will go wrong, everything will make go wrong. Hope for the best, expect the worst and find a body. Again, this is a good rule for life as well but to quote government digital services, user research is a team sport. You will need somebody to support you and also take notes. So let's recap. Oh, more slides. Great. So let's recap what we learned as how the reader team never did any user research and how sorry they are about that now and how you can not follow their steps and do some of the user research to help yourself get it right. So this is my last advice to everybody who may be inspired by this talk to do something by themselves. Allow yourselves not to get it right every time and also allow yourselves not to get user research right at all because when you talk to a researcher, they ask, we will be obsessed with like methodology and questions and perfect phrasing. And this is, and it's very easy then to be obsessed about maybe I'm not doing this right and this obsession stuck and then research feels overwhelming and it doesn't have to be because in reality it's, you're helping yourself. And again, if you come with an engineering mindset, you mostly thinking about quantitative and sample sizes and where is the statistical significance in this thing and again, we are not trying to prove anything there. We are not trying to extend the results on the population and we are definitely not trying to get published. What we are again trying to do is to minimize risks and maybe, maybe add some, it's a bit like gambling honestly. If I were to have a metaphor for the user research it will be gambling and trying to kind of help yourself from the future. So it's also very hard but I think we all should take some skills to practice so if you're building something, if you're thinking of user research in the end of the day, get out, find some people and talk to them, show them what you're building and maybe do a tiny interview. If this approach feels terrifying to you, I assure you it absolutely is. And as I still feel dead inside when I need to do this type of research when I need to talk to a person and show them some website and I'm in a cafe. But in the end of the day, people are pretty great and everybody wants, will be helpful. So, yeah, and having some data however small or however imperfect it will be is better than having no research at all.