 Okay. Hi everyone. So this talk is about user testing. So thank you so much for coming. Okay. So I'm Jen. I'm a UI designer and usually I my job is laying out screens and working on workflows and things of that nature but something I've become very interested in is user testing and user research. I design board games and as a hobby and a big part of that is user testing and play testing. So that's the way I originally got into it but at my job I've done a few micro tests and a user interview so I've started to learn more about the user research world. Okay. So we're going to be specifically talking about micro tests today which are short simple user tests but there's some other options if you are hoping to gather user feedback on a product you're working on. So some examples of that are you could write a survey, you can do user interviews or you can do a more formal test. So just before we get dive into it I'm gonna give you some pointers about how if you wanted to run and test yourself how you might be able to throw one together but it's always a good idea if you know anyone who has a background in this to ask them for help. I personally have one of my co-workers is really knowledgeable about this and she helped me understand the data that I got because if you're not really a statistician it can be difficult to you might come to an incorrect conclusion so it's always good to ask an expert. Okay so what is a user test? A user test is a way of testing how well your products is usable by by people who you hope to use at your users and the three the three terms that people like to use are usable, learnable and satisfactory so that's kind of what you're testing for. So what is a micro test? A micro test is a simplified user test. You have fewer participants and the main the main goal that you would be looking for is early problem identification so the idea is that you would do this early on in the development process and find out if there's any large problems that you should address before you start getting really deep into development. So you can use a micro test to test any sort of products hardware software even a workflow or an experience so you can use this for all sorts of things not just software. So okay I think there's a lot of designers in here but I think it's really important to emphasize why is it important to test software with users? Because who are we developing software for? People! We're not developing software for robots. We need to put it in front of people to see how people react because you never know how people are gonna react because people are weird. Okay so test early that's a big part of the software development process in the boss community which is really great but I feel that the user tests and putting things in front of users is often left until it's released and so user testing is something that you can have the same ideology for. So and the big reason for this is the whole sum cost fallacy because people are a lot less likely to go back and try to make changes if they've already put in a thousand hours into building it. If you build this beautiful feature it is really complicated and then it turns out that it's just super unusable. It was really difficult to make and the engineering behind it is beautiful and amazing but no one wants to use your feature. You haven't really built anything very good have you? So it's better to do this early before you put in the effort you're much more likely to get a better product and be willing to be agile and make changes. Okay so for this talk we're using the example of fedora land so there is an amusement park and if they have an app and the app helps users locate rides within the park so this whole talk is going to pretend that we have this app and we're going to write a user test for it together. So the point of this app is that it helps the patrons find different rides in the park. It shows the location of the rides on the map and it shows the user's location and we can imagine that they'll be walking around this park and we're gonna be testing this Android app with them. Okay so selecting testers who's going to represent the user? So who are the final users of the products? What is their level of domain knowledge? Do they know a lot of information or do they have background on the scenario that you're going to be putting them in? Have they use other apps that are similar? Have they use products like yours before? So what you do is depending on the type of test you're going to be running you can create some criteria based on your goal. Okay so for this example for fedora land the minimal requirement is that a tester is unfamiliar with the park. At the very least you don't want to be testing with someone who's very knowledgeable about where the rides are because then you won't be able to tell if they they're just finding a ride space on their own personal knowledge or if they're using your app. So that would be the minimal requirement. A better requirement would be can we find some testers who might actually visit the park and might use the app? Some people with smartphones so you might try to hone in on that. Then once you've found a group of people who fit that criteria it's a good idea to try to find some diversity within that group. So for example you might want to find people from different demographics maybe a parent with their small kids or a teenager someone who is in a wheelchair and finding their way around the park. So you're giving you're you're using your you're testing your app with a with different groups you might find different problems. Okay and also in addition to that you don't always have to test the entire population who might use your product. You can do targeted user testing which would be choosing a subsection of the population. So let's say for Fedora land I only wanted to test if this helps people with people with parents with kids and I wanted to see if how that population was able to interact with the app. I decided I didn't care about if teenagers could interact with the app and so I could run a test just focusing on that population. So in my personal experience techies can make really bad testers and I think it's something that's really not people don't realize until they've experienced it but this is something that I've run into in a few of my tests. It's very funny if you a lot of times people get very caught off in the technology that you're testing and rather than act as a user they'll ask you a lot of questions about how it was built and what the hardware is and stuff that the average user likely won't have to ask. That's what the caveat to that is if you're designing something for a technical person they should be the person you're testing it with. But a funny example of this was in the show Silicon Valley I don't know if anyone watches that but one of the main characters sent his product out to all of his friends and everyone said wow this is the most amazing product ever it's gonna do great and then they released it and everyone hated it. A ton of people downloaded it but everyone stopped using it immediately turns out the reason is he sent his product to a bunch of people who have the background knowledge that he did he sent it he basically tested his product with his demographic so it was totally a useless test and it fails and it's all funny but the point is if you're just testing your software with people who are exactly like you you're not actually learning anything so it's a good idea to put it in front of the average user. So how many testers should you have? For a micro test for a small problem identification test by users can be enough. Ten is great. Obviously the more people you test with the more likely you are to have correct results that said since micro tests are more about problem identification you don't need a hundred people to tell you you have a problem. Once you see it the first time you have an idea you have the problem once you see it a third time you're pretty sure okay this feature needs to be reworked once three people get caught on it you don't need a hundred people to tell you. So interacting with testers when you're working with testers it's really important that you you're trying to run the test if you are not there. Unfortunately it's usually not possible to do that but it you want to the perfect world you wouldn't even be there so you don't want to influence the test and a big part of that is staying neutral and that is with neutral language I don't know how many people have experience with neutral language I'm going to throw out an example for you guys and I want to see if you audience participation if you could pick out which is the neutral phrasing of this question. So first phrasing A do you like this presentation phrasing B do you find this presentation boring phrasing C how do you feel about this presentation okay which is the one that did not lead the audience yeah three okay so yes you want to be phrasing things in a way that doesn't leave your tester so if you keep saying things like how do you like this feature the word like implies a positive attribute to the feature you say oh did you get confused you're implying that they they got confused and it was difficult so it's better to try to phrase things as neutrally as possible. Also it's just a good idea not to emphasize that you were involved in creating a product if you usually testers aren't even people who are involved in the software development process usually they are their third parties but in this case you you're making a product you want to test it it's totally fine to do that but you just don't want to keep emphasizing like oh I have to go back and develop change this feature now because people will be less comfortable giving you negative feedback and you want negative feedback okay so designing a user test let's start with this example that we were talking about and we'll work backwards so once you have a user test you will have a packet of this thing you've written and this is an example of a page from one okay so we're going to do the first task of the fedora lands app test okay so the first task select a roller coaster so I give my everything green is stuff that you would read out loud to the testers and everything else is just notes that the user would be the tester would be referencing so selected roller coaster is the task that we're doing the overall task on the setup we would let's say we restart the app and make sure it opens on the landing page of the app so then the task instructions which you would read out loud to the user you are visiting fedora lands and are excited to ride the roller coasters the first coaster you want to ride is hopper kitty show me how you would view this coaster on the map and then for yourself you have these different ways that they could complete that that action so let's say within this app there is two different methods two different workloads they could go down to complete that task they can do it in two steps here or they can open the hamburger menu and do it in a few other steps what you'll be doing is recording what they do each of their tasks each of their tax on the screen and then after it's usually a good idea to ask them how easy or difficult we'll go into the rest of it later but basically the first task is a good idea to give the user background kind of give them a setting tell them who they are and what the scenario is and keep it simple for the first one so this one was literally just I'm gonna select something on that it's not trying to find anything yet it's not trying to dive deeper it's just you know getting them comfortable and teaching them what the flow of the test will be like okay so with user testing isn't really it's really important to be specific you know you're not just giving throwing the app at a user and watching them use it that's something you can do but it's not necessarily what we're trying to do here we're trying to be a little more scientific about it think of this as an experiment and you want it to be repeatable also you don't want to influence it's really important to think about influencing the test accidentally by variability so if you want to you know get rid of as much variability as possible okay so yeah okay so every time you make a test you're going to be thinking of an overall goal for that my process so what is the main thing that you're trying to learn by putting your app in front of users so an example for the example we're doing I was thinking about with the pivotal feature of the app was and was there any features that I thought might be confusing with the app and those and with that knowledge I wrote out a whole so for the goal for this one is kind of user with no knowledge of the doorland find a ride by using the map app that's a very specific thing I'm trying to learn I'm just trying to see if the app is useful for them in navigating I'm not asking about other things that said we're going to be breaking every goal you have to break it into objectives so objectives are the steps that takes to build to the larger goal so navigating the park might be is one thing that is built together by a lot of complicated steps so it's one kind of more complicated goal that's built by doing more simple steps excuse me so okay they're basically the basis of the tasks are going to walk through and it's the workflow of how you achieve that overall goal so breaking down the goal we talked about the doorland objectives would be can a user select a specific ride and find it using the map can the users switch between different rides on the map so those are two things that are going to be important for a user to be able to achieve the goal of the test so we want to make sure they can do those smaller tasks okay so secondary questions you know these are not the main point of the test though while we have the app in the hands of users these are things we want to look for and just check on and see how if those features are resonating with users some questions that are just in the back of my head that I want to answer early on that might affect future development so some of the secondary questions for this fedora land app are what are the users preferred map orientation do they know how to access details about the ride how long does it take them to find a ride do they use the zoom feature so these are things that I might not write a whole scenario around but I'll be paying attention to when they're performing the scenario and kind of marking down if they do those things so we've been talking about scenarios those are the questions the things you'll read aloud to the user and the tasks that they're going to walk you through so and these are written based on your objectives so the smaller parts of your goal the portions that the workflow that builds to your goal so you know you keep it simple at first and you work up some more complicated tasks okay so my objectives here were kind of users like a specific ride and find it using the map can we use your switch between different rides on the map and then the scenario I wrote based on those objectives where you hear the hyper kitty has a really long line but that the tux loom is empty you want to head to tux loom show me how you would view this new ride on the map so now I'm learning if they're able to switch between different rides on the map and then the next scenario might be please take me to tux loom and so then you might follow them around the park and and see if they're able to so ease of use is a temperature check that it's a really good idea to do after every scenario every time the user performs a task and they've given you any feedback a way to start the conversation essay how easy or difficult was it for you to perform this task and then you have a rating scale from easy to difficult and they rate it and then you can ask them to talk more about why they rated it and so you just kind of can take notes and and also over time you'll see how different how many people rated the same tasks differently you'll kind of get an average of the ease of use it's not technically not statistically accurate but it gives you an idea yes so I think I'll show it later but they're the ideas that you've written this whole document of the test things that you will read aloud things that the questions you're going to ask and then also notes for yourself of how the app needs to be set up before each scenario and one way of doing it is printing out a packet and that'll be your note-taking packet and each each time you have any user you'll print out a new packet and you're able to you'll be marking directly on that so I'll show you I'll give you some specifics for that later but for example if I got up to the usability scale for one of the tasks I would just circle like let's say they said someone easy I would circle it and make a note right underneath okay so in addition to the general temperature check there might also be some open-ended questions that you want to ask your user so some of them might be probing feedback questions specific to the tasks they just completed but it's always a good idea even if you don't do them do have specific questions it's always a good idea to end with the question is there anything else you would like to share about the task they might say nothing but they might give you a little more information and might start further conversation remember these should be phrased neutrally like whoever before so did it occur to you that you could zoom in and out of the map that's a neutrally phrased question how helpful was the directional arrow when finding the ride that is one that is leading because you're using the word helpful a better way if you wanted to neutralize that question you might say how helpful or unhelpful was the directional arrow if you're going to use a positive it's always good to throw in the negative but if you can just remove any positive or negative words from your questions that's always better you know at least look in here I did a test with a similar scenario and I accidentally did ask how helpful what it happens okay so I'm just I'm gonna go over this kind of quickly just to let you know there's a thing called sauce you can do more research on it later it's it's a ten item questionnaire that you ask at the end of the usability test it gives you a number at the end it's not a percentage but it'll give you a number that you can base on other usability tests to give yourself an idea are you above average usability or you will low average usability the questions are crazed and it's also kind of similar to the ease of use scale where they're answering these questions on a scale of strongly disagree to strongly agree it's questions like I would like to use this system frequently I found the system unnecessarily complex things like that and so they'll rate each of those and you can put it into a calculator and it'll give you a number back I think you're looking for above 70 above 68 is considered above average it's it's a it's useful I but you don't just because you get above 78 or 62 or whatever it doesn't mean that your product is usable necessarily it means it's above average by by these metrics but there can still be problems with it so it's one thing to use amongst many other things okay so running the test so we talked about this a little already but you'll be writing a script that you follow with each user which will eliminate any variability you want to be you want to make sure that you're saying the same words over and over again it also reduces the chance that if a user behaves differently from one test to another that it was because you influence them it's probably just because users are different people so the good thing a first thing to do for your script is have an introduction it sets expectations for what the test will be like if they've never done a test before and you know kind of sets the tone and the speaking style and makes it comfortable and gives yourself an opportunity to answer any questions they might have so I'm not going to read along a whole long but thank you for giving us some of your time today to improve our products we like to obtain feedback from end users during development that is the purpose of today's activity and you would keep going and say today we're testing for the doorlands app you know this app is supposed to help you navigate the park we're looking to see your thoughts and expectations around this app so you would you would write out a thing describing what the goal of today was so the test packet that includes your script the setup for each task the steps to complete the task and whether they've passed or failed either of those these are used scale and on space to answer any open-ended questions so you'll be taking a lot of notes during your tests you're going to be encouraging the tester to think out loud because sometimes people might just you know be walking through the task and you can watch them but it's more helpful if they're saying what they're they're surprised or if they get caught on something kind of probe them and see what's bothering them or what they don't understand and you can take notes on what they're saying so this is another example of what a page from a packet might look like and the red is examples of notes that I might have written so this instruction we went over this one already it was a task where they you had the switch from the touch spoon to the touch spoon so let's say these are the two different methods that they could have used to succeed at the app they let's say they went down this path so they select the correct right I circle yes they did that they did this I circle yes was there any assists I say zero I'm going to go on to assist in a moment and then I asked them the usability scale they rated it to I circle to when I asked them if there's anything else they said they wanted a back button so I took a note of that so anything that they were kind of saying throughout the task I might have noted on this page next to the different steps okay so assists are kind of confusing I think they can be difficult but the idea is that you're testing you're not testing the user you always want to let the user know that I'm not testing you I'm testing the app but you're you're testing to see if they succeed at different tasks so you kind of put down a scale of cast and fail so if you are a middle task and the user is taking a really long time to complete it and you can tell that they've gone down an incorrect path and if there's no way that they're going to come back and do the task correctly you might want to assist them and put them back on track but if you do that you need to make a note of it because they wouldn't have been able to complete that task by themselves so also if they give up or unable to continue yes yes so you would add a you might want to make a note if they eventually are able to do it but you would give them a fail for that because the idea is to see if they're able to do it on their own if you weren't there so if you have to help them they've they failed it and and that's for you to say later okay this feature might have been confusing so I might have to go back and do it and you want to make note of the number and the type of assists so let's say they were looking at the map but they got confused and were they had selected the wrong ride and they started walking towards the incorrect ride they had failed the task is it worth it for me to let them walk across the park and and fail before I like tell them no it's probably okay for me to stop and assist them and put them back on track but I just needed note that they would have failed does that make sense okay okay I realized I missed with the the neutral talk before we go to this is another thing to worry about we talked about neutral language neutral body language is another thing to to think about when you're testing users you don't want to accidentally point at the screen and kind of show them hints of where they're supposed to tap if you're doing a test that involves walking like this task with because it's a navigating app you don't want to walk in front of them and kind of pinch which direction they should be moving in that can be very difficult you have to just kind of stand behind them and try to be a neutral party also you can't show any emotion based on their ability to complete or not complete a task you don't want them to feel like they're failing you you know you don't want that you're supposed to be a robot you're not even supposed to be there so you don't want to make them feel good or bad about their ability to complete the task it's really your testing the software's ability to do it so okay so evaluating the test so now we've walked through all these things and I have let's say I've had five testers I've recorded all their ease of use and all their feedback and their build their passes and fails so now I just have a ton of data what am I going to do with it so this probably could be a whole talk about making spreadsheets for data but I'm going to go into detail with the idea a way of doing it would be to set up a spreadsheet and for each column write the task and the steps that would take to complete the task and just reported they passed or failed each task and then you know see the average see if certain tasks end up having more fails and others have more passes and you can start to pinpoint and get an idea of which tasks have a lot of fails let's say one task 50 percent of people failed or needed to have help to complete the task you know that you need to rethink that feature so you want to look for trends like I was just talking about and also you want to see if there's any repeated user comments so maybe everyone was able to complete a task but three people mentioned that they were confused efforts by a certain icon you might want to you know you took a note of it they were able to do it but you want to look into that feature also you might notice that some tasks have really high ease of use and people totally were able at 100% of your participants were able to complete that task and they already did it highly you know you probably shouldn't touch that it's probably good that's it also an important thing to know because sometimes when you're developing things you don't really know what's successful or not successful you don't want to go in and put your finger somewhere where it's actually perfect you should leave it also you can check if you're so high yourself so based on that trends you can take note it you have successful features and pain points that you need to rethink so we'll have all this data and now it's time to take your favorite brainstorming method and come with all the developers and designers and come up with a solution thank you I want to this take I mean any questions and also if you had any products that you wanted to maybe do a micro test for and you want to talk about how you go back so I think the difference between a micro test and a larger test is the scope so it's you're still going through a lot of the steps you do for a normal test you're still very thorough but you're doing it for a smaller subset of features maybe you're only concentrating on one main feature also you're doing it earlier in the process with the goal of problem identification as opposed to later in the process with the goal of general usability also you probably have fewer participants so I think a larger test usually people look for 20 participants or more in this case maybe you only have five so you can I think that would depend on your personal I was hoping to get new people who might not be super familiar with testing to give it a try and if all of this background like reporting and you know having all this large setup you don't need it it could be helpful for looking back at the test but it's not requirement also for the online thing I think you absolutely absolutely could do it online with this shared screen might be a little more difficult but I could probably get the same information yeah I think a big part of testing is using the user into your app they might likely never used it before so and also forgetting using them into using the app you're also using them into the testing process so they've never been a part of the test before you want to give them an idea of okay first I will give you a scenario you will do the task I will ask you ease of use and so they kind of get the steps of each task and it's it's better to do that I mean the first one is almost a throw away it's usually a task that doesn't even matter a whole lot I mean it's probably something so easy that you would assume that a user would be able to complete it but it's just a way of easing them in and then yeah I do like to build up because sometimes you're you're looking to see into the specifics that they're able to complete the smaller the smaller tasks that build up to that larger goal so I usually want to stop them between each of that smaller tasks so even if the task was clicking the hamburger menu selecting something else and then bringing it up on the screen I want to stop them and make sure that works for them and there's any issues in that before we go on to the larger test which is following them around the park to see if they can find the roller coaster any other questions does anyone have any products that they were thinking of testing okay well thank you that one more quickly than I thought if anyone ever wants to contact me if they need help with testing they want to run a test on my information up here I also have some cards if you're interested and I would love to help you if you have any further questions thanks