 Welcome to Agile Roots 2010, sponsored by Version 1 Rally Software, Virio, Amirisys, Agile Alliance, and X-Mission Internet. Discount Usability Testing for Agile Peeps by Ben Terry. Okay, so this is discount usability testing for Agile Toots. Everybody hear me okay? I've been there. I work at Rally Software. I've been on Agile teams, often on different teams for probably been a little over eight years. So I've done just about everything that you can do on Agile. I'm certainly not a specialist in any area. I've done a lot of development, a lot of testing, a lot of architecture, a little bit of interaction design. I've been a product owner, I even did some sales work, I did work out so good. I've done quite a few things. I've also worked at, I believe Rally is the 14th company that I've worked at. I've been there for two years, it's the longest job I've ever had. It's a great company. I work in the services division at Rally. I don't build a tool. So I do that and I hope customers get started. Mostly working with new teams that are picking off, sometimes doing Agile team jobs. That's pretty much what I do. At least by day, by night, I have a holding company called Helber Kinshore for all my experiments. They will probably do, I'm glad you can share why I have it. But I've done a couple of different things. You've never seen a digital cruise that's an information radiator on top of a team's integration server. That was an event that we were talking about before. And a bunch of other little side projects. I'll show you one of them as we go through. I'll show you a little test of that a couple of hours ago. Just grabbing somebody to say, hey, come get this out. It's pretty bad, so I'll let you get it. So let me start off with a little bit of a story from a couple of years ago. I was working at a big healthcare software company. This was before I went to rally. This was about three years ago. I was working on this great team. It was a pretty big team. It started off as a waterfall project with about 300 people. That failed pretty bad, so we switched over to an Agile project, which had about maybe 100 people, which is pretty big to start with, right? But we had great developers. We had great testers. We had great user experience. We had some great partners over there working with us, too. It was a great app. So we were working on something that looked similar to this. It was an electronic record application. It was a very cool, very fun app. Again, great team. The groups that we brought in brought in some people that did usability testing. I had never done usability testing before. This time I was an architect. And pretty much what we did was they adapted to our Agile cycles. They came in and they would pull me to a room, and they would ask them to do a handful of things. We would watch and talk about what happened, and we would have that in the form of a backlog. That's pretty much all we did. It made a pretty big impact, I think, on the product that we were working on. Now, not long after that, this product went to go to market at a certain time, and it wasn't going to make it with the right amount of functionality that it needed. So they kind of hedged our bets. We bought another product from a different company that wasn't nearly as flashy and shiny. They kind of worked, although not totally, and it was meant for tablets, right? So it was meant for tablets and zebras. And that was the core differentiator of this product. You would see, in our minds, the physicians would walk around and they would use their tablets just like this. Well, the company that had built this application had a lot of subject minor expertise in house, but they didn't do any interaction with their users at all. Very rarely would they do anything other than go to trade shifts. So, obviously, well, it didn't turn out to be such a great company. What I found was, right away, if we brought this app in, we had a good employee. So I went out to a couple customer sites, and one place I went was in the middle of nowhere in Corbyn, Kentucky. Has anybody ever been there? I didn't think so. It's this really interesting small town where Kentucky Fried Chicken was started. Salt Lake. Was it a relative? The Salt Lake was the first KFC franchise. Oh, we got it. It's a lie, too. Colonel Sanders started it in Kentucky. It was turned into Kentucky Fried Chicken, and we got it with Pete Harmon in Salt Lake and opened the first KFC. I had an education from working there when I was in high school. So now you have a whole story. I was going to say, there's pictures, or there's statues of the colonel everywhere around the board. So maybe they were under a lot of pressure. Interestingly, it is a pretty, or it is definitely an interesting place. Whenever I went there, this was what I saw, where instead of using the tablet, like a tablet, people were using the tablet like this. They were pulling out the pen, and they were poking at the screen, so they were hitting the menus across the top, and they were basically interacting with the application like that. Now, this whole, this entire application was built around the concept of eek, right? The ability to have hand-writing recognition. The ability to work well in that form of fact. But whenever I went there, this is what I saw. And I thought, well, this is kind of a strange place. Maybe it's just these people that use the application like this. But as I started to go visit more and more customers, I noticed that everybody used the application like this. And if you talked to everybody on the team, would you go visit other customers? I don't think anybody ever saw anyone use the tablet, which is kind of interesting, right? Because whenever you went back and you looked at the team that they were originally using the software, whenever they tested it, it was all on tablet. So that's kind of the vast difference that I have in my background, to seeing what the differences are and just being involved with the users. So whenever I think back on that, what could have made that application better, I think, well, talk to your users, right? Interact with your users a bit. It's pretty simple after you've done it, if you've done it. To understand that that's important. So I started to think, what's the simplest thing that could possibly work? Because most companies that I visit, not going to see a different company ever meet this employee. At least three companies a month have done that for two years. Very rarely do I see people getting out and engaging with their customers. And if they do, it's somebody like their product owner or the managers. A lot of times they use their experience before there are any. But very rarely is it anybody else in the team. Who's a delivery team member, developer, tester, strong master? Keep your hand up if you've ever been to a customer. That's great. That's more than I think I've seen since I've been out. So it probably says something that you hear at this conference. But did you notice great, did you notice weird things whenever you saw your users? It's always an eye-opening experience. So I think that for most of these teams, these big companies, whenever I say take your developers for your testers or whoever, and send them out to go hang out with your users, they say no, that's not going to work. We don't have the budget for that. They just don't want to hear. So I started thinking, what's the simplest thing that could possibly work to get people working involved in this path? So if everybody here that went out and visited a customer, it just won't shake their head. Yeah, you had pretty changing experiences. Very interesting things. So I think that my assertion is that the lightest way that you could do this is you could do something like very simple, discount-type, light-weight usability testing. Donnelly Norman in the design and battery day thing says this quote, I think this is great, says there's a big difference between the expertise required to be a designer and that required to be a user. And the word designer is often because experts in the device they are designing, users are often experts at the task they are kind of performing on the device. I think this is very true. Whenever I think back to that company, they designed it, they performed back when they designed it in the device. But even though they didn't have any designers or developers to design it, but it was designed that way, we can very rarely be looking at the users. So my point of view is this, and this was from my submission for this talk. So whenever I sent this to Kay, I said, you know this, this was from the college speakers, the corporate papers. But this, I really latched onto this thing that more importantly, I don't know if it's inspiring to me, it was for great software products that capture our user's imagination and succeed in the market and create value for us and our customers and their overall life. My response to that was, I don't think this is true. That's what this conference is about. But I don't think that is the reality of Moose Software. I think Moose Software sucks, especially from the users. If I look at, like my Twitter schedule, for the past two years it's all about all the interactions that I've had with lots of different software. And it's all back, very rarely do I say, this is a great application. And think to yourself, when was the last time that you used an application when you said that's a beautiful experience, I just sat with that app. Do you walk away totally smiling? Does anybody do that on a regular basis? You do tell me because I love to use the app. Because I don't even use it. So I think that by doing this discount user testing, by usability testing, by getting in there and just seeing what users are thinking and interacting with users, we end up with much happier users. We will end up with much better software. And we'll end up with a lot better experiences. So looking at why shouldn't we do this user testing, I'll show you what it is shortly. And the first thing is because it helps and it helps a good bit. I was just at a conference where the speaker was talking about usability testing in general. And he said, having usable software is kind of what I'm having set up for you. And I think that's very true. It's not that usability is your endpoint. But it's a great starting point. It's not necessarily usability in and of itself doing this user testing, usability testing isn't necessarily what's going to get you great software. But it certainly starts you along that path. So think of this like your journey, right? This is your base game. And usability testing can start you there and push you off into open directions. And the reason that I think usability testing and the way that we're going to talk about it is so powerful is because there's a large degree of disproportionality and the amount of effort that we put into it and the effect that we get out of it. So are there any tests for a minute? Are there any people that are infected with being test driven? Test driven development. We're trying to get out of it. Well, I see this like TDD, right? TDD is a very interesting concept because once you use TDD, it's not just about the testing. It ends up being about all these other things. And I think that this user testing will vary somewhat. And I think this is also true. So Tom Peters recently tweeted this. He said, unintended consequences far, far exceed. The intended consequences. So I think that whenever you go down this path and you start having a regular cadence of user testing, I think you get all these other benefits. And the biggest one of those is having true empathy for your users. I think it's extremely important. It's the only way that we're ever going to get to great software. And this is one. Has anybody ever used an empathy map? Do you have any idea what that is? Anybody ever seen one of these? They're not very common. The idea is that on a flip chart like this over here, we would draw a picture of our users. This may be just a goal in the application. And we would write things like, what the user thinks. What the user sees. What they say. What they hear. Maybe what they feel. The idea is that you're trying to have more an empathetic standpoint of your users. You're trying to understand them a bit. And I started doing this with the teams that I'm currently working with. And what I do this is that teams do this. When we get through the exercises, just fine. And they say, yeah, this is who our user is. This is our system admin. Or this is our whatever user will or adopter in your opposition. But what I've noticed is that once they start to go out and they start to adopt usability testing very lightly now, this changes completely. If they would go back and look at what they came up with here, these are all assumptions. There are assumptions that you can think are right, but they very early are. And so you can help users a little bit. So it is a whole different world. From being on the team, you know, you're thinking from the developer standpoint, right? You're even thinking somebody like a doctor doesn't care about the people, right? They're not buying you anymore. They're buying the software. No. No. No. No. No. No. No. No. Is this true, I think, from those cases? It's, I mean, it has everything to do with our education to how we learn to develop software at the first place. But we've been bought into certain mental models that tend to force us into cubes, right? There are seven cubes all day. Unfortunately a lot of people, right? So we got these walls around you but I think once you get out, when you start interacting we start realizing that, you know, there's a whole more to these things. So I think that whenever you get out there you start interacting, you move from this model I think of it as Ivan and for her. But, you get, from this point on, you start out in your cubicle walls, right? You're thinking about yourself. How would I use this software? How would I do this action? How would I perform this task? And then when I'm beginning to an agile environment, you start to move your version to this them phase, right? Your stage. So you're looking at roles, you're all working together. You guide now the cubicle, you're quickly out of your team room. And then you're looking at roles, a typical user story, as a whatever role. And you start thinking about the roles. Unfortunately, I think this is certainly much, much better than this I face. But once you get into that them phase, then you're just locked in the stereotypes. And that's what is usually wrong whenever you start that. Whenever you start interacting with the users more deeply, you get into this kind of where you start thinking about real people. You start referencing back to the past that you've seen happen, and to the people that you've seen recently. Anybody seen the, what is the browser video? You, let me show you this. I'm from New York, to find out what is the browser? Search, I'll do it, I think. I call it search engine, that's what I call it. Browser. What is a browser? Search engine. Browser. What? It's where I searched through. Like, I searched for this, correct? What is a browser? Good one. I'm looking at what it is. Browser reviews are looking at internet web page. Do you know what the difference is between the search engine and the browser? Well, not exactly. I have to say what? Yeah, I don't know. The search engine is when you're searching for something. There's some of the browsers that we've got, but I'm not sure what that is. So do you know what the difference is between Google and your browser? Um, no, no. What browser do you use? I use Firefox. What do you use? Switch to Firefox. My friend came over to my house and erased all my other browsers and installed them and said, you're using this now. What browser do you guys use? Do not. I have an email. We go through, uh, uh, Rob and Google things for, I think. I'm not good at that. I might be the wrong one, but what browser do you guys like to use? Is Google? I use Google. Google predominates the market, obviously, but, uh, I can't go out and go to the app and just get in some business. Yeah, yeah. So I use it as a browser and all of this is on Firefox. Oh, uh, is that your internet browser? Uh. Do you know what a Google browser is, bro? Should I use it? Have you heard of Google Chrome browser? Chrome browser, it's the one. So 8% knew what a browser was. Oh, wow. That's kind of surprising. Not really. No. It's surprising? But I mean that, it's true, right? I mean, does anybody in here, I mean, maybe I'm wrong about this too, but I would assume that everybody in here knows what a browser is. I don't know how to put it. But it's a different, it's a different group, right? I would say we're probably hitting 100% on Chrome. Yeah. And we're hitting 8% on the street. World, bro. So it's a whole, it's a whole different world. So, as part of doing what I'm gonna talk about there, user testing, you're looking to get real feedback through your products, and you're looking to get hot bandwidth information, you have feedback. Watching people's facial expression in their body language makes a big difference, right? So, usually I heard somebody say, well, think about how you, if you had to write down words, if you had to think about writing an email to somebody and explain to them how to define a circle. Explain to them what a circle is. It'd be pretty hard to do. You could just draw a circle. So just watch users. You don't have to go through document after document, or have people write and tell you what's wrong with yourself, right? Just watch them, usually. Much higher bandwidth communication. So, Facebook. There's a guy that lives out on the other North Carolina of it, who, his name's Brian Goodbyen. He's an experienced designer for Facebook. And he has this great great story about, whenever he joined, he's just out of school. His first task was to redesign the sign-in page to get higher conversions. And Facebook has a lot of users that can improve the conversion rate by one percentage point. We're talking about a whole lot of users. So what they were seeing was they wanted to improve the conversion rate. What they noticed was, as soon as people went through, went through the, I think it was three or four pages, and then there were information. Whenever they got to this page, right? This page that shows your recommended friends, right? People that you might know. Everybody had these experiences that looked like this, right? People were so excited. They were, you know, some people started, you showed me the videos of the user testing, you showed me the photos testing. There were people crying whenever they saw this. They think about that, that emotional connection, 10 years, you know, somebody you haven't seen for 10 years. So what do you realize right away after watching those videos and you realize it too yourself? What you want to do is you want to get to this as fast as possible. So the anatomy of a user test. Pretty simple. You just need a few tasks. There's tasks related to goals. You need a few users. You need a few observers. And after that, you need to debrief over a few years, which usually works well. So you're not going to produce anything like this. You're not going to have these big documents. Usually what you're going to end up with is a list of three recommendations. You're using them really fast. You should inform the backlog. It may just be information. The varied most, you may want to write a page after you use these. These are pretty light. As far as the things that you want to test, you may be testing the stories, but more often than not, you're going to test some type of scenario or multiple stories. You want to test some task that the user is going to be doing, which may or may not be a story. It may be one story, it may be multiple stories. So really, you're talking about a user case? Not necessarily. I mean, it all depends on how you would, how you could be, in this case, certainly. But if you're, it depends on the level of detail that your stories are different now. So if you have a huge application, you have stories that kind of are on the edges of whether or not they're vertical slices, then you're probably going to have multiple stories that you're testing in some sort of test that we hear. If you're a smaller shop, and you're using a very hard way, I mean, if you have some great people that turn out to be really fast, then maybe we'll do a test with you. So this is pretty much what I think it should look like to get started. Now, I don't, I'm not going to tell you for sure this is exactly what you should do, but I can tell you is I'll give you this as a prescription to start with. I don't have a usability factor. What I noticed was that doing this made a huge difference in how I related to customers and how I see teams really. So we start off with the cadence, right? That cadence may be your iteration values. They may be sooner than that. But say you're doing two iterations every two weeks. Why not pick three users? Maybe they're real users, maybe they're representative users. Have three observers and have a facilitator. A facilitator's, you have a few examples like your UX group. As far as the observers, anybody should be observed, right? Developers, testers, stakeholders. Bring everybody in. Have everybody see what's going on. It'll change them as much as it will change you as you go through what's just happened. So we've got our cadence. We're doing this all the time. The good part is that the feedback, this isn't like traditional usability testing where at the end you get a big document and a bunch of recommendations that you could apply to the next product. It's just like a retrospective versus like the product is more of, right? You're doing this all the time. You're constantly responding where you're building. Hopefully that's informing the backlog and changing to help you build better software. So pretty much how you do this. Bring in a user. You ask them to do a few tasks. Say maybe three. This may take 30 minutes, 45 minutes. Probably no longer than that. The facilitator is gonna ask them to perform tasks. Pretty simple. They're gonna keep them talking. We want them talking about it. Kind of want them narrating their thought process. This isn't perfect, but it's pretty good. And you're probably gonna have observers observing. So we'll just pretend it's our developers and stakeholders. Yes, sir. Everybody. So now you've got, you know, quite a few people that are watching you. And four of them are watching the other way. How do you keep that guy from not feeling like he's an official? Right? I don't know. Well, there's a couple of things. There's software that will help, right? You can just stream it to somewhere else. There's one idea. The other thing I would say is if what you're doing is just trying to get started, don't worry about it too much. Yeah, it may affect your testing a little bit, but at the same time, some information is better than the other ones. So if somebody who has often done the facilitating part, part of that job with some facilitator is set it up in such a way that you're mindful that, so that this person is comfortable in whatever way they choose to set it up. So it could be just making sure that they're comfortable with their conversation or that set up or whatever way makes sense for sure that the other person feels relaxed and can actually focus on it. Great. Then do you bring it into your environment or do you need to bring all of your time to their environment because? Well, it's really, there's, if you can, it's great to go to their environment, right? I would, but it's also good enough in many cases, especially if you're getting started, right? So the point of view that I'm coming out with this is, let's just get started. Let's just get exposure. So I would say, yeah, if you can find a little conference room, book it and bring somebody in, right? Even if you have to go down, you know, the hallway to find somebody, bring them in, even though they're not completely representative. Little information in most cases is going to be better than known. So you go through your three test groups. They need the three tasks, the facilitator, the observers get together, discuss what are the three biggest things that we could work on, and that ends up in there in one piece of work, or changing the backlog, or performing the backlog. So that's pretty much all I'm putting forward, is this little process. The magical part is they can do it with these things. That starts you on that path. Now I don't think that you're just limited in software. You can certainly do this with paper prototypes. You can do this with competitor sites. Maybe you do it with patterns that you want to implement. So you want to do progressive disclosure on the website. Why do you have to carry it up? Find somewhere similar, test that. It'll give you information. The only difference whenever you do that is a lot of times you focus on what's good versus what's bad, because you want to see the good parts of other stuff. So pretty straightforward. This is what it might look like. What I did was earlier today, I just went and grabbed somebody from downstairs and said, hey, I've got this website. Could you do a couple of things on it? He said, yeah, I'll look at it. I said, you've got it. He said, yeah. I said, great. So I have one of the websites that I have on the site. Just a couple of months ago, I decided it was too expensive for me to play golf, because I'd play golf a lot, and I'd drop it in the workplace. People here want to do it. So I created a website where I could golf or do these, so that I could call up the golf courses and say, hey, I'm writing these names for golf courses, and I'll play them for you. This actually works very well for me. So I've got $800 in golf in two months, which is great. It's kind of fun to write your reviews. But the interesting thing is that I'm my really target reviewer, right? But at some point, I would still like to see, you know, I don't want to create a portal application. I did this overnight, and I was down it. I was at South by Southwest, and one night, I just got this idea, because I had enough golf game and basically just put together a walk. So it was good enough to get started. I thought I could get it out there and get it back. And this is the site. Actually, whenever we get through this, think about some of the obvious things that you see that could use improvement. And we'll talk about them for a short period of time. But don't be pretty obvious on me. Yep. You guys get to the moral. So looking at the navigation, see what the website does. So I was going to list different courses. So in a way, I can subscribe. I would be creating an account in a bunch of seconds. What I'm down to here is the point is I just wanted to look around for your saw, right? So this is 212, like I said, by the Bay. It's created with stars, with pictures, linked to their website. Yeah, I did this with Silverback. And you actually changed the font size over and over. It was trying to use it. You used it to see? Yeah. It took a little bit of a second. And there's another one down here. So if I went to this, it appears to be a blog. It was just entries, articles, and another author. I've noticed Ben, there's guy who's the cabinet. So he's a couple contributors. Recent post-sports reviews, range reviews, with new courses, also driving ranges. So it's new, because it's the new one in Silverback. It uses sterile links, we're looking at each blog. It has pages that give a lot of feel. You want to status page? Yeah, right now. OK. Seriously, what pages are you going to do? So the first thing I'm going to have to do is sign up. There's really a lot of names. Keith and Todd, right? You've been wanting to hear what they're doing, bro. You can see, the video is not very big, but there's times that you'll see them squinting their pages. I'm always asking, what are you looking at? Because I can't tell if it's like the images or the text. Let's see what it's on. So I'm looking for tag wire. This is over here in the right hand side. I just asked him to find a review for tag wire. It's a pretty decent review of your life. It's got four stars. So you never do a review of that text. You'll need to review the text. I was looking at the image, the course overview. It doesn't really tell you the options into how pretty it is, which is. So his task here, what I asked him to do is to look at this site and tell if it's going to look at this to review and tell me basically to figure out if it's a course that he would like to play on an upcoming golf course that he's taking over. It's pretty nice. It looks like B. Is that a new third-party lit? So of course, it's good. It's one of those two new balls. The other thing that's interesting is you can do this with concept, right? I mean, it's basically to tell you the content of the photos that were taken for the reviews. I'm still not concerned about the notes on this. So I'd have to look. So I tried to go to the website for the golf course to see how much it cost. And so I got a floor for air. Just floating the whole ball in. Now, I didn't consent for him to find this, right? I just never clicked that link. So you didn't see anywhere to go to get the price? No. Yeah, interesting. Like he mentioned, he wants to find the price to figure out if he's going to play. And this was really interesting to me because I tested this on a few people. Nobody has ever said anything about the price. But it's why I started this site. I never thought about it. In any other music site, he never says a price. Because I never really know it. So this is given around the line. It's about pretty much what it looks like, right? Somebody's going to be doing the facilitation, just asking the questions. Can you perform this task? Or should we back to a user goal or user need? And basically, you watch them do it. You keep them talking. As observers, it would be something similar to what you guys were doing here where you're going to observe. And we also would like to report it. Yeah, we do this research. I've seen a lot in the last couple of weeks. And we used Silverback. I can't do it myself. Maybe now I go back to the videos very often. But we've now been able to see the upload devices. It's quite hard to get pretty much forward around one person with a small off-device. If you do any software that's similar to Silverback and I've built my own devices, I think going back to those, I'll go back a lot more often because I can see them a little better. So the question was around like, I don't know how do you get a video. The only thing that I've seen, as far as actually testing an application, is using one of those GorillaPod-type things to just mount a camera on. I'm going to hold it down. And it's just videos down. But certainly not. The one part with that is the phone, is how they hold the device. Actually, it packs their... It's very similar to my story at the beginning. You don't get out and you don't see how people are using it. It's not that you're going to find everything. I'm doing the typical. If you're laying your phone down and you're poking at it, there's a little video camera right here and people standing around you. You're probably going to find everything. You may find some obvious things. Yeah. Has anybody used an app that was good for that? Anybody in here? Well, I know there's... I don't know if you have a mouse or a mobile device. It's just... The face is a little... It's on the top and it's a little... Yeah. Yeah. It would be a little better. We don't have to look at it. I wonder if there's just a recorder to record the screen of the app. That doesn't work. Because when I do my testing, I don't need to record the face. You just record the audio and the screen. And then keep them talking. Is there a reason that you don't include the face? We just haven't found it necessarily that useful. Because when you say, like you just had to ask him, what are you looking at? And since you're not doing eye tracking, you can't tell anyway. So if I... That's why. And I'm using... So we just decided to skip the extra expense and use WebEx. So I used WebEx for... to do the remote. It's just cheap and easy to mount. Also, whenever we... Yeah. Whenever we get to the end here. I'll talk... I'll see if anybody has any recommendations or any tools they could use or things that you'd like. Okay, so the first thing I want you to do is to think about the current website that you're working on or the last project you were on and think of five to ten things that you might want to test. So I'm not going to have you go through goals and task analysis of what this is, but just think at a high level, if you were going to test five to ten things, what would they be? So what are things like the major functionality of the site? Maybe things that you've lost sleep over. Maybe things that you could have other input. Or things that you think are a use of only problem. Think about those things and right now, five to ten things, I'm going to give you two or three minutes to the ten things. Ten things? Yeah. If you ask other people in your company or on your teams, do you think they would have a pretty good intersection of the two? You think they would have the top three things that you could probably come up with? Yeah. More than likely. Yeah. It's pretty easy to decide on a task whenever you first get started. It's pretty easy to look at from a standpoint of what are the big things that we should test? Those are the things you want to test. So the next thing that we're going to do is go through this discount of testing with your system. Typically what this would take is if you look at this on a calendar, this is maybe four hours worth of effort. To get the task ready to find the users, to bring them in, to record them and do whatever it is that you can do and then to bring them on. So that's the discount part of this discounting ability. Very, very lightweight. The idea is that you're just increasing feedback as fast as possible and doing it without a huge cost. The advantage is that you can do it very easily. So what I need is the tables to have a 5 or 16 fit. So whenever you open these up, there's a couple of you're laid out a couple of different ways. The usability test script at the front. I want you to pick a couple of different people. So you've got instructions on your table. That's what I passed around earlier. So what you're going to need is you're going to need a facilitator. You're going to need somebody that you're going to be doing the testing on. Then you're going to need an exception. And then you're going to have at least three examples, right? Everybody else. So this this one right here is a usability script. Everybody else can get one of these on the table. Whoever it is that chooses to be kind of personal to the test subject is a lack of a better term. You get the gift card. Oh! Okay, so let me explain this to you. This first sheet this first sheet it talks about, it says if the topic starts with usability test script this is from a book called Rocket Surgery. It's easy that it was just released, not too long, probably like two, three months ago. It's a great book. You can read it in like half a plain size. It's straight out of that book. Now, what you're going to do whenever you start out, a facilitator is going to talk to the person that's doing the testing and they're going to basically just read through the script. I would read it pretty much verbatim. It even says in here that you're reading through the script. There's some things in here that won't apply like it says this will probably take an hour and I'm going to start the screen recording software but you're not going to do it. But you just read through this. As you read through it, you'll get to a point near the end of it where it asks you to go through and do the tasks. So to start off, before you do the tasks it will have to ask you questions like how much time do you spend on one? How much time do you spend on that? What percentage of that would you say is using email versus browsing? What are your favorite sites? Just a few things to get the people talking. This is just a facilitation technique. We'll talk and we'll talk more. After that, whenever it says to go through and actually do the scenarios you've got two sheets. One is a blog entry on a WordPress blog so the task is that you're going to write a blog entry and the scenario talks about your context. This is why you want to write it and you're going to basically go through and do a blog. Set up a WordPress blog last night and you can use your name and password on here. Everybody let me know if anybody doesn't have a laptop to have internet access. So do you guys want to come sit over here and watch me bumble? You also have a second that you test. As you go through and do the other just write as an observer write down what it is that you're observing think of your biggest name, your top name and then we'll do great. Are there any questions about this? I want to read more about our blog. How was it? Anybody want to go start a WordPress blog tomorrow? I know. So let's talk about what observations did you have? I'll do that. What works for you? I have an observation I was one of the person doing the testing that was doing me. I tend to be somebody who will speak out. I like to think out loud and you can tell by how I use my hands and just kind of do it. And that was probably helpful for the group because I did it. And I don't know if you run into personalities that don't like to think out loud and you bias your application toward those who do. Yeah, I'm like that. I don't like to think out loud. I'm kind of a bad person to do the test. But from the facilitator standpoint, what they should be doing is just prodding. And it may be that occasionally you can test people that just start doing it. You can test them somewhere. But if you go through and do that test it may not be a good personality for it. That seems to be fairly rare, honestly. Most people can adapt to it well. So we still we as observers have money to help. And we wonder how much the facilitator helps. Not at all. Not at all. So it is very hard to watch it. I can tell you guys a little bit about whenever I go out and work with customers sometimes they use our software as well. So I don't know how to use our software pretty well. I also do additional advertising. I train on my software as well. So I know how to use it pretty well. Whenever I watch people use it I always make a conscious effort to not say oh, you should do it this way. I always wait. Usually 5 or 10 minutes. I don't want to do it too long. But I'm going to observe what's going on. So if I see somebody taking a path to get to hear that there's like this I observe it happen. And I talk to them later about it. It's very hard, especially if you're somebody that's involved on the team to not say oh, go over here click that. You don't want to do it that way. Or to help guy. You don't want to guy. What you want to do is observe how people are using the application without you setting these up. Is it okay for the facilitator to ask things that the testers are not saying? How are you feeling about this? Or do you feel interfaces to clutter? Things like that. But it's just asking things too. A lot of times you have to be careful of what you ask because you've got to answer. So like if I said like on the Murrow-Gofford you say if you look at the top across the top the taxonomy of the app Thursday says like a home, course listings about us and subscribe. If I would say something like where do we think you would subscribe? I know that right? Maybe I have an open question like how are you feeling right now? I think those are great questions. How are you feeling? What are you looking at? What's troubling you? Very open-ended questions and you want them to narrate their way. Because what I did is how are you feeling? He said maybe the rest I would like this to be bigger and the rest is cluttered. So then I asked him is the clutter like keep bothering you? So then I narrowed it because he said I think so. Because it was you were building up the words that were used. So I don't think that's kind. Something in that scenario tell me a little bit more about that. I find this all the time is trying not to bias the responses leading the witness. So it's something I'm constantly falling into. I'm always catching myself on. Yes the list of instructions that was there and I guess we were on the other side. It seemed a little confusing versus the sort of thing we want you to do this, this, this, this. That's the feeling I got from it whether it was there or not. I remembered the last thing you said whenever you do that may not be the case because those scenarios they have a good bit of detail in that case if that's what you're hearing then I was short. So it probably depends on what it is. It would be better to walk through step by step for where would you go to go to a new post and then wait for that as a facilitator and then be like okay well let's click there, go to page, let's give it a title. So like the granularity of the task you could certainly break it down a little more especially like with the WordPress one right if you haven't used WordPress before it's pretty confusing to do all that because the interface is pretty confusing. Yeah you could certainly break it down more I would just try it see what happens and then adjust it and see how from one test to the other if you get changes or if you notice frustration or the other thing is as a facilitator whenever you go through and after you read the scenario you want to give the sheet to the person. You always have a physical sheet so they can go back and reference to it because like you said you're going to remember the last day even if it's only a couple of seconds. Might be off topic but what would happen if you receive you got a website and you receive an email with someone saying I always get stuck here I would like it to be like this would you take that as a sort of I mean with the same importance and someone who you observe the usability test or is that of lower importance? Because you weren't I don't know that I could answer it really without more context I mean it depends on so many things that the things that you observe are going to be higher bandwidth probably what I would do is if I saw that I saw that that was fairly frequent then I would go and figure out what to do I would put that as a scenario of this kind of test a good example of that is for instance the application I talked about earlier from the tablet a large number of the support costs came in and they talked about flopping laptops we had no idea what that was what was happening was because of the leverage of being pushed on the top of the screen the hinges were getting out so whenever that happens what do you do? I don't know if it's ever got implemented or not but whenever we sat down and we did some paper prototypes just kind of with people within the company what could this look like you can come up with great solutions with good enough solutions pretty fast and in most cases for instance the user experience group got around it what they came out with a couple of hours later is what can we move the menu to the bottom of the screen so there's less leverage to give us some more time to figure out how to fix the problems because that's a partner problem you can come up with incredibly brilliant solutions just here let me run through the where next piece about 10 minutes left iterating on the feedback I think that my personal opinion is that in the agile community we've lost the passion around iteration as far as iterating not iterations but iterating on products making products better and using feedback to help inform what we do with the product as larger and larger companies adopt agile I see this all the time they still start off with a big requirements document and what's been called a backlog and they expect it all to come out in the end they're not iterating on top of what it is to do so I think we need to think about this a little bit this process can help drive that and help the iteration but if you're not going to change anything then what gets the feedback right the value is the feedback that comes out of this process this isn't interesting for as well as far as thinking about how to fix things so even though a piece of duct tape covering the hole in your pants might not be pretty it's still better than the hole in your pocket surgery made easy it's hard for me to latch on to this idea how do we do that from the development where whenever I was a developer it took me a long time to figure out how to do the simplest thing that could possibly work and to be comfortable with that from a design standpoint it's hard for me to take that into because I want to make things that are I want to make the next news chair I want something to be beautiful but at the same time you have to balance that so just doing something to help improve the problem and iterating your way and having emergence come out of that and emerging into a great experience is very valid as well so do the minimum that you can to fix the problem if you think about this in terms of tweaks tweaks instead of redesign this is another definitely bothers me the templates came straight out of there a lot of these quotes came straight out of there in very action but this is what Steve says about tweaks cost less they require less work they don't want to enlarge or break up families which is a very big point you're not going to hopefully be working with 80 hours a week trying to get more than a tweak small changes are more likely to happen the break isn't much stuff people don't really like to agree to talk about this anyway the arm is risking you is not as important when this is going to require lots of money so as a chair you're just saying you put in a lot of tweaks to it and you kind of have a new product it wasn't in your face all at once you can certainly iterate your way to a new product depending on the adoption of your product for instance like eBay at first I remember hearing this story about how they started out as like a yellow background one day they switched it to white and they had a huge jump over because everybody didn't know the site was still the real site it's over a period of a year that changed the hex values it's a white so I don't know if that's a true story but it is really I think you can iterate your way there that's an extreme example Facebook has a lot of users for our applications most of the time slow intentional changes but not big bang I think experience another thing I want you to think about is that you listen to the people that the observers when you talk about it think about thoughtful reduction your first instance and I saw this whenever I was a developer as well to fix a problem is to add so you go to the admin and to fix an interface or a usability problem or an experience problem you go and add more things under the action many times way more often than we actually do you can probably subtract so whenever you have a problem think about if there's things that you can take away to help fix that problem so John made over the book called the laws of simplicity that's excellent he has a law in there called thoughtful reduction that's a very powerful thing to think about to think about subtraction power of subtraction you don't leverage that a lot applications on top and on top of the total a lot of times you really know the value that we're delivering you can take things out a lot of times you don't need buy-in just buy a pizza or ask people to do tasks you've got to list a task right there if you do need real buy-in and you need money frame it as an experiment whenever you talk to people and you try to get money to do this or you need some type of approval to do this this is an experiment I would say to do this with everything as well start thinking in terms of experiments we're going to try it we're going to see if it works if it doesn't we won't do it anymore and you just find something if it does then we'll go forward everything is an experiment think of this as an experiment as well the second thing is to promote it as minimal cost because it is if you don't have a big graph like a lot of information it's a very useful word or it's a national you can not send anything else but minimal cost and you also have this for proportional returns you'll see more in the group whenever you look at a you look at the number of users if you need to test to find high percentage of usability problems this is it's an interesting curve it can certainly exploit so if you look in this range the three to five range you're getting a lot of stuff and you can build a lot better software by finding this number of usability and you have to know where it got it's numbers yeah, he's got a very big research I'll give it to you yeah, it's on this cycle maybe I should bring in to test bring the five to six he tells the assumptions behind the problems the important thing as far as getting started I think is that if it's zero users you get zero information so keep that in mind too so my point of view on this is we need to really stop building stuff we should build better software we can do that through empathy and the specific actions that we can take for this is you have a list of five to ten things you can take that back to the people if you work with it and just find a few people to test it's a pretty easy thing I found somebody this morning in the session find a few people to observe that's one review anyways and then to brief on these observations see what you find out figure out where you're getting in so these three steps today from the top we've got enough to get started with this method the benefit to this hopefully is to a place where we realize that we need to understand that we have more empathy towards our users that should lead us, just this technique as a tool in and of itself isn't going to get you into great software it's going to get you into a mental model where you're thinking about your users and understanding your users things like adopting a contextual inquiry it's very very powerful to go watch and observe users a place where they work in their natural environment I don't want to discount that at all I think that's wonderful, I think you should do it many times I think that companies that aren't doing anything this method will get you started very soon as soon as you start to bring in stakeholders when your stakeholders watch your users do something like you just did with adding in your post to WordPress they're going to be blown away wow that's hard so the last thing I want to talk to you about hold up your hand if you're on Facebook or Twitter there's a few people that aren't certainly like 80% of the room this is really interesting how is that the South by Southwest conference not stumbled into this into this room where somebody was doing a presentation on what happens to your social life when you die your social media life which was a really strange topic I just happened to walk into this room now listen to this thing this is really strange so whenever I walked out probably about two weeks later I got this become a fan email from Facebook this is whenever Facebook had a fan versus life and it was from somebody that had sent it from their brother so this was very strange I had a friend request a fan request so I went to the page and I looked at it and started digging through the page and there's all these emotional walkers I remember this whenever we did this I had so much fun when people were talking to the person they were still around it was in its own way it was kind of disturbing and kind of beautiful but at the same time and then whenever I started thinking about that the other piece over here is this is my daughter Hailey I'm a recent dad my first daughter I started thinking about her life is a lot different than my life what happens whenever her kids are around if you live your life on the social networks there's probably tons of information that all these people can find about you to say my granddaughter or grandson probably figured out everything that I ever did because it's somewhere between Facebook and Twitter in most cases so the thing that I don't want to have happen is I don't want to have my last walkers to be been carried into building a city software so I think that we have a I think that because we create this stuff we have a gift that you can give to people you have a gift to create it we're actually creating stuff we can create better experiences you're part of what we did we can give these things out to the world we can make happy users we can make happy people and I think we can build better software because I really think that we have the most