 this is my usability goes to 11, a hacker's guide to user experience research. So just give a little background. I started out as a script kiddie, DEF CON 17, just run around, pop and shit. Went to grad school and became a UX researcher, did a couple internships at Mozilla and Palo Alto Research Center and my master's thesis focused on how to redesign the tour browser bundle so it would be a little more usable for non-experts. Basically my focus was these things called privacy enhancing technologies and we'll get a little bit into the whole privacy versus security terminology later on. And currently I am a staff technologist at the Center for Democracy and Technology in Washington D.C. That basically means that I do hill visits occasionally doing some lobbying but mostly I work on web standards and serving as an in-house expert for our advocacy. So to just give a high level outline we're going to talk about why usability is important, why is usability hard, why is usable security especially hard and how we can evaluate our tools to see whether or not they're usable. And these all sound like relatively simple things but they get hard very quickly. So to just sort of set us up terminology wise, when we talk about security we're usually basically talking about the CIA triad. We're talking about confidentiality, we're talking about integrity is this data modified in transit and we're talking about availability, non-repudiation, things like that. Privacy is just control over your personal information in general. If you don't have security you're never going to get to privacy because someone else already has that control for you. And so these privacy enhancing technologies are just any technology that lets you control your privacy. So something like Veracrypt that lets you encrypt your files that could be a privacy enhancing technology. Something like TOR is usually considered a privacy enhancing technology. So at the high level why even encourage pet adoption? Why shouldn't these people just RTFM? Why does making privacy usable matter? Well we've extensively documented the existence of mass surveillance. And studies have shown that mass surveillance chills free speech. It chills what we talk about. It chills what we look at. This chilling effect has an effect on our democracy. If we want to have a republic we have to have debate. Period. End of story. Why encourage adoption of these technologies? The math is good. If you look at the current surveillance practices the so-called 5Is are mostly focusing on bulk collection and injection. We have, great I don't have the notes, but these NSA systems like Tempura for example, well that was GCHQ. But when you're sucking up massive amounts of data, if that data is encrypted it's not as useful. If the transmission of that data is using HGPS or some sort of integrity check it's much harder to modify and transit, much harder to shoot malware in there. We don't have any evidence that the more common encryption standards like AES or PGP have been subverted. There was the NIST, interesting NIST. So basically we know that our underlying technologies are solid. We need to resist this urge to follow into what Mika Lee called security nihilism and focus on making sure that the tools that we have are used by as many people as possible. And thirdly we have the idea of network effects. Most of you may have heard of Metcalfe's law, the idea that the strength of a network is end to the power of two and being the number of users. This is especially true for anonymity systems. Again, in the community sometimes we'll use the phrase RTFM when we're talking about something like Linux but when we're talking about a privacy tool, a privacy tool is best when the most people can use it. So for example with Tor, the bigger that swarm is, the less chance I have of knowing who you in particular are. On the other hand as one of my friends from Pittsburgh said, nobody wants to download a special application to talk to Greg Norsey when I ask them to download a signal. Luckily with certain recent events in the news people have been more and more willing to use encryption. So why encourage adoption of these technologies? Because nothing to hide is a load. Everybody wants privacy. If you don't think that you want privacy in your life, see me after this talk so I can set up a nest cam in your bathroom. So we all want some privacy. We've set a floor. What we're talking about is the ceiling. So that brings us to our next point. If usability is so important why don't we just add usability to these products? And that when I was making these slides reminded me of this Neil deGrasse quote Obama authorized North Korean sanctions over cyber hacking. Solution there it seems to me is to create unhackable systems. Just like I think we in this community know by now that you can't simply bolt on security after you've designed a product. Similarly you can't just come in at the end, throw your tool at a designer and say hey can you make this usable? It's not going to work. And usability is hard. We can define when a system is secure. If you can if you can if I send out a packet we can we can do these entropy calculations we can see how long it would take to decrypt this given the key space yada yada. There's no real way to know is this piece of software usable. It's just a continual and iterative process. And even when we ask users for feedback it might not be helpful. I'm sure if you've ever developed an open source project you get tired of complaints like this. Your usability is bad. I can't tell you why it's bad but your product is hard to use and I don't like it. Now you can do two things in this scenario. You can just double down and say RTFM and not try and actually improve your product or you can use some sort of systematic way of improving the usability. And the other thing is that usable security is even harder than just trying to define something usable. Look how long we were listening to music with all sorts of weird technologies before we finally settled on the iphone or the ipod. That was a very usable easy way to have a ton of music at your disposal and it took many, many years. And there's was this very relatively old paper called why Johnny can't encrypt that was put out by Alma Winton and JD Tiger back in 1999. And basically it was the first time that one of the first times that we said hey we take this theory from HCI human computer interaction which is an academic stream of research and let's apply it to a security tool and see what pops out. And so Johnny can encrypt according to Winton and Tiger if he's made aware of the security tasks able to figure out how to perform these tasks doesn't make any dangerous errors and feels sufficiently comfortable to continue using and that last one is key. There's a phrase they use in the human computer interaction literature it's the gulf of execution and a great example might be when you're trying to set the timer on your VCR and you know what your end goal is. You want to record the latest episode of France and you know what your starting point is. You don't have a timer set but you have no idea how to go from that start goal to the end goal and if you can't logic your way towards it you may just throw up your hands and say you know what I guess I'm just not going to see friends this and Winton and Tiger operationalized some properties about usable security that make it especially hard to do and I'm going to give each of these its own little slide to really delve in. The first is the unmotivated user property and we had some projector issues I wish this was bigger but that is a tweet from Swift on security that says tell me more about the crypto you invented with Taylor Swift leaning into the children. The unmotivated user property talks about the fact that security isn't a primary task that users aren't lazy they're actually actually quite rationally. These are rational actors security is not something they want to do their time is of value and it is limited so they are going to expend the least energy possible on your security system. This is why they're writing their passwords down this is why they're using dictionary words as passwords this is why they're doing all the bad things it's not because they hate you it's because they just have other things they want to do they want to shop on Amazon they want to send snarky tweets and anything that gets in the way of that is an obstacle to be overcome they're trying to hack your system. Second is the abstraction property programmers deal with abstraction all day. These concepts like public keys and signing and things like that they're very intuitive to people like me but they're not for our users so for example when a user goes to a coffee shop they might feel secure because the coffee shop has a password on their Wi-Fi what they may not understand is that that is operating in pre-shared key mode so therefore anyone else who has that password can just snarf up the packets and decrypt them. Third is the lack of feedback property. When my security is breached I may not get any feedback that it's happened until my credentials are used or something bad happens. So for example most people don't know they were on the wall of sheep until somebody logs into their system and messes with it. And finally the barn door property. Just like after the horse runs out of the barn it's futile to lock the gate. Secrets once leaked remain so forever. Small errors irreversible consequences. So for example with Dread Pirate Roberts the guy who ran the Silk Road. He plugged the Silk Road at one point using his real Gmail address which had his real name in there. The Feds started monitoring his physical mail and he had some fake IDs shipped to him with a bunch of names other than his legal one. So they go and talk to him. He says some suspicious stuff like well how do you know I ordered those. There's this site called the Silk Road that somebody could have sent them to me from as like a prank or something which seems like a bit of an odd defense. So they put physical surveillance on him and eventually they followed him to the San Francisco Public Library where he when he logged into the administration interface for the Silk Road they just yanked the computer off of him and imaged it. Finally the weakest link property. The network is only as strong as the weakest link that's the user and the attacker only needs to find one unpatched system and that unpatched system may be the person who clicks on a phishing link. So how can you make your privacy software more usable? The first is the idea of what I heard about when I was interviewing at IBM Design called the T-shaped Designer and the idea is to have a depth in one area and breadth in two. And the three areas they're talking about are user research, design in the more artistic sense, and software engineering. And usually what you'll see happen is when we're hiring engineers we hire purely on software engineering. We don't really care about these other two qualities. And especially on open source projects you know you may be the solitary developer or only one or two developers on a project so you may have somebody who's a crypto expert who's doing the crypto, you have a networking guy who's doing some networking modules and everything's just sort of going to fit together at the end. And that's just, you know, usability is not just a module that you can have somebody off coding in a corner and bring to the table at the end. So let's say you decide that you want your project to be usable. But you're experiencing this gulf of execution. You have the will to make your project usable, but you don't necessarily understand how you can do that. That's what the majority of this talk is going to be about. And there's two different techniques you can really use that we're going to talk about. The first is no. Oh, wonderful. Clicker decided not to work. There's two techniques you can use. The first is called a cognitive walk-through. Really? Okay. The first is called a cognitive walk-through and you can do that just as a solitary researcher or you can do a full-blown lab study. And I'm actually going to talk about both. I actually have a case study that we're going to share from some of my own research. So cognitive walk-through is pretty simple. You're just going to sit down. You're going to think of some of the tasks that you would want to do when you're using this software and ask yourself, are the users going to be able to produce the effect this action has? Are users going to see the control that they need to click on? Once users find this control, are they going to realize that it produces the effect that they want? And after that action taken, will users understand the feedback they get? So basically you just come up with a set of tasks that you need to accomplish. You go through the interface, thinking critically about how you could go, what could go wrong or what would be confusing. And you just sort of document the process along the way. And you can usually find most of the high-level usability issues just by doing this. But it does have to do with, you're probably the person who designed the system and you're going to have some blind spots about what may be hard to use. So to give you an idea of when we say core tasks, there was a paper by Jeremy Clark and a few others called Usability of Anonymous Web Browsing, which was the first examination of Tor usability. And this was the study that led to the creation of the Tor Browser Bundle, that they looked at all these different ways you could set up Tor, Prevoxy, Command Line, et cetera, et cetera. And the conclusion that the most usable way of setting up Tor was to have some sort of integrated browser that you just started up and Tor is encapsulated within it. So their core tasks were just to install Tor, configure Tor, confirm that the traffic is being anonymized, which is always a good thing. And then successfully disable Tor and return to a direct connection. Because back then everybody was really big on the Tor button extension. The idea was that we're going to be browsing and turn it off. We now know that that's kind of dangerous. We now know that's not the best idea, and it's better to have a separate browser. But that was the state of the art at the time. And that was actually considered really usable that you could just click something in the UI and not just be messing around on the terminal. So then if we can get 80% of the way there without having to do a big user study, why do one? Because whenever you come into the rest of your developers and say, I came up with this, you're going to get like, well, yeah, it's just your opinion, man. You think we should make these changes? I think they should read the manual. We got two opinions. 50-50. Which way should we go? Here's a hint. They're probably going to want to go the way that doesn't require any more coding. And then you know, you're an expert. If you understand the system, you're going to see things differently. So for example, Charlie here has a box of hornets. And he's saying, you know what? I'm going to put a little H here on this box so everyone knows that it's full of hornets. And these are the kinds of errors that you can make if you as an expert try and design an interface because you have this intimate knowledge of the system. So we're going to talk about a case study. This was actually a master's thesis I did when I was in grad school. I know I'm not a big fan of school either but when it comes to doing usability, it's actually really useful to have some background in experimental psychology, anthropology, things like that. I'm not sure if doing a PhD was the best move. Getting a master's and dropping out probably was though. So just to give a really quick overview, just in case not everybody in the room is familiar with TOR. It's basically this anonymity service that uses onion routing technology. It was originally developed by the Navy. The Navy realized if you have an anonymity network that is only used by the Navy, maybe if I don't know which particular person the Navy is using TOR, we know someone in the Navy is visiting my website and that might be problematic. So they decided to expand the pool. It got taken over by the EFF for a while and funded that is. And is now its own 501C3 nonprofit. And the basic idea is that you're routing your traffic through three nodes, the entry node, middle node and exit node to mask your identity. And you should note when using TOR that your content is not connected when it's exiting the network unless you're using HTTPS or something. So, you know, you're just going to get a list of TOR nodes from a directory server. You're going to go through your path. And then every 10 minutes you're actually switching circuits and each website gets its own circuit. So if I'm using the Guardian in one tab and I'm looking at the Washington Posts and the other, you don't necessarily know that I'm looking at both those websites. There had been some previous research on TOR, but they basically just did this cognitive walkthrough, said put it in a bundle and then did the whole mission accomplished and fly off in a helicopter. Typical of academic research. So the original TOR browser bundle at the time we were doing this was a custom Firefox build. They also included Vidalia that was doing the proxy management and then it had the TOR button extension built in which allows you to turn the TOR browser bundle on and off. So just to sort of give a timeline about when all this went out, we did the first of the two studies back in 2012. We ended up at this little conference in Spain which I did not know this until they showed up, but most of the TOR developers went there as well. So when I was giving this original research I had like Roger Dingledine sitting in the front row. So then we spoke with TOR, they actually ended up making some changes to the UI and then we developed a custom extension to do some of the changes that weren't being put in. And then we actually did a second usability study where we just verified because at the time I was in grad school it wasn't good enough to just improve an open source project, you had to make it like a full blown scientific experiment. So, you know, and we did find out that the, well I'll just get to that later. So yeah, two studies about the same number of students download and install the TOR browser bundle. At the time the second study, the TOR browser bundle had done several changes. Most importantly they had dropped Vidalia which as we'll see later, having Vidalia in the mix really just create a lot of weird issues and rolled that functionality into the TOR browser bundle so that there was only one window for TOR at any given time. And the way we went about this was actually pretty simple. We passed out consent forms and study sheets. This was an IRB approved study and I'm not going to get into that in detail, but basically you know, it's not considered ethical to just take somebody's lab section and say, congratulations I'm going to experiment on you. We get consent for the experimentation and if they had not wanted to participate in the experiment they had to be free to do something else instead so we let them write an essay on how TOR works or something like that if they didn't feel comfortable doing this. We briefed them that this wasn't a typical lab. So normally if we're doing a lab for this class they have some sort of task they have to work through on their own so we consider cheating to just ask the person sitting next to them for the answer, ask the TA for the answer so we tell them that each time you encounter an issue just raise your hand and I'm going to come over and tell you how to fix it and then when I did this I would say please just write down the issue you had because we want to catalog all the issues that the software is encountering and then this is the really interesting part because how do you move from a list of complaints to something that you can work with and you have this thing in psychology they refer to as coding responses but they're talking about coding them in the sense of coding them in the categories so what you do is you get two people to independently come up with a list of categories you agree on the intersection of those or the union of those becomes this final category list after you talk a bit and make sure you're not overlapping and then both coders independently read through each issue and decide to assign it to a category and then you can actually generate what's called Cohen's Kappa which is just basically a fancy way of making sure that the amount of agreement between the two coders was more than you would expect from random chains because you don't want a situation where we're both just completely going wild and so we came up with a bunch of issues but it's really hard to read that graph so I've also got a pie chart and the big huge takeaway was that about 57% of the issues boil down to three distinct usability issues you change these three things and hypothetically you can cut the usability issues in half and we're going to go in detail into each of these the first was the concept of long launch time where's that pointer else oh so you'll make a laser but you won't go forward interesting so on the old version of tour when you clicked to start tour you would be told you're now connected to the tour network and at this point users would go yay I'm on tour I can be anonymous the problem was that only the tour browser bundle window was anonymous nothing else was so let's say I had previously had Chrome open and I'm being told I'm connected to tour okay I'm going to hop into Chrome, log into Facebook and organize a protest and probably get shot because I was not anonymous when I did that the proposed solution was to we actually didn't think high enough we said you should just alter the lag so that you didn't have this situation where Vidalia was popping up and then there was a long delay until the window opened and the tour project actually just rolled all of this functionality into the tour button extension the second issue we encountered was browsing delay and then this becomes a little bit harder because people are basically complaining tour slow there have been entire DEF CON talks about the fact that tour is slow and sure as he's working on it and the thing about this is that you're just never going to be as fast routing your traffic through a series of nodes as you will if you were doing a straight shot so then it becomes more of a communication issue how can we make users aware of this trade off you know there's been some research that users are more willing to do some sacrifices if they're explicitly using privacy enhancing technologies and also it's just like in general like you know if you're in a coffee shop you might not be as angry you can't use Netflix then if you were at home it's about setting expectations so we came up with these warning dialogues that could pop up when tour was experiencing a large amount of latency and say due to security and there was research that basically shows if you tell somebody we're doing this for security they'll tolerate all sorts of stuff proof of that for example lies within the TSA so tour knows you're experiencing delay we apologize thank you for your patience hypothetically if they're being shown this they're not going to be as angry and then the third thing was when the users were window discriminability so these users aren't sure which window was the tour browser bundle well at the time it was a custom firefox build in that they even still had the firefox logo so if you've got firefox open you've got the tour browser bundle window open you're not really sure which is which if you go off and do another application so they end up developing this really nice logo really sleek I really like how it's not focused directly on the US directly on the trying to show the entire world a nice little artistic license there so what happened after we made these usability did this study and so between the two studies the number of people who just said no problems because I guess I should backtrack a little and say at that point one guy got to the end of the thing and he's like I didn't have any issues I'm like are you sure because if you didn't have any issues that's a valid data point we don't want to discourage you we just didn't want a situation where the entire class said oh yeah I didn't have any issues and then walked out after two minutes and we probably would have just discarded that result but there was double the amount of people after we made these changes who said oh I'm not actually experiencing any usability issues I've had an across the board reduction in these usability issues so long launch time when weight down but everything went down at least a little this went through a lot quicker than I thought so just to kind of summarize usability is this uniquely hard problem you should try and hire these T-shaped designers who also have experience with UX research and UX design and that doesn't mean that they have to be experts it just means that they have to have a passing familiarity cognitive walkthroughs can help sess out usability issues even if your project has minimal resources and you should talk to your users usability is not a one-time operation you need to revisit this periodically and finally you may be sitting here thinking about what you can do next and I would recommend that you read the paper why Johnny can't encrypt there's not the enemy by Angel which we were talking about earlier you read why Johnny can't blow the whistle which is the paper that I was just discussing there's an entire human-computer interaction bibliography that has a bunch of free papers and also just you know come up with a few things you want to read on your own the thing I really want to push in this talk usability is not a checklist I think any DEF CON attendee who I pulled out of the hallway would probably if I said we're sick of security being a checklist they would nod furiously we don't want this to have the equivalent of the PCI standards for usability where we're just checking boxes we've done this we've done that we did a cognitive walkthrough we actually want to have usability I'm just going to back up because I thought I accidentally skipped something but I guess I didn't so I just moved through these a lot faster than I intended but that does mean we have some extra time if anybody has any so go ahead so the question is if I've heard of anything that would make dot onion addresses a little easier to read a little less fishable because if I can know this is the site I'm going to I haven't seen anything specifically related to dot onion addresses I've seen some really interesting schemes for example Mozilla had a thing up a thing it was like about a month ago where they were talking about this idea of using emojis as ciphers that like you can use chunks of characters equals this emoji one chunk of characters equals that so that could be something that you could do you could try and translate this long string of characters into a series of pictures or you know it could be as simple as trying to display it in chunks like there's been some research that people usually can only remember five plus or minus two seven plus or minus two chunks of information so if you can split the onion up you know just adding dashes you know if you can just visually separate that out a little bit so that people can check for at a time that can be helpful I mean generally like if I had the perfect ideas like you have like a general idea of what your software product is going to be and you know you can write back end stuff before hand but if you would want to have like a general at least a general idea going in of what the flow is going to be and that doesn't necessarily mean the interface specifically like you know you can write back end stuff before hand and that doesn't necessarily mean the interface specifically like but if I was going to write like the next veracrypt I would at least know a basic flow of like somebody's going to you know create a container put things in the container open the container things like that that you would want to try and come up with these flows early on and then yes you could try and develop a little bit in parallel because I can think of another a number of scenarios where you're going to be working out the kinks in something on the low level and then be able to use a general UI but you can at least be planning it out Well, sort of, if you build a security system or protocol or whatever that's very secure but it's completely apparently unusable like you don't have a, well you don't have a research solution, the usability challenges it doesn't really help it so like that's sort of like any way I was coming at it is like should I constrain the way I'm developing other parts of the protocol by usability research what you can do is you can, it's always possible to offer like a simple UI and then have something that you can click on to go into these sort of advanced options for the user who wants to because the problem is like what you present to the user will be clicked on so if you bury some of the more complicated stuff in the UI the things that you know only the experts are going to need you know if somebody is truly an expert they might not even be doing these things in the UI to begin with they might be doing these things in the UI so these were two different semesters so this was like you know the fall of I forget when but it was like let's say fall of 2012 fall of 2013 it was two separate classes yeah so signal and tour are both good trying to think you know vericryps doing better you know not perfect but doing better than a true crypt used to do is that we didn't really go into in this talk because I thought it was going to be pressed for time was there was a lot of security warning confusion and you can do entire papers just on how to design one security dialogue but that can be confusing for people you don't see this error as much but when people would get mixed content errors so you would have it where parts of the page where HTTPS encrypted and parts weren't and you would often get those when HTTP and the rest of the page wasn't so there's a lot of times where you'll get these really advanced warnings that it's just I don't know what's going on I don't know what this warning means so there's a lot of room for improvement there but it's one of those things where you know they've gotten very very good UX and the remaining problems are really hard ones yet how do you deal with people that when they get you know release one and then release two and they're expecting to configure privacy settings and move around based on that user research how do you handle that when it's not really plausible to maintain both code paths? Sure so that's an interesting point because then basically you've got this you know the bacon I forget what the technical term would be like something like for example when Microsoft Office moved to the ribbon you can the ribbon is probably more intuitive for somebody who's never used Microsoft Office before but for somebody who was used to a very specific questions to get very specific responses having to relearn the UI I think that you know you basically have to decide that there is going to be this sort of switching cost for lack of a better term of you know switching to a more usable UI and you can make sure you have some documentation available so that if somebody is having trouble finding their tasks that they're not just left out in the cold maybe not just you know and really you know like I think before set out the browser mandolin when they drop the daily and things like that they do like a blog with every release the details what's been changed and if you're doing major changes you can try and have some help.