 Since yesterday, I had to start late. I thought today I'd start early to make up for it. So good morning. I'm Julian Harty. Some of you have met me. If not, chat to me later on. I'm in the bug bash for the Sony M3 stuff. I've been working mobile properly for about 10 years. But this talk is more general about user experience on web and on mobile. And I'm happy to take questions through, but I have got quite a lot of material. So sometimes I'll move your question to the end. So the first thing we want to do is to understand what does UX or user experience mean for people. I picked a couple of examples. I think these are personant for many of us in our lives today. I suspect that some of you now use Uber and Ola cabs. And you're familiar with booking taxes in the good old days, when we normally phoned someone up and said, please can have a cab to take us to whatever. And of course, you've got the tuk-tuk's and all the rest of it around India, which help us to get from place A to place B. One of the things that's very clear to me is that the companies such as Ola cabs are transforming the market. And they're transforming, in part, by changing the user experience. Gone are the days when I would fly to a foreign country and try and work out who the heck's going to take me in a cab, how much they're ripping me off, and just which route they're taking me to the city that I thought was five kilometers away. We seem to be driving forever and ever in the wrong direction. Because these days, we've got mobile devices. We know who our driver is. They've been rated by someone else. You know, they've got four point something scores. They've got 57 reviews. We can actually see in real time where we're going. If it's a problem, we know how to complain and who to complain to. So this is transforming our user experiences. And it's transforming it more for some of us than others. And by that, I think that some people have been treated worse in general by taxi companies than others in the past, particularly women traveling alone. It's something I've seen and observed quite a bit in life. But the people are more vulnerable, tended to suffer more. But the industry is now moving on. So the feelings, the emotions, and the user experience is what we personally feel. It's based on using something. We can't have a user experience without using something. So you have a user experience at the conference. You have a user experience for those who stayed yesterday evening and listened to the music. And some people played the musical instruments. And it's essentially what we do and always using the system. It involves a number of different factors. So for instance, it's doing things like credibility. So I don't know what companies you work for, what your systems are. But one of the reasons people use your system is because you're credible. If you weren't trustworthy, people wouldn't use your systems. It's to do with desirability. So how many of you have iPhones or Samsung edge phones? So part of the reason we do this is because they're desirable. They look good. They reflect well on our status by having these devices. So if we're standing in line at 6 AM waiting for the Apple Store to have the next device coming out, we're doing that because we desire the device. Otherwise, we just order online and arrive next week. Who cares? But no, we actually make the effort, even this digital age, to go and stand in line and buy stuff. It's holistic, and it always includes our perceptions. So it's clouded or affected by who we are and what we think. It's affected by our expectations. So with the taxi examples I gave you, we now expect to be able to rate the driver and the experience. We now expect them to turn up on time. And for us to be able to see, look, the driver's three minutes away, two minutes away, one minute away. I had a friend fly over from California to visit me in England in London. He'd booked a cab using one of these services. And the cab driver remained 15 minutes away on the system. The little dot said, the driver's 15 minutes away. And he got really upset about it. Now, in the old days, you just went to a black cab. You paid your $150 for the joy of sitting in a black cab to leave London Airport because it's a lot of money, and they did what they liked. But now he got upset. He complained, et cetera, et cetera. So his expectations had changed his experience. Our mood changes our user experience. I'm happy today because I won a prize. OK, so whatever else happens today, I'll be in a good mood. And that means if I'm using your software, I might be in a slightly better mood than I would otherwise be. On the other hand, if I just had to pay my tax bill and the tax bill was much more than I wanted to pay, I realized now I haven't got the money to buy my new iPhone, I get sad and grumpy, and I use your system, the user experience is likely to change. The situation affects what we do. So I rode a motorbike with my daughter around Europe. I use a GPS to navigate. I use an iPhone. Guess what happens with an iPhone when it's raining? The touchscreen stops working. You cannot actually get it to respond to touch it. So then I was in this rain because there isn't much when you drive a motorbike trying to get the bloody thing to work. So I could see the GPS to see where we've just gone wrong on the road. So very, very frustrating. So the context changes our expectations. The next thing is that we can look at user experience two different ways. One is the digital experience. We can essentially measure that from software reporting things automatically. And the other is the analog experience, which is essentially feedback from a human being. The human being actually communicates and is part of this process. So in terms of measuring user experience, some of you will be familiar with terms like click-through rates. So if you have an advert or there's something on a site and people are going, yes, they're clicked on it, you may have things like dwell time. How long did the person spend on whatever the hell it is in front of them? And the software is a web page or a mobile app. And we say, they only spent half a second on that. Too short. What does too short mean? Well, we're not quite sure. But we think it means it wasn't engaging. So we start to infer things from the time. Sometimes people spend too long. And we infer things because we believe too long is bad. Because perhaps they didn't comprehend the page. They didn't understand the content. Does all make sense? All right. Usage. So again, we measure usage. We have sort of one-day active, seven-day active, 30-day active, which simply means that if a group of users, all of you, started using ConfEngine, the app for the conference, yesterday, how many of you are still using it today? Not many. OK, so the retention rate is already dropped. Now, some of you may use it later on today or tomorrow. So you would count within the seven-day actives if you haven't used it earlier on. So this is a standard measure in the industry. And we like to see retention. We like to say that retention stays high. And when it stays high, we assume the user experience is better. Then we've got this sad old world of crashes, particularly when we're dealing with mobile apps, the device will sometimes say to us, sorry, the application has stopped or the application is not responding, close. That we infer gives a bad user experience. And then finally, we're looking at the usability measurements, including things like task completion. So how long does it take for the user to get from the beginning of something to the end of something and complete it successfully? And by that I mean they've navigated each of the stages. That may be a three-step process. Log in, add something to the shopping cart, pay, check out. Make sense? The other part of the measurement is correctness. So did they do the right thing? So let's go back to filing taxes. Did we provide the correct information or not? We could have got to enter the form and then whoever checks the form deems that it was incorrect. So correctness is important as well as task completion. When we're looking at analog user experience then, as I mentioned earlier on, we're dealing inherently with human beings. So it could be that we interview users. We could do that in little user communities. We could ask them into the company, whatever it is. We could record them. Some recordings are explicit, so I'm explicitly being recorded by a camera over there. I know that. When you ask questions, you're probably implicitly being recorded by a camera because you're not really paying attention. Till suddenly you look up there and think, oh my good, my head's this big. You know? Oh yes, luckily I'm still pretty busy or whatever it is. So we have the implicit recording where the apps in particular are using the camera to record us or the software has got some little bit of JavaScript running in it that's recording what the user's mouse is doing and where they're hovering and all these things. So we've got the implicit recording. And typically you've agreed to it but you agreed to it on page 37 of those terms and conditions you just clicked through. So I would argue that we didn't really agree to it. We're just so fed up of companies asking for our first child that we clicked through the agreement and say yes, continue. The next thing then is we can have surveys. So I expect many of us have seen these little pop-up. Would you like to rate us? Would you like to give us two minutes of your time? No, I wouldn't. No, I really wouldn't. Just please go away. Okay, open the background. I'll kill you later. You know, whatever it is. Oh, but you get a free $20 voucher off. Okay, click, click. Yes, yes, yes, yes, yes, yes, yes, finish. Yay, $20 voucher. Okay, so it's important because the feedback we get from human beings is often biased by various factors. Like, I just want to get this out of the way, rather than I truly want to give you feedback or like our glorious vote in England two days ago in UK, we asked two insane questions. Not the sensible one is can we change what we're doing? But do you want to stay or do you want to leave? Well, I don't actually like either question but I have to answer one of them so I answered one of them. But the question wasn't well formed in my view. I'd prefer to have a different question. And then we've got this concept of net promoter score. If you work for sites that sell stuff, you'll typically be asked, or video sites, things like Netflix, you'll be asked for a net promoter score which is essentially measuring the likelihood of you saying either good things to your friends or bad things to your friends about the service, the app, whatever it is. And there are three buckets. There are the buckets of people who don't have a strong opinion and though we're not going to say anything to anyone. The bad news is you don't count at all now. You're not important to net promoter score. The ones who are strongly against something, you'll go and actively say to your friends, don't use ConfEngine because it's a pile of crap. So you're important because you detract from the app or the service. And conversely, we have the net promoters who say, brilliant, love this stuff. You must use it, you know, when you're running your next conference, definitely use this product. And it's important because of the network effect, the human effect. So that's the net promoter score. And then finally, in the mobile world, we quite often have in-app feedback and similar things exist in websites that says essentially, would you like to provide us feedback? And they then sometimes, particularly in apps, bifurcate what happens next. If they think you only have a positive review, they direct you to the public website. Go on to the public website, write a five-star review because that helps our ratings. Versus, you're going to complain? Talk directly to us. We'll take care of you because they don't want the bad news to be public. So talking about app stores and reviews then, 4.4 was the score for our Android app. For one of the companies I was working with, it dropped to 4.3. Is this material to you? Would you still download an app if it was a 4.3 rating? Probably. So you think, eh, it's a bit annoying, isn't it? But what the heck does this actually mean to us? Well, what actually happened for us is we noticed our revenues dropped between 5% and 10%. Roughly on that day. I thought, what the hell? How could it go down so much for what seemed to be so little? And there were several factors that happened. One was in the app stores, it's essentially got a search engine in it. And the search engine gives results. When an app gets a lower score, it dropped down in the rankings. And again, you'd think if I typed olacabs, the first result's gonna be olacabs, isn't it? If I typed eBay, the first one's gonna be eBay. But if I typed something like dating or photos, I'd get a generic category. So we'd drop down in one of the generic categories. The second thing is that the users of the app, I think implicitly and unconsciously, noticed something was wrong, which is why this out of providing more negative reviews in the app store, and also spent less money. The cause of this seems insane. You sort of look at this and think, what the heck was happening? And this is a sort of mock-up of the screen. Essentially, in the background, I've used purple and yellow here. We had some images shown to people. And we had a little dialogue box that appeared in the center. And you look at the dialogue box and it seems kind of central, isn't it? What's wrong with the dialogue box? Well, what was actually happening is, if you look here, this is the English, tap on the color you prefer to continue the game. This is the mock-up, by the way, not the real words. But in German, do you notice it's a little bit longer? And here, the French being proud of their language, use even more letters to say the same thing. And I think this is Tamil, isn't it? So this is Tamil saying the same thing, I hope. Google translate, so help me if it's wrong. So what was important is if you notice here, this is a screenshot of the equivalent dialogue box in German. The text is actually wrapped to three lines. Now, again, you kind of think, well, no big deal. But what actually happened is, we had to raise condition the code. It said, draw the dialogue box and get the localized translation from the server. Sometimes it got the localized translation first and it drew the dialogue box around the contents. So it put it in the center. But sometimes, it drew the dialogue box first and the text arrived afterwards and we updated it. If the text was still small enough, the dialogue box didn't change in locational size. But if it did, it sort of expanded and dropped it down a little bit on the screen. And that seemed to be enough for us to get the lower app store reviews. When we fixed it, the scores went up. It's a bit hard to read story about the text. It's lots of words. I want to give you an example of a review of an app that I found in America. It's something called M&T Bank. Can you see up there the customer ratings? Current version. What's the score of that? Yeah, one and a half stars. And compared to all versions, which is two and a half stars, we can see it's going downhill. Now, how many of us, if we were sane, would install this app? Who wants to install a one and a half star app on their device? Well, it's a bank. So it either is your bank or isn't your bank. If it is your bank, you're kind of stuffed now, aren't you? I either have to use this app or no app. I'll go back to the website. And you read the feedback here. And the feedback here, oops, this is very sensitive for some reason, anyway, the feedback here is essentially customers complaining. But from a testing perspective, quite often, these are things that we can catch in testing in future. So it's good for us to know what's happening. Most of you will know of Kindle. So it's an Amazon device. It's also an application. I was in the States. And this was, I think, around the 30th of March this year. Again, you can't really read the detail of the text, but hopefully you can see the headlines. So feedback number one says, perfect and keeps getting better. Five stars. Brilliant. Two, how to break an app. Nine, awful. 10, update ruined my Kindle on. Four, fab app. Five, good but dreadful after probably a recent update. 11, oh no, awful after update. 12, one star, latest update. 13, not working after today's update. Six, what did you do? Brackets, Amazon. Rubbish. Buggy updates. Updates broken my app. Now, we hope that someone's paying attention, don't we here? So the good news is that less than a week later, Amazon had updated the app. I took this screenshot on the 6th of April. So it's just been launched. Did you see what they do? Performance and stability improvements. Customer ratings on the 6th, so one day after the new release had gone out, had gone to three stars. Not quite as good as the historical score, three and a half. But we see that a month later, the app's actually at four and a half stars. So Amazon had done something right. So this is a success story. We see something went wrong. And basically they should have perhaps fixed something in testing and found it in testing first rather than push out the buggy one. But they fixed it within a week and we now see the results of it. Staggeringly though, 96% of the companies who can provide feedback on their apps don't bother. So you're a company, a bit of detail I'll take in a second, but if you're a company and you're not paying attention to the feedback on the app stores for your apps, you're in deep trouble. Because otherwise you get users who get grumpy. And what happens is users perceive that you're not responding to them, they get pissy. So the user experience will inherently decrease. Now the detail is that on Android, on the app store, on Google Play, you can provide feedback as a developer. So you complain or write good news or whatever. I can go as a developer and write something back to you. So it becomes a conversation and a dialogue. On iOS, you can't provide feedback, it's sort of listen only. So you can't respond directly to the users. So there is a subtle distinction, but on Android, 96% of people don't bother responding to the feedback from the users. And it's a massive difference because you can actually help improve the customer experience just by providing feedback and saying, yes, I understand there's a problem. We're dealing with it, we've got our best team on it, we're just waiting for the final testing to go through and for the app store to approve it. And the rest of you look at that and say, good, they care. They're dealing with it, I'll wait a few hours. And then I'll try again. Rather than if you look at it and you think it's an imperious company who aren't listening, I gotta make sure they listen. I'm gonna tell them another one style, another one style and we all get grumpy. And these have a long effect on it. The next thing we do then in terms of digital analytics is looking at our digital experience is using analytics and heat maps. These are feedback cycles for testing, or in this case an app, but it's fairly similar for web applications. The difference being that we wouldn't tend to have an in-store app feedback. So we have an app, we have some sort of testing. And the feedback cycle in most teams is fairly rapid. We're perhaps in the same room, the same team, the same building, even if we split in the same organization often. And we have a sort of common trust within the organization. We've all signed confidentiality agreements. We can talk fairly openly about what's good and what's bad. And that feedback hopefully helps us to improve the app for future releases. If not, what's the point? The big narrow circle is social media feedback. We can go on to social media, Twitter, Facebook, whatever it is, Instagram, and we can write things. We can go on YouTube even and record videos, what we like and dislike. This feedback is important, and we can respond to it. And again, companies tend to learn to do that. The app store feedback for a mobile app between 0.1% or 1,000, and 10% of users provide feedback, so one in 10. The average is about one in 100. So how many of you have downloaded an app on a mobile phone? Some haven't? Okay, most of you have. How many of you provide feedback? Okay, so we're looking at about a third to a quarter of you provide feedback on at least one application. So the majority of you haven't provided feedback on any app even though you download and use them. So I don't know whether you like, dislike, tolerate, love, because you don't provide feedback. You're the silent majority. So the in-app analytics is where we add a little bit of code to our application or our website, and it captures digital sort of fingerprints of what we're doing. It says the user's just gone to the search page. The user's completed the checkout, whatever it is. Quite often it's provided by a third party company, so it may be provided by Google or Flurry or any of the other companies that are out there, and we go and look on a reporting server and get the data back. So how does this help us? So one of the important things is in any software team, and I used to work at Google and eBay and Salesforce and all the sort of latest high-tech companies, great innovative developers, but inherently within a team, we're limited by what we perceive and what we think. So for instance, before I came to India, which was about five years ago, the first time, I had no clue what being in India was like. I knew a lot from talking to people and working with people from India, but until I came here I didn't experience it. So if I was trying to out an app for the Indian market, good luck. I just don't understand enough about things. I don't understand the norms, I don't expect what you want, what you dislike, colors, choices, all these things. So we need to get the feedback, but teams can't get the feedback unless we get it from someone who knows. So given that with the app stores or any other feedback, we're getting feedback from about 1% of the population, we're ignoring the 99%. With the analytics data, we're now getting data from roughly 100% of our population. I say roughly because sometimes they're behind broken networks, oops, or behind firewalls, or in flight mode, or somewhere where the network's not available, and so the messages aren't being sent from the app or from the web application. And we may have to wait and get them later, or they may just say, no, I don't want to send feedback or use some sort of blocker to stop this traffic going through. Again, we block much more on the website than we tend to block on the mobile devices because people don't really know how. Low cost of operations. Once we put the software in, unless we're a very high end customer of the analytics services, it's typically free. If we're a big brand company with billions of transactions, then we'll probably pay for the service and support. And hopefully this real world usage can help drive our testing and analysis and bring more realism to what the testing we're doing. To give you an example in the company we were working with, we had a team from all over the world, and we were based in Soho in London. And we were looking at the analytics and we suddenly realized that we had this hotspot of users in Paris. Paris, France, for those who come from America. So Paris, France. Now what's key about people from France? They tend to speak French. What were we testing in? English. Now it wasn't because we didn't have people who spoke French, but we hadn't thought, we hadn't gone, duh, we've got a bunch of users here, we should make sure the app works well for the French users or the people who speak Tamil or whatever it is. But because we got data from analytics, we could start to do this. Again, an example from mobile apps, I think the web is fairly similar. We've logically got roughly three layers of software. We've got the user interface, we've got some sort of application logic, and then we've got the operating system. So the analytics I'm talking about is ideally suited to the application logic. It's kind of what the app is doing, what the software is doing, the purpose, the intent. It reflects a little bit on the user interface, but it doesn't tend to tell us whether it's the right colors or the user's like they look and feel and the rest of it. But again, sometimes we're getting fur things about the GUI, the click-through rate has dropped, perhaps the problem is something to do with the GUI. We don't know, we're gonna investigate. The GUI side of it then, something called Heat Maps, which is available for web and for mobile, helps us to understand more about that. And finally, the operating system, unless it's instrumented, by instrument it has extra code that records what's happening, we don't tend to get much, but again, we can infer some of it from the analytics. This is an example of a heat map from a company called Apsi. They're one of the leaders in the analytics market and in the heat maps market. And we've got an example of some uglier sin, terms and conditions. And you've probably all seen something like this. It's full of capitalized text, whether it's blurry or not, it doesn't really matter, we all don't understand it, we don't want to look at it. All we know is we have to do something at this point, don't we? So what do we do? We thump on the little bit at the bottom, don't we? Go away, just okay, whatever it is. But the heat maps told them that something was going wrong. In fact, what they noticed is that the drop-off rate on this screen was around a third of all users stopped at this point. That's quite a lot, isn't it? So there were two problems. The first was that they actually had to tick the little, I agree, to the terms of service. There's a little tiny tick box to the left, but a bit like the way this screen resolution isn't perfect, users with a fat finger of device don't always hit exactly in the tick box. And this meant that they didn't necessarily tick it. And you can see from the heat map there where people were actually tapping versus where the size of the button was, on the tick box. The second was, quite often, they were just pressing the button. And the logic flaw was, if they just press the button, the app didn't respond. It didn't say, you need to tick the tick box. It just sat there waiting for the users to tick the tick box. Now, functionally, if we're sitting here doing functional tests, we might say, yeah, it's passed the functional tests. You need to tick the box and then press the okay button. Yeah, tests passed. But fundamentally, the problem was one of design and one of area. So the app developers, having seen this data, did two changes. The first is they made the whole, I agree, into a live area. So that meant if you hit anywhere on that, it ticked the tick box. So in other words, the users who saw the tick box and wanted to answer yes, or had to answer yes, they were able to do so. The second was, if you just press the button at the bottom, like many of us do, and you hadn't ticked the tick box, they had a little dialogue box saying, do you agree with terms and conditions? Please tick this box to continue. And again, the users now knew what they had to do and could tick the little box and move on. And as a result of this, they got a significant improvement in the retention rate of users. Microsoft have a big research arm, and they're some of the world leaders in researching about software development. So this means that this is from Microsoft work from 2010, and they were looking at analytics for software development. And this was looking at things like the past. We look at the past quite often, don't we? And we look at trends and defect reports and the rest of it. But we may not look much more than that. However, we can start to do some analysis of how and why did things happen? How and why did these bugs creep into the system? How and why did we see these things happening, these effects in the world? So we can look then at software quality models and say, how do we improve how we develop the software? Perhaps we can fix things that way. Today, we've just launched a new version of our software. We're getting some sort of alerts. We can hear complaints from users and feedback. So we need to know what's happening right now. How many users have upgraded? Are there any problems? And we need to know that because sometimes we need to make a decision. What's our next action we need to do? Do we need to pull a broken app from the App Store? Do we need to revert? Do we need to have more people on customer service because we know we're gonna get more complaints from now on? Do we need to do this, that or the other? Do we need to buy more servers? Whatever it is, because most of us now order servers online. We're going to Amazon web services and say, yes, three more machines, please. And two minutes later, we've got more capacity. Then we can look into the future and many of you in companies who do the sort of forecasting. How much do you expect the business to grow this year? Oh, well, it's gonna be busy in November. It's gonna grow by 20%. But as importantly, we can do some sort of modeling and simulation and look at things that will change the model rather than just persist the model. So there may be a transforming effect that we need to deal with. Right. Any of you use analytics in your companies? I'm intrigued, did any of you evaluate alternatives? One of you. So by evaluate alternatives, I mean you looked at Google and you looked at Mixpanel and you looked at Flurry, you actually did some testing and moved on. So a few of you do that. But many of us, particularly when agile teams and we're on a one week or two week sprint, you get a little story and it says implement analytics. Oh, bugger. What does that mean? Let's do a quick search on Stack Overflow or the web. Yeah, what's everyone else using? Oh, well, this, my mate's using Google, so I'll use Google or I'm using Android, I'll use Google or my friend's using Flurry, so I'll use this. So we make that decision very quickly and then we try and worry about how to implement and then some lucky fool worries about how to test it. So we need to deal with two fundamental challenges. The first is something called precision. Precision means is it repeatable? If we see this as a series of inputs, do we get the same outputs? Accuracy is looking, are we on target, even though we may have a bigger spread? And ideally, we get both of them. A different way of representing these, by the way, these come from Wikipedia. So you're very welcome to look at Wikipedia and all the material will be available. So there's the other way of looking at it then. It's precision is the area under the curve. If it's precise, it's a very narrow area under the curve. Otherwise it's a very broad and flabby area under the curve. And the trueness is how far we are for the correct answer. This is the heat mapping software. It's from a French company. I created a new account. I hadn't used the service. Do you notice anything odd for a new account where I've never used a service before? I've got two users. I've got seven sessions. I've got, you don't see it here, but there's 20 there. There's seven somethings there. I've got trial variants. What? Now, if it doesn't work at zero, if the numbers are accurate at zero, how much trust can I give it when it says you've got 20 million customers? So I actually spoke to one of the leaders of the company and said, these numbers seem wrong. And the answer was, well, our big customers like it. So I look at that and think, huh? Well, it happens the world over. So this was an example where I was working at a company and they had two analytics libraries built into the software. The numbers didn't agree with each other. So because they didn't agree with each other, they said, what should we do? Let's get a third piece of software and add that. Of course, a sensible approach because hopefully the third one will agree with either A or B. Well, guess what? They didn't. So we now have three sets of divergent answers. Right, so let's take the one we know is wrong out. Make sense? The trouble was that was the one that gave the most positive numbers. Now, if you ever worked with a sales and marketing team, you'll know that quite often their bonus depends on, increase the market share in India by 35%. If you take away the thing that's most likely to give them that answer and they're being paid $100,000 of results of achieving that, funny enough, they get grumpy. Because the numbers of the three systems don't correlate, they don't agree with each other, you can't just sort of extrapolate from one of the other two and say, oh, it's 10% more than. So we were left in this ugly bind because no one had actually thought to test it at the beginning and make sure that whichever libraries they're using were accurate. I mentioned that it's running our devices and sometimes it's working implicitly, so we've got a dark side of analytics to think about. When we're capturing information about users, we have a responsibility to those users. And there are several examples of that sort of a little bit naughty. So at the top here, we've got 50 million Android users were being tracked for a flashlight application. Now, if you've got a modern iPhone, you probably know you can just go and turn the flashlight on your phone as it is. But in the old days, you had a little app that did that for you and this is an Android app. And essentially, they'd release it in the app store but essentially it was a bit of spyware built into it. But iOS isn't immune either. So here, this was October last year. The apps were using something called private APIs and they were using that to harvest personal information from the users. Now, the apps are people just using them but under the covers, the app company had used a third party library to get the analytics data and that library was misbehaving. But people hadn't checked that. To be fair, it's fairly hard to check these are security vulnerabilities but nonetheless no one had caught it until some security company found it and report it to Apple and then all the apps were pulled from the market. And one of the things you probably know is that you can give your app's permission to do things like make and receive phone calls. If you have Skype, you've probably done that. It may send text messages for you. Well, in some cases those messages are cheap or free. They're part of your bundle with the telco. Other times they call expensive premium rate numbers in Bogota. You probably don't know where Bogota is but it's $2 a minute. So your phone bill is now being eaten at $2 a minute. Silently, you don't know anything about it but the app is doing this for you. Not very friendly. Interesting company from Israel called Safety K are actually analyzing reverse engineering of these libraries to look at the real behaviors and then reporting on them. This is an example from academic research. They were running a multi-year trial across millions of users and what they did is deliberately reduced some of the accuracy on the data they were collecting. And what you may notice, we've got two US cities here, Dallas and New York and the differences in color are the difference in latency in the network. And I hope you can see that there's clear differences in behavior even on a 10 kilometer by 10 kilometer grid that's enough to act upon. So all this wonderful technology and gathering information about the user experience is necessary but not necessarily sufficient. And here's an example for you. Again, it's from academic research. Do you see there's a sort of red rings which the keyboard and the little sort of dotted blackish rings are network requests. Do you notice there seems to be a correlation that when the user presses on the keyboard a network packet is sent? Yeah? And it seems to be a one for one pattern. Now, you're the testers for this project. So you're looking at this and you say, is this what it's supposed to be doing? Is it supposed to be using lots of network traffic or not each time they use the types? Yes or no? No? Who thinks yes and who thinks no? Okay, this is a bit like the survey for the UK voting. So the real answer is it depends. Because if this was something like type ahead where we're doing searches, it's auto completing and I'm typing B. So Google's trying to guess what the hell's B. Okay, well it starts with some B words. Ah, ah, bah, bah, bath. Maybe he wants to find a bath. No? Ban, ban? Okay, and eventually it'll say, oh, it's in Bangalore. Okay, let me give Bangalore as a result. So that's what's happening perhaps. Perhaps that's a good thing in the application. Perhaps it's a bad thing. We don't know by ourselves. We need to go and find out more about it. So some things that might help us to improve the user experience. The first example, this is again an academic paper from a mix of university people and a company called Shopify in the States. And they were looking at the 10 most frequent complaints. And here we can see things like functional errors, feature requests, apps crashing in order. But then they looked at most impactful ones. What you may not see very well here is that privacy and ethics was actually the number one even though it wasn't often complained about. Again, in similar research, what they discovered is that app store feedback is one to five stars. One being low, five being high. Paradoxically, it wasn't the ones that carried the worst news at the most serious. It was the two stars. Because it's very easy to write a one-star review. Hate the app, dump it. But that may be based on 15 seconds of looking at the app. The two-star reviews tend to be more considered and more thoughtful. I've been using this for a while. I can see there's something fundamentally broken. Here's what's broken in my view. So I write to you and I tell you in the feedback. You say, ah, okay, now I'm going to deal with that. So we need to look at the two-star feedback and pay more attention to that. Of course, paying attention to the other ones as well. The next thing is, I did a talk about, I don't know, eight years ago. Now 2008, it's recorded online. There's a YouTube video of it based on something called Six Thinking Hats. It's an idea that's been around 40, 50 years. Debono popularized it, if you've heard of him. He's the guy who's known for lateral thinking, which you've probably heard of. Anyway, and this is six different thinking hats enables us to perceive and direct our thinking. The white hat is all about facts and data, what we know, what we don't know, what we know we don't know that we need to get answers to. The red hat is about emotions. Now, when we're dealing with that user experience, we're typically dealing with human emotions. So perhaps from a testing perspective, we need to think about testing where emotions are included in them. The yellow hat is looking at the positive, the best possible outcome. The black hat represents two things. One is the negative aspects of things, so bad feelings grumping us. The other is what happens and what can go wrong if we achieve what we want to achieve. So we do everything right. What still can be a problematic with it. And quite often, we have side effects of what we do that we don't really think through until the end of it. So we have a promotion and the promotion encourages more usage. More usage then causes the service to crash. So it is an unintended consequence of what we're doing. Green hat is for creative thinking and the blue hat is for the big picture to make sure we're not losing track and focusing too much on the weeds and anything else. Many of you have heard of Alfred Hitchcock making films. So he actually had two scripts. He had the blue script, which is the facts, which is act A and act B are in a dialogue on the side of a mountain or whatever it is. But he had the green script, which is about emotions. How he wanted the audience to feel at this point in the film. So if we think about our testing, how do we want our users to feel when using the application at this point? And of course, what do we want them not to feel? We don't want to feel frightened, confused, lost, pissed off, et cetera. Most of you have heard of this company. Many of you use this company. So what Facebook realized, fun enough from usage in India, is not the whole world has LTE. I noticed that Bangalore now has LTE, so that's good. But many parts of India is still on GPRS. So they actually have this day in the company when they simulate a GPRS network for all the people with their iPhone 6s and the rest of it. Oh, I mentioned iPhones. Well, most of the developers actually had iPhones and iPads. So they actually said, right, you need to go and get the same devices users have and again in India, many of the users have Android devices. So let's make sure that we're actually using the same devices to give us a closer representation of the user experience of the real users, not the developers who are living in their Silicon worlds. And I have all these devices and I rotate them every few months. I go from Android to iOS to BlackBerry to Windows Phone to try and make sure that I get the different user experiences. I've yet to explain, but I'm actually doing a PhD research on using analytics to improve software quality and improve testing. So part of what I'm looking at now is ways to do test recommendations based on the data we're getting. I'm very happy to chat to people afterwards in future, if you're interested in this work because it works well with me and the research I'm doing. And I can hopefully help you in finding ways to recommend tests to run in future based on the data we get. I can tell I'm running out of time, so I'll be fairly quick. One of the perennial questions we have in testing is how many devices? If you look at mobile, how many tests, how many web browsers, which version do we need to run? Well, if I ask many of you, do testing, you'll say, I've got five devices in the drawer. We've got 10 in the cupboard and those are the ones we use. Okay, fine. Well, how do we actually work out which ones are enough? These little pre-pictures, the top two are Android devices. People install an app and it basically does a little bit of network monitoring and they've got in the order of a quarter of a million users. So not very many, but across this quarter of a million users, they've got something like 40,000 different devices reporting it. The most popular one, the top left hand corner is a Samsung Galaxy. If we look at it by manufacturer, the big dark blue is Samsung. So Samsung is 40% of all the traffic. The second one in red is LGE and I got Novo and Hawaii and ASUS, et cetera, et cetera. And then down to this little manufacturer, who knows where that came from? So, but it's some device. So when we're doing testing, if we look at data, it may help us decide which devices to test with. This is across a population with an app, but you may look across your company's data and say, even if that's true for the world, here with my project, my company, our users, our patents are different. For those of you sitting grinning because you do iOS apps and iOS is easy and there's only a few devices, this is a matrix of all the different iOS devices and the different features, the different versions of iOS they shipped with and the different versions that supported. So even for iOS testing, it's now got fairly complex in working out all these permutations. Another way of looking at it, this is from academic research, is to say, well, let's focus on the human feedback, the apps or reviews, because if we get good reviews from these users, then everything's happy. So here they looked at how many devices we need to test with to cover 80% of the devices where people write feedback. And what they found is they compared free and paid devices, free and paid apps, sorry, and they found that in the order of about eight devices was enough to get about 80% coverage. So again, it's based on analytics and data rather than just based on guesswork and what you happen to have in your drawer. That's enough for feedback. So we may want to think about enough for the bugs or for the usage or locales and languages. So all these are important factors to consider. We can move beyond pure user experience and look at things like customer experience. This is from a chap called Bruce, who I met in the States, his website details are below. And here he's looking at customer experience and focusing on transactions, bit like digital, and relationship, the human sides of it. So well worth looking at. Finally, then I'll move on to the good UX is good business. Many of you know of this company, they're trying to build up business now in India and open their own stores here. Every store is designed. They're not accidental with Apple. The whole user experience and the moment you walk up to the store is intentional rather than accidental. This is the flying. So here was basically Delta and they now respond very promptly to people who are flying, particularly in the US, that's where the major market is. And let's say that you're flying and you land at Detroit. You need to take another aircraft flying to JFK. You can actually tweet now in real time. They'll say, your gate, Julian, is D3. Your flight leaves in 17 minutes. So they're actually giving feedback in real time on social media, which means we get a good user experience. The one on the left was just a company I responded to. I couldn't find a way to contact them. I tweet and a few minutes later they respond with please callers. Virgin, I was flying with them from the States and I was speaking to the crew and the crew essentially said, all airlines are roughly the same. It's the experience you get in the airplane that makes a difference. I talked about retention. This is an example from a chap called Andrew Chen who does quite a lot of research with his company. And the retention for the top 10 apps we can see is significantly better than for the average across on the left hand side. So in other words, good apps do get more usage and retain their users for longer. Couple of other things to consider. This is the UK market. I think the same is holding true in India. In banking, we're familiar with traditional banks. But many people in the world are unbanked. That means they don't belong to a typical bank account. And also for things that banks use to charge me a lot of money for, like sending money internationally, there are new startups who are competing against them. So for any traditional business, the world is changing around them and startups are nipping at their heels, trying to take business from them and to build a new business. In the UK, now by law, I can change my bank account automatically within seven days. I simply go to the new bank and they send a message to the old bank and the old bank must give them all the data within a certain number of days. And my account must be transferred without error within seven days. Now, you used to think of customer retention of banks being decades. But now I can just go in today, whatever it is Saturday, I can move my accounts away from my old bank, which I've been with for 30 years, by the way, in a couple of days. Pensions, similar in the UK, you can now take control of your pensions, transportation, I've mentioned Uber and older cabs and Google Maps. So we keep transforming the businesses and the worlds. So it's important that if we think about anything we're doing to do user experience, that we can't just sit still. We need to find ways to innovate in how we do our testing, how we deliver our software. And ultimately, we need to care enough about our users to change how we work. If we're not doing that, then we don't really care about user experience. So final thing then is, this is a book that I co-wrote with someone from HP, those nice people who are helping sponsor the conference, on using analytics to improve software development and software testing. I've got a few print copies that we'll be giving out sometime during the conference. The PDF is available online from the book's title. So the mobile analyticsplaybook.com, very welcome to download a copy and send me feedback. This is me, Julian Hartick, gmail.com. Thank you.