 I'm assuming that all the people who've got a mobile phone with you, how many of you want to test an application in this morning? On a mobile device? Three, four, five, about half of you. And the other half are here because you don't like the other topics? Right. Do you have your own applications here that you like to test, or would you like to test a generic application? Do you have your own? So, own applications you want to test? No, no, we don't have an application. You don't have an application. So, who has an application that I have to test this morning? Anyone? OK, none of you have your own application, one person. Does anyone else you want to test just our application that I say you test its application? OK, good. Right. How many of you don't have Android? So, you're on iOS or Windows Phone? OK, that's three against the rest of all the Android. Are you willing to install the Qwix on the way? Yes, sir. OK. Do you have a Wi-Fi password? Yes. So, a Wi-Fi password? Let's just get to that bit. The Wi-Fi password is at the top there. I've taken that from the sign outside. So, what I suggest you do is on your mobile devices, you log into the Wi-Fi, get as far as Google Play, and download the Qwix application. So, you can do that in the next half an hour or so, and then when you get to the testing side of it, you'll only have the application installed on your devices. Or, if you were here yesterday, so you don't want to talk about Qwix. So, a little flatless, you see the verb of the next on the end of it. Right. How many of you are involved in testing mobile applications at the moment or in developing and testing mobile applications? One, Brave Soul. Two, anyone not involved in testing mobile applications? Anyone I'm starting from? Okay, about half of you. So, I can start fairly slowly on this topic and try and build on it. Here I have about 30 to 40 slides. I'm going to try and give you some work to do so that you're involved. I have another 300 or 400 slides on the topic. So, at the back of time, I'm going to pack in my slides to bring some material in based on your feedback and questions you ask me. I hope that's okay. The problem with that is that if you don't ask me questions, then it's going to be hard for me to know what's missing in the slides. So, please, all the way through this session, put your hand up, ask a question, if I haven't given you a natural opportunity, and I'll then do an account to help you with the testing aspects of things. Because everyone who needs it got the Wi-Fi password there. Anyone need it? Does it work? Yes. Okay, so I have to be logged into the room account, so I'm using a different way of doing this stuff. That's just a wrap from the beginning. Right at the beginning, like yesterday, you're free to use my slides. You're free to ask for a copy. I've already sent yesterday's slides to Naresh and asked them to put them on the website. But if you want, you can just ask them for me. The only condition of these slides is that if you do want to modify them, you make that work available for other people. So, that means that if you're in your own company, by the way, and you modify my slides, you're also obligated to make them available publicly. Just a little warning for you. This is a legal licensing agreement that gives you pretty much any permissions providing you conform with that. For some reason, my footage is just hung beautifully. All right, you know when the good days happen? This isn't the good day. This is the end seconds. Press the power button now. Press the power button. Well, thankfully, it's a Mac. And a good day, a Mac boots in about 10 seconds. So, we'll see if the Mac comes back. We'll see if he was happy. And if not, we'll do a much more interactive workshop with no slides at all. We should be fine by me, but perhaps not with you. It's got the grace of the death with those people who've never seen a Mac sitting on their doing and not wearing much at all. Yes, good question. So, the four or five people who haven't got Android, one of the good things about being a workshop is I want you to work with other people. So, it's a good opportunity for you to compare and contrast user interfaces and see the differences. If we're testing across platform applications such as a Twitter, then we'll be comparing, contrasting the functionality. And whereas in Estonia, we're testing and we've tested with Skype, and we've found massive differences between the different applications. So different that the team didn't actually recognise their own application. Unfortunately, I've lost a sound guide. I'm getting echoed. Do you get echoed? When the sound guide comes back, I'll see if I get it to tune in. I have now my Google login. See, something's happening. It'll do the Mac recovery. Right. Install the gimmicks app. Yes, I think install the gimmicks app, if you don't mind. It doesn't have to be much in terms of permissions. Can you tune out the echo? This is the best one you get. Please, Simon. Can you tune the echo out? I get a lot of echo in the speakers. Just to reduce the echo if you can. I'm not sure it's low or not. Can you hear it in the back? Let's look with that. Right, here's the powerpoint. Good to get with this one. Thank you for holding with me then. It's been a few minutes. The first thing is, we're at an agile conference. So, hopefully you've heard this magic buzzword of agile testing. This work is fairly long in the tooth now. Got a claim? Oh. Well, Mr... Can you do the focusing? I think it's the main job. The slide, wasn't it? All right. A contrast for the lack of quality of the slide. So, what we've got here at the top is business-facing. At the bottom, we've got technology-facing. So, testing tries to address these two different directions. The business-side of the data is a feature useful. Is it what we're trying to deliver? Technology is the implementation working well for us. So, we need to consider both aspects in terms of testing. And then we've got the other left and right sides. We've got critiquing products. We've got to be asking questions of the product. And then using tests to support the team. And Lisa Crispin basically took this concept based on familial work from the people in the agile community, and created the concept of quadrants. And her book on agile testing is a good start in terms of understanding testing. And what we can see here is that basically, it's supported by auto-radar testing at the top of the project two, which is where we're looking at supporting the team of business-facing mixture of automated and manual testing. If you're looking for a team of product that tends to be manual or attractive testing, and then down here with performance, low security testing, et cetera, we're looking at a tool that is testing test automation. I'll thank her on most of these during this workshop with you. All people will be doing a lot of performance or security testing today, because it's in practice doing it in a short workshop. Do you think how automated testing and tool-based testing are looking? Tool-based testing are things like using analysis tools that scan the code base and look for common problems, for instance. Performance testing, you may be driving the automated test, but you also then need some sort of instrumentation or other information that you get from back in times. So does that answer your question? I do apologise, look back when I come back. Does anyone have a computer? This is not very well. It's 5 minutes and 11 seconds again. There's a combination of... I think we're going to have to do this, so I apologise for this. First time I've died like this on me. It's two years old, but it was working yesterday. Just repeating one time. I was thinking it was just a bit tight. I don't know if it's just because these are the UF5 bin. How do you know in terms of installing Qubits? Who's got Qubits installed? While we're sorting this out, how many of you are going to access? You've got no computers on that table. No computers here. What I'd like to do then, is to wait for the next 5 minutes. I'd like you to explore the Qubits user interface. Initially in English, you'll find a little menu structure. Don't worry too much about down a user in file that you should find the initial point as it asks you a little help page that will get you to download a small file which is called RayCharles. So basically just explore the user interface first in English and I'd like to change the settings to your preferred language and do similar testing in terms of your language. For about 10 minutes, we'll have plenty of time to get the material coupled across here. Sorry? Yes, it has a little link in the middle of this and if you are willing to install Zenfali, RayCharles is 2.4 megabytes. So it's tiny and you can download a deep north slow Wi-Fi connection. You're welcome to pick a larger one. Hindi, if it's available, is about 700 megabytes. So upon the mobile Wi-Fi, that may well take the rest of today. So you probably won't have a smaller one than that. You should be able to move down because of a problem. Yes, I did. By the way, if you're able to write the slides it's a back-up, back-up, back-up, they're here. So you're very welcome to take them now and the problem will be to just in case I fill up some else's computer which actually will be right back. Oh, you don't have PowerPoint in here, do you? No, that's fine. It should work with preview, hopefully. Can you... I'm back up, aren't I? Right. Well, how's your laptop? Okay. So, I don't know if there's a curse here and I apologise, of course, because it's affecting the workshop but I'm back to my laptop. We'll see if he's going to continue and if not then we should probably just realise that today's not a good day for us and go and do something else. One of the things we're trying to do in terms of agile then is to reduce the feedback time and the feedback cycles with mobile applications this tends to be a much longer feedback cycle and it would be for applications with web-based technologies. For instance, we've got to be able to deploy the software on some devices we've got the whole challenges of dealing with our users who are typically a long way from us and the fact that the users have much more power now they have the upper hand so in the old days when we sold software if they complain they complained through channels that we controlled for the most part with social media people may complain social media but now the app store rating is probably the thing that's most important to companies in terms of the success and viability of their applications. So one of the first requirements for us is to look at continuous integration which is a development process which helps us to take work done by any of the developers and typically in development teams you've got more than one developer working on an application and to be able to integrate that work into a cohesive whole of the software and given that about half of you haven't much experience of this I've just got the raw ingredients so the first thing of course is to have some sort of source code and we have a code repository Now, scarily I still meet developers who use source control and they're just working on a local machine and sort of wishful thinking hope that when they build in their IDE everything will be fine that they press the magic button and it copies the software onto their phone or whatever but not realising that they risk losing everything as we've just seen it's very easy for computers to turn against you at any moment and particularly if you're on these modern laptops like my one which has SSD it's working fine to catastrophic failure and normally it does it fairly quickly it's normally a sort of it's not working but it should be okay it's not working I just rebooted end of this so, you learn to make backups so, anyhow I'm having a good reminder of that today so, thank you Apple anyway, so we've got a code in source control repository Git tends to the flavour of the month these days now, just as a code appears in the source repository doesn't mean anything happens unless there's a trigger or something that says, okay a new code has arrived I'm going to do something and doing something then says I'm going to run a build tool for those working on iOS using something like Xcode build Visual Studio I haven't done builds for five years or more in Visual Studio but it always used to have ways to be able to do command line builds so something similar Android certainly we've got techniques to do this we've run everything from the command line easy to use in the ant system or the Gradle system and that basically takes all the source code it takes the build files and says, right, I'm going to build the application and it kind of dumps it in an area of memory on the hard disk so, what happened to it next? well, we built it so we've kind of done the integration side of it but we probably wanted some automated tests against it which means you have to write these automated tests there's also going to the source code they also go into the code repository they also trigger updates when they're written and built and then the tests are run we then have some sort of runtime environment and there are two main runtimes environments there's things called emulators a thing called simulators when you're working on iOS development then you're actually running something called a simulator which means it looks like an iPhone or an iPad but it isn't it's basically Intel software running on the Intel chipset which happens to be your MacBook the real device is running an ARM based chipset so it's entirely different compilation processors different compilers different deployment different code that's running at execution time for Android we're running in something called an emulator which means it's actually emulating the whole operating system so your software isn't supposed to know the difference between being an Android virtual machine or in a physical device and then of course we have the devices which we then connect up perhaps using a cable perhaps connecting over wifi so we deploy the software to that and when we're looking at sort of agile projects at scale so we're trying to do more than one device at once on the big challenge is how do we run the tests on a tablet and on a phone and how do we run on main devices and on the go Android has much better support for parallel cost devices there are a number of open source and commercial products that enable you to deploy in parallel run the tests in parallel and then get the feedback at the same time so I talked about the time to use some feedback and one of the first challenges for us is where are the licenses where do the delays happen so what I'd like to do is ask any of you to suggest where are delays in the process of getting feedback given about half of you already working with them I'm hoping you've got some practical experiences and then after that I've got my impressions, my slide on the different sections but I'd like me to have a little suggestions on where you think delays happen so I'll take it from the beginning here I am I'm a programmer, you're the programmer so you've written the code and you've pressed ctrl save or command save or whatever it is your favourite idea is to build the code and you want to test it so where are the delays and how long does it take any suggestions from people from the network so we've had to get the software to get feedback in terms of testing so you think one delay is in the network how long do you think the delay is in the network is it a millisecond a second 10 seconds a minute, an hour, a day have you printed in on the back and reflected in on the base ok, so let's just get to your suggestion of the time and then we'll come on to build size so maybe a minute's taken by the network now a person I don't often see the network as a major factor because normally it's me with a high speed network, I need a network I'm connecting the phone or whatever I'm deploying to locally or running the simulator locally but we may have a minute there and then you said the build times the big size so and can you give me the idea of the build size that you were working with so that they were packaging a large amount of data instead of a base of about 89 megabytes and that was adding to the delay times ok, the Qwix application is a cross-platform environment so when you check out the code from source storage or github you basically get a lot of code you run a configure file at the beginning and then that says ok I'm going to build android and I'm going to build iOS not iOS sorry, I'm going to build OSX or I'm going to build just the desktop ones or just the servers or just android so that takes maybe about 30 seconds then run the dependencies and one of the first thing it does it goes to the network and says this little file and it combines lots of C++ libraries into the build I was trying to do this last week I was in Sri Lanka and it took me the whole weekend because basically we go out on the hotel network the hotel network wasn't that reliable and I had to go and back log in again so it took the whole weekend just to get the build process to that point but even on a fast machine, on a fast network it would take maybe 10 minutes for that anyone else but in suggestions so we built the code now what? are we testing yet? so what else is between the two? can you give feedback if it's really working? that's right, we need feedback so we've got a start to build so what happens between build and feedback ok so we've got deployment and how long is deployment a page? well a few seconds in one case any other suggestions so why we might have a different time why would it be a minute or two or a couple of seconds so you're talking about downloading over a network so if you're doing it with a wifi yes, I can see it being a minute yes on Android over USB on this on a good day this is an i7 processor with 16 gigs of RAM so copying over USB 3 it's still downloads about 3 megabytes a second so there's a 10 meg application 3 seconds it is 100 megs, we're looking at more like 30 seconds so a couple of minutes ok so we've got the software on to the device but what if we're using an emulator or a simulator we may actually have to start that running and that can easily take a couple of minutes so again we're looking agile and you're probably familiar with this whole concept of you run the tests and you maybe do tests first or test driven development we're now looking at minutes for feedback or coffee or whatever your favourite drink is go and check your Facebook status oh come back an hour later has the build finished oh it failed with an error, never mind so we're looking at that sort of stuff so here are some suggestions then we've got the build times for Qwix it's long enough now that I give up I no longer wait for the automated tests to run because it does this whole sort of stuff in the background and typically it takes never less than a couple of minutes it can easily take 10, 20 or even an hour if it's something I haven't built for a while and it's downloading extra software we've got the I'm calling the commissioning runtime environment which means getting the runtime environment set up which could be a phone it could be an emulator or simulator and getting the software on it ready for us to use and with many of the android test automation frameworks you're actually installing two pieces of software you're installing the app itself and then the test automation is actually a headless application so you can't see it in the user interface you can't click on it it's actually an app that's running there on the device when we're looking at IOS test automation what happens is we typically modify the binary the source code in fact and then we build a modified binary and deploy that and it actually connects over TCPIP and HTTP requests as a way a test automation works so all that needs to be set up at this point a deployment we've talked a little bit about that and in fact it can take a little bit of time over a network when you're doing test automation with iPads and iPhones very often you're doing that over Wi-Fi so it may sound silly but that means that your computers need to be on the same logical network so if you're running a desktop computer connected to read the net it has to be on the same network as the Wi-Fi network otherwise they can't see each other and the test automation relies on that communications for the test automation to work then we've got the whole app store approval so if you're wanting feedback from users we're not looking at seconds anymore at best we're looking at hours if you're looking at say Android and in worst case we're looking at months if you're looking at Apple on a bad day where they're taking a while to do approval cycles or to reject to the feedback I've got lots of slides on this side of it so if you want that material just email me or use it after the break and finally we get the feedback from the fields which is where the users are using the software and we find things like crash dumps any of you used to put the developer consoles for Google Play or the iPhone app store do you ever use these things? so basically when you're an app developer you have an account you pay for it on Apple or Android I think it's a small amount of money on Windows phone that's probably about $500 and you get access to be able to upload your applications do your screenshots like the words about it etc you can then say I'm going to release my application it gets released and you get feedback saying you've had one download you've had two downloads you've had 10 downloads, you've had 1,000 downloads you've got 6,000 downloads you've got 10 million, 100 million, whatever it is number of downloads and the app store rating and for Qix it's currently about 4.39 out of 5 and we've got about 50,000 downloads and about 35,000 active installs which means that someone hasn't removed it from the device but we also get is a crash information so at the moment we've got three types of crashes coming through and they're all exceptions it's Java based so we get some exceptions saying something went wrong and here's a stack trace now we haven't done anything special for this it's just part of the app store process is that the devices collect this information send it back over the network to Google, Google make it available to us and similar techniques exist for all the app stores so we start to look through the feedback and say oh stupid me I forgot to check that it can cope with right to left to character sets okay I'll fix a bug I'll upload the changes but we'll take a look at a long cycle time and it also gives us the feedback now the qualified things are failure so a failure means that we've got something like an exception it's stopped working and defects may be that users start to get grumpy I can't find the menu option it doesn't seem to do this for me anymore you remove the feature that I used to love Starbucks is a good example they have a mobile app they like to have a customized coffee or whatever it was and you go into a Starbucks store and say this is my customized coffee and they remove that from the app and users got really grumpy saying this was my favorite feature but they haven't communicated that well to the users users wanted the old app but unfortunately you can't get the old app app stores only move you forward they never take you backwards again so that's an example of a defect in use we didn't know about it if we were Starbucks now we need to deal with it so moving on then to time to use the feedback in terms of interactive testing now this little matrix comes from a chat called Jonathan Bach how many of you are familiar with this concept of exploratory testing most of you heard this buzzword it came from a couple of people including Ken Caner and James Bach who coined the phrase exploratory testing more than a decade ago John Bach is the brother of James Bach and works in the software industry doing many software testing and software quality so Ken it was a nice little idea which is we want to maximize the testing and we want to minimize the time taken to be bug reporting and setup so setup is getting the system ready to do the testing bug investigation says ok we have found something that seems to be a problem do we have to take screenshots do we have to write little nodes do we have to go and copy files off the device or can we continue testing so it basically says you maximize T and you minimize the BS in the testing work that we are doing so a minimized T it may be that we want to email the files to the phones for Android phones to enable to the developer mode then the phone will line you to download the APK file which is how Android apps are bundled straight from the device so I email it to you you click on the little link a couple of seconds later it's installed on your device it works quite well doesn't it can't do this iOS I don't think you can do it on Windows Phone you may just be mailing a URL to a link to a link to download things and if you've ever had to type these in insanely long URLs you'll know just how nice it is to have it available to you having some sort of workstation that might include some math if you're working on iOS to deploy software makes it a little bit easier for those of you lucky enough to use Windows you'll know that every time you plug a new device into Windows it says you're installing a device driver and if it can't find a device driver or it needs a network to get it you're stuck until the moment comes that you get the device driver sorted someone here has Linux I can't remember if you have Linux so Linux you have a bunch of configuration files and custom rules most of the time you don't know about this but just occasionally you'll find things like I have a nook with me which is an Android tablet and I had to edit a special file for Android to allow it to see it and work with it and then I file it's sort of hidden away that you have to find it add a little hex code and that then enables you to use the device certainly with the Mac on a good day the Mac just works on a bad day it doesn't and then we've got this whole concept of being able to download the apps for testing so your build server if you can make the builds available on the build server and someone can just log in with the same old phone download the latest APK put it on the device then my is a bug investigation stupid simple things like screenshots I still travel with the camera and one of the things I do with the camera is take pictures of phone screens and say look this is what's wrong with it because it's very hard to tell someone let's say I'm working with a developer on Qwits who's in Switzerland and I say look someone in India has had a problem with this say good luck what can I do about it oh here's a screenshot of it okay I've got a clue I've seen the one that's from the app store like you all are now if it crashes on your device then providing you allow it to send the crash information we'll actually get that in the developer console a few hours later so we can then go and investigate the problem and deal with it from a interactive testing perspective learning how to use the logs is really useful so with iOS it's part of Xcode so if anyone working on an iOS application you need Xcode installing your device so you're looking at 5, 10 gigabytes of content plus all the attendant setup fiddling around with Xcode Android using a crime call ADB which is part of the SDK tools some people have found in quite a way just to package ADBs so you can do this and a few other tools and then we've got to remember that just because we've got a problem today so here we are on Saturday you may be reporting bugs in Qwits which is very kind of you but it may take a week or two before someone looks at it so all the context around it needs to be captured as elegant as we can do that so in terms of maximising testing then there's a couple of people Jonathan Cole and a company called Mwlia who are based in Bangalore who've come up with heuristics and these are the heuristics and first of all I'll explain what a heuristic is so a heuristic is a fallible guideline so it may be wrong but it tends to be useful to people so a heuristic for me might be if I'm wandering around a city and I don't know the city find the railway station or follow the river because the river will normally go somewhere close to the centre of the city so occasionally I end up going the wrong direction but not very often and so far it's worked well for me so these heuristics each letter stands for a form of testing I'll give you an example shortly and helps to prompt our head and think oh okay what the heck does the I mean oh yes that means this so let's go and look at the eye aspect of testing the other topics I'm dealing with at the moment is this concept of antifragile so antifragile stops us from ending up being in a fragile situation where things are going wrong and suffers breaking or the process of the breaking so from an agile perspective we need to go to adapt to the context what's happening at the moment and just because the context has changed we shouldn't just give up so we need to find a way to keep going and these heuristics help us to keep going does anyone know about the black swan book okay just a couple of you so the black swan book is written by someone who traded stock in America and he said one of the worst problems we had to deal with as a society is the problem happens very infrequently but when it happens it changes everything so the crash of the banking crisis in America in 2008 affected the whole world no one knew when to predict it they didn't know the scope of it and so his book was written from that perspective and his newest book is on antifragile which is saying that things like the human body we actually improve through stress and load so my legs get stronger because I do exercise and if we don't do exercise and don't use our bodies we'll get weaker make sense? so the challenge is can we write software that also improves as it's used so that's the concept and now we're coming to the agile communities so that's what I'm talking about so this is the pneumonic for mobile testing what I'll have to do for you because it's impossible to read on the screen which I apologise is look at the web URL for you when we do this in a few minutes so what we're looking at is a thing called cop flung gum which is enabling us to remember different aspects of the testing and what the testing sees for communications orientation and platform so communications we've got an application like Qwix most of you have downloaded Qwix and one chap at the back says do I need to install the ZIM file? if you install the ZIM file where does it come from? it isn't on your device so it's using the network so one way you could test if you think about communications is what happens if I put the phone into flight mode disable the wifi what does the application do? does the application have a nice sort of error message say I think I've noticed the wifi's not there please can you enable wifi and then try to move the file or does it fail with an exception saying no network error or even worse error in file network.cpp do you just cannot do this? page cannot be displayed it's a great error message in fact the application Qwix on Android has something called a web view it has the main content of the display so most of the screen is actually a web view now this became important because the platform changed with Android must you have Android phones? how many of you have got a KitKat device? KitKat is your ring system ok so that's Android 4.4 so how many of you have got jellybee devices? most of you? ICS, ice cream sandwich? ok I feel sorry for you and then gingerbread which is 2.3 so a couple of people got gingerbread gingerbread is on the cheap low end phones the ones you can buy for $30 ICS came out a couple of years ago it normally means you bought a specialist device that no one supports anymore with rare exceptions jellybee is probably still shipping on about a third of the devices and KitKat is now on more than 40% of the devices so one of the fundamental changes that happened with KitKat is that the gluten team finally changed the web view and they changed it from one that was based on webkit to one that was based on the chromium browser now it might sound silly I'll try and show you this interactively so this is Qix and I think this will be in Hindi if I pinch and zoom can you see I'm pinching and zooming so with the older KitKat based browser what would happen when I pinch and zoomed is the text would automatically reflow and resize now it doesn't so when KitKat first came out about a year ago this was a 1% problem 1% of users had this so a lot of people in the world were stretching their heads and saying dear Google you should fix this you fools how could you do this to us well there's some good reasons why Google did this so there's basically a long thread on one of the Google bugs saying we'll not fix working as intended so what do you do from a development perspective the developers Android said we're not going to do this anymore for you good luck so there's article after article after article and posts on Stack Overflow anyone trying to fix this and they're saying jet JavaScript do this do that, set it into compatibility mode blah blah blah blah we'll fix the problem so we've gone from a 1% problem to a 40% problem because these days more and more people are getting newer devices which include KitKat so the problem is getting more important to us and that's a platform problem in terms of these heuristics those of you who don't know if you go to theandro.com site Google updates statistics about every two weeks on all the different screen sizes and system versions and a bunch of other useful criteria that means you can go and look there and say ah we've noticed that now there's 40% of our users or 40% of the users in the world are now on KitKat so we want to make sure that we've got at least one KitKat device to test with ah we've noticed more and more people using the extra high density screens we better make sure we've got at least one device with an extra high density screen for our testing and we may want to test on that first but now I'm sorry to say we don't test on the ICS device first because there's 3% still using those so sorry but you're also using Linux you're clearly a geek you can probably fix the bugs in our code make sense? so we can use information that's already available to us from sites such as Google to help guide us in terms of what we're trying to do at the bottom of here is the very very long URL that's the short URL you'll find it very quickly with your favourite search engine and search for cop flung gung now I'm looking around I can see there's a laptop here there's a laptop here and a laptop here hopefully a couple more people have got laptops so would you be happy to do some testing based on cop flung gung for about 20 minutes? ok there's what I suggest then if there's a table with two people with a laptop whoever's got a laptop moves for the two nice people here if you can go and find a table without people with a laptop then you can share the laptop I think it's going to be the best way you can actually look at this and what I'll do is I'll project on the screen the cop flung gung as well but it's not going to be very good because of the resolution here do you have any questions at this point? you have one question there sir oh you're happy ok thank you so about 20 minutes then can I watch so I'll check in with you at 11.40 yes so if you've got QT installed on device and hopefully you have on a mobile device then I'd like you to use this cop flung gung and pick one of them initially sorry I've got a bit of echo so if you were to pick communication you might focus on the communication aspect you may want to pick the orientation to do with the rotation of the screen to see what happens so do you get updates and messages like that ok so you have 20 minutes of testing using this heuristic cop flung gung if you don't mind briefly sharing whether you think it's useful or not have you learnt anything from doing it any concerns or suggestions you might have for doing something different in addition to this so perhaps if I ask the two of you who are sitting here at the front do we have a mic by the way hello phone people hello do we have a mic see mobile phones are addictive hello thank you so thoughts on using a heuristic in particular this one for testing mobile apps I think it's pretty exhausted and thorough than what I initially thought it would be going through each point they have covered a whole lot of stuff which is new to me and I think most of this stuff should be automated because it would be very challenging to go through each single time maybe after a single feature if you have to test all these things it's very exhausted and one thing I like about this heuristic is why you are testing your application you have a focused approach of covering that particular heuristic for us and then going through the other things if you just go through this you have an existentialist of test cases yes so you have an existentialist of test cases I would suggest a heuristic like this would give you enough to test for between an hour and several days and the several days is having lots of different devices and testing in different contexts but it's enough to run through fairly quickly and think ok what does that do in terms of communication what are the problems I get if I play around with the communication or change the settings or experiment with the gestures so you could sort of spend five minutes on each and work through this fairly quickly also actually going to spend a bit more time focusing on different areas would anyone else like to comment on that actually I have been developing the app for the last six months and we will be releasing it next month I came from a web-based background never knew what mobile development is about all I knew is I used mobiles and I used apps but never developed them so what I learned from this diagram is something to do with how an app should be designed what should be considered by designing that is not serve as a testing heuristics but I think if we follow some of these rules when designing that because what happens is that an app is about communicating information to the user we should not over communicate at the same time we should not under communicate we don't have the liberty of a big screen like a laptop in case of a mobile so what we communicate to the user has to be really precise and these heuristics to some extent help me understand what are shortcomings that my team has developed yes thank you from a different perspective when going through this diagram so it really helps but at the same time I feel it doesn't talk about anything related to functionality what is the value that it adds to the user but I don't know if that is really the scope of this diagram so a good point a one that I want to build on is that no heuristic so how do you automate things like gestures how do you automate the observation that the user interface is no longer working as the user would like it so it may be technically correct it may still have the elements there but it looks ugly or they've got to scroll left and right too much to read the text so I think you are testing important things so how do you automate so I think you are testing in portrait mode and basically half the screen was just a big grey nothingness and the text is all squashed across on the right so could we improve a layout of it automation won't tell you that so the interactive human testing really does help now someone was testing on a particular phone said to me how do I report a bug because he'd noticed that the menus the little stars and stuff at the top changed from landscape mode to portrait mode so he was testing five different icons to three different icons was it you sir? now I've never seen this bug before I have no idea what phone you're using and I've used keywords on plenty of phones so here you see some of the importance of diversity of devices because I think it was a table back there everyone's using a Samsung phone they all look identical so you could all be testing in parallel but you're not going to find the bugs that he found simply by having a different device and over here we had a 2.1 Android device so old and up in system that Google Play doesn't support 2.1 devices Google have publicly stated the minimum they supported was 2.2 because so few devices use that anymore and there is a way to install the software the keyword software from the developer site but we wouldn't expect many users of 2.1 devices to want to do that so it's these sort of problems that it's worth knowing about but the other more if you're shipping your app on Google Play there's no point testing on a 2.1 device because they can't get to it so once you know that it saves you a little bit of time when you're testing any other comments? if you pass them, go and get the mic first can you just get to this gentleman first he was before you he does have 90% of the power though so I found this guide very good this is actually very useful for the regression testing but for the exploratory testing or say for a story testing I would not use this guide as is what I do is I'll do the exploratory testing and find which category it falls in so that this information this bug falls in user function or something like that this would help me in triaging the bugs but I will not start one would be different one would be bottoms up another would be top down top down and bottom up so far I'm being repetitive but say for regression I will go through this guide go through the feature category and do the regression but exploratory I'll do the testing first and then fit in what category it falls ok thank you I'll pass the mic over to this gentleman I'll use this as the last feedback and I say I'm really burning to say something nothing more just feedback we've had an app out for about a year and a half and I found a defect right now ok thank you so found a bug and that's been shipping for a year and a half good because of my laptop playing up a little bit I apologize we're well behind in the materials so before I forget I'll make the slides available I have another several hundred slides available on topics so you're very welcome to email me and say hey Julian please send me the extra slides and I'm happy to send them to you they're all under the creative comments agreement which means you can use them on a share like basis and give them to your friends colleagues etc so I'm going to just skip through very very quickly I've got about two minutes left to say these are the topic areas I want to cover and you can pick them up later there are videos by the way of me talking about these topics that you'll find online so things like testability helps make it easier for us to test and that testing could be in terms of the interfacing so how easy I interact with the application or it could be in terms of how easily I obtain the information back from the application one of my favourite examples and I think someone was suffering from this at this conference is when you have a user interface with a username and password and testing whether it correctly handles someone mistyping a username and password combination typically what the app will say is the combination of username and password doesn't match our records please retry in the old days people used to say you've mistyped your password or we don't recognise your username but the trouble with that is that hackers realise very quickly that oh I've got the username right now I've just got to guess the password and they use software to very quickly try many different passwords the most common password or football team names or pet names or whatever it is and then with a programme you can do this 10 million 100 million times a second if you're running at a high speed computers so what the application does is it says the first time you mistype the username and password I'll just display the dialogue box and say please try again but if you do it wrong the second time might wait 0.1 of a second now a human doesn't really perceive 0.1 of a second as any time at all but for a computer programme trying to do it a million times a second now we've really slowed it down haven't we the third time we get it wrong might wait two seconds the fourth time it might wait five seconds so if we're testing the functionality of this sort of login screen we may from a testing perspective want to know what algorithm is being applied is doing the back off algorithm by the way is it stage 3 which is where we're now doing this incremental back off stage 4 might be the application shuts down for 10 minutes for instance so testability means that we can see this information inside the application as if it was speaking to us and saying oh look now this is what I'm actually doing at the moment because they're going to be rich bugs in that area so in terms of design for testability we can use hooks and write code in the application to make it easier to get the information and to test automation I wrote a paper that's published on the Logigear website, Logigear's a testing company about test automation interfacing that goes into lots of detail about the whole interfacing challenges so one of the challenges then in terms of agile and yes they were arguing a little bit about velocity and story points and stuff at the conference so one of the challenges is to be able to provide value in terms of the application and where does testing fit into this and in terms of software development the cost isn't the cost of writing the software, the cost is the life of the software over how many years it's been available and the gentleman there said he's actually been shipping for a year and a half so when the developers wrote the code it was 2013 wasn't it? Summer 2013 or maybe they wrote it even earlier could it be 2012? The code is still there so we look at the cost of supporting it and updating the application with iOS 8 and new devices you can't just sit on your code base and say I shipped it two years ago which should still work on old devices it's got to still be updated and modernised so we can actually find that the initial cost of development may be dwarfed by the maintenance costs and we have some trade-offs to think about here and testing may have some impact of this so one big question is when and where to spend money on testing and the voters are a smallish company in London and I think they're now in Berlin a couple of other places in Europe and they basically develop Android applications and they quote to lots of companies some big that you've heard of, some small companies and say well we can develop the application for you this is how much work we think it's going to be and how much it's going to cost you and we have price A which is no automated tests and price B which is with automated tests so this price is this one is 1.6 lakh so it's about 60% more now as a client, which one do you pick 1.6 lakh well, we're in India aren't we so I can tell the answer we're going to negotiate me down and say I want that for 1 lakh but on the later what they found out is that you don't recoup the money if you extra 60% of the costs on the first version of the app you're iterating round as that gentleman is doing and saying ok we've got the app we're shipping the app, we need to update it and we're running automated tests it reduces the sort of internal approval cycle we think we've got a build that's good enough to put in the app store from several days in their case to about a day so still some interactive testing being involved but the automated tests help them to do the testing faster and get the feedback sooner I'll skip the details there because of the time one thing I want to mention is that testing isn't the only way of getting the answers in mobile as in the other world we've now got a number of specialist companies who will basically say give us your software we'll send it to our team of crack testers from around the world and get them to give you some feedback and you may give some guidelines and say I want you to test using Copflung Gun or I want to test based on looking for functionality to test on different devices I want someone who speaks fluent Arabic or fluent Hebrew to do the testing because we don't so you may be doing this sort of testing learning to look at the logs typically you can plug in whatever your device is using USB cables that's an Android tablet and we've got an iOS iPad-y thing here so plug it in and use the relevant software tools and look at the logs and what you'll find out is that your applications on your device are very chassis they're all writing in there so if you're using say Gmail or Google Maps or Facebook or Twitter they're all writing to system log so even though you're focusing on your testing you'll learn things about the apps you're using and then it'll give you some idea of how can we improve the logs that we're writing from our mobile application so that we can then do a better job of diagnosing problems in futures I mentioned crash dumps crash dumps are really useful to help understand we ended up here and here is where it went wrong what was all the stuff that was happening above it now how do we get to here look at the crash dump and it gives us a better idea of where the problems are analytics is used a lot for marketing there are tens of companies who make analytics available Google Analytics has been one example you've probably heard of and you have a little library that you include in your main application and when the application is shipped it will typically be sending extra little messages from the application under control of the program as we wrote the extra code send a little message for this thing and it'll do that when the network is available and that's collected by say Google or maybe your servers and we'll then review this information and it'll give us some good ideas of how the application has actually been used in the field so since you showed me your application let's look at rotary clubs, wasn't it and the distance from where we are now maybe you have analytics that says by the way we've noticed that users are doing searching and ordering by distance or they're searching and ordering by feedback so people prefer this one or don't like this or they're searching based on price maybe you have to pay to go to some and not to others so that might be useful in terms of looking at extra features or redesigning the user interface because perhaps there's something that you think is more popular but the users are having trouble getting to it so you put it on the home screen of the application so analytics can help that with app stores but one of the challenges with that is when someone says something bad about the application it stays around for a long time so some of the smarter companies are now saying what we'll do is we'll have a little in-app feedback and that comes directly to us so it bypasses the app store and we get the feedback immediately also we can have more flexibility in the data we ask people to provide and maybe you can ask them to take a picture and they'll take a picture and send this along with the feedback so you can use that as another way of complementing testing this is the android UI automation viewer tool there's something similar in iOS instruments that allows us to see as green structures you have a question can you just grab the mic so before the app goes into the app store does the app store does the validation of the app does the app store does the validation it does some validation it depends on which app store we're dealing with so in the early days of Google Play or what we used to call Android Market they didn't seem to be much validation taking place whereas with the Apple app store there seems to be a lot of stuff happening and there's also some carrier app stores so say T-Mobile has an app store and they may go through their own approval process so typically each app store will have a series of internal guidelines that says check for the app store check for these things make sure you can log in with the app maybe have a human being actually using the application and sometimes with the app stores you now have to send them at least it's wise to send them here's a login account to try the application so they can then try it and make sure that the app doesn't do bad things so if we look at a fairly blunt example the app doesn't include pornography if it's being rated as mild so how can they test that it's very hard to automate that so get them to log in and they do that as part of the app store approval cycle Apple will publish a fairly extensive guideline on the app store approval process and I think somebody Windows with Windows Phone provides a very good guideline that's saying these are the sort of things that you want to make sure you've got sorted before you send the app to us because you're not going to make it through so those guidelines are really important as part of it and to test all the machine tools we're getting screenshots quite often the app stores need screenshots and you'll want them to encourage and entice users so automation is a great way of collecting those things so this is your automation viewer it's been in that Android I think since Ice Cream Sandwich it's either that or Jell-O-Beam and what we can see here is a visualisation of the curex application on a tablet in portrait mode and over here you can see a big blue bar that's saying that we're in a text view and where the dice is at the top is a little tiny red dotted sort of square and that's the actually representing that dice and that's that text view there which has got a text link of random article now why is the text link useful? well if we're using the speech audio feedback say for a user who's got visual impairments so they're blind they can actually listen to it and it'll say random article user interface is in English that's a very good thing to have isn't it because that's what that icon does who changes user interface to say French does it have that translated or not Tesla automation also needs a way of interacting with the software so maybe it's looking for that label of the element as a way of finding it so rather than just saying blindly go across 600 pixels to the right and then 20 pixels down and do a click which as you probably realise is very different if we're in portrait mode the landscape mode because it's now over here so that's not very robust in terms of Tesla automation so instead we go for a labelled element and anyone who's used tools like WebDriver or Thvernium for web-based Tesla automation would be familiar with this sort of stuff I hope and I'll repeat that these tools are available for different operating systems so we can use it for any of them so mobile-like analytics again I'll do it very quick on this I think there's a recorded keynote of me doing this topic in Finland a couple of years ago or in Estonia so the idea is we've got applications here they're saying data for some sort of collection some sort of database typically the content is then filtered some of the analytics engines make sure we remove personal identifier information like usernames and passwords we shouldn't be passed as part of analytics and finally we should be logging to a web server and look at this information there's a sort of mechanic of it so we have an app, we actually combine into it some while also software typically using third party software these days that live is normally given a source code so you can look at the source code and make sure you're happy with it that's in part because of a reaction to the industry where this code in the early days did extra things that caused problems with privacy or used too much network bandwidth or whatever it did so now they publish this software so you can look at it yourself and then eventually it sends the information over and sometimes it collects the messages and puts them all together and sends a single sort of summary message used with 23 searches today rather than these are the individual searches because it reduces the amount of network traffic it sends and we can use these sort of questions then to ask questions this is actually worked from Microsoft and they use it to look at the whole development process so we can look at things like what's been happening in the past so reporting, so trending defects we've noticed a number of crashes are happening more often now we've also noticed the number of devices that KitKat are going up is there some sort of correlation between the two or is it just random it seems to be a correlation but I think there's causation which means this is causing that okay we'll go and investigate that so analytics can help us find those sort of problems looking backwards alerts are great at saying maybe we've got a new version of the app or overnight maybe a new device like an Android 1 device is shipping now in India that wasn't available a couple of months ago so we're starting to see new notifications saying something seems to be going wrong let's go and investigate we may have a problem on our hands and then we can do sort of future extrapolation prediction based on the analytics information so I think thankfully that's it so I'm just about on time I'm supposed to finish so thank you very much my contact details I think you all have it you're very welcome to the slides and I'm happy to take questions it's your lunch break now thank you any questions John McGrath Mike this is about QX rather than the you are using the web version of Wikipedia why not the mobile version of Wikipedia which I think has a better layout for smaller devices so a great question which is why aren't we using the mobile version of Wikipedia the QX project's been around a few years now and it is the offline Wikipedia reader and although it was done by volunteers it was done with the support of Wikimedia Foundation and we worked together at Wikimedia Hackathons for it the Wikimedia Foundation decided to do their own application mobile that has some similarities with the QX application and one of the things to the team to think about the volunteer team who support QX is other things we can do to improve the user interface based on what's happening with other similar projects so it's an open question and a great one to ask and it's an example where you sometimes need to compare between products it isn't enough to say well my product meets the specifications or the requirements or whatever it is if someone else is shipping a better product well that's kind of the end of your business now for us we're free we give the software away we compete against free software so it's less of a business problem but on the less we want to make sure that our software is as good as it can be or maybe it's time for us to stop working on this and maybe we should work on that software instead so we may change our focus now one of the things that QX has done is that they've now allowed us to put videos in the content and saying we're going to put it in this single cohesive file so you only ship one file and all the contents inside it and people will watch videos etc etc and of course companies can use it you can use it to distribute content to people with mobile devices with android devices or desktop computers so it's a great question not sure what the answer is and it's good that you're comparing between the two different versions another question at the back what are the quadrants of testing when it comes to mobile applications what is your opinion on quadrant 4 vs quadrant 2 test like would you I'm sorry quadrant 3 and quadrant 2 test would you recommend a lot of automated tests for the UI or would you in your experience have you found that exploratory testing is better because in this case I think you were doing more of exploratory testing with the mnemonics it is possible to do quite a bit of UI testing using test automation most of the test automation tools both the commercial ones and the open source ones focus on testing through the user interface so they're interacting with elements on the user interface such as clicking on that button typing that text box whatever it is and that automation can help us feed up some of the feedback cycles in terms of testing mobile applications but it doesn't tell you everything that's wrong with the user interface and you'll find that the test will miss more than they find by design you don't tell a computer to go and look for stuff it's not supposed to be looking for so it'll only look for the stuff that you tell it to look for and only check very limited features about it so do I see the answer appear on the screen yes, test passed now it's written white on white that's not the problem of the automated test that's a human problem the human looks at it and says I can't read it so there's a talk I've done and the sites are available online you'll find a talk eventually my blog is blog.bettersoftwaretesting.com which talks about the fallacies of automated tests so you might find that useful in terms of looking at these comparisons I find that the human interactive testing is really important for mobile apps and every time I do some sort of workshop at conferences on this we normally find 100 plus bugs of shipping applications and that's from given we've already got testing and other people have done testing in these apps we're still finding problems just by having the rich interactions of human beings so given that I use the even richer as a population than we are we're mainly male with some exceptions we tend to be the more passionate people we care more about technology our users are not like this so we need to make sure we're considering the user aspects of it which is why I look at analytics as well because we're collecting data from in the field of how the software is actually being used rather than how we think it's being used so I don't know if I've exactly answered your question but I've talked a little bit about GUI testing a little bit about complementary techniques such as gathering information by using analytics thank you any other questions you want to ask a question I don't know if so just checking you just might be polite you're probably our polite alright, thank you I'll call the quits then, thank you very much and thank you for being with me when the computer misbehaved twice and thank you for sacrificing your laptop it's been working since hasn't it so good, thank you then