 Let's see if we're in the right place test automation for Drupal 8 and mobile devices and tablets Who am I you saw my name on Astasios, thus galopalos Originally trained as a Java developer. I've been doing software testing since 2003. I started off at Nokia We all kind of remember Nokia, right? I've been working and testing in actual mobile devices in numerous Operating systems not just Nokia, but also in Windows. We all kind of remember the Windows phone I haven't been having very much luck, you know in Windows in mobile devices But what we're going to talk about today is software testing Test automation specifically For specifically for mobile devices, which is something that's in its infancy And just to give you something about my experience I started at Nokia. Like I said using the quick test professional tool. I've also worked in robot framework Doing some night watch and phantom.js. We've already seen a Example of that And also using the codeception tool and for our purposes we're going to be using Appium Appium is basically the Selenium tool specifically for Mobile devices So we'll we'll have an example of that and we'll also be doing some testing of our own too Okay, one thing I'd like to do is kind of take the Web and mobile devices and mobile telephony and kind of see where they come together I'm just to make sure we know what we're talking about the brief history of the web We all kind of know Where the web started way back in ancient history in the early to mid 1980s We have CERN developing its own its own like mini mini web and From there we got to mobile devices, which were completely different things and the web basically existed for like 20 years or so just in mobile devices Interconnected networks were supposed to function just on large computers and later on on just you know desktop computers It was only in the late 90s early 2000s that the web started to appear on mobile devices And of course, that's when the first problem started to appear Now let's talk about mobility. Let's talk about mobile devices In the 1910s roughly a hundred years ago public switch telephone networks started to become problem Well, it's already come. Yeah problem, but also common Radio communications 10 years later started to become popular and 20 years after that Basically telephony and radio kind of came together for the first time and it was only in the 1990s that Cellular telephones really became popular developed by Martin Cooper a name. We'll see later but the thing about mobile devices is that they've kind of You know always been a little Problematic, you know technology has been difficult to Predict and we'll see how difficult technology has been in terms of predicting what's going to happen I've got a few ironic true quotes about the development of technology that will go through really quickly Let's see what people thought about technology will be reducing will be filtering down into technology In that that worries about like what we're going to do Thomas Watts and president of IBM in the 1940s back when computers were considered like big room-sized devices said there is a world market for maybe five computers even more For what we have to do. There's no reason anyone would want a computer in their home Olson from digital equipment. This company was bought out a few years ago, but I think you look Packard or someone like that Martin Cooper himself the Developer the inventor of the cellular phone said cellular phones will absolutely not replace local wired systems in 1981 The point of this Martin Cooper had to take a very cautious approach to development Mobile telecommunication the invention of mobile telecommunication was just as much a political challenge as an engineering He had to convince his own motorola executives that the mobile phone would be a good idea and Luckily he did More funny predictions about the web Robert Metcalf the founder of ethernet himself said I predict the internet will soon go spectacularly supernova in 1996 catastrophically collapse Right wrong The point is what we're going to talk about is old paradigms die hard and the old way of thinking about technology Still exists even to those who are at the forefront of technology Let's talk about mobile devices In 1997 Apple is already dead the CTO of Microsoft He himself said Steve Ballmer there's no chance that the iPhone is going to get any significant market share We can kind of predict, you know, what happened. There's the iOS 10 recently Released and what's Steve Ballmer doing today big business, you know business wishful thinking is a big impediment to technology Okay One more prediction in five years. I don't think there will be a reason for a tablet anymore Tablets themselves are not a good business model We'll see about this one tablet shipments fell by 12 and a half percent And one thing I'd like for you to to keep in mind of like as you go through Drupal con see how many people are using mobile devices versus how many are using tablets the phone versus the tablet Who's going to win and what should you be develop developing for? Here's something that somebody actually told me in 2004 a Nokia manager said when I was working on a device I was testing a device where the screen was remarkably small For the web, but the actual web itself was he became even smaller on the screen And when I tried to file a bug report for that The Nokia manager said why would anyone want to look at a website from a cell phone? Okay problem As I said here old paradigms die hard like even to managers at Nokia one of the biggest technology companies in the world At the time I should say They really thought of the mobile device as a telephone Nokia wanted to make a telephone Whereas Android iOS saw what you're holding now. It's basically just mini computers that can also make phone calls Okay, that was the point of the previous slides But let's talk about test automation for mobile devices in terms of bugs The cost of buggy software is much greater than we think Here this statistic the National Institute of Standards and Technologies part of the US Department of Commerce Calculates that software bugs cause 59.5 billion. That's billion with the bee every year just in the US So bugs are expensive and that's including your technology also Bugs are an integral part of every software, but the point is Proper testing early testing Will hopefully reduce the cost Bugs are cheaper and easier to fix as far To the left as you can possibly go during the requirement gathering and analysis and design phase During development the bugs that appear develop in development also get kind of more expensive and they're put That by the time you get to the traditional testing in QA area The bug fix process has gotten much more expensive and as we go to the right The bugs get very very expensive to fix and I've been unfortunately a lot of software projects where Basically There's too much like a B testing. There's too much like post-release testing. So you're waiting too long for the real fixing process the real testing process to really begin what we want to do is push that That big high part of the graph on the right as far as possible To the left and where we get like a nice more parallel line instead of instead of the big Big zoom up that we see in this slide Okay, let's get quickly to test automation. Let's look at the web. Let's start at the prehistory of the web Okay, we're gonna go way back into the 1990s here and when did the web and mobile devices Intersected it basically intersected in a device like like this This is the Nokia communicator This kind of model didn't last has anyone ever seen this kind of a phone that kind of like the book open model Yeah, that's kind of died out. Did you know the whole idea of the flip phone is kind of Kind of dying away, especially this kind Let's look at the web circa the late 90s Here we have a different model of the of the Nokia communicator with Yeah, Wikipedia the thing is the reason why I'm showing this this slide is because Even today like 20 years later after this kind of mobile web design happened the same problems still exist What do we see here? We see Layout problems. There was no Responsivity here. Responsivity didn't exist at this time So what do we see we see like buttons on top of text fields text fields over running text forms running out of the Running out of the frame and you also had to you know do a lot of scrolling you had to do like Horizontal scrolling vertical scrolling to see what you had to see And so why was this the case, you know back 15 plus years ago You really had to have custom code developers had to write special code specifically for mobile devices And this limited web development because you had to basically train and hire mobile specific developers And also, you know at the time the low resolution that really a text-based system and slow loading times At some point later, yeah problem solved we now have mobile browsers that can do HTML CSS PHP Java So instead of really developing like I say for a stripped-down web You can now develop for the web that was normally expected on Yeah, on desktop and laptops And we finally do have big screens that are light portable easy to use easy to read And what should we do we have to keep in mind that you know people do surf on the mobile web I'm 60% remember I yeah, there's a lot to keep in mind here like 60% of mobile digital time is spent on Devices, you know just the web time is spending devices most online time now is spent on mobile devices, but at the same time we still develop for laptop and desktops and Numbers to keep in mind. This is from 2012 but I think the number is still applied today Yeah, 74% of visitors are likely to return to mobile friendly websites Over 60% won't go to a or won't return to a website that isn't considered mobile friendly And yeah, 70% almost are likely to make purchases from a mobile friendly website So now let's talk about these things Why are people on mobile devices we can ask yourself right now? Why are a lot of you now looking at your phones? Is it really just me and this presentation? Or do you have like immediate and easy web access you can look at your phone like right now? And also the devices are discreet Who knows what all of you or some of you are like really looking at right now? And you can surf with small easily concealed furtive web browsers And also it's one thing I've noticed people are mobile and therefore want to use mobile mobile browsing And also just one thing I've noticed that you know web mobility is a habit, you know People have started the habit and it's going to keep growing and it's also a completely reflexive Instinct activity, you know people sit at a bus stop on a bus at a restaurant getting a cup of coffee And they immediately pull out their phones and start surfing So it's this kind of world that we have to start developing for Yeah, and more things to keep in mind. Let's do some actual testing here Test automation I say is just one part of testing A Lot of things that you see test automation is often marketed as a sales device what this often ends up Unfortunately for what I've seen is the automation of buggy software Test automation alone will not solve your quality assurance problems to do really good quality assurance you have to have You have to start off like I said way over to the right. You have to do like Code checking linting peer reviews unit tests. It's all like one big package You know, you can't just automate your tests and there go your bugs Yeah, keep in mind keep in mind even more One thing to really keep in mind mobile test automation is still at its early stage I gotta admit this isn't easy a lot goes wrong here starting with Android is my suggestion if If we bring in iOS, there'll be just too many problems. So I recommend yeah starting with Android As we'll see any even more to remember plan testing in QA at the same time. They are planning development You know, we have to stop leaving test planning design and execution to the end with the whole idea of DevOps With the whole idea of continuous development. There's also continuous testing going on at the same time So it's all part of one big cycle And like I say, you know bug finding is good. We need to celebrate finding our bugs throughout the system and Yeah, how to start this just informally Learn to spot anomalies everywhere in daily life What is wrong? Just look around like right now. What is wrong? Is there something unexpected? Yeah, with this room go outside see what's unexpected and Avoid confirmation bias often people don't see bugs in what they develop because they don't want to see what's in We don't want to see problems with what we develop Okay, but you can get around that by you know looking for problems look for bugs. You're always there Okay, and let's take a look at our two different approaches to the mobile web testing We've got basically two things two devices that appear on the web one We can look at web-based mobile apps versus native mobile apps think they require, you know Two different approaches to software testing two different approaches to how we test these things And Let's start off with the mobile devices the Drupal Drupal on mobile devices Drupal can function on mobile devices in two ways. We started With Drupal 7 fairly recent activity, but Drupal 8 Expanded a lot more than what Drupal 7 was able to offer in terms of mobile development and since this isn't This is more of a testing in QA related Presentation rather than anything revolving you know evolving development. I strongly urge you to look at this Drupal.org docs look at version 8 and mobile Okay, we're going to talk about this little more as we continue desktop development versus Mobile development, what should you develop first? The trick is I guess we're trained at this. It's a it's a habit We basically still develop on the desktop first but start think about thinking about developing a Mobile first mentality and who thinks we should have a mobile first mentality Let's look at this quote if I were to start Drupal from scratch today I'd build it for mobile experiences first and desktop experience second Look who said that a while ago Is that what's going on still should we be going in this direction I think yes, but yeah It's up to your development But always keep in mind that remember the users will look at your pages and mobile mobile devices And they very well might see your pages first in a mobile device So the big questions the big question asked before mobile QA begins. Do you have a mobile specific web? website or a native mobile app If it's mobile specific then ask is there a different markup and layout than in the desktop site Okay, do they look any different? Have you really developed for different markup and layout? Have you developed for responsibility? And if the answer is yes, your test should validate that the mobile markup and layout Should first be tested and then the rest of the site for functionality and layout because if people first look at the at the device at the web In your device at your website in a mobile device if it just looks wrong chances are they'll go away immediately and not really look at the functionality or the content Okay, often the answer could be no then the website is basically a website For the desktop that's kind of like scrunched down into a mobile device then you have to check for Responsivity layout And then the functionality do things do what they're supposed to do and also look at the content And quickly to native mobile app testing And according to the Drupal 8 website Drupal can be used As the back end for a mobile application the website itself says Drupal can contain your content Business logic and management. This is Drupal 8. I just want to make that clear And so your app can be the front end that communicates with Drupal on the back end And if you're talking about back end testing you can use a number of other tools This robot framework tool has a Drupal library and it's very good other libraries that can test the back end of something Okay, so here we have the the back end that's used for storage and native Validations and a few questions to ask you run numerous repeated tests And you should test for this does the native mobile app communicate with Drupal back end Is expected data there are the files in the expected format and do they have the right data? Okay, so let's talk about Testing with actual devices, okay, we have emulators versus device devices the emulators the advantage to emulators is that you can actually Run your tests on an actual Website that look that's designed to actually be like something that looks on the phone The emulators allow you to debug the flow as you test the apps. They're good for initial testing The thing to always remember when testing on an emulator is no end user is going to look at your website on an emulator People don't just don't just look at emulators Things to look at for the best ones that I've had the most success with is the ios on the xcode Which is yeah the ios development environment and also the android sdk Um after the initial tests on emulators, you should continue your testing on on the actual devices themselves You can actually plug your device Into the computer and run your website on test automation all the way through Just in the device itself Okay, but what challenges are we talking about? One thing we don't really talk about is what is a bug The funny thing about the whole idea of a bug is that there's no real Definition of what a bug is that everybody agrees on Um a bug as I say here is any unexpected or undesired result And let's talk about what software testing is Again, there's no real Definition for this that everybody agrees on But as I say it's the active pursuit and reporting of software errors that lead to unexpected results And to me this is an active and creative process Just when you're finished developing you can continue your creativity In the testing process And um another thing I call it it's an active formal and recorded exploration to get information about the specific software under test And there are two things to look for now. This applies to manual testing test automation Um any kind of testing anytime you look at your At the device anytime you look at the software you've developed think about this. There are two kinds of bugs. There are software bugs Versus user expectation bugs. So let's talk about the software bugs first As the Whatever the number is Report says a software bug is an error that leads to a fault that leads to a failure Okay, what do we call an error is a human action that presues that produces an incorrect result In other words something that you do That causes that causes an error And the error leads to a fault Which is a manifestation of the error in the software And the funny thing is is you can call this you can call the fault you can call the Manifestation of the error by any number of names I know that in in the us If we look at the first word after known as we're not allowed to use this word anymore You know, you don't produce this you don't say this in fact. We don't even use the second word anymore Everything's an issue instead of a bug report. You have an issue report because You open yourself up to legal problems supposedly it looks bad in the contract So everything's an issue And this issue as we call it will lead to a failure Yeah, it's a deviation from the software from its expected delivery or service And we have user expectation bugs these things are just as bad as like a crash caused by a software error This isn't caused by any error itself, but it's something that appears wrong In the software things like typos text field overruns things that run out of the frame and very often Developers call these minor defects Okay, in my opinion, there's no real there's no such thing really as a minor defect everything has to be fixed because this Statistic this number I got from Hewlett Packard Hewlett Packard claims that 40 percent of its customer complaints are for what developers would call minor defects Okay, so people complain about minor things all the time Um, let's get to our examples. Um, oh, yeah two other jobs of quality assurance is validation This is what you have to ask yourself Um, is this the right system very often You're developing some nice software that looks really cool that works really well, but it's not what the customer wanted Okay, so make sure you're developing what the customer wants and also verification does the system work right? Well, let's get to some actual examples As we see here, is this a problem this this actually existed this was from microsoft This was from about 10 years ago But Is this really a problem? In fact, how many problems do we see here other than the weird english? Um Is there another possible problem you see this actually did happen and it caused a lot of complaints because You get the idea of what they want to say here But what's the expected result if you press yes? Presumably whatever you were doing in wordpad would be saved Well, what would be the expected result if you press no? And what would be the expected result if you press cancel? The problem is we don't have clarity here And so we have a bug That's really nothing that was caused by the software, but The user the end user will say, you know, what the hell and we you want to avoid those wtf moments in your software Okay, but what can test automation help? Um This is often the problem. I've noticed with test automation. It's a good tool, but it's often used wrongly Like yeah, you can hammer a nail with the crescent wrench, but that's not really the purpose Um Test automation is a useful tool, but wrongly used Um, and one thing I want to say in mind one thing I want to keep in mind is test automation does not in itself find bugs And this is one thing I want to say. It's not really test automation Doesn't find the bugs, but it's the tester's creativity and coding skills that find the bugs themselves Let's have a few more examples. This was actually a result of my test automation here Let's look at this. Has anyone ever seen this? Does this look familiar? The thing is when you look at this if you go to www.drupal.org and try to log in on your device Um, if you have android 7 This is what you get Is there a problem here? This is the thing you want to ask This was an actual run in a test bug in a test itself that found what I call What I see at least like at least two different kinds of bugs. Is this is this okay with everybody? the first problem I have is um How many login buttons do you guys develop? How much how many is enough login button? Do we need two login buttons? Why is there the text on one side? As opposed to the other side of the login button And also is this is the little login button in the middle responsive or not Just actually um go in and log in on your devices and see what result you'll get You'll probably get a surprising result One thing that I do in in any kind of test automation is screenshots tons of screenshots And examine the screenshots okay going on quickly. Um As I say test automation is just beginning In the end mobile test automation doesn't really differ from any other count mobile test automation doesn't differ from any kind of Any other test automation Test automation Is more think of it as a development project rather than any kind of testing project Mobile realities The hardest part this is where people run into problems with test automation Once you actually scripted the test The most difficult part is actually Maintaining the test that that's the true Genius the true talent of a successful test automation effort Is actually making sure the test can keep going um test automation works because It provides results over time You can keep your test automation going You will always have regression bugs the point is you need those regression bugs to tell you if something is going wrong Not just right after you run the test And after you script the test but as the the tests keep going week after week after week Don't have this attitude of just automate your tests But think of it as you want it to keep how can you automate your test to keep them going over time? of using Jenkins Travis whatever system you need Okay Let's keep going test automation Test automation like I said earlier but want to reiterate is is it's not a panacea by itself and it will not solve Your quality assurance problems Let's not just automate our tests think of automation as a big plan in itself. You have to have a separate automation plan Okay, how do we do this? I always think hand over your app for testing Your software has bugs, but often we don't see our own bugs. We don't see the bugs that we developed itself Um independent testing is very important for catching bugs And we have to think of development think of dev ops and continuous integration continuous development means continuous testing in its own cycle Okay, so how do we begin as we sum up? um Think of yeah the test automation here start automating the most stable things first Like the login that we just saw so the more stable things are at the bottom of the pyramid And as testing and bug fixing continue after each iteration The more stable areas kind of keep up and fill fill up the triangle Or the pyramid here Okay, what we do have what we have to do first is find the bugs first Okay in my opinion and from what I've seen in the Scary to think about 14 years of software testing. I don't think that anything will really replace Just observation of the system you created Um find the bug first and after you have a stable system really only then can automation begin Things to remember um test the automation tool itself does not find the bugs um One thing I hear often throughout the years is test automation Itself is often criticized When it's really the approach, you know test automation the tool itself isn't there to be criticized It's often the approach the philosophy the creativity the coding itself um As I really think the person writing the test is the one that finds the bugs No test automation tool can be better than the tester who's creating the test So there is no one best tool And as I say here, the best automation tool is the one that you're most comfortable and confident in using The one I use now Is apium Because it is well, it's basically the Well, the most Stable of the test automation systems And the easiest one to use And here we are just finishing up. You should test the flow of the operation Um You have to keep in mind the three things the three points below You have to keep in mind before you write your tests and also just in general. What is the flow supposed to do? What's supposed to happen? Is something preventing the flow of operation? And does the flow produce any unexpected results? And speaking of which know the expected result, what is your software supposed to do? Does the actual result you have to code your tests? So the actual result matches The predefined expected result If no, then you have a bug and report it And also you have to keep in mind The act the entire screen itself is the expected result Not just the exact same thing the one thing you want, but the whole screen is the actual result And on our last notes, yeah, I want to keep in mind, you know finding bugs is good Um to me I've I've often yes been the annoyance to the software testing project because Um to me a successful test run is one that find but that finds bugs too many people want every all the nice little lights to turn green I want to see reds there A good test is one that finds bugs Um, so create design and create tests that find bugs uses many assertions and verifications as possible to find Anomalies anything unexpected and and yeah develop that spot the anomaly life Approach to everything in life Okay Last things Your tests have to communicate with you Your tests have to inform you If something goes wrong Okay Know the difference between a bug in the software and the bug of the test script often It's our test scripts that are buggy and not really the software And know what your tests are doing and what they're finding And one thing I have to think of yeah, keep in mind think of it like this test automation is like the autopilot in an airplane When when the captain is flying an airplane and he sets the autopilot He doesn't just go into the back and start drinking the little bottles of booze, you know They're always they're watching the instruments. You have to monitor your tests You have to pay attention to your tests know what your tests are doing if they're finding bugs if they're passing And you have to update them Applications of test automation really quick Smoke tests which means like right when something has been built you test if the build succeeds Did the build succeed and is it worth continuing to test? And also regression tests you can regression test After every individual build During each sprint and also at the end to end You know for the final regression test Um How to start the process? Um Yeah, like I said earlier with the pyramid of start prioritized by importance What's the most important thing and what's the most stable and keep your tests short and for a single purpose many Tests end up being hundreds of lines long. Keep them very short Um Keep them with three parts Set up single purpose and assertions and value and evaluations Um So what do we do just ending up? Um Does anyone has anyone tried this test driven development? This is one of those things where everyone agrees is a is a good idea. It's a great idea, but it's very Rarely really used As it's supposed to A good idea something to experiment with is you create the tests first And then the feature is developed so that it passes the test. It's going to be done at all levels This can be done At the unit test level at the functional test level and also with acceptance end to end testing Okay, this way, you know, like I said the qa testing process is baked right into the development itself And to end I always say test early test often Use as many tests and assertions and verifications as possible And also be creative. You're trying to find bugs look for bugs And as I say here, um move fast and actually really break things breaking things is good So keep that in mind and now that's all that's all I've got So thank you and continue testing