 In the in the program you might see that it says I work at campaign monitor. I actually didn't pass probation a few weeks ago with that company so I can tell you a story about where the quality coach role didn't quite work out but I'll be starting a new job when I get back to Sydney for mobile testing so my specialty has been mobile it's one of my passions and I want to show you a robot that I built in a two-day hackathon today so there was a there was a competition to name the robot first of all put your hand up if you've done an Arduino project before cool how about you know involved with mobile testing on some level there we go so there was a competition to name the robot during the hackathon so it was named Taffy McTap folks I'm gonna kick you off with a demo first this is Taffy over here I'm going to exit out of my browser go to my IDE or my terminal I'm going to start a node server on my local machine there we go go over to here you can you can see the there we go back over here and I will just type in this function so that's just a demo of Christmas Carol I built this robot as during a hackathon just before Christmas so it was themed with Christmas but maybe you can recognize the next next tune it's a little bit more complicated I just heard a bolt come off at least at least my prayers to the demo gods were answered and nothing exploded but I lost a bolt where did that bolt come from I don't know I have to find out later anyway so now that I've got the demo out of the way I'll go through the rest of the presentation it's an open source design based on tapster 2.0 it's all 3D printed you can access the source code was originally designed by Jason Huggins one of the founders of the selenium you know we wouldn't be here if it weren't for the amazing team that helped out with this you can see version one of the tapster in this image the configuration is a little different it's actually they're playing Angry Birds as well so you can use it to play video games if you like all the parts are 3D printed we had a at my previous company we got a 3D printer for to give away as a prize at a conference we were sponsoring but they wanted to allow the engineers to play with the 3D printer so instead of just buying one to give out as a prize we bought two one for the office and one for the conference so I was able to use the printer in the office for this just before the hackathon I also it took me about 20 hours to print most the parts note to self do not put things on the edge of the printer or else you might get some mistakes so they were printing off fine and this took I don't know 45 minutes but it took a long time to find out that it had started to go off the edges a little bit I also went to my local hacker space out in Sydney to laser cut some of the larger parts the base and the and the clear parts were laser cut rather than printed so if you have the opportunity to go check out your local space it's an amazing community and they're always full of really helpful people who happy to show you how to do anything there users and Arduino on the top and our three server motor configuration the three server motors are arranged in a Delta bot config this is often used on the industrial line maybe in a chocolate factory where you have to pick things up and place them with precision over and over again so that's why this design is is useful for this one users magnets and ball bearings for joints I'm actually just going to disassemble Tappy here and pass around some parts for you oh that's where the bolt came off I have to reattach the server motor later that's not going to work so Tappy is now armless you now have a piece of it in your possession it took about two days worth of build time I spent some time after work building it once I printed all the materials and I spent a day in the hacker space but it was surprisingly quick I thought it would take a lot longer than I expected to build it cost me about 300 dollars in parts which that's an Australian dollars that's about 15,000 rupees but I was trying to follow an American build of materials I ordered a lot of the parts from American stores and that increased the postage I probably paid about a hundred dollars worth of postage for most of these parts but now that I've built one it would be easier to I know what materials I'd need to find locally so I should be able to build that a bit cheaper and I don't know what the prices are for electronics are like here in India or if there's a market for Arduino or local suppliers for a lot of these things the play piano algorithm we hacked away in one and a half days I pay programmed with the developer while we did this exercise I thought I'd have a lot more issues with calibration it didn't take as long as I was expecting to build I thought the first thing out of the fence was was going to be just trying to get it to touch the screen and but most of the hard work had already been done runs the standard formata protocol on the Arduino chip itself that just allows you to translate whatever language you want to run on your local machine such as node or other languages and it just translates it into the Arduino machine code or assembly like language I don't really know this in details it's just the magic of the technology it was pretty easy to get up and running I'm running a node server on my local machine and it also uses a Johnny 5 API which is a JavaScript API for robotics internet of things it's been around since 2012 I haven't seen if there's been any local commits to it lately but it's all open source this is part of the if you go to Huggins's source repository for this this is some of the math sets involved for translating the rotations of the motors into XYZ coordinates so I didn't actually have to do the maths myself it had already been figured out we have you pass in your coordinates you do a little bit of maths either reflecting and rotating and then basically just palm out those angles to the to the server motors using a map function but this was all done beforehand so it was a lot I didn't have to figure out all the maths and the tricky bits myself if you know this is a piano layout and we're mapping each key to a area so for example we have the F3 key here and the black one here is the F3 shop put your hand up if you come from a music background we're going to have like a really brief overview of music theory in this talk as well so automation and music theory go hand in hand right so for that F3 key we map where it's pressed and where it's lifted because we want the robot to move from above the key before it presses and then back up before it moves to the next one it moves to the lifted phase and then presses and then back up so we've just mapped out the XYZ coordinates of where those keys are we did that through a lot of manual process using that go function we would just pass in XYZ and figure out where the button was on the on the key and and then just coded that into our source so it's it's currently hard coded it's not collaborate calibrated for the screen and so it's just just off my based off my iPhone 6 screen size so it doesn't accommodate different resolutions just yet but I did have a pixel at one point and the keys were close enough that it still worked I used the S here after the F3 to represent that black key the sharp key and that has a slightly different area this was the iPlay trombone so this was the sheet music for the Imperial March you can see that we have a lot of flats in the key signature that means I had to transpose this music into the keyboard on my screen size I had to transpose it into a different key I also had to convert all of the flats into sharps as well because I've written all the notes as sharps a sharp and a flat are the same thing it's just a different way of talking about it for example that that F we talked about the F sharp is also a G flat so they're pretty easy to convert you can see here that it gets a little bit more complicated and if you remember back to the demo of the robot it the timing wasn't quite right it can play the slow notes quite well but when you try to introduce more complicated timing it tends to stuff up a little bit and it was hard to get this to work I think the next phase for this robot would be to figure out how to get the timing more right because we're just basing each each bar has four beats and we're breaking each bar into sixteenths and then we're just mapping them back to the to the robot but you play a note for one sixteenth that's not in the same ratio as a quarter bar a quarter note for it and then we map each song into its own function and we pass in a string that represents the the music so for example here we've got a method called play advance and this means play the F3 key for four sixteenths of a bar this one here means play the G3 sharp key for one sixteenth of a bar so that's how we've converted music into code for this robot so how can this help you test in the grand scheme of things probably not that much I felt this robot as part of a hackathon to demonstrate how robots could be used for testing but I still haven't found any really good valid use cases other than bringing it to conferences and giving really good demos I was a contractor at Google Maps at one point and there was one feature where it could have made sense to do this type of testing so I was helping triage bugs that came through for public transport and one of the local teams in Sydney was working on a new feature that if you were walking up to a bus stop or a train station you'd get a notification alerting you to the talent table of that bus stop or train station within the next five or ten minutes so as you're walking around the city you'd probably get notifications of different commute notifications and the first way to test that is to try and fake your GPS location but the Android system underneath is too smart for that you can't just fake your GPS location and have this feature work it's a bit trickier than that and the way Google Maps models the world they have these beacons everywhere and so sometimes you actually have to walk into a beacon and then out of a beacon to trigger this feature so it introduced problems with testing they also wanted to do battery testing with this feature too one of the ideas I put forward with with how to test this on over a large scale because you needed the robot or you needed the phone to be moving if we just attach this robot to a Roomba and let the Roomba drive around the office for three hours getting in and out of access points while this interacted with the app I think that would have been one way to hack long-term battery testing for this feature if we wanted to but being able to do the same test at the same time on two different devices I guess would have been a bit tricky too but I imagine that's been one of the few cases I could use this robot for testing. I'll go through some lessons learned through building this the instructions to build were a little bit all over the place I don't think they were in a central repository on the internet I did have to find some other ones and there seemed to be a little bit of a mismatch they might have been based on the previous version of the tapster robot I probably should add some of my own instructions on the internet but I've been too lazy contributing to open source and and what not is is not high up on the priority list at the moment unfortunately but I printed too many parts but now that some of the parts are broken I'm actually glad I printed more parts but it meant that the builder materials didn't have quite the right list of things I needed to get going and the bolts that I ordered I had issues with the bolts I ended up because it was using an American build of materials all the bolts were in you like inches and the threading per inch and sizing and what not and we don't really you can't really go to the hardware store in Australia and get those nuts and bolts because of metric so I but I did just end up getting some bolts nuts and bolts from the local hardware store which were close enough anyway but it added that extra layer of I wasn't too sure what to do any questions on that I do have some bonus material that I would like to go through cool we are up to bonus material so now you can build a robot and you can automate whatever you like hopefully it's inspired you to play around with a little bit of Arduino my next Arduino project will probably be one of those autonomous gardening systems that can do the weeding and the watering and whatnot I think that would be pretty cool but so you can there's this question of sure I can automate all of this but where do I start how do I start my automation and go from there so I'm going to walk you through a visual risk-based framework that I developed with one of my previous teams to help answer that question so I will exit out of this one the previous company I was working at was called Tyro I'll just give you a little bit of context they do f-posts terminals so if you go to your cafe or your florist you use a terminal to pay for a card a card to pay for your transaction or a coffee whatnot they have a microservice backend they utilize test-driven development and pair programming was a huge part of their culture there so collaboration was pretty much part of their culture just as an example every machine was set up for pair programming so you'd have one machine two monitors two keyboards and mouse mice not what's the plural for I don't know but each each work station would be set up for pair programming and often the way this would work someone would drive someone would have the keyboard and mouse and someone else would provide feedback and then it might switch and change it's a really interesting environment to work in if you ever get the chance to so I'm going to get into the framework first you can break your app into flows for example say with a banking app you might do registration my video not working I guess not let's see oh no there we go I just needed to go back and so Darth Vader is logging into his banking because even Darth Vader has to charge on credit cards and he's going through the two factor authentication setting up a password setting up a pin now is in his bank account he has five hundred and twenty dollars you might want to transfer funds have a look through your previous history just to make sure that a transaction went through you might want to send off a contact us request if there's something happening you want to question something you can send off the contact us request and Tyra will give you a phone call at a convenient time you might want to change your pin might think your pins compromised so you went to your old pin you went to your new pin twice and logging in so we've analyzed all of the flows of our app unfortunately most apps out there don't quite have six flows but now that we've had a think of all the different ways throughout we're going to map these flows onto a risk board so on our x-axis we're going to put our frequency of use if something low frequency on the left hand side high frequency on the right side we're going to put our impact if broken on the on the y-axis and draw a nice little graph like this break it up into three areas and we're going to start with number one registration has a fairly low frequency of use it shouldn't happen all that often but it has a really high impact on the user if the user can't register something is gone terribly wrong so I would like you to pair up with someone you say hi we are networking at this conference after all and I would like you to map out the other six things that are on that sheet with someone there are some of these worksheets you can use up at the front you want to come on down we're friendly people we don't bite that hard oh so on the up here we have some sheets you're going to be using doing the same sort of thing that Sam has up here on worksheets up here to repeat your instructions Sam to move the sticky notes to do risk analysis on the flows we're just going to map them into where we think they should go do we have sheets up there oh yeah yeah all the sheets are up in the front here okay how we're going with that exercise hopefully by having a conversation with someone you realize that we have different understandings of risk we all apply our risk radar differently so it is perfectly fine for you to have a different answer than I do because the context of when I did this exercise is completely different to your context you're going to have a different understanding of where these things should go but for example transfer funds is probably a higher frequency of use of registration and also a pretty high impact if you can't get money out of your bank account you can't run your business you might want to view your transactions but maybe you don't do that as often and there might be some sort of work around you can also access previous transactions on the terminal itself as well so maybe you don't care so much if it's broken in app contact us you might even question why you have this in the app in the first place because you can generally access our our phone number from our website and give us a call whenever you need but there there might be regulatory reasons why that feature is in the app being able to change pin you should probably put your hand up if you've ever changed the pin on your bank account surprisingly large amount I've actually since I've been banking for 15 years I actually haven't changed the pin on my my bank account but it has a pretty high impact if you're concerned that your pin is compromised and it doesn't go through correctly that's a pretty major issue and being able to log in it's the first thing you do before you do anything and if you can't do this then shit has hit the fan to use an Australianism you might want to map different areas of risk as well so we'll give the stuff in the upper quadrant three stars to say this is really important we might want to focus our automation on these areas first because we think these areas are going to impact our customers the most if there's a bug in this area we want to find out about it first before our customers do this area you might give two stars you might go we might be okay with releasing stuff with bugs in this area as we as we iterate but we would like to know about them at some point and here you might question why bother testing this on a at least from an automation point of view what value does it add to have a UI test for the contact us request other than it gives you at coverage you might want to map other elements of risk as well this is only a 2d representation you might want to put like a sticky note or a color on something that might have security implications so number five change pin even though from a user's point of view it might have a low frequency of use it might have security implications so this might drive your influences around we might put more automation here even though we think it's a low-use feature because of security risks and here's an example of the one I used a Tyro it grew out to a bit more than six but this was just as we were launching our banking platform and starting to do our Android app as well we started on iOS but having this next to the board and when we were developing a new feature and going where do we think this fits in the grand scheme of things would help us determine what type of automation we might want to build for this before we've built that so it helped facilitate those conversations I've also used the key to represent others potential risks as well things that might have a financial impact or illegal impact from running your own business next we can break these flows into tests but I'm going to check how I'm going for time 20 minutes left okay I'll continue going thank you and I'm going to walk you through a whiteboard exercise to be able to think through your testing we're going to number our flow here for example our change pin we're going to walk through this one we're going to put an orange sticky note here to represent this one next we're going to go through our setup for the test what type of data do we need beforehand we're going to use the pink sticky notes here to represent that we need to be registered and we need to be logged into the app before we start this test so for example we need to be presented with the home screen the next thing is the going through the tests the steps and the checks I'm going to use blue here to represent a user action so for example we click on the hamburger side menu we click on the options tab and we have some sort of expected outcome I'm going to use a green sticky note here see menu list we see the screen and I'm going to continue through the process here so next I'm going to use blue we're going to click on change passcode we're going to click here or tap so shouldn't say click because it's an app and we should see the change passcode screen so from here when you change your pin in app you enter your old pin once and then your new pin twice what are some of the other things we could test in this situation the entering the same pin trying to change it to the same pin yeah session timeout is another good one incorrect old pin what do we do with that situation so hopefully you realize that there's a whole bunch of stuff we can do we could enter the correct pin and continue down the path we could enter incorrect pin and stay on the current thing we could enter a pin after a session timeout we can enter the phone in airplane mode and ensure the next network call fails and see what happens in the app we can also press the back button and go back to the previous screen without so all of these is it's it's really good for brainstorming for exploratory testing to do this type of walkthrough to go at each point in each screen what can I possibly do and and go it doesn't make sense to automate all of these things but for example trying to put your phone to sleep and then relaunching the app app you yourself doesn't really support this feature very easily or the X codes UI automation test suite doesn't support this type of testing either as soon as you put your app to sleep your test stops running and you it's it's kind of hard to relaunch that test but in terms of manual testing that type of test is fairly cheap to do you can invest a lot of engineering into doing that type of test but you have to question the value it adds as a we're going to go do a segue down more of a non-happy path case if we enter the incorrect pin we will see a notification saying that the incorrect pin was was entered if you continue then to enter it incorrectly two more times you actually lock the pin so through doing this exercise I've discovered a new flow I'm going to call it number seven lock pin I'm going to go back to my sheet there and and map out where I think lock pin fits into everything else but through doing this type of analysis I've discovered a new feature but this this test we're talking about here is about the change pin so I'm just going to take a note and come back to that later and I'm also not going to go all the way down the error path but I might just merge this enter incorrect pin once and then enter correct pin just to get that coverage over UI I think your UI automation should be just testing the flow through the app and this comes with some caveats of it does mean you have less tests they do get slightly more coverage but it makes them harder to debug and in my team we weren't reporting on the number of tests we wanted to reduce our build time our build time for our iOS app was taking 25 30 minutes and that was just too long for a fairly straightforward app and I wanted to try and reduce that down to maybe five or ten minutes through having this that our UI automation was very on point unfortunately the refactoring to do this type of testing it was never prioritized due to product development because developing new features is always always gets a high priority and then we continue down that test there just to just to remind you of what the colors here mean we've used the orange here to map back to our risk board we've used the pink here to represent data we set up before or during the test we might have to do a back-end config set up or change while doing some of these tests we've used blue here to represent a user action and green for some sort of check or success indicator and here's an example of going through that test in a little bit more detail I've used fat green arrows here to represent the happy path and the smaller black arrows to represent divergence or non-happy parts and then we continue from there but that's the the segue I wanted to take you down but yeah so have fun with testing and hopefully this risk framework gives you a little bit more of a framework or analysis tool before you deep dive into automation taking that step back and thinking about what do I want to automate first awesome thank you Sam