 Welcome, everyone, to this exciting session called It's the 20s. So why are we still testing like it's yesterday by Justin Eisen and Sharan Baghmar. We're very glad that you could join us today without any further delay. Justin, over to you. And thank you, everyone, for joining our talk today. Myself and my colleague, Anand Baghmar, are going to talk to you about it's the 2020s and why do we still test like it's yesterday. So a little bit about myself. I'm a senior customer success engineer or software developer at Apple Tools. I've been in the QA automation dev realm for a little over 20 years now, many, many years of writing mobile automation and web browser automation. And so let's begin. So today I'm going to talk or have a quick history of what about Appium, the challenges that we face in building, maintaining our mobile tests and frameworks. Why have we tested the way we have for so long? And then I'm going to introduce you or Anand and I are going to introduce a new tool, ultra-fascinated mobile grid and the benefits of it. So Appium 2012, the beginning. So I actually was at this conference, the Sunium conference in 2012. And I saw Dan Crayar's lightning talk about iOS Auto, which touched me on many levels because it spoke about all the frustrations we're having at the time, trying to automate our mobile devices. Really the two industry standards at that time were UI automation for Apple and then Robotium for Android, which were, again, two different frameworks, two different languages. So a lot of complexity, a lot of time and effort just to automate two different frameworks. And so having seen the lightning talk was just open my eyes and saw the possibilities. And so I believe, so did Jason Hogan shortly after. Dan and Jason got together, collaborated, and built Appium. Thanks to Sauce Labs, they then backed that development. And with led by Jonathan Lips and a bunch of developers there, they brought Appium to scale, shortly adding Android and then all the other capabilities until what we have today. So we do owe a lot of gratitude to these guys, to Sauce Labs, for making Appium what it is today. So what are some of the challenges that we face in today's automation for mobile? So as you probably know, there are lots and lots of dependencies that have to be installed for both Android and iOS. You can run into versioning conflicts with whether it's your operating system, your Java node. Could be the Xcode command line tools. It could be the Android, APIs, SDK levels, Appium version, maybe beta, maybe in two older version. So lots of conflicts that you can run into, which over time have gotten a lot better. The Appium devs throughout the world actually have made a tremendous amount of effort into making this as easy as possible, especially with Appium 2, just getting the required dependencies has become a lot easier in a lot of libraries that they've had to make that. And especially now that it's plugin-based, we don't have to necessarily install all these dependencies. We just need to install the dependencies needed, depending on the plug-ins that we want to use. So that's great. We're heading in a great direction there. Device flakiness. I think device flakiness has actually been, since the dawn of mobile automation, it's always been a problem. Especially years ago, when it first started out, it was the Wild West, and devices were constantly changing. And we had new app vendors coming out, especially for the Android side, or basically Android side. So I think in the last several years, that's worked its way out as these frameworks, and even the mobile devices themselves have matured. We have many, many different resolutions now, and form factors with phones, tablets, tablets. So we have to account for all these in our automation code to be able to automate across all these different resolutions. We have app testability issues. I could probably have a conference talk just on this topic alone. Traditionally, or a lot of times, we're just given these applications, and we're told, go ahead and automate it. Well, it's not always that easy, because some apps are very complex. So we really need to work with our app developers to make our apps more testable. That could include adding APIs or some type of hooks to help with the automation tests to change the state of UI, maybe have hidden elements that only the automation framework can click and interact with to give us options to get the state of the UI to a certain point where we can automate it correctly. Consistent locators, that's always been a problem. It's always been a problem for a web as well. That's getting better, as I think more and more development teams and automation engineers are starting to use the industry best practices, such as accessibility labels to handle that, but it still can be a problem and a challenge for automation. Mobile devices or mobile apps utilize lots of gestures these days, and some of them are very complex. But then when you factor in all the different resolutions that we have to handle with that, it may work fine on one device, but the gesture has to be completely modified for a different device, so that adds more complexity. There are challenges to our testing. Then test scaling, obviously the most efficient way to scale your tests is through parallelization. And so that can lead to more challenges as well. Test environment issues. This is another topic I could probably have a whole talk about is a lot of times as developers or automation engineers or testers were given terrible test environments where they're completely flaky, maybe the code's not there, maybe the machine that's running the test environment doesn't have the resources that needs to fully run the way it needs to, so it results in flaky tests. In accessibility, more and more so now, more companies are requiring that apps and even countries that their applications must be 100% accessible for users out in the world that have disabilities. And so it's important that accessibility is tested and, but again, it adds another layer of challenges to do that. So let's talk about some scaling challenges. We'll go through some examples. And so what are some of our scaling options? So we can certainly just execute our tests over and over again sequentially on every device, not very efficient, but sometimes, depending on how your framework's set up, you might not have any other choice, but of course it's gonna take you forever to test it across all your different devices that you support. Ultimately, the best and most efficient way is parallelization. And as I mentioned, I can add more challenges to your tests, especially when you consider applications that may have user sessions where you can only log into one user session on one parallel process or thread and another thread that can't use the same user, so then you have to have multiple users. And of course, you don't wanna have your test set up where one thread changes something on a page that another thread of the same test needs because if one thread deletes something in the UI and the other tries to use that, it's gonna fail. So that's gonna cause your failures of flaky tests, but of course with proper parallelization setup of your framework, you shouldn't run into that issue. We have the Selenium Grid, which we've had for many years and the awesome OS developers or open source developers for it have continuously improved it, especially now, I think Selenium Grid 4 is just about to be released or has been released and all the effort that Diego Molina and Simon Stewart have contributed to it and all the other many open source engineers has been a great benefit for us and it has been very benefit for many years. A lot of companies actually rely on it as their core architecture to bring them business. Docker images have come into play, last three or four years and that's even growing now with like Kubernetes and we're having these big ecosystems to help scale our tests using that. Then we have many cloud servers, test servers is out there to help scale our testing as well. And then finally, if you can ultimately just go to the cloud servers themselves, such as AWS, Azure, Google Cloud and run your test from there using a wide array of different architectures, such as Selenium Grid, Docker images and so forth. Even AWS has what they call AWS Fargate, which is a serverless-based service and a lot of people test their mobile and web apps using that as well. So here's a standard execution of Selenium tests or sorry, an Appium test running locally from your computer. You have your test code, your computer, your Appium server, iPhone or Android, whatever it is and an app server if your app requires it to send and receive responses from your app to publish the results of the UI on your device. So say we want to run a simple test and we wanna run it on multiple devices. The first option is we just run it seriously. So redundantly running the same test over and over and over again. On different devices, which is not efficient again, it takes some time, but basically this is a basic example of a test. We launched the app, it goes to the UI, we assert something that yes, something's displayed or it has this text value or it's this color or whatever assertion we want just to make sure we're at the right place. We click a button, we may add a weight here to wait until we've gone to the UI state has changed and then we verify several different assertions to verify that the UI is displaying everything we need to or the few things we are looking for. But then we run into problems such as this. So you can see here those assertions that I wrote on the example before on iPhone 12 worked perfectly fine. So however, when we ran the same test again on iPhone eight it only got the first assertion, the remaining assertions it couldn't find because it couldn't find the element to get to the next few. And of course, many of you on this that are watching this probably have seen this error as we have many times before. So what happened in this case? Well, we obviously had an issue with our test kit. So this was one of the points of failure that we need to account for to now fix it which we simply can do by just adding a scroll to element. So it's a simple fix, but again, it's adding more code and more logic to our framework. And of course our scroll to element may need to it could get more complex depending on all the different devices that we need to use in our resolutions. So let's take it a little step further. Maybe we want to run our test in parallel and we want to scale it across either a cloud service, the Selenium grid, Docker containers, essentially it all comes down to we're gonna spin up multiple Appian servers which it's either gonna be a local or it's gonna be on some type of cloud service. It's gonna connect to each device that we want to test with. And again, that's the same flow but now we've just cascaded the processes. And if we move on and again, we're redundantly running the same tests across the same or across different devices to get the results back for testing. And so here's a quick, very simple, same test but now we're gonna scale it. We could do this in multiple different ways. We're using testNG, JUnit, Mocha, RSpec, whatever it is and all the different language findings have different ways for parallelization. And this is just a simple linkless creating a thread for each of the capabilities I want to test, so the different devices. However, what can go wrong? Well, there's many things that could go wrong. Just the scenario alone, just tested on three different devices, potential 42 different failure points. There could be a failure with an App server. So like we talked about, as I spoke about before flaky test environments, maybe the communication between the App server and the phone got interrupted. Maybe the device that we're testing with either locally or on the cloud service, it had issues and it didn't connect. Or the Appian server crashes. There's just many points of failure. So essentially every icon you see on the screen and every single error, arrow back and forth, there's a point of failure. And so what if we traditionally done here? Well, we'd have to go in and dig in and start looking at the logs, maybe looking at our App server logs, watching the videos in the cloud service if that's where we're running it and kind of figuring out what happens. But ultimately, nine times out of 10, we just run the test again and we'll cross our fingers and hope that it passes. And if it passes, we just say, okay, great. I guess it was a fluke. But it's a lot of time spending into debug. So why have we tested this way as long as we have? Well, it's been the maturity of the native ecosystems or probably a lack thereof. As I mentioned earlier, it was sort of the wild west. You had multiple devices coming on the market. Some of these vendors that created these devices don't even exist anymore. Apple was constantly making changes to the market and just things were not stabilized. But the last few years or so, that it's becoming more and more stable. We want to ensure that our applications run on all these different devices, our application support to ensure our customers can successfully use our application. Of course, it all becomes monetized and we want to make money if that's what our app's intention is. And historically, this was always the approach. We didn't really have any other options. There was no real alternatives until now, which I'm gonna kind of show you. So introducing the Apple Tools Ultra-Fascinated Grid. And so it's the standard local approach, like I showed you earlier a few slides ago. All you have to do is run it on one single device, either an Android, either a simulator, real iPhone, right now we only support iOS, it will soon support Android. And we just run it once, but we get the same results of all the parallelization we talked about before, all within the time of one single execution. And so here's an example test that we did before. If you notice at the top of the test, it's just we've removed the assertions because now we're gonna implement visual checks. And the power of visual checking is it checks everything on the page. So every single element and every visual attribute of every single element is automatically asserted for us, such as the text, the color, the location. So if anything changes, it will be caught without us having to physically actually look at the device or the application itself. So no longer do I need to pick and choose to write certain assertions, something that's displayed because if it's displayed in UI, visual check will happen. Next, at the very bottom, you can see I could just set all the devices that I need to validate my application on. And finally, in the middle there, we can, if we wanted to, we can write an assertion in a single assertion to verify that everything is visually okay and everything executed correctly on all the devices on the ultra-fast grid. And so here are the results. You can see all our tests here, all the devices that we executed on, it executed within around 37 seconds. So the execution of one gave me the results of 12 different screenshots across all those different devices that I configured. So very efficient, fast way to test a script running on a single device instead of redundantly running the same test or run over again on multiple devices. I'm just showing you some examples. You get full page screenshots. Actually, if I go back here, take note that you could actually set it in landscape and portrait, multiple different capabilities built within the configuration. So perhaps let's maybe we want to take it a step further and let's kind of test with a bit more modern app application, not such a basic app that we had before. And so this is just an example to show you everything you can test all at once, both functionally, make sure it's visually correct. You also test localization and accessibility. So in this case, I'm going to just start two different Appian servers, have two different phones, but then get screenshots across all these different devices that you could see here in the middle, as well as visual checks across. And then finally at the very end, I'll just go ahead and start the video. At the very end, I can write an assertion to test that it was visually passed and that it actually passed or failed accessibility-wise. So for the time it took me just to run it once, I get coverage across multiple devices very fast and efficiently. And let's just go ahead and take a look at the results. And so you can see here, now I have 72 total tests captured. Two different languages across nine different devices and four different screens giving me 72 tests. And I could sort my tests by localization. So I could see all my English screens and I can then see all my Spanish screens across all the different devices. And so now I could go ahead and inspect some of the UIs that were captured and verify just as an extra incentive to make sure that they are, they look correct. Maybe the layout's correct. I could take a look at, now I could test the localization, or sorry, the accessibility. So now I'm gonna verify the contrast. So here we have a failure. I have the option here to just ignore that failure if I wanted to, because maybe I've talked to my product team or whoever it is in my company since it's not an issue. Or I could go ahead and just create a bug. And if I want to, I could simply just create a Jira ticket or a rally ticket or a few other bug tracker systems that we would use throughout the industry. And simply just message my developer saying there was an issue that was detected. So what are some of the features and benefits of Ultra FastCrate? Well, it's a simplistic test authoring basically you write and run once. So no longer do I have to consider or think about all the different resolutions or all the different devices and have all these different conditions within my code saying if it's an iPhone 8, then I need to scroll or if it's an iPhone 10 and do this, I just write it for one single device, one UI, one resolution with that in mind. So it makes my complexity of writing my testing much, much easier and much easier to maintain. Everything is gonna be visually perfect. So I know how visually my application looks and I have baselines based off those applications and the different resolutions of those. And of course I could get full page screenshots. So if we have applications that have long UI views that you have to swipe up to get see the entire view, it's all captured forming less code maintenance. And so inherently just with visual testing adding visual testing to any automation framework for functional based testing, it definitely reduces the code that you have to maintain. So no longer do I have to maintain all these page objects all these functions to assert different attributes on the UI or gestures to get to do these assertions. The visual test doesn't for me so that all that can be thrown out and it's much faster and more efficient. Fast fit feedback loop, especially in today's CI CD world, we need everybody wants as fast as possible feedback loop. So now we can, because everything we're gonna be doing with these checks is gonna be across multiple different devices at once. Less points of failure. So no longer am I redundantly running the same test over and over and over again on different devices. I'm just running it once on one device and so it reduces the point of failures. Of course, there can always be failures of the device that you're testing with locally or on your cloud service or your Appium or your code. There's always gonna be points of failures but in this case, there's less. Easier data management or test data management. So no longer, as I mentioned before with parallelization, it adds complexity of having many different user sessions or accounting that one test is not messing up another test on a different thread, that's out the window as well because no longer do you have to worry about that as much. Of course, we could get contrast validation, which I showed you and everything is asynchronously parallelized. So as it's running the test, everything's running on the Apple tools ultra fast grid in parallel without you having to worry about any type of connections to any type of servers. And then security, some of the benefits that with this offering, you don't have to upload your application to any service, any third party. You don't have to open up any tunneling, firewalls, proxies to get, to have the service talk to your internal application server. And so you also then don't have to worry about any of your data or sensitive data in that matter being left on any third party systems. And so with that, I am going to turn it over to my colleague, Anand who's going to talk to you about the ultra fast grid for mobile web. Hey, thank you, Justin. I'm quickly gonna share my screen. Okay, so hello everyone. I hope you're having a wonderful APM conference. And quick introduction about myself. Like Justin, I've also been in the quality space for more than 20 years. And I love dabbling with code. Quality is my passion. And I work in various different roles to help make the product quality better. So with that, let's continue from where Justin left us talking about scaling challenges for native applications. All the things that he spoke about for native applications we have also seen for our browsers and it could be desktop browsers or it could be mobile web as well. This doesn't stop just at browsers or native apps. Hybrid apps would also fall into the similar category. The challenge is not really in validating the functionality across these different devices or browsers, especially when it comes to browsers. It's not really about validating the functionality. The way the pixels are rendered in the browsers that is still controlled by the browser's rendering engine. The browsers are W3C spec compliant. So functionality would work fine, but the rendering can still be different in these cases. And this could be for various factors not just the rendering engine, but also the devices where the pages are being rendered that also contributes to this fact. So what it really comes down to is how do you ensure your application is working as expected? It is seen, the user experience is as expected regardless of what type of device is being used by our users. It could be a phone, it could be a tablet, it could be a tablet, it could be a desktop browser. It does not matter. It could be portrait or landscape does not matter. Each device is going to render the pages in a slightly different way. And you need to ensure that everything is going to continue working fine as expected. The sizes of these devices are also varying a lot, which I don't really need to explain. Everyone is aware of these aspects. What this does mean though, is how do you make sure the experience that the users are getting continues to work fine. Likewise for the desktop browsers as well, or laptop browsers, I don't know many people who would be using desktops anymore, but the challenge is the same. So the traditional approach that Justin was referring to for native apps is also of course seen in the browser landscape as well, whether it is mobile web or in the desktop browsers. How do you really scale your execution to make sure your functionality and your user experience continues to work as expected on each of these? So the traditional approach, what you would do is, if not all, you have an identified subset of tests, which you would want to run on each and every different type of device that you can get your hand on and ensure everything is working fine. This is a tried and tested solution and it works. It works reasonably well or as the scale really increases, it starts falling apart. There are also a lot of different techniques that you can use. You can build your own infrastructure or to support the testing of this or you can buy or use the cloud solutions to test this as well. However, the challenges really are that there are far too many combinations over here. This is very slow feedback and it is error prone. So what can be done differently from a mobile web perspective to handle this situation for a mobile web? And that's where the same solution that Justin was referring to, the Apple Tools Ultra Fast Grid, can give us a lot of value by reducing the footprint of how many tests you're executing on each supported device or browser. And what I want to do to showcase this is I'm going to keep my fingers crossed and do a live demo. What I have over here is an Android emulator that is running. I have a simple APM web test that I have implemented and I have included a lot of different visual checkpoints using Apple Tools over here. I'm going to run this test. As you can see, there is just one test that is running and so far it seems to be continuing well. So maybe the demo gods are with me. We see that my test is actually loading the convention website for APM Conf. It is going to click on got it so that the cookie pop up disappears. And that really is my test now. Oh, actually I go to the schedule as well. I forgot what the test was. It's been a long time, but this is what my test is really doing. I want to make sure I'm able to get to the schedule page as expected and see the results out of that. Now, what happens at this point is in the apt-tools dashboard, you would start seeing the results arrive for this test. There was just one test that ran, but actually over here in the dashboard, we see there are 20 tests that are running. And the speed of execution is going to be different for me compared to Justin because Justin is sitting in the US. That is the cloud server that I'm pointing to for apt-tools. I'm in India, so I have to go all the way across the wire, across the waters to get to the server. But if you look at this test over here, the single test I ran on my emulator, but I'm actually now seeing the results of that on a Safari browser, on Chrome, Firefox, Edge, automatically, and it is highlighting to me what the differences might be there on the screens that have been captured. And this is not just about the desktop browsers. I have got many different types of devices that I have grouped the test by, whether it's Galaxy Note Plus, OnePlus 7, iPad Mini, in Portrait and Landscape Mode, and automatically I'm getting the visual validations happening, the functional and visual validations happening from a test execution in a very easy fashion. So the solution gives you a lot of different advantages from the test execution and scaling perspective. Because you're running the test in your own environment, you are able to execute it really fast. You're able to test the functionality if it is working as expected, but with the ultra-fast grid scaling approach, what you do get is automatic validation of your functionality and the visual rendering or the user experience from all your supported browsers and devices. And for that matter, I did not really need to run this test on an emulator. I could have very easily run it on a desktop browser and from there, I could have still got the results from mobile web as well. So I hope this one aspect of scaling can be useful to you. There are places where you might probably not find this useful and that could be if you're using some custom components that are developed for different browser types, custom UI components. And if that is the case, then you want to definitely run those on individual browsers as well. But that would be a very small subset of tests that you're repeating on each and every browser and that is supported with a mobile web or desktop. And the rest of the tests, you can get good value from this scaling approach. So with that, we will pause over here and see what questions you might have and try to address those. I will stop my sharing. So Justin can also come on screen. Okay, so Justin, we have some questions. Do you want to start taking that on the top? Let me take a look. See, I don't, I still don't feel I understand how it's different running test multiple devices, browsers, saw slides or browsers that can be elaborate. Yeah, so it's a different technique, different architecture or a different way of looking at it based on sort of if you looked at how the slide that I showed you a little while ago where everything's asynchronous. So I run it once. Our application code is sent to the altifast grid and then rendered on different devices. Whereas if you test on browser stack or sawslab or several other cloud providers, you're redundantly running the same tests across multiple different devices. And so this way you run it once on one device and get the same results much faster across all the different devices that you would normally test on in a redundant fashion. Next one is visual tests are great, but how do you ensure interactions are working as expected on different device browser combinations? You want to answer this one or none? Sure, so based on what functionality you're trying to validate, you need to map out on what combinations of devices whether real or emulator, you need to do that validation. In fact, the mobile test pyramid is a very good reference to follow in this case to think about your test execution strategy, what type of test should run on what type of devices or interfaces? Can I run something in mobile emulation mode? Can I run this something on an emulator? Do I need to run something on a real device? So based on the type of test that you have and the impact it has or potential impact of the underlying operating system it has since we're talking mobile operating system versions, custom OS versus stock OS, in case of Android especially, you would want to categorize these subset of tests and make sure you've got a good spread of platforms, real devices that you actually test this on, not just emulators. And that could be the way to go, but it's taking very conscious decisions about which tests need to run where and what's the risk of something not working. I hope that answers the question, Nuresh. And it looks like Nuresh has another question here. So it's based on your experience, what percentage of bugs, defects fall into visual rendering related issues versus logic or data related issues? I guess I can answer this. I mean, it really depends. I mean, logic and data related issues is always plagued our industry for testing. I mean, really we want to make our test environments or test deterministic as possible. So that a lot of times includes setting up your tests to be successful and deterministic by seeding your data using APIs or databases to make sure you get deterministic data. However, with visual testing, that can impact visual testing if you're not making your tests deterministic. However, with at least with Aputool's visual testing, we have different tools or different annotations to handle dynamic data such as layout. So if we have data that's changing, we could apply layout regions to verify that that same type of data is there such as an image or text and not fail in the future because the AI algorithm is gonna verify that same type of object or type of object is there. To add a little bit more to what Justin said, there is also another very interesting aspect what visual testing can bring in the validations. A traditional test automation approach is a very deterministic approach, right? I know what actions am I doing and what are the validations that I want to do as part of that? What assertions I'm going to have in my test. But our pages are complex or our screens are complex and because of data issues, because of logic issues, it could be in backend or in the rendering logic that might be there processing the data. We might see unexpected behavior on the screen which could be very easily lost in a traditional test automation approach based on assertions because our assertions are checking only for very specific behavior and you might miss out on other areas where the errors are manifesting and that's where visual validations can give you a lot of benefit where all those differences will be highlighted to you and in combination to the different algorithms, you could use it and tune it correctly to highlight the issues that are really going to be meaningful for you to take further decisions. So that's where it's not a direct answer about percentage of bugs or defects, but the coverage that you get with visual validation increases the value of your test tremendously and hence you need to write less code, less tests, which Simon very beautifully referred to in his keynote earlier. This automatically helps you give that added coverage while reducing the code footprint that you have. Next one is did we specify, did we specify on which devices we want to run that script? Yes, yes, certainly. So during one of my slides, we have an enum on the iOS device type. You could set all the devices that you need to test on. Same thing for both the mobile web as well. So when you test on mobile web with the ultra fast grid, you could set a configuration. You could tell it all the different browsers that you want to test on. We also support IE 10 and 11 and we have several different versions. And so I just want to, and this going back to the first question as well, what's the difference between other cloud, traditional cloud services is, again, you just run this once and for the execution time of just running one test or maybe several tests, you get the full coverage instead of redundantly running the same tests over and over again on all these different browsers and different environments. Same thing applies to the native mobile as well. All right, thank you so much, for the introduction and Anand for sharing your experience and for the super informative session. Thank you everyone. Goodbye, take care. Goodbye.