 Alright, thanks everyone. Thank you for attending my talk. My name is Dan. I am a software engineer at Sauce Labs. I've been working on the Appium open source project for two plus years now. And probably my two most notable projects is I've done a lot of work on Appium desktop. So for those of you who don't know, that's our UI client. And then what I'm here to talk about today is the Appium espresso driver. So before we jump into discussing the espresso drivers, specifically I'd like to briefly touch on how it is that cross-platform automation works in Appium. So as most people in this audience probably know, Appium implements the WebDriver protocol. The WebDriver protocol is part of the World Wide Web Consortium, so it's an official standard that will be around for ages. And in the words of the standard itself, it's a remote control interface that enables introspection of control of user agents. So some examples of implementations are a WebDriver or ChromeDriver which automates Chrome, and FirefoxDriver which automates Firefox. Now, Appium as probably most of you know already is special in that instead of just implementing the Web browser it takes the WebDriver protocol and brings it to various other platforms. So mobile apps, desktop apps and possibly more to come in the future. So the question is how is it that we bring the WebDriver protocol to these platforms? So we have automations for several different platforms, so Android and iOS. Those are the most notable ones and the ones that we support at Sauce Labs. Recently we got a Tizen driver, that's the Samsung operating system. Windows is supported through the WinApp driver and U.itv is also supported and I didn't include it but we have a Mac driver automation here. Now Tizen and Windows are actually special in that they are actually built and maintained by the actual vendors. So that was a pretty exciting new addition to have fully native vendor specific drivers. So Tizen was developed by Samsung, Windows was developed by Microsoft. iOS and Android on the other hand we don't have such implementations of WebDriver agent. Apple and Android don't develop WebDriver automations for their devices so we have to take the Appian Core team has to take their native user interface testing platforms or frameworks and we have to bring the Appian standard to those. Now how do we do that? So we wrote a sort of wrapper around these native test frameworks. So for iOS we have the XCUI test driver and included in that is a Xcode project we have called WebDriver agent. We use that to basically delegate to Apple's native XCUI test framework. Android we have a couple of drivers, well we have three but two that are currently supported. The UI Automator 2 driver which uses androidx.tests UI Automator 2 framework and what I'm going to talk about today the Espresso driver which uses Google's Android X Espresso framework. So now it's helpful the whole point of having a standard it seems like the whole point of having a standard should be we shouldn't care about what is being used under the hood like what native framework is being used because ideally we shouldn't care because it's supposed to hide the implementation details from the tester but we should just know how to use the WebDriver protocol and we shouldn't care what's under the hood but anyone who has some experience with Appium knows that it doesn't quite work this way we try our best to implement it as faithfully as possible to the standard but there's always going to be differences in feature support, differences in performance and then differences in the ways that the drivers work and it's quite helpful to understand some of those important differences to help write better Appium tests and so that's where I'm going to talk today so I'm going to talk about the Espresso driver and firstly I'm going to do a comparison of Espresso and UI Automator 2 so these are the two frameworks that we use that we saw in the previous slide we'll do a quick outline of how these two differ from each other so those differences affect the Appium drivers so let's give a brief overview of the two UI Automator 2 is black box style testing so that means you can automate apps and the device and you can automate any app not just the app under test and they describe it as functional UI testing across system and installed apps and it also has device level operations so things like toggling the Wi-Fi pressing the back button setting the orientation Espresso is what they coined a new term as gray box style testing so gray box style testing framework it's a functional testing for testing the app under test and gray box can be described as it's mostly black box except it has access to the internals of the android application so this is a bit of a double-edged sword having it access to the internals because it can now actually help the stability and performance of your test but obviously you're violating the principle of it being a black box testing framework because ideally we want Appium to be something that models faithfully the way a user uses your application but I'll go over now two of those specific features that I think are interesting and then if we have bonus time one more so feature number one Espresso has the ability to wait for the UI to be synchronized before performing operations feature number two it can access elements that are outside of the viewport excuse me using data matchers I'll explain what this means and why it's advantageous to testers starting with the first one UI synchronization Espresso waits for the UI to be synchronized before it performs its next operation what does this mean and why is this advantageous to us so to test this I'll show let's go over a very basic test case so for this test case I want to open up a pop-up menu and select an item so I have screenshots here so I want to click the button that says make a pop-up clicking that button will cause a menu to open up and then I want to click an item in that menu sounds easy enough so I'm going to try this really basic Appium script this is just pseudo code and I'm going to use UI Automator 2 first of all first off so we're just going to locate the make a pop-up button and click on it locate the menu item and click on it so I'm just going to run this test here let me just mirror my laptop ok so this is the test ok so it actually passed the first time but failed the second time the first time I've ever seen it pass so but at least I know we know that it's flaky so it failed the second time it didn't find the element so can anyone tell me why it didn't find the element in the menu item so element not found exception so I kind of forgetting that kind of sounds like the consensus was you're all correct that it didn't wait enough time for the UI to render the menu item ok so to demonstrate what happened I'm just going to go over this basic HTTP network chart so we requested to click the make a pop-up button we dispatched the button click and the pop-up menu starts rendering and the duration it takes to render is represented by the green line then we respond with the button click success so we didn't wait for the UI to finish rendering because the UI Automator test doesn't care what happens after it just knows the button was clicked we can respond with success but then here's what happens next we request to click on the menu item and as you can see here it's not done rendering yet and it's going to respond with a failure like what we just saw so the element isn't there yet element not found error so what are some fixes for this well simple solution just add a delay so this is the pseudo code solution here we're going to do the same algorithm locate the pop-up menu button and click on it wait for three seconds so if you're doing Java code that would be like thread.sleep which as we all know is a code smell and locate the menu item and then click on it so I'm just going to change this ever so slightly so I won't do the demonstration for it but I'll just show what what what does happen is that same thing request a click pop-up menu button dispatch the event pop-up menu starts rendering respond with button click success again it doesn't wait for the UI to stop rendering except on the client side now as represented by this tan line we've added a delay there that waits until it runs the next command and then we we request to click the menu item button and we have success this time so the pop-up menu is there we can click the menu item now what are some problems with this well how long should the delay be that's the big problem so we don't want it to be too short because if it's too short we don't give it enough time to wait for the UI to finish rendering and we're stuck with our original problem we also don't want it to be too long because even though it may be long enough if you're leaving too much time in your delay you're just adding a bunch of idle time to your tests and you're making your build needlessly long so you need to find this kind of Goldilocks zone of what the perfect thread.sleep is and then throw into it the problem of what if you have some emulators or devices that are more performant than others then what might be an adequate thread.sleep for one that won't be adequate for the other so what's another solution so another solution is to do retries when locating an element so the algorithm would look something like this locate the make a pop-up button and click on it try to locate the menu item if the menu item is found click on it and we're done if the menu item is not found wait for 500 milliseconds and return to step 2 if a certain max number of retries was reached throw an exception and just give up and say we didn't find the element so this looks a lot better than the thread.sleep proposition this is what it's going to look like now so same thing as before but now we're just going to do a whole bunch of retries to get the menu item until we found it and we got the menu item and we're done so this technique introduces a added problem though because as you can see introducing this fix adds quite a few HTTP requests in this example here we have 1, 2, 3, 4, 5 6 round trips if you're running Appium locally that's not a huge problem but if you're running it in a cloud service like Sauce Labs or a Sauce Labs competitor it won't perform as well because you're going to be hitting a remote end point over and over and over again so yeah and also another problem is we're kind of left with a similar problem of the thread.sleep in that we need to pick an optimal interval period and an optimal number of retries which again might not be a one size fits all solution for everything alright so that was me running the test using now I am going to run the test using espresso so the test is the exact same the only difference was I set the automation name to espresso instead of UI Automator 2 so it works and then let's try it again so we know that it's not flaky so they work this time though it should work it's worked every single time I've tried it and it finished in 11 seconds so it clicks a button on the menu item it works so it always works what's different about espresso versus UI Automator 2 that's allowing this to work now so we did it without needing any retries or intervals so to explain what espresso is doing under the hood let's look at this again we're requesting to click the pop up menu button dispatching the button click and the pop up menu starts to stop rendering we respond with success so it still doesn't actually wait for the UI to stop rendering it responds with success but now what we're doing is we're going to click the menu item button and what espresso is going to do is it's actually going to wait for the UI to be idle before it performs the operation so the operation will be blocked until the menu item is ready and so it sends the success once immediately after so as you can see compared to the thread.sleep and retry interval this is a perfectly optimal solution because we're clicking the menu item immediately after the menu item has been rendered there's no delay after and there's no flakiness because no matter how long it takes to render the menu item it's going to click the button right after it's done rendering so why does it do this well espresso does three things that ui2 doesn't so it waits for three UI synchronization conditions to be met before it performs an operation so espresso will not perform an operation until number one the message queue is empty that's some data structure and android that I'm not familiar with but I read it in the documentation number two it waits for async task any instances of async task that are currently in progress waits for it to finish async task is like the I'm a javascript person but I know it's like the java equivalent or the android equivalent of like a promise so it's something that waits for a non-blocking operation to finish so like a network request uses async task and then thirdly it waits for all instances of idling resource to be idle so the first two I won't talk about but number three I'd like to go into more detail on those idling resource what is an idling resource an idling resource is defined from the documentation as an asynchronous operation whose results affect subsequent operations in a UI test so we can synchronize user interface in an app in espresso in our android applications by registering these operations with the idling resource object so just to be clear this is a application development data structure this isn't a thing for writing tests so this is something that developers actually need to add into their application an idling resource object has two states either it's idle which means espresso is allowed to run so long as there's no other idle resources or not idle resources sorry and as long as the other two conditions are met and then not idle which means that espresso is blocked until it becomes idle so if you have one idling resource object in your android application you're blocked from doing anything like espresso blocks from doing anything so if you have an idle resource one or more and you try to run an espresso operation espresso blocks it until that idling resource is made idle so I think we can better explain this with a coding example so this is called simple idling resources resource so this is java code so line one we instantiate the simple idling resource object and the object has the initial state of idle line number two we set the state to not idle using the method set idle state false so this means that it's not idle and anything that happens from now will block espresso from running step number three we do some kind of UI operations so the example I used the operation would be rendering a menu item you can think of this as just any UI operation that can't be interrupted and it's just like a critical UI operation area should be locked by an idling resource so we do the UI operations and then we set the idle state back to true and then that idling resource is no longer is officially idle again and espresso as long as the other synchronization conditions are met is permitted to start running again so anyone who has any experience with multi-threaded programming this pattern actually should look pretty similar familiar to you this is similar to how a mutex works you can think of it as like line number one is sorry line number two it's locking the mutex line number three is the critical area and line number four is it's unlocking the mutex and then so another couple idling resources there's counting idling resource which has a default state similar to that of being idle and then it has an increment and a decrement method so when the number goes up it becomes not idle and when it goes back down to zero it's idle again and then we have URI idling resource which blocks when there's pending network requests so what are some of the drawbacks or concerns with this well I said before how the pattern is similar to multi-threaded programming and there's a danger of your test becoming deadlocked so if you have an idling resource that never gets released your tests are going to be blocked forever and it requires changes to the app code which requires coordination between the QA team and the dev team which doesn't always work efficiently and then number three it still doesn't provide a guarantee of UI synchronization idling resource you could possibly mess it up and implement it incorrectly or forget to do it you may still need to fall back to using retries but I guess the idea should be that it's something that's there to help you make your tests more stable and less flaky so with that complete I am now going to move on to the second of the two features I'm going to discuss and that's with espresso you can locate elements that are outside of the viewport so again I have a test case here I want to find an element from a large list and the element is called search view so the search the item I'm looking for is outside of the viewport so it has to scroll down until it finds it and then clicks on it okay so let's do another coding example that just I'm just going to find the element with the content description search view and then click on that element yeah so it didn't find the element and the reason it didn't find the element is because the element we're looking for is outside of the viewport so why isn't it being rendered so why is the search view not interactable like even though it's outside of the viewport why can't we just they just render it all well up on the left here I have the list of the menu items and then the view hierarchy so you can only see everything from animation to game controller input the search view menu item is the one that we want to see and that comes it's alphabetical so it comes way after that yeah so it only it only renders the visible elements why is that so let's to understand this better let's look at how androids list you object works list view android list view widget is an instance of what we call is an instance of an adapter view an adapter view is a UI object that has a object usually a collection of data in memory that represents the entire represents the entire contents of that data structure so list view is an adapter view I think there's like grid view scroll view lots of different things so that adapter view will only at a given time display render and display the items that need to be made visible to the user of your application and this is what the list of adapters look like so in this case so this actually this is a truncated version of what I got from the menu view it's a list of plain old javas objects that define the attributes of the element so the content description the title and the intent so in my case there's 11 visible items the rest of the items are not visible and the item that we want to interact with is in the not visible portion why did they do this why not just render all of the list items well the obvious reason is performance so fewer components you render in a user interface the better performance is going to be because rendering user interface is expensive and the more components the more memory is going to be used the more CPU is going to be used so the better the app performs and so this is this is good for users of the app because they're going to get a better performing app and this by the way this pattern is you see this pattern in pretty much almost any application design so iOS, web yeah it comes up all the time so it's good for the users of the app but bad for the testers of the app because there's no easily way to locate elements that are off screen or outside of the view port the solution if we were to use uitemator to is to use a scroll action that scrolls through the component scrolls the element in the view and the algorithm will look something like this try to locate the element if we find the element return the element if we do not find the element scroll down and return to step 1 if we've reached the bottom of the scrolling then we just give up throw an exception and say we didn't find the elements kind of similar to how retries work so I'm just going to show another coding example here scrolling, found the element, clicked on it and that took 21 seconds so it works it's a pretty reliable method but what are some problems with this fix well likewise a lot of extra HDDB requests similar to how retries work a lot of HDDB requests to perform the intermediate scrolling operations so that case we scrolled like three times if you have a really long component scroll dozens and dozens and dozens of well no matter depending on how big the component is it can just take forever to scroll so that's lots of HDDB requests and that's lots of different scrolling events so it's lots of multiple scrolls instead of one smooth scroll so ideally it would be nice if we could just instead of having those janky multiple scrolling events if I could just scroll to that item in one smooth motion also there's potential flakiness so maybe we have a large device that we don't need to use a scrolling technique on and our tests work without scrolling but then when we try on a smaller device it breaks and yeah so that's where espresso comes in with espresso we can write locators that actually target the adapters instead of the actual views themselves and we automatically scroll them into the viewport so this is the object we want to target here so if you remember from before the list of adapter items we want to target the java object so not the view but the java object within the adapter that has the title I have it in bold here search view how do we do this so this is an espresso example so this uses the espresso framework it doesn't use apium so this is independent of this espresso has a thing called on data or data matchers that are you and they use hamcrest matchers hamcrest matchers are java matchers that are used to target objects so it's like a querying strategy for targeting java maps or strings or just any kind of collection of objects it'll target an element or an object or multiple objects so in this case we have a hamcrest matcher that looks like this so we want to find one that's an instance of map.class so these are all maps this is just a readable representation of the java map and then we want one that this is hamcrest here has an entry where the key is title and the value is search view so this is how espresso works this is how espresso does it as I said before apium delegates the apium espresso driver delegates to espresso and we now have a data matcher selector that can actually use that so here's how the new data matcher selector looks so here it is in kind of a java script like code so find element using hyphen android data matcher and then we take a format that takes the hamcrest matcher and you represent it in json format so you provide the name of the method and the arguments in this case the arguments are title and search view so as you can see side by side these are basically equivalent with each other they do the exact same thing so this json object gets sent over the wire the espresso server receives it then packs it executes the hamcrest matcher so now what we're going to do is I'm going to run the same script that we did before without any scrolling but we're going to use a different selector this time so here's what the script is going to look like this is java script we have the data matcher we find the element by data matcher click on the element so we're running the script again you'll have to pay attention closely because it happens pretty quickly here it clicked on the element so it actually if you happens pretty quickly but if you notice it does one really quick smooth scroll so it scrolls to the element that we want to click on just did one operation instead of doing multiple scrolls and then checking it's just one vast scroll and if you have a look at here the times so with uitemator 2 it was 21 seconds now with espresso we've reduced that number to 13 seconds and on top of on top of that being more performant it's also less flaky so it's a less flaky way to doing it because you can target the views on just any screen resolution and yeah it doesn't matter yeah it doesn't matter like if it's a large screen or a small screen the data matcher will find elements that are in or outside of the viewport so I think I'm going to blast through this last section pretty quickly so I can do a question period how do we write these data matcher selectors well because you won't have access to the source code espresso source has an object called adapters which contains the list of adapters that you can apply the hamcrest matcher to and so you can just cut and paste that from your adapter so you can either dump that into your console using driver.source or you can do an inspection session you can inspect your app in Appium Desktop and you can cut and paste it from there as well so that you can use that as your basis for writing data matchers and so in conclusion for this why should we use data matchers well so number one I talked about performance already it's also less code it's just it's one data matcher it's one selector that scrolls to and finds the element scrolling is done automatically so you don't need to write that it's a lot faster as we saw UI Automator requires multiple scrolls and then locator request so that's multiple round trips this is all done in one HTTP round trip and yeah it's less flaky as well because scrolling there's a risk of if you do the scrolling yourself there's a risk that you could scroll past an element and then never find it or there's a risk that you could be finding an element on a larger screen and it works on a large screen but then it doesn't work on a smaller screen so screen size and resolution don't matter okay so any questions or comments I know it was a five minutes long yeah I'll have a bonus section if there's we get through this quickly though so any questions but we if we have a big amount that's that we are already using UI Automator 2 is it easy actually to switch into the express or there is a like big look? I mean there are some incompatibility so the biggest incompatibility and actually what Rajdeep here is talking about in the next or actually I know you're talking about the back door in the next yeah so you can't actually the biggest incompatibility is with the espresso driver you can't access apps outside of the app under test so if you have system apps that you need to be automated that won't work but otherwise it's mostly compatible thank you yeah but I can't promise it's going to be a seamless cut over hi actually I was asking whether if you have a custom view would that be more like expressive would be effective in that also a custom view as long as it's an instance of an adaptor view yeah it would you be able to use that so it doesn't have to be a view that's native to the Android component library as long as it implements adaptor view then it'll work thanks welcome questions hi my name is so I have two questions to ask one is on the long list let's say we are in the middle of the list and the element like the item which we are interested a look is at the top will espresso be able to find out yes it will so it doesn't matter if it's inside or outside of the view port it'll find the element so the direction doesn't matter it can scroll up and down which wherever the that's right yeah so if you scroll down already it'll scroll up to find the element at the beginning so it doesn't matter where it is okay and on the second thing is like let's say we have a huge list of products and these products get rendered as we scroll so in this case does let's take an example of a HTTP request where we make a request and we get the list of items in the screen let's say we have some four items displayed as we scroll up the next set of elements get rendered so does espresso handle this as well I don't think it would handle that because if you're loading that would mean you're loading the data remotely yes so it's not going to have an adapter until the data actually gets loaded there's a possibility maybe if you have like an infinite scroll that loads some of the adapter data ahead of time you might get you might get pretty like further down the list then like outside of the viewport but if the item hasn't been downloaded yet then yeah you wouldn't be able to access it thank you welcome more questions hello I just want to ask that in our application use so many recycle here is it content as adapter as well is sorry what was it recycle fuse recycle fuse it's android fuse android fields what about a few recycler view oh yeah so that actually is we don't support that yet so these are adapter views yeah recycler views I'm aware of those we don't support that yet in espresso I am aware of those that espresso the framework has a way of handling recycler views but we haven't implemented it yet so but that's possibility okay thank you