 Um, first before I'm going to start, who was here at the awesome talk of Angie? Because she stole my thunder. I only got one slide for you left for the people who were already here. So, yeah, but now you know the advantages you can have with building or kind of like doing all the images compare image comparison. I'm here to tell you how you can create your own module. And I'm also going to tell you I'm going to explain you all the challenges you might face when you want to create your own image compare module, especially for native apps. But first, let's take a look at the slides. Am I working? Where's mine? Yes, today is not about me, but first of all, I need to give you a little bit of context. So, yeah, let's, this is a slide for everybody who was in the previous session. I'm a solution architect at Sauce Labs. You might also know me as the blue guy because of my picture in the open source community. You might have seen me last year at the first Appium conference where I explained the research I did for a customer I was working for. What I should use when I needed to automate a React Native application in the end, yeah, we came up with Appium. Otherwise, I was not invited. And last but not least, you might have seen me during SeleniumConf last year. Here also in Bangalore where I explained all the best practices about how to speed up your test cases with Appium and React Native and kind of like getting the coverage you really wanted to have. And this talk is going to be part three. And I hope in comparison to movies that my sequels get better each time. So, let's start. Start with this story. I'm going to throw away my clicker. I started at Tele2. It was in 2017 and I was in the team with developers with me as an automation geek and we had one manual tester. And even though all my tests were passing, my colleague was still finding some issues in our app and basically they were layout issues. And yeah, we got more and more features also in our app and he was kind of like stressed out. Kind of like the guy you see here doing all the tests and it was really, really stressed out. So I thought is there a way that I can help my colleague and especially with automation because yeah, I was the automation geek in the team. So I pitched the idea for doing visual comparison and got some time to do the research. So I started my research and normally when I do a research I do that based on some requirements. So I put down some requirements on paper which we kind of like discovered together with the product owner and the developer. And one of the things that at least our visual comparison solution should have, it should have native app support. It should also be a cross-platform solution and the reason for that we were using React Native. We were writing something with JavaScript and in the end it was compiled to a native iOS and a native Android app. We also wanted it to integrate with WebDriver.io because that was the tool we were using and already written all our tests in. So it would be very nice, kind of like very useful if we could reuse the code we already had to get to some specific stages also in our app. If we wanted to do the research and we saw some paid tools, yeah, we also wanted to have kind of like a trial first. In our case we said, okay, we wanted it to support blockouts and to support element screenshots less bounding boxes. And you might be wondering why should an image comparison tool need to have that? Well, basically for these three reasons. Initially when you take a screenshot of a native application you will get a full page screenshot or a full view screenshot where you have your status bar, you have your tool bar if you're having an Android application and if you compare that to a web screenshot you will not get that. You will only get your viewport. So you want to focus on the parts, and Angie already explained it before. You want to focus on the parts that really matter. Secondly, if you would look at animations, if you're taking a web application it's quite easy to disable animations. It's harder to do that with native screenshots, with native applications. Last but not least, we also had the dynamic text. For example, with a build numera version, app version, that kind of stuff. On a web application you can easily kind of like put static text in that and then take a screenshot, put it back. Well, cannot do that with a native app. So now that we know where we need to look for it, so let's take a look at the research we did for all the tools. And basically one requirement we had, it needed to support native apps. And then we just did the research. The research was quite hard and the reason for that is that there are not a lot of tools out there to support visual regression for native apps. We found these tools. EpicTools is there. We've got screenshot tests for Android, FizzWays, Nicole, a lot of tools. What we now, what we did after that is kind of like create a list. And based on that list, we looked at our requirements. And one of the first requirements we had, it needed to be a cross-platform solution for a native app. So we could already narrow the list down because the three tools that got across were only or for iOS or only for Android. Then in the end, we also wanted the tool to be maintained that if we found something, it should actively be maintained because if there was an issue and it was an open source tool, for example, we wanted to use, we wanted to be able to add a PR and hopefully get it fixed as soon as possible. But yeah, Nicole's end of life is not maintained anymore. So in the end, we had FizzWays and ApplyTools and FizzWays already draw our attention because it supported WebDriver.io. But in the end, it did not support the blockouts. So in the end, we had ApplyTools and we're like, okay, this is awesome, but we had one challenge. And the challenge was that ApplyTools did not support WebDriver.io for native application at the time we did the research. So we contacted ApplyTools to support and we got the option to use just the JavaScript Appian bindings and kind of like created ourselves. So we're wondering what should we do? And you might also be wondering, what have we done? I think you might already know the answer, but I'm just going to explain it. In the end, we did what every JavaScript developer currently is doing. We build it our own. And the reason for that was that initially we said we wanted to have something that we could reuse our code, our WebDriver code already. And if we would then compare the amount of effort we needed to put into creating a new framework, implementing parallelism, it was kind of like the same time we needed to put in creating a new module. And basically what we also wanted to do from a Teletube perspective, we wanted to create something that was open source that everybody was able to use. Because if we were able to use, it was awesome if more people could use it. So we said, okay, let's do that. And I got some time to do that. So, yeah, you need to start somewhere. So where do you start? I thought, okay, if I'm going to start coding, I'm trying to or maybe I'm going to get a complete mess. I first need to know which requirements I have when I need to create my module. I wanted to create something for WebDriver.io. So I divided logically the module into two parts, kind of like the module itself that only needed to kind of retrieve some information and a comparison engine. And a comparison engine would retrieve the information, do the comparison for me, and give me the results back. Basically, this would kind of like create the module we wanted to have. And then, yeah, we've got some requirements here also. If we would look, and especially for an open source comparison library, it should also be actively maintained. Yeah, also the blockouts should have been supported. But what was also very important for us, because we wanted it to be an open source module, you could easily install that we did not have any environmental dependencies that people who were using Node, for example, in their Jenkins instance or whatsoever. Did not need to install extra dependencies. And last but not least, it would also be nice if we had some comparison options, if we could kind of like change some stuff. So in the end, I did the research. And like I said before, it looks like JavaScript developers like to develop everything for themselves. Because as you can see, we've got a lot of libraries here. They're all pixel by pixel libraries. And if you would then look at the requirements we had with one of the first things we said, okay, it needed to be actively maintained. That means that we can already put across to four libraries. We're not going to select them or do a proof of concept with them. Then we also said we needed to have something that should support blockouts. Again, to narrow the list down. And as said, we did not want it to have extra environmental dependencies. So in the end, we did not look into Node OpenCV, even though it's really awesome. But as said, it should just be a module you should install like you do that with WebDriver.io. So it should just work out of the box. So in the end, we came up with PixelDiv and with Resomble.js. And I've been using PixelDiv in the past. It's a very nice library, but in the end I found out it was not accurate enough when it did the pixel by pixel comparison. So in the end, I ended up with using Resomble.js. In the end, as said, I started using Resomble.js. And if you would then look, we've got WebDriver.io, we've got Appium, and we've got Resomble.js. And you would now wonder why are the images not aligned here? Well, we're missing one important ingredient here, and that's some blue magic. Because yeah, we need to stitch them all together. So if you would then look at the module, again, I'm not coding here yet. But if you would think about a module, what should the module do? First of all, it should be able to create a screenshot. Secondly, it should be able to compare those screenshots with each other. Then we should be able to do a save element. And last but not least, we should also be able to do the compare element. So now let's dive into each method, again, before we start coding, and just check what do we need to do. This one was the hardest part. Save screenshot. Because yeah, what should it do? It should save a screenshot. Let's go to the next one. Compare screens. I think this was kind of like the most difficult part. Because as said, if you're going to create a screenshot, you get everything. So where could it go wrong? Well, basically on these three points, those three points can cause flakiness. And I'll kind of like dive into them and show you how you can dynamically fix that. Because it would be nice if we can do that dynamically. We might know the device we're going to test, but yeah, there are different devices, different screen sizes. So first of all, if we would look at the status bar. For the status bar, if you, for example, compare Android and iOS with each other, they can already differ in height. But if you would then also look at iOS versions or Android versions, they can also differ in height. And what we just wanted to have is a cross-platform solution that our module was able to determine the height and width, the position of the status bar itself, so we could just block that one out during the comparison. So let's use one of my favorite tools, and that's Appium Desktop. Because Appium Desktop is kind of like the way to show how the UI hierarchy looks like, visualized with a screenshot next to it. As you can already see here, this is on an iPhone X. We can determine the position, which is quite easy, 00, but we can determine the height and the width here of the status bar. That's awesome, but if you're going to do a research, you should not rely on one phone, you should test multiple phones. So I'm not going to bore you here with all the phones. Just going to show you a smaller phone from Apple. That's the iPhone SE. It's a smaller screen, and as you can also see here, we have the option here to determine the status bar based on an element and also based, we can get back the width and the height. So for Apple, this time it's very easy to use Apple and to determine at least the status bar. So now let's take a look at Android. And I'm already taking a breath. This is cool. With Android, this is on a Samsung S10. We already got the status bar here. Awesome, we can determine the position. So as said, we need to do a research, we need to check multiple phones. So I checked the Google Pixel. And as you can see here, you cannot find a status bar element. You cannot find a navigation bar element. So what should we do here? Because it's quite hard now to dynamically determine the position at least of the status bar. And I think the following research costed me some precious minutes of my life because I needed to know what the difference was between the XE UI test driver and at least the UI Automator 2 driver. And during my research, I stumbled upon the Settings API. And I think this is the best kept secrets of Appian. Who here knows the Settings API, by the way? Ah, guys who work with Appian. The non-contributors or the core contributors to Appian. Who know it? Well, basically, we can all read it. Basically, it can help you over, kind of like, it specifies the behavior of the Appian server. And if you would dive into the documentation, you would also see that there's a difference, for example, between the previous UI Automator driver, UI Automator 2 and also with the XE UI test driver. Because it does not show you all the elements. It does not give you back all the elements. Kind of like it's smart enough, call it AI, call it machine learning or not. It's only going to give you the elements that is, or relevant, or are on the screen. So how should we solve this now? Should we use the Settings API? Yes or no? Well, I think I needed to use it. So what I did, and I'm going to show you. I'm going to stop the presentation. We're going to dive into a little bit of the Appian server. It's here. What I'm now going to show you is not only for my module. I think it's also something you can use when you're going to do automation. So please keep paying attention. It might help you. First of all, what I wanted to show you is the status bar and the toolbar. But just let's go to a nice form. And in this form we have a button. We're not seeing when we're opening the screen here. Let's refresh Appian desktop. And in our UI hierarchy, as we can see here, we can search elements. We are not finding something that looks like a status bar or something. But also, if we wouldn't look at the buttons, we would only now see a text view here. We do not see the buttons. But for example, I would now refresh my page because I've got the elements in the view now. Keep paying attention. And yes, we've got the elements now here. This is basically what you also might face when you need to automate your native app. You cannot see if your element is there. You need to scroll it into view. Okay. Now let's do some magic. First, get this back. I'm going to hide the elements again. And yeah, they're gone. There's also something new in this screen. They released it with, I think, Appian desktop version 1.12. Does anybody see the new awesome feature here? No? Okay. It's this one. It's the Action tab. And if you click on the Action tab, you've got some options here, for example, to select an action. You can kind of like do something on the device and on the session. Well, we're going to demo the Settings app here. So we're going to do it on the session. And again, we've got some options here. And let's take a look at the settings. And what you've got here, you've got a button called Update Settings. And if you click on this one, you get the possibility to kind of like post adjacent object to the Appian server. And if you would then refresh your page, you would get more information, hopefully. It's a demo, but we would hope to get more information. So let's start doing that. And one of the commands would be allow in visible elements. By default, it's set to false. We're now going to set it to true. And we're going to execute the action. If I would have made an error in my JSON object I was posting, I would have got an error here, but I did it quite good, I think. If we would now go back to the Source tab, we would still not see our element because we need to get the page source again. It needs to be refreshed. And if we're doing that now, I'm now going to make the screen a little bit bigger. Ah, we already now see our status background. And we also see our navigation bar. So now, if we go back to the module, I got the option to dynamically determine the position, the height, and the width, at least at the status bar. But also get back to what I think would also be useful for your automation to see, yes, even though the buttons are not visible, they're now visible in the UI hierarchy. And even with an option, they're displayed false because they're not in the view. So maybe you can also use this when you're doing your automation for Android with UI Automator 2. Okay, having said that, let's go back also to the module. And I'm going back here. Please go back here. Now I've already shown you that you have the possibility to at least determine the status bar. But there's also something extra. We've got the toolbar. The toolbar itself from Android can also cause some flakiness. And I've seen this with the latest Galaxy phones where there's kind of like a gradient in the software navigation bar of Android or Samsung. And it could also cause some flakiness. So I also want that kind of like to be blocked out automatically. And as you can also see here, the status bar itself or the navigation bar itself can also differ in height. And you can also have some devices that do not have a status navigation bar. Take for example, what is it? I think the Samsung and Galaxy S10. It has a hardware navigation bar. So if we would now kind of like use the API again and would now look for example at the Samsung S10. Initially we did not have the navigation bar with the Samsung. Now we have it if we're going to use that one. So this is quite easy now to make the module itself dynamical. But there's more because as I said, we've got the navigation bar. We've got the status bar, but we also got the handle, the home handle bar from iOS. And yeah, this is also something that can cause some flakiness. And I don't know if you ever used an iPhone X series. When you're watching for example a YouTube movie, you're going to kind of like put your phone from portrait to landscape. You would also see that the handle bar kind of like fades away. So the behavior, kind of like the fading could also cause flakiness if we're doing the comparison. So we also need to find something here, a way to block out the handle bar. And I think we have a second challenge here because there's no place here in the UI hierarchy to determine where we can find the handle bar. No element here to determine what the position or the width is. And yeah, I think if you would now think about this finding a solution, I think we again can be grateful that we are automating Apple. Because Apple is quite limited in the amount of devices. So we can now just figure out a different solution for blocking this out. And one of the ways is determine on the width of the screen. We can determine if it's an iPhone X series, even though we've got two different phones, we've got the iPhone X and the Max. But then we can still determine the width and if it's an iPhone X series, yes or no. So now just with the static block, we can just block it out because we can tell is this an iPhone X and if it's a large phone, yes or no. So I think we now covered at least three points that could cause the flakiness. So the image comparison for the screenshot itself, I think it's done. We can kind of like build it. But then we also have the safe element. It would also be nice if we can do that. And if you would now look at the WC3, W3C, I'm always missing that word up. If you're now look at the W3C specs, you would see that we initially have the screenshot, but we can also take an element screenshot, which makes it quite easy for us. We do not get the noise. But not bad. You can get something like this. This is kind of like a login form. But what if, for example, a user of the module is using an old version of Appian that might not support this? Or if you want to do something extra, there's a second option, how you can create an element screenshot. I think it's basically almost the same as what I did with determining at least the position of the status bar. We can use a method also from the W3C specs to get the element rectangles. And when we create a large screenshot, we know the position of the element on that screenshot, and we can use the elements rectangles just to crop it. It's quite easy. So we kind of like also have a fallback here. But when you're going to use that, you can also provide some extra powers. And extra powers is like saying, okay, if I have an element, I want it maybe bigger, provide some margins, or I want to be smaller, provide negative margins. And that will give you, for example, the two results you're seeing here. So you've got some more options here to create larger or smaller element screenshots. And then last but not least, we've got the compare element. What else you could do, yeah, basically the same as what we had with comparing screenshots. So it's not that difficult here. There's only one thing that I found out when I wanted to kind of like block out elements on the element screenshots itself. I've chosen not to do it because it makes the logic more complex. I didn't have time enough to do that. And normally if you have a lot of complex code, it could cause flakiness in the code itself. I just need to have a vacation to fix it or something. But for now, it's not in the module. But it's, yeah, I think you've now seen at least how you can determine it. Let's take a look at the use case because it's nice to know now how to build it. But yeah, it's also nice to see what you can do with it. And I want to get back to Tella 2 and get back to the real use case where we were developing, why we were developing this module. We were at a point where we needed to migrate, we needed to upgrade React Native. We were working with React Native version 0.54 and we needed to migrate to version 0.57. Wow, that's a very large migration. I executed all my functional test cases and again everything was successful. But as you might remember, we had our colleague who was always checking all the layouts. He was kind of like freaking out. I was like, okay, let's start using this module. And I don't know if you already see a difference here. Well, the difference is here. This was just an indentation issue. It was a little font issue. But this simple test case already proved for me that the module was working. So I just released it on all the test cases I had. At least specifically for the image comparison. And we found multiple issues in the end. Issues that could cause flakiness for the customer or unexpected behavior. We had issues, for example, also with a login form that certain elements or certain characters, especially for the password field, were just shown. We needed to fix that. It's kind of like a security thing. We had issues with our radio buttons as well. These were kind of like the things we had with the image comparison module and also we found during the upgrade. So for us, it was really useful to use this within our pipeline. And now you've seen this all. You might think, okay, this is really, really awesome to use. And I must agree. But there's also a disclaimer. And I want to start at least with the cons and then start with the good things of the module. Well, first of all, as you might have also heard this morning, this is just a basic pixel by pixel comparison. So it might can give you some results that might not be an issue. Yes or no. On the other hand, it's open source. It can also have an impact on your project size. And the reason for that is this module does not have a dashboard. It does not use a cloud service. So you need to store the images somewhere. Basically try to store them in your project. There are some limited blockouts, as I said already. And yeah, we do not have the dashboard. And I must say it's really awesome to have a dashboard because it makes your life a little bit easier. But yeah, there are some pros. First of all, it's open source. So you can just use it. You can use it with WebDriver. It's only a few lines of code and you can already use it. The module logic, as you've seen, is not that hard. And if you want to create it your own in Java and Ruby or Python or whatsoever, I think just take a look at it. It might help you creating your own module. And if you're going to do that, please share it with the open source community. But what I think is kind of like the power also of my own module is that it has an easy cross-platform device blockout logic. And I'm just going to show you a simple part of piece of code. And what I mean with this is normally when I need to use something where I have my validations, my checks not in my code itself, but I need to visit something else like a dashboard. And I read my test. I do not see what's happening in the test. And basically here I can now just see what's happening with my test, what I'm blocking out, what I'm not blocking out, what I'm validating, what I'm not validating. So it makes it easier for me at least to see what my test is doing. I like to see that. So I hope you now got a very good impression of what you might need to do if you want to create your own module or what you should not do or why you should not do it. I also showed you that even you might not see your elements in your UI hierarchy with Android. There is an option to kind of like find your elements. And yeah, if you want to know more also about this module, well, the link is there. I already shared the PDFs, the slides, so you can check the links also. Check the logic if you can use it or just check if you want to use it. There's a specific version for version 4 of WebDriver.io and there's also a version 5 of this module. And just to give you a heads up, I just released a new version of this where the blockouts in the previous screen you saw with the Tella2 app, there was a black line. I thought, yeah, you just want to see what's happening behind it, what was there. So I just created a new way of blocking out the elements. So you could now at least see what's blocked out, yes or no. For now, I think we have a few minutes left also for Q&A. So are there any questions? Meaning that you can block out a specific part of that element? Yes. If I understand it correctly, there is an option, for example, with this module to determine or to provide extra margins. So if you find the element, it has kind of like the height and the width, but you can also say just add 50 pixels on top of it or a negative margin. I hope that answers your question. Okay, so my question is related to graphs. It's not only we do most of the time image comparison. So what we do, many times we provide some random data and generate graphs to test it out. So how can I make sure the graph which is generated for me is correct on the X and Y coordinates? Yeah, you're already kind of like if you're doing the image comparison, you're comparing it with the baseline. But if you're baseline, if you're kind of like the data you're constantly putting in it is fluctuating, yeah, with this module, it's quite hard to do the proper image comparison. Resumable JS provides something called mismatch percentage. Yes. So how do you determine what should be the mismatch percentage so that I can pass my test? Because I always see that there is a mismatch percentage even if the screenshots are similar. Yeah, and what I would always advise you is not to accept a mismatch. And the reason for that is if you're going to do that unattended, you're going to run it at night. What will be 1% mismatch? Is that allowed or not allowed? Because you do not know which element gave the mismatch. So I would always advise not to use a threshold with the mismatch percentage. Always just accept zero percentage because otherwise you might, especially if you're going to look at screenshots for iOS, they have a, you have a higher resolution, meaning you've got more pixels. And then 1% of what is it, 300,000 pixels is still a lot, but you do not know where the mismatch is. So you might be accepting things that might not go to production. So I would advise not to use it. Yeah, as said, that's kind of like one of the downsides of pixel by pixel. Yeah, you're doing kind of like all the calculation on your server. So I must say you need to have kind of like a good server where you're running this on and not like the most slowest one because it will slow down in the end your comparison and also your test execution. I do not have real data on if you're going to run things in parallel. You have 100 instances and you're going to do the comparison how it will take. As said, I do not have those numbers. Yeah, you're mentioning memory leaks with it. I don't know what the parallelization amount is that you're using. I'm running this. I'm using Resomal.js in multiple modules. And I also run at least weekly a check build with what is it? I think 12 or 13 concurrent sessions. And I do not have any issues with that. But yeah, it could be that if you're running 20, 30 or 50, yeah, it's consuming a lot of memory, which might lead to memory leaks. That was it. I think one question. And then let's get to lunch. Yeah, so I'm not that familiar with Resomal.js. Okay, but my query is like when you're working on an Android device, as per your flow, you would be taking a screenshot, then taking the screenshot of like saving the elements, right? So one time if you take the screenshots, will that be applicable across any of the Android devices? Or for each of the device execution, the flow would be saving the screenshot elements and go on? It depends on how you set it up. If I understand it correctly, you need to create a screenshot of each device. Each device. Yeah, each device. What I have kind of like your requirement is that I need to have Android 8.1, 7, and 9. But I also need to execute that on five different devices. And that means that you have five times three is 15 screenshots only of one element, because you cannot reuse them, at least not with this module. I don't know how fancy all the rest of the paid tools are. But basically if you want to do proper image comparison. Across all devices, you'll have to take the screenshot. Okay, thank you. Okay, but then I would like to thank you for being here, for listening to me, and hopefully it was useful for you, that you can also use this or the module or the logic, or even the new, not so new, the secret API of Appium for UI Automator 2. Thank you.