 Good morning everyone. My name is Angie Jones. I'm an Automation Architect and Senior Developer Advocate at Aplitool. Also the Director of Test Automation University, which is an online platform with free courses on all things test automation. So if you haven't checked it out, definitely do so. I blog about Test Automation Strategies and Techniques at AngieJones.Tech and you can find me on Twitter at techgirl1908. Today I want to talk with you about automated visual testing. But first we're going to play a little game of who done it. Clearly, somebody in this room murdered Lord Smythe, who at precisely 3.34. Let me start it over. Clearly, somebody in this room murdered Lord Smythe, who at precisely 3.34 this afternoon was brutally bludgeoned to death with a blunt instrument. I want each of you to tell me your whereabouts at precisely the time that this dastardly deed took place. I was polishing the brass in the master bedroom. I was buttering his lordship's scones below stairs, sir. What? I was planting my petunias in the potting shed. Constable, arrest Lady Smythe. But how did you know? Madam, as any horticulturist will tell you, one does not plant petunias until May is out. Take her away. Sorry madam. It's just a matter of observation. The real question is how observant were you? Uh, action. Clearly, somebody in this room murdered Lord Smythe, who at precisely 3.34 this afternoon was brutally bludgeoned to death with a blunt instrument. I want each of you to tell me your whereabouts at precisely the time that this dastardly deed took place. I was polishing the brass in the master bedroom. I was buttering his lordship's scones below stairs, sir. But I was planting my petunias in the potting shed. Constable, arrest Lady Smythe. You probably identify as Tester, and yet you miss the dead man get up and walk away, right? Of course you did, because we all suffer from inattentional blindness. And what this means is we're still focused on a specific thing that we don't see new and unexpected things coming to our viewport, right? This is especially true as you are testing applications. You're so focused on the feature, you may not notice other things that are occurring in the application. True story. A couple of weeks ago, this woman told me about her application. She said that they were doing manual testing on it, and they were running regression all the time. And they didn't even notice that the entire color scheme of their application changed until a customer told them so. Classic case of inattentional blindness. So how do we solve for this? Well, typically, we go ahead and automate the script, right? So we make sure that we're taking the exact same steps every time. We make sure that we are asserting against the same things. We have the same verification points each time. And it seems like it might help with inattentional blindness, but actually what you're doing is coding in the inattentional blindness. Because your script are not able to look for anything other than what you told it to look for. For example, let's say your script is supposed to make sure that some text is there. Well, sure, it does that, but what about if the text is upside down? What if the text is bleeding off of the page? What if the text is hidden behind some other element? You would never know this because your scripts wouldn't tell you. You wouldn't find this out until some customer complained about it. So they're actually worse than we are. Our scripts are worse than we are as testers because we might notice this. We have a better chance of noticing that sort of thing than our scripts would. Now you might think, hmm, those seem like edge cases, seem like nice to have sort of things. But let me tell you, I see visual bugs in all sorts of applications. Such as this one, this is Cineworld. Cineworld is the second largest cinema chain in the world. They had a visual bug on the page where you store your credit card. This was the bug. So you see the radio buttons, you see the labels, they're not aligned. You don't know which one of these labels corresponds to which radio button. Maybe it doesn't seem like a very high severity bug. But when you think about this, I don't know about you. I wouldn't store my credit card on this page because I think that it hasn't been tested. That's exactly what Kim did. And she got so frustrated she tweeted about it. So now they're embarrassed in front of the whole world about this bug. I have a theory, though. I think they did test it. In fact, I think they automated the test for this. The automated script would never catch this. The labels are there. The radio buttons are there. This will continue to work. The stuff is in the DOM. It can certainly click the button. But from a user perspective, our script wouldn't catch it. This is OpenTable. Did you all have OpenTable? You used it to make a restaurant reservation. So I went to North Carolina where I used to live. I visited. And I wanted to have dinner with a lot of my friends. It was going to be seven of us. And North Carolina is famous for its barbecue. So I went and found my favorite barbecue restaurant to make a reservation. And I picked the date. I picked the time. I went ahead and clicked the button for the time. And a modal popped up. Now this modal, it took me a little while to figure out what am I supposed to do here because there was nothing in the middle, right? And as I kind of scanned the page, I noticed those buttons in the upper right corner, the two select buttons. I didn't know if there was an error to have two select buttons. I noticed that they were not aligned. And more importantly, what am I selecting? There's nothing that tells me. So I did what any of you would do. I opened Chrome Dev Tools and I started digging around to figure out what are these buttons. And once I did that, I saw that the labels actually were on the page. I just had noticed them because they were all the way up in the upper left corner. And so I got to thinking, this is a pretty bad bug. I have a reservation of seven people. We're going to spend money. We're going to buy drinks. We're going to have a good time. But most people wouldn't know to use Chrome Dev Tools to be able to still make this reservation. And they probably would have exited. Maybe they would have tried the mobile app, but guess what? The bug was there, too. So that would have been a reservation loss. How many reservations have these people lost because of this? Now these aren't just one-offs. This happens with your favorite tech companies as well. It happens with Amazon. It happens with Facebook. It happens with Twitter. It even happens with Google. Great news is that you can actually automate these visual things. There are tools available to allow you to add this to your automated script to remove that inattentional blindness. The way that visual validation works takes a screenshot of your application and it's in the state that you like it to be in. And then during regression testing, it takes another screenshot and compares those two to make sure everything looks like you want it to look. Now visual testing is not something that's new. There are older techniques that are still being used as well that compare the pixels of the images. Now this is kind of giving visual testing a bad name. In fact, someone stopped me as I was coming on to the stage to tell me how flaky they heard visual testing was. This is true. If you're using the pixel to pixel, right? Let me show you. This first picture is the snapshot when we were happy with this test. The second picture is a regression test. The regression test fails. Who can see why? The button is bolded, right? Well, that button isn't just bolded by default. It happens to be that way because the mouse is actually hovering over that button during the automated script. So this test fails. You as the engineer, you go and look at the failure and you realize this is why it fails. You're annoyed at this point, right? Yes. You don't want to go back to your script and try to figure out how do I write code to make sure my mouse is not hovering over this button. Here's another one of the tests. This one failed. Who can see why? Cursor. Exactly. So after the word bad, the cursor is solid. Guess what? Cursor is blink. So if it took the first picture when the cursor was solid and the second picture when it's not, then the pixel to pixel would fail this. So this became really, really annoying to people and they thought, okay, visual testing may not be the way. Now, there is a newer technique created by Apple Tools with this API that's called i. Now, the i's API only highlights what matters. It uses machine learning to look at the images just as we would, just as the human i would, just as our brain would and only pick up the things that we would care about. Here's a typical foster difference game. You all don't get to play because you didn't do well at the first game. You let the dead man walk out the door. So I'll save you the trouble. I ran this through pixel to pixel and everything was highlighted because it's so sensitive. Even like small adjustments with white spaces. So this is really difficult to tell what actually is different. And then I ran it with i's and you can see how it picks up the things that we would have picked up if we actually played this game. So enough of me talking. I actually want to do some scenarios. I'm going to show you how you can add i's to your Appium framework. So the first scenario that we're going to look at, we're going to, oh, let me show you the app first. So this is the app. It's a simple to do app. And you can click a button and add a task. Let's just say task. And you can select any of these colors here. So you can start grouping your tasks by different categories. So maybe purple is for your personal task, blue for your work task, and so on. And then we just click this to add the task. And we see the little color right there. So this first scenario is asking us to add multiple tasks of different categories. So let's do that now. So in here, I have already set up my Android driver and everything. And I have a page object class you call this task screen that is mimicking that task. So we're going to add our first test. And we'll call this add task in different categories. All right. So let's go ahead and add some tasks. Let's say task one. I need to maybe wash the car. Task two. Y'all give me some tasks. Grocery shopping. What else? What else? What else? Do what? Find who? The murderer. Yes. Okay. And one more. Me time. Oh, I loved it. Okay. So we have our fourth task. Let's go ahead and add the task. So we have this add item in our page object class. And we can just go ahead and say add the task and we can pick one of the colors. So we'll add yellow. And we'll do this for each one of those. So let's change this. Pick some more colors. Blue, purple, and pink. All right. So we've added them. Now we want to verify, right? So we add these four tasks. We want to verify that the task is actually there. Let me get the task first. So I can do a list. We'll call it task. And we can say, give us the tasks that are on the screen. And now I want to make sure each one of those tasks are in the list. So let's do some assertions here. We'll assert true that the list contains each one of the tasks. And if this was real life, I'll put a little message there. Don't think I'm trying to skip that for time's sake. So we'll make sure that each one of the tasks is there. What if we added a task and it showed up multiple times, right? So let's make sure that there are only these four tasks there. So let's do an assert equal. And make sure that the size is four. And then of course, since we added these in different colors, we want to make sure that those colors are actually right. Well, that's where we get into a little bit of a problem. So this is a native app. So I can't use the call to get the CSS value unsupported for native apps. When looking in the inspector, there was nothing there that identified those colors as the colors they were. Everything was just called task color. So there was nothing that said that this is purple or this is yellow. Fortunately, visual testing can help us with this, right? So I'm going to add visual testing to this test. I won't add it right here at this level. I'll add it in the class that we're inheriting from so that any test can use this. So let's go ahead and set this up. I'm going to make a new object called eyes. And this is the eyes API. And after we go ahead and get everything up and running, I am going to make a call to a new method. Let's call it initialize. Get it? And we'll create this new method here. Initialize. And what we need to do is go ahead and create a new instance of eyes. And when you create your free applications account, you get an API key. So you need to put that in your test. So I can say set the API key. And I already have this save as a environment variable. So I'm just going to give it the name. I called it aptitude API key, right? And that'll set everything up. Now for the actual check. I'm going to make another method. Let's call this one validate screen. And what I do is say open your eyes. So I call this eyes.open. Give it the driver. Give it the name of the application. And then give it the test name. Now, since we're putting this at a higher level of abstraction, I don't know what the test name is going to be. So I'm just going to use this as a trick to say, give me whoever called me the name of that test. After we open the eyes, we then make a call to check windows. And then that's it. We say close your eyes. Voila. And we can use this in our test. And this will verify the ass screen. The colors on there, right? Now here's the thing. It's not just verifying the colors on there. It's verifying everything on here. So it's verifying that this test name is correct. It's verifying that there's only four of these tags. So I no longer even need these assertions. So now I'm adding additional assertions and yet I'm deleting codes. In fact, now that I don't have those assertions, I don't even need these variables because I only use these once now so I can get rid of more codes. And not just the code. In my test, I can get rid of things in the page object model class as well. So now I don't have to worry with like getting IDs and things on all of those elements because that's no longer relevant. So my test becomes a lot shorter now. All right. Let's run this guy. Oh, and we all know that mobile test runs low. So I brought us some music while we wait for it. So there's I dash four. And we can see the status is new. If we open this up, we see that it took a screenshot. So this is the very first time. Now what it's going to do is save this as the baseline. I didn't even need to come here to this page. I only came to show you, but this will be saved as the baseline image. So every time I run this test again, it will compare it against this image. Now there are some pros and there are some cons to this. The pros are that we are verifying everything on this screen. So like I said, we got to get rid of some of the assertions. We don't have to think about a lot of what am I checking? We've removed the inattentional blindness. A con is that it's validating the entire screen, right? So not only is it verifying these tabs, it's verifying the title. That might be a plus. It's verifying this bottom footer. It's verifying the plus sign. It's verifying this header. That's where we're going to get ourselves in trouble, right? What's going to change on here? The time is going to change. The battery might change. The Wi-Fi. So if I were to run this again, it will fail, all right? You want to see it or you take my word. So I'm going to run it again. Just so I show you a failed one. I want you to really understand what you're getting yourself into. So we'll run it again. I know that this test has some differences and it gives us a link. Here's the link. We can see what the differences are. Notice this is marked as unresolved, not failed. Which I like. I'll tell you why. This is coming here as a human being. This thing is using AI. We often ask ourselves, is AI going to take my job? And then people say, no, no. AI will assist you. This is a case. So this is AI assisting you saying, hey, I found some differences. I'll come in here and say, oh, yes. What difference did you find? And it says the time. And I say no, silly. I don't care about the time, right? So we can solve this in a couple of ways. We could scope our test down. So we don't have to use visual validation on the entire screen. We can scope it to a certain region or a certain element. So if I got the ID for the container past, I can say, just scope it down right there. And we could do that in our code. Right here, instead of saying check window, we could make a call to this check element, for example. And pass it to selector. That's one way. Another way, there's these different annotations. So I can use this was called the ignore region. And I can say, I want this entire region here. Ignore. And so every time another test is ran, it will not pay attention to the status bar. And then I can say, okay, AI, I made the decision. This is now what I want to be my base. So I save this as my baseline. All right. So that was the first scenario. Let's do another one. We're going to add multiple tasks on them. So let's look at our app to see what that even looked like. If I want to complete one, say I went grocery shopping, I just click it. And this turns to gray, and this is crossed out. Again, this is styling issues. So I can't even get to those and verify them with like my testing G or whatever on this native app. So good thing we already have visual testing plugged in. It should be very straightforward for us to add it. So let's go back to our class and we'll create a new method here. We'll call this test task complete. Let's go ahead and make a couple of tasks. Only probably need like two. Let's say we're going to what are we doing? Let's do our taxes, maybe. And what? Pick up dinner. And then I'm going to go ahead and add these tasks. And we can just call our screen and say add item. And that'll add both of the tasks. And then I need to cross one off, right? So I can say screen.completeItem and pull one out. So let's do the first one and then I can do a check. So we'll validate the screen and let's... before we run this though. This is a new test. This is going to be a new baseline. We already know we've been down this road before. What's going to happen? The time is going to change on the next one. I'm going to have to go in. I'm going to have to ignore region on. So I don't want to do this for every single test that I write. Fortunately I can do this programatically. So we go to this validate screen we have set up. Instead of calling check window we can do what's called a fluent check. So just say check and then we say, yeah I do want to check the window. However, I want to ignore a certain region. And then I can just give it the locator for that status. And since I have this at a higher level this will automatically do it for all of the tests. So I already had this locator defined. Now it will automatically ignore this status form. Now I can run the test. So you don't have to worry about storing them yourself but yet we see that it actually ignored that region for us. So perfect. Alright, what other scenarios are we going to tackle? Some long tests. Make sure they don't go all off the page and stuff. And it says we can do this for any of the categories. So we'll pick a few just to make sure it works everywhere. So make a new test and we'll say this one is called long. We'll do this list again. And let's add some really long ones this time. Let's say I need to buy a flight from UK to India then buy a flight from India to U.S. Alright. And with a take my imaginary kids to soccer practice don't forget to buy next and check out Appium 2020 submit abstract for a great talk. Alright. So there's our test and what I want to do is I want to pick any random color like color doesn't matter but I want to put them in different categories to make sure that it at least works in all of the categories. So let's just go ahead and get all of the colors. I can do give me all the values of the colors and then I'm going to do for each and then I'm going to say go ahead and call screen dot add item and give it one of these colors. Let's just pick a random one. Alright. So this will add paths in some random colors. Now we're doing this random colors thing. We don't know what color is going to be and this is going to change every time we run this test. So fortunately eyes is really flexible to set what's called the match level. Now the match level we've been using by default is strict. And so that's comparing everything. We can change the match level. Basically we are saying change the way you're comparing this. I want you to just compare the content and what this does is look at the text and everything basically go colorblind ignore the colors of it. So I don't care how many categories there in, I just want to make sure that I'm exercising different categories and making sure that none of them are broken for long tests. So after I do that I do my validation. So let's go ahead and run this one. Run it again so we have something to compare it to. Keep the party going. Y'all think the colors are different yet it's still passing. That's because we use that content match level. If we wanted to see what this would look like in another one we could still do a preview of any of these other ones. So the exact is the pixel to pixel, the strict is the default one. So if I did the strict one I would see that it highlighted these colors here as being different. And so you see that that will work. Now what are some pros and cons here? Pro of course is that it's pretty flexible. So in a case where we don't care about that one, we could utilize this and still be pretty flexible in our tests. A con is if any of these colors were wrong, you would miss that. So if it's adding it in a different category, this would not help you. This would fail. So this is where we come in as the engineers. This AI stuff can be as flexible or as powerful as you need it to be. You still have to use your brain. You have to control it. You have to figure out what's the best use cases for it and how to utilize it. Don't let it just take over. So we have one more scenario to do then. We're done. This one says verify dynamic lists with random tasks and random colors. So now we have colors, we have text everywhere as in no prescribed order. You may think you can't use visual testing for this. So let's go and make a new test. We'll call this one this dynamic test. Let's just go ahead and I want to shuffle these actually. So I'm not going to use that one because it's immutable. So I need to do this old school way. And I'm going to add some tasks here. So let's add maybe like three tasks. So I need to do laundry. I need to finish slides for some conference I'm sure. And I need to probably unpack. We have these. What I want to do here is I want to shuffle this list. So I'm going to say shuffle the task so that every time this runs this is in a different order. And then I'm going to just copy this random color stuff so that we get that too. And then instead of using the content level I can use this layout. And what layout does is just make sure the layout of the screen is as you want it to be. So we use this at Twitter when I used to work there. We had the tweet. We don't know what the tweets are going to say. I don't really care about the text. I just want to make sure structurally they're not overlapping and that they look fine. There's people who use this for news companies. You don't know what the latest news is going to be. But you just want to make sure everything looks right. So we can use that layout one for that. And we're going to go ahead and run this one. If I was verifying some things on the back end and I just needed to make sure the front end looked okay this might be good. Or for smoke test or something. Automated visual testing. Any questions? Thank you all so much.