 Let me start my timer. All right, as I mentioned, thank you everyone for joining. My name is Michy Winig, and today we'll talk about visual regressions. I've mentioned mobile as one of the platforms, but overall today we will actually talk about multiple platforms. And to clarify, first I'd like to make clear what goals we're trying to achieve today. So maybe you will go to another talk if you figure these goals are not for you. So first and foremost, we will talk of course about visual testing as a whole. And what it is, how to do it, why we even need to care. Then I'll talk about visual, obviously the test pyramid as a way to structure your tests, but with a new perspective applying the visual testing practices to the well-known pyramid that you've probably used before. And last but not least, I'm hoping for you guys to either adopt the concept from today's talk or bring some tool to your workplace so you can actually improve your day-to-day testing workflow. So as I've mentioned, you might be asking yourself, again, what platform I will be talking about. From the title of today's talk, it's pretty clear I'll mention mobile. But nowadays mobile is not just iOS and Android. It's obviously things like responsive applications which would adjust the layout depending on what device you use website. And also hybrid apps if you're using something like React Native and similar frameworks. So we'll talk about all of those. But mobile as a whole, especially mobile native, is pretty broad and I'd like to clarify that throughout today I'll actually be focusing on one particular OS, mobile native OS, and it'll be Android. And the reason why is just by showing this data, this is a market share for the past five years and the growth of that market share. You can see the Android is all the way on the top and iOS is still kind of down. You know, comparing in actual data, it's 80% or so Android market share in just 17 or so for iOS. But as I've mentioned, we're not only gonna talk about native, we'll talk about the web apps as well. So I'd like to ask you a question. Do you guys actually still do manual testing? Just raise a couple hands if you're doing it. That's great, that's awesome. And obviously do you guys write automation? I hope I see some nodding, that's great. Hands to you guys in the conference. Serenium Conf. But if you think about your tests and how you write the automation in the first place, from the top level, what would lead your tests? What would you keep in mind? How would you organize your QA effort? And you know, to think about it, obviously there is a small hint I can give you. It's something that has four sides. And obviously that's the testing quadrants. Or it's also called agile testing quadrants. To be very clear today, obviously it's not me who came up with those. It's from the Agile Testing Book. So today we'll really focus on the first two quadrants. I'll talk about component and unit tests. And I'll try to focus on approval tests, scenario tests, you know, end-to-end functional testing. But you know, that's pretty brief. They're just the top level. If you go to more details when it comes to testing, there will be something else that will guide your testing practices. And that thing has like only three sides. And obviously it's a test pyramid that we all know and love. So the first level, obviously the unit test, as you can see from how wide that level is, you should supposed to have the highest number of those. It's supposed to be low cost, high speed, but low fidelity test. Meaning that on their own, they're not giving you enough confidence in the product. But they're nice and easy to write, pass to run. Integration test that will involve multiple units and end-to-end test. Obviously it's a single task or end-to-end flow that you're trying to test and you should have the fewest number of those tests. Because obviously they are the highest cost, lowest speed, but the highest fidelity as well. So the highest confidence in your product is given by that level. So in terms of the, you know, theory that's so fun, but I like to be more practical and I would say let's actually test something. And the good question to ask, especially now if you go outside, is anyone thirsty? I'm generally thirsty, so especially here in India where it's quite hot outside. And so first thing I do, I find a drinking glass. And when I have a drinking glass, I obviously examine, does it fit in my arm? Is it made of inner glass? Is it slipping from my arm? And so this is how I will start. Then I will pour some water in the glass and I will have a glass full of water, hopefully. And then obviously I'll drink it. So here we have kind of a flow and if you were to apply the test pyramid to it, you would see that the first test that I've done to the glass examining whether it fits in my arm, it's made of glass, that's a unit test level. When you add water to it and you can actually see that those two units, water and the glass are combined and function together, that's an integration test. And last but not least, if you go all the way from having empty glass, full glass and empty again, that will be end to end test. So ultimately again, we apply the test pyramid to this scenario. Simple case, but you might be asking yourself, are we done? And I would say unfortunately not. We'll be very short talk if we were. And imagine you have a glass like this. It's a bit cracked, but you still can pour water into it. You still have water. So functionally, that glass is still operational. You were able to pour water in that glass. Although visually it's not perfect, it has a crack in it. And last but not least, you're still capable of drinking it. I mean, you're probably gonna get some glass in it, but ultimately you were able to complete the task at hand functionally. It's still performing its duties of helping you to actually drink the water. So you still have that unit test, integration test and the end to end test. That's all great. Again, test pyramid in action. But we're obviously missing something. And if you were to compare the two, you can see that even though they're functionally the same, you're still capable of drinking water with either of those glasses. They are different visually. And the problem here is if you were to extrapolate a bit more, that visual crack that you see on the glass actually has underlying problems where some water is leaking out. The same way some visual regressions that you might have might be a sign that you have something going really wrong or something that will eventually break like this crack glass would eventually break. And so you can also think of it less abstract terms and more of a user experience. If in a full glass, in a fully visually perfect application you have glass full of user experience, in a cracked one you have very limited number and amount of user experience. So it's leaking out. This is what the visual testing is trying to solve. So that functionally working glasses are there but you have to test something beyond that to be able to detect those visual regressions. And so here we're basically trying to tackle the content, the visual content that we have on the page whether it's a color, it's a font that you might have. Visual animation is a common problem whenever you test, have a visual testing in place. The page layout, I believe the talk before mine was talking about layouts quite a bit but that's essential for things like responsive applications which especially when you're changing it from desktop to tablet to a mobile phone in between those state changes often you have visual regressions as well. So this is what visual testing is trying to accomplish to make sure you have those gaps covered. So I'd like to make a statement that obviously functional manual testing is difficult. You can deny that. But what I would also like to say is that visual manual testing is nearly impossible. I tend to say it's impossible but I don't like talking extremes. So let's put it nearly impossible. And the way I'd like to confirm that and clarify that is by using a little bit of science. There are many, many papers out there written on this topic. Obviously I have to read this but if you're interested I can give you some links but in short, there are many terms about this called partial blindness, change blindness, special recognition, there is like a lot of those in a sense that humans just are very bad in detecting changes in the images especially tiny one. If you remember as a kid you probably had a chance to play with those fine 10 differences in the two images. Those were simple but what if we're talking about single pixel offset that was caused by some CSS changes by another developer? You cannot possibly detect those. And so obviously the manual testing is out of the question. So the conclusion is that visual testing requires automation. So with that, we now ready to talk about test pyramid and where would the visual testing actually fit in our test pyramid that we've showed? I would say that it actually fits on every single level. It doesn't just fit somewhere in between unit integration and end-to-end tests, it actually has its own visual test pyramid. So if you were to present it in a way it will be something like that at least from my perspective. So in the bottom layer you would have the CSS DOM tests, then you would go up, it will be visual component tests and last but not least you would have visual page tests. And I'd like to approach it today as if it was a journey as if it was a hill that we're trying to climb. And this is what we'll do today. We'll start with talking about the CSS and DOM tests which will, they will work at the same way the regular unit tests function, low cost, high speed, low fidelity. Then we will move on to the visual component tests and we'll wrap it up with the page tests which the same as end-to-end tests would fit this description of high fidelity, low speed, and the high cost. So without further ado, I'd like to start the climb. And obviously you have to start the climb with the first step and the first step will be the CSS and DOM tests. Okay, so with those, before we go any further, I'd like to clarify that for today, I've created an app which is fairly, fairly simple but I've done it in a, as a responsive application using React. I made it as a hybrid app using React Native. I wrote it in an Android, native Android using Java and iOS using Swift. So we will, throughout today, as I've mentioned with target platforms, we'll try to exercise visual testing with all of those. All right, so, but for this first, you know, the DOM and CSS tests will only talk about the responsive apps and hybrid apps. All right, so demo time, the snapshot testing. So when it comes to the snapshot testing, let me go and open up, where is my this thingy? Great, so for the snapshot tests which I really, really like, we will use something called Jest. I'm most likely pronouncing correctly, but basically something that was built by Facebook and something I really enjoy as a developer. The whole point in the special, that middle statement they make, it's an instant feedback. The unit test to me, it's all about giving developer feedback really quickly so they actually run those tests. If tests take 10 minutes to run, no developer in the same mind would ever run those. So this is why Jest is so great. So if I were to demonstrate to you Jest, let me see which one would be just, I think this is the one, this is the one, great. So let me make sure I open presentation mode so you can actually see it, then you do this, then you do that, great. All right, I think you're seeing, great. So we have a simple application. Actually, let me show you the application first. Yes, so I live in Canada and so I basically usually say hello in Canada and I have a flag in the middle, I have a text on top and I have a button to switch countries. So because Canada is so close to America, I often go to the States. So if I switch the country, I can say hello America as well, very easily. So this is the app basically. You have three components. You have text, image and the button. Doesn't get simpler than that. So in terms of the application itself, if I were to look at it, you would have a couple of components in place and you would have something called the change country component, which is actually a button. So when it comes to the button, basically in React, we don't have to look at it too closely, but in a sense, you can see it's very similar to the regular DOM you would define. You have a button, you would define a class, you would define a style. Obviously in line style is usually bad, but in React this is how we do it. So in terms of CSS, you would define it this way in a sense, you would specify color, you would say how many, whether it's supposed to have borders and so on and so forth. So this doesn't really matter to us. What you have to see is that I was able to define the class name, style, actions and the label. So just is very lightweight and the only thing I have to do is basically run NPM test. If I were to point my NPM to the just library for testing, it will basically generate, sorry, before I even do that, I have to write the test first. And the way you write just tests for snapshots, it's basically you obviously define the test, you name it some way. So for example, I say country change button. Then I initialize the component. I say, you know, basically have the arrange for me creation of the change country button. Then I would basically format the result and last but not least, it would try to save it and match it with the existing snapshot. So this basically, this test is almost the same for any test you would write for snapshots. But what you have to remember is that it basically creates those snapshots and compares them and ultimately you have something like this created. Snapshot folder with a snapshot saved right here. And so basically what it does, it, the thing with these responsive apps and new responsive frameworks, is they create something called virtual DOM. If you've written any Selenium tests, you know that it's one of the problems with those virtual locators that you have to navigate and catch. And virtual DOM is generally kind of pain to test. So in this scenario, what it does basically, it renders that individual component for you, saves that DOM as the, you know, basically the test file, and then compares it up to the next run. And so ultimately you can see it has a button, it has a class name, it even has a CSS. So if I were to change the component and quickly run this test, I would see that the snapshot test has failed. So that basically would be it. So for example, if I were to quickly change, I don't know how to change these two something different like zero, zero, zero, zero. Okay, like this, all the way. Yes, and then I were to run the MPM test. It would run it for me. What I like about these new frameworks is that it actually gives you the output and kind of explanation where it failed and what it failed exactly. So it tells me it failed in the match of snapshot, obviously. And it tells me it used to have this color, now it has this color. The font size also changed as well, which I actually changed beforehand. So, you know, very quick DOMs, it checks the DOM for you and CSS. And again, it matches our description of low cost, high speed tests. So that basically snapshots test in a nutshell. Okay, so if I were to move on, obviously you might be asking, is it enough? And obviously no, since it's just the first step in our journey, and let me actually show you why. And the reason why is because if imagine I had that responsive app in the React. As I've mentioned before, let me make open up developer console. Okay, so if for example, if I were to change it from this desktop view to let's say an iPhone. You can see that the way it structures this web app is that it will get rid of the flag and only has a button. So it only changes the label for you. But for example, if I were to have something in between, if I were not to catch all the ranges for all the devices. And then the point where it changes the state like the platform of your app, it might be missing something in this scenario for this device, I didn't take into the account there is the resolution as a result. I'm basically not showing the flag anymore but I'll still show the button and show the label. So those in between stages to test them, this snapshot test would not be enough. Because again, usually you can't see it but it still takes space. And so to tackle those, to tackle those. You have to go beyond that and you have to go to the next level, to visual component tests. But before we talk about visual component testing, I'd like to talk about the component development in the first place, which I kind of touched basis on. The component driven testing or development, my apologies, has plenty of names. It's also called the pattern libraries. People also use it. It basically old thing with a new name, nothing more. And in the sense what it's done, it, you develop the components which are usable at its core. They are all about the encapsulation of logic. What I like about components that they often exist, I mean ideally they should exist in isolation. They shouldn't be sharing anything with outside world. They are composable because if you're trying to build an app consisting of multiple components, you should be able to combine those components. And last but not least, they are often function as semantic library for your app. You know, if it's, if you have a button component, it's most like a code button component. If you have an image, might be an image component as well. To give you an illustration, looking at our app, which consists of only three components. The one on the top, imagine we're writing an Android code here. The one on the top would be a text view component. If you've ever done Android development, basically text view that Java, which behaves the same way everywhere and it's reusable, but I can change and define different label. Image view is a different component, reusable, and the same time composable because I can combine it with a text view and just change the image behind it. And last but not least, it's a button component. Again, reusable, but with a different action and the label behind it. And all together, it creates me an app that I use and test today. So something I like to use with any component testing and development, it's called a storybook. It behaves as a library for you. The testing ground as well. It's a style guide at the same time because you can define different states and have a nice documentation right there. And also you can even treat it as IDE. And to give you just a quick glance and how to use it, how it's actually created, I like to always refer to the Airbnb storybook. Storybook is an open source so you can easily spin it on your own and write it for your local app, but I really like how Airbnb done it. So for example, they think it's big enough. So they have a component called date range picker. And basically it looks like this. So when you open it, it shows you it's in English, you can operate and it's fully functional. Then you can also change it right here because you can define different states your component can take and write it in Persian or non-English locale. At the same time, it's still functional and operational right there. So nowadays there are many frameworks out there that test the storybooks, Upritools is one of the enterprise. I believe there is a chromatic and other solution out there. So people really like testing these components in a storybook and I personally use it a lot for component development and testing. So, but going back to our application and testing and actually looking into visual component tests, here we will talk about all four types of apps that I've developed. I will try to test all of those and apply the test frameworks to all four. So, first of all we'll start with the screenshot testing and all screenshot testing frameworks follow the same pattern. They have a view loader, something that loads your application or the component you're trying to test. They all have an action driver that performs click, swipe operations and something that will compare the screenshot that it creates and also give you the threshold for failures, reporting and so on. So this will be comparison driver. So if you were to look at the oldest tool out there that's mostly deprecated but people still know and use at least in the past, Phantom CSS. So as the view loader, there will be Phantom.js which is currently deprecated. So there will be Phantom.js that they will use to load the app. They would have a Casper.js which would perform operations on your website. And last but not least they have something called resemble.js which would compare your three shots and depending on the threshold you set will give you a failure if any screenshots actually fail to be correct. So if you were to look at it in action in this old tool, for that I like to use that simple example. Many people used in the past. I just used the calculator in HTML5 that basically you can do one plus one equals two. Doesn't really matter. What's important is if you were to write a test for it, calculator, yes, that's the guy. All right, so if you were to actually write a test for it you would do something like that. So first of all, obviously you define the test hand then you would initialize your Phantom CSS which is not important for us at the moment. Then similarly how you would do it with WebDriver you locate the website you're trying to test. You set the resolution. Here you can actually again test your responsive apps if you wanted to. Then this is not very essential to us. What it does it's basically looking for a body in this scenario and trying to take a screenshot and if it fails to take a screenshot it will log the exception. So here ultimately I have two use cases. I'm basically testing that the arrangements of these buttons is correct and also that operation one plus one is working. So if I were to run the test really quickly and I would do my favorite MPM test you can see that currently my screenshots folder is empty. So if I were to just run the test fortunately again it's fairly quick and it tells you that look, okay is it your first time running this test because you don't have screenshots yet. So now you would actually have screenshots created in your folder. And as I mentioned before, first test was to check the body and the second is that one plus one operations actually working. So that's easy, that's easy. But imagine instead of having the normal calculator you had it completely rearranged. So technically speaking it's still the same operations. You can still do one plus one but usually all buttons are messed up in different locations. So functionally it's correct, visually it is incorrect. So if I were to change the calculator in a test and I were to do rearranged and rerun my test really quickly it would basically complain. It would complain and it also tell me that there was a mismatch of N% so in this case it's 429% difference that found and would create even more screenshots in this case. If you were to look at the screenshots it would give me the difference. This purple color, if you ever work with a visual testing tool people like this purple color it doesn't have to be that. You can change it to anything but that's basically the common one people use. But in short you can see that it tells you one test fails because it was rearranged but the functional one, the one plus one is still working. So again that visual aspect is what you're trying to test and what it shows you by simply testing that single component of the body of your calculator using just Phantom CSS. The simplest one out there. Okay so if I were to go forward, yes. Next one I like the most actually, Backstop.js. What it basically does as the view loader it uses Headless Chrome. As the action operator it uses Pepiteer which basically has an API to interact with Headless Chrome and last but not least Backstop.js itself that performs the screenshot management, comparison and all those complex logic. If I were to demo it, what I like the most about Backstop.js is that it's really developer friendly because it's configuration driven. What I mean by that if I were to show it to you. Okay great, if I were to show it to you right here basically it does, there are just few operations you have to do. If you were to run Backstop, actually see what, yeah. If you were to run Backstop in it it will create a configuration file for you which I already adjusted a little bit. Basically how it looks like you, yes this one, great. So you have a Backstop test so you name it some way then you have to specify what plattons you're running it for. You have a tablet, you have a phone, you specify width and height. Then you can specify scenarios at hand. For example in this use case very simple I'm going to my app and I just wanna take a snapshot of the whole app. If I wanted to be very component driven I would say just take a snapshot of the American flag only. And if I were to run it, just run Backstop test. Again it's fairly quickly, yes allow. Of course allow and it will create. So, okay, two pass, two fail, I don't know what failed. All right, switch country. Anyways, okay let me make it larger a bit, great. So, this is better, this is good. So first time I ran it, it all passed. So you can see that when it just takes a whole app it's just the same as it used to be. So American flag, hello America. And for the last where it uses a phone and you're trying to find American flag it doesn't find anything of course because as I've shown you before we don't show the flag when you have a mobile phone itself. If I were for example to change something. If I were to change it for instance, let me change it really quickly. If I were to swap, yes country for example if instead of this three I had this three, okay. And I were to run my test, of course allow it to create. You can see that now it tells me that the reference that the test you had before it was American flag. Now it's Canadian and there is a huge difference while still passing the last test because the flag is not showing for mobile. The gist of all of this is that you can see it's really the configuration driven visual testing that you can really easily deploy and quickly spin off especially if you have a small app. I would say this framework is perfect for just a tiny app that you might be developing for the scale. Obviously you might need to look at the enterprise solutions. But moving forward to more of a native screenshot tests. You still have the same pattern. You have a view loader, you have an action driver and you have a comparison driver. So if you talk about native for Android it will be something called screenshot tests for Android. Very descriptive name. I believe Facebook team created it. For iOS you would have something called iOS snapshot test case. Also created I believe originally by Facebook but currently maintained by Uber. As I've mentioned when it comes to native we will be mostly talking about Android. And so for the screenshot tests for Android for the native if I were to demonstrate that it would be all right let me do this view, enter then destruction remote create. So again you have your app. Let me see my application. I don't like that it always screws up this thing. Okay, anyways. Anyways, so you have your app. The same, this is basically similar app. Let me show you actually on the, I don't know why the resolution here is messed up but on iPhone basically looks like this. So similar way you just change the flags, change the countries. But ultimately if you were to write the snapshot test for native to be in the same location as your class your app itself. So then you basically you set up the context which is fairly simple. It's always the same. You specify what layout you want to work with. And last but not least you just take a snapshot and you record it. So that's basically if you were to write it for the whole app, for the whole application in hand. If you were to test just the certain component the American flag for example you would do it the same way all up to specific view that you're trying to tackle. In this scenario I say just focus on the American flag. You can tell the reason why these tests have to be located in the same place as my app is that I can navigate to the individual views by references, these are class. So hence I have to exist and my tests have to be located at the same place as my core app. So this is more of an Android developer focused and hence I said it's a native screenshot test that you might be writing. So if I were to actually run it it's a simple gradle plugin to run it. It takes a while to run especially the first time so ultimately it produces something like this this kind of report. So if you were to just test the flag itself it would show that you were targeting the image view with a certain height even if it was to fail to tell you about it. If it was an entire app to be similar gain the label and the flag at hand. So this is basically quick demonstration of that app test framework. As I mentioned it takes a while to run the first time which I don't really like especially as a developer but if you really want to have a native framework that's your application. So if I were to move forward. Okay, Android, great. So if you talk about enterprise there are plenty of different solutions. There's obviously screening.io one of the solutions out there they have a nice support for React not React native yet. Flowlessapp.io it's the small company I believe they are based in Ukraine that mostly focused on iOS applications and the visual testing of those iOS native visual tests of those iOS applications. Chromatic as I mentioned they are focused on storybooks and persi.io also use the screenshot testing but as part of its continuous integration. And as I mentioned these are all enterprise which I won't be focused on too much today. So overall from this component test the best practices are obviously take care of the screenshot names. Don't call them screenshot A underscore copy. Always go components first never go the whole page because it will just be very slow. Git media is a nice extension I like to use as part of my Git flow because it basically doesn't store your screenshots and your Git repository. It stores it or whatever you want AWS Azure or any computing cloud service that you like. So it's nice ways to arrange and organize your screenshots. And last but not least still adhere to the standard test pyramid. Start with unit tests. Don't jump to component tests right away. So the conclusion overall is that the first level the DOM and CSS is not enough. You have to have the component tests. You can see that component tests are really you're trying to bridge this gap between developers and testers and improve that feedback loop you're trying to give to the developers. Make the tests run fast even if it's visual tests. And you can always choose the existing solution but if anything I've shown today with those action drivers comparison driver that there's so many solutions out there for the small use cases that people tend to build their own solution but at the same time think twice the return of a measurement might be minimal but you still can do that. And as I've shown you the pattern is always the same. You loader action driver and comparison driver. If I were to go to the last part of the uphill journey to the page tests obviously first we'll have to talk about the responsive testing where you can manipulate the window sizes. You can still operate with DOM and CSS. Transition states is what you're trying to kind of focus on quite a bit and last but not least you can still take screenshots even if you're using something more heavy. So if you want to look at the responsive testing obviously Selenium is one of the common use cases it's common ways to test the applications which are written in a responsive way. Please don't log. Okay, great. All right. Let me open it up. Nice. I believe that's the one source main test. Okay, great. Any of you presentation mode, great. So again, fairly simple, your common web driver obviously you have a setup. You have your teardown where you quit the driver if you were to test your desktop lag. You first, as I mentioned, you can manipulate your window size which for just a regular desktop application platform if you're testing, as I mentioned, those responsive changes of platform how your app reacts for desktop, tablet, and mobile those are the good things to test still with the Selenium. So you can set it for desktop, you can set it for mobile and also what I like to test is the transition state. So if you have 374 as a mobile, try 375 for the transition state and make sure that in this scenario I compare the CSS values because they in the case of my app control whether flag is showing up or not. So you can use those to actually test your responsive app. You can have a couple of those like a smoke test but I wouldn't go too crazy about these scenarios because again, these are very minimal CSS based and still heavy because it involves Selenium. I won't be running it because it's very simple but that's basically the gist of it, how it operates. If I were to look further and actually do some native visual testing I would obviously use Apple and Apple tools. In terms of visual test frameworks, this one is an enterprise solution but they are stable, I personally use them a lot. Like even though I showed you the backstop JS if you scale up, you have to use them. I have to maybe a strong word but I haven't seen the other solutions out there that on a scale might be as reliable. And again, there is a lot of traction around this solution. So if I were to actually show you an example and actually want to run it, enter distraction mode, exit distraction mode, great. So if I were to actually show you to native, yeah, that's native. It will be very similar to how you would run your regular driver test if you've seen how sit app is almost done. So you would have your driver specified. You would have capabilities. Capabilities is what you have to control because it would determine what platform you're running it for. In my scenario, I'll be running it for Android because again, I like Android in terms of testing. So we'll be basically specifying capabilities, saying this is what platform we're running these tests for. We have to initialize eyes. Again, this API is fairly nice because very semantically correct. And for the test itself, if I wanted to test the whole app, it will be as simple as that. So I'll basically check window and close the eyes and the tear down. I have to make sure I abort if not closed and quit driver. This one I actually want to run. Okay, exit presentation mode and let me run it really quickly. Okay, great, let me run it. Okay, so to clarify, I already run it ahead of time. So I already have a successful test. So, you know, sometimes emulator just messes up the layout a bit, but ultimately it kind of started the app behind this thing here in this emulator and it will take a screenshot. And ultimately, unfortunately, it uses Wi-Fi, so you can see it disappeared, but it took some time. And if I were to show you how the app retools handles it, so I have my own account. If I were to refresh, let's hope Wi-Fi is good here. Okay, it's still loading, but ultimately, if I already ran this test, it would save me the original screenshot. And yes, to show me that I still have some unresolved tests that were run. And you can see that they failed. The reason why they failed is time. Because I ran it like two hours ago, so the time piece was off and as a result, my tests fail. That's one of the part of complication of visual testing because the maintenance cost can be high. Fortunately enough, the app retools allow you to kind of ignore certain regions by just selecting it here. So it will be ignored and it won't be failing anymore. So this is why, again, when it scales up, app retools are quite nice. So to wrap it all up, enterprise solutions. Obviously, app retools, as I've mentioned, the browser stack and source labs, those two solutions help you to scale up and just run your tests in the first place. There used to be something called test object, now just all part of the test source labs. But ultimately, browser stack and source labs will help you to run and scale up your application. So here we're actually reaching the summit of our visual test regressions in our visual test pyramid. And the main question to ask is how would you fit in this uphill journey and how would you try to utilize these principles I've mentioned today? And basically the call for action for today is that you really need to start the climb. You need to start with visual unit tests, go further with maybe component tests and of course, page tests, if you have some smoke tests or just small number of those full app automations. So with that, I'll go to the Q&A, thank you, page. So are there any questions? People frequently ask me, for example, for again, as I've mentioned, maintenance costs usually the highest, the most complex. Things with visual testing. Just that I've mentioned with the snapshot tests, especially being complained about by many developers, personally, I use just a lot for legacy systems. If I wrote an old React app, I would take a snapshot, save it and make sure that it doesn't regress ever. That's great for legacy applications. So the just kind of tests and people don't complain about maintenance, those are the best. One thing I wanted to mention, especially because of talk before me, visual testing is just part of so-called approval tests. If you're working with text, you have something called I believe text test framework, if you just wanna test text. Visual testing is more focused on images, but it's all ultimately goes to this approval test, which involve intelligent assertions and validation by the engineer, by the test engineer, by developers and so on. Any other questions? Those are the common things that you asked, but great. Thank you. So this came to my mind, like in Applatory Tools, there is an option to make the false positives, right, to ignore and all. So in open source, is there any way in order to have that false positives, to ignore those kind of false positives? I would say you always like the way to avoid this complaint of maintenance, it's a legit complaint though. And this false positive to avoid those, because ignoring is one thing, avoiding them is what I prefer, is start component testing, always do component. As soon as you go to integration, you can have one or two of those, don't go beyond that, don't have, as soon as you test a single component, you will have very few, hopefully, very few false positives. In the scenario I showed you with Applatory Tools, where time was an issue, it would only occur if I test the whole app. In reality, in the real use case, I would only test individual component for the most part, and have maybe one smoke test for the whole application. But I wouldn't ignore false positives, I would try to avoid them by respecting test pyramid. Test pyramid is the most important guidance that you might have the easiest and the simplest to adopt. Yeah, and the second question follows like, so even I am using the backstop JS, even I used to compare them with the resemble JS, all those things. So I used to do the page level test of entire application with the backstop. So is there any way like, can we do like components test using backstop JS? Oh yeah, yeah, I believe one of the test I showed was, maybe I wasn't clear, but yes, this is one. So if you do backstop JS, so I had two scenarios. First scenario was the whole app, second scenario was American flag, just the flag component. And what you do, you use selector attribute, and you specify what selector I wanna test. You can even like, backstop JS also supports button clicks. If I wanted to test like human flow, you know, I open my app, I click a switch button and I go and select a certain component. Backstop JS supports all of that. But again, this is, I would say, it's to get started kinda, to get a fast feedback for fast iteration company, but if you scale up, you might need to go to other solutions. But yes, once we question, you can do the selectors. Yeah, I used on before and on ready selectors. I need to try this option, maybe. Yeah, yeah, yeah, it's very easy, quite nice. Yeah, I'll connect with you offline. Thank you. Thanks, Dimitri, for going us through an extensive and exhaustive core structure. Is there any chance this is available on GitHub? Yeah, yeah, I will share it with the slides. I believe we updated to the main content app, you can upload the slides. I will attach it there as well, so you guys can take a look. Thanks a lot. Thank you.