 Hello, my test. Thank you guys and good morning, everyone. So today my talk is about automating Q8 through visual regression testing. So I, Andrew already introduced me, but I'll do some quick introduction. So I'm a WordPress solutions architect at Pantheon and I'm a community contributor from the Philippines and fun fact about me, like Andrew who said I proudly named my three daughters from operating system. That's eunuchs, rose linux, and Kali and Ironically the name of my wife is Max. So they all like jive together. So So discussion points that we'll be doing today is automation intro and what is visual regression and Backstop JS basics and what it's what are the limitations and WordPress? Related VR VRT services out there and we'll do a recap and if you have some cute questions You could ask me later on so To start I guess we we all in our career became this cowboy coder we started Without version control, I guess you know we do SFTP and We don't really do Q&A. We are Yolo coders. We do directly to the live site so Once in our career, we did that I guess so but now we are is we are Coming into this era like what we call like web ops or DevOps era where teams evolve into bigger teams and new requirements or processes arises During that and as much as possible we want to avoid downtime on our live site every time that we do deploy so That's and we also want to automate those repetitive tasks and Yeah, as you have not heard yet what the hell is web ops culture. We already have like DevOps what's web ops what we want to Like share is like web devs In this culture we want web devs to focus on their development and not too much on the tooling Content managers can focus on their Content instead of like the the whole whole process itself and stakeholders will not worry about if there's like Downtime every every time that there's like code changes and their website and Importantly even though that the traffic has a high spike We want to make sure that the site won't go down And the why why do we automate? We want the bots to do the boring or bots or AI to do the boring repetitive tasks for us and What are the things that we could automate? First of all the one the most favorite that I like is the automating of PHP coding standards and also compiling of CSS and JavaScript you can automate the minify and combine also the cross browser testing so You won't manually do that As well as PHP unit testing Also the image optimization if you're not using CDN or the edge to do the optimization you could also automate it on your build process Also for the page speed scoring Instead of you manually doing the core of vitals you could automate it that and As well as the SEO score audits everything that you do a build you could also put that in your CI And one of the most common thing in the WordPress ecosystem is core plugin team updates because it's also one of the like most common breaking point and in your testing process So what are the benefits if we do automate automation? Yeah, the stakeholders love this it saves us money and for the developers it frees up a lot of developer time instead of doing Man manual QHX so it also helps big team to do frequent deployments in their website and You want to also do early bug detection even and avoid to do or see it on the live site and Guess if you I guess you already have encountered this in your in your life that you try to fix bug one and then There's a new bug to that will happen and after you fix bug to The the first bug is reoccurring so it's one of the annoying part when when fixing bugs, so we want to minimize their reoccurrence of those bugs and We could more focus on more features develop instead of like bug bug hunting or bug in those bugs So CICD how can automation can help? We could put in continuous integration work out Features in parallel if you have like big teams you could this developer one developer two working on feature a developer two and four working in a feature B and They could work in parallel with each other and not worrying about Stepping each other throws when doing development we could also insert automation and test in specific environments and Importantly we won't impact the live site while doing development and On the continuous delivery side You could help this part could help ship the code from your CI pipeline to the Live site and without now downtime, of course and we could do a lot of features in this way and Importantly multiple dev developer collaboration So Previously in the early days of CI CD People do their own DIY QA process or the CICD processes and they use Jenkins cruise control final builder to do that and they have to Like build the server up do their scripting from there and it's a painful frost process previously But now there are CI CD SAS companies that arise from there like circle CI and Travis No server setup needed necessarily They just put in your scripts and CI see Circle CI or Travis and they could be able to help you out from there But now a lot of version control host has integrated their CI CD like get lab Bit bucket and get hub actions So you won't need like separate services in doing your automation You only have like one provider for that one so for for easier Scripting So yeah, let's let's Check first sometime if you haven't heard of visual regression yet, so regression as defined by Lexico It's like a former or turn to a former less developed state So you fix you fix something, but it went back to the older self that has like issues as defined by Lexico so internet and When does visual regression happen? Especially when you do CMS core updates plugin updates team updates and modification and adding new functionality That's the usual um Causes of visual regression And PHP version updates because sometimes take when you do this changes You're gonna see like errors in your code and atop it and push us your site or the Your site to like shift a couple of pixels down So that could be considered visual regression because it's the the image shifted So What are the VR VRT tools out there that you can use so? We have rate That was developed by BBC news so that's the link if you're interested to check that out it has 5,000 stars and get up 400 forks, and it uses open-source Apache license and It does like screenshot and comparison test the live site against the staging site and There's also this Gemini testing It has one case stars in get up 150 forks and MIT license and it's developed by Yandex and And This one there's this backstop JS They have six thousand stars in get up 500 forks and Using MIT license and they have the school scrubber thingy that I'll show you later on and And You could also the the feature that I like with backstop JS is you could be able to test in different Viewport or screen sizes like that the width and height and then you could be able to as mobile Tablet and Desktop and you could put in as many as you want and for That the visual they have like a GUI web visual difference inspector you could be able to see what That's the difference of your test and your live site after you push your changes And if you're gonna be putting it in your CICD There's like CLI reports if you you could automate if it will pass or fail given on a specific threshold like 5% or 0% that depends you could be able to put that and You could also do the omx Exclusions that say you have a slider And don't you don't want to test it you could exclude that so it will not be included in the testing so Let's go to the setup of backstop JS it requires no JS 6 plus and about the JS 6 above and and Installation is pretty easy if you have already no JS gonna just do npm install G for global then backstop JS and You need to initialize your installation Backstop in it and go ahead and edit backstop Jason. So backstop Jason is like a basic file where you put in the configuration and Every backstop JS you could be able to put in the ID of the test and like what I have said earlier you could be able to put in Different viewports on that section where you could label it Let's say you're gonna use with 320 by 480 and you could label it as a phone tablet and so on so forth and For scenarios, this is the parameters that you're gonna put if you're gonna be like for example, you're gonna do live site which is your URL or or Live site is the reference URL and the staging site is the URL so you could be able to put in multiple URLs in there and This are the rest where your The past the path for the Reports that is being generated in this one. So this is the path that you will configure in your CI Where to check those results in your test? so basically To summarize the basic backstop JS or backstop that Jason file required paths is the ID viewport label within height Scenarios and the label for this one. Let's say home page the URL of your home page and The URL of your staging site So with this basic configuration the basic required parameters in your backstop JS You could be able to do a basic test So for the view viewports you could be able to put at least one as well as in the scenarios But you could be able to put in like as many as you want like hundred and Yeah for a demo for initializing your Live site because what we want is we want to check first your live site Before the changes are made so we're gonna do a backstop reference command and it will take screenshot of those Pages that you have configured in your backstop Jason file So if there's no error No red red flags and and there it means that the Screenshot process is successful and Then if you're gonna be testing the changes that you have in your Staging site you're gonna execute the backstop test So after that one there will be a pop-up of the you staging URL site that you Configured in your backstop JS file like what in this screenshot and yeah This is the GUI result so as you can see the report there's like two parts of the Report so there's a GUI and there's also Jason file so in in the test that I made you can see that To test past in that GUI and you'll be able to see I put in my configuration that if it's lower than 20 percent difference from the Live site it will pass so from here you can see also see 15 percent difference and then the Site shifted septic pixels on the y-axis So that's for the phone viewport and this is for the viewport of the tablet So in your CI you could also check the reports Jason file the section for your viewport viewport and the phone test it passed and in your Tablet it also passed So via CLI when you also execute the backstop test Command you could also capture the error response from here I simulate that a test failed so you could be able to catch the response code from there So this second test that I made I I Changed something and it shifted the the site you I'll do a demo later on But if you see something like pink stuff like that one It means that that part of the site on your live site got broken so Yeah I'll show a demo later on So They they have this backstop.js have this scrubber thingy that they call so when you try to test your Live site and your Staging site you can see that the scrubber tells you this is the original site and after the changes That's the like pink part that change in your site so That's what they called the scrubber thingy All right, so okay What is the limitations on on this kind of testing so by default If you have like dynamic contents since we're doing screenshots and it could only capture one part of it at a time Dynamic contents like social feeds which is sometimes random or if like Somebody tweeted and then it shifted the the layout It will be like flag. So it's one of the limitations as well as like the image sliders sometimes when you do a screenshot and then they The screenshot tool took a different timing on there It might be also flag as a failure Also random add sources because sometimes add We it's unpredictable. So it changes sometimes. So As well as like moving items like you have a lot of like animation happening in your site It can also be flagged as a failure and as well as videos because yeah same same as sliders and and Reaction after an action. Let's say you have some changes in your menu and You need to hover for it to see the difference You need to do some advanced scripting for you to be able to capture that one but by default The the VRT Process the default VRT process will flag that as default because you need to do some advanced scripting to capture that reaction after an action so In doing this in your automation be it on get of actions or Bitbucket or GitLab This is not That an opinion on how to set it up You could do it on many ways. There's like a lot of ways to do that to solve a problem. So Usually what I do is live site. I I clone it For our team, let's say you have the multiple development team You have developer one development to and we have like an environment for plug-in core update. So while developer one is doing some features developer two is doing their own feature We could do a VRT or plug-in core update on a separate environment and then let's say we schedule Mondays or Tuesdays to like merge our codes and We do the final integration final testing before Passing everything to live. So if all test passes That's the time that we push it to live instead of doing that In the live site, so at at the moment visual regression there are If you don't want to do the automation by yourself You don't want to like code those stuff because they're already Software as a service in the market that is using this that technology Like Percy chromatic apti tools vis-vis Staging pilot and many more Yeah, you could just use their services to Do the VRT on on your end as That is claimer Staging pilot is acquired by Pantheon May 2019 so This was their gooey when before they were Acquired they have the site you could be able to choose which Pages of your site you want to update and then You could automate it or you could manually check it first before Merging those changes in your live site. So they're using the backstop JS for that one What are the common features on those VRT companies that offers that service so live versus staging difference like that screenshot they shows What's the difference of live and staging they also provide an API for you to be able to programmatically do it on your end? And Yeah, multiple screen break break points that you could set for mobile 320 by 720 Tablet you could define what are the break points that you want to test and They can also have version control integration. Let's say you push into get hub. It will be sent to your hosting and error reports, of course and In the WordPress space There are also in the WordPress ecosystem, there are also Existing services that specializes in WordPress that utilizes visual regression testing. So Yeah, surprisingly, there's a plug-in for that. So there's this visual regression test plug-in by bleach so You can solve a plug-in and then you set your URL on your site that you want to test and It will be sent to their server for testing and you're gonna get You can do comparisons if it if it fail or it it will show you if it fails. So The free version this I guess will let you only test three pages at a time I'm not sure about the paid pricing for WPN the difference our friends in WP engine also uses this WP engine smart plug-in the manager They do automated backups You could also set the frequency on when to check the updates like daily nightly and The changes are applied to you could choose not directly online you could choose if it's in test or like the environment that you want to do the test and If you if you want to do it online, they have a feature to Automatically roll back the the changes if it fails So our friends in five will also have that same Feature it's an add-on you set the frequency when to check You let's say you want to do the updates at night. You could also do that They have like random paid checks like I think 20 20 pages And they automatically they have an option to put the site in maintenance mode before doing the updates and Yeah, update light same same as WP engine the updates can be done on live or in the staging and roll back if the VRT checks failed and WPM you dev same same technology back up checks the screenshot of your site before the update and then before the update begins there's some checks that you you could do and If the site became down or there's some error it will let you know and it also Get some screenshots up after the update and will also give you a report and of course pation also have this kind of service you could set the frequency you could configure specific pages and What happens is automatically clone live to new environment first so you won't mess with the live site and You could also exclude certain parts of the Site for the test. Let's say the the slider or the videos if you know the deal met DOM Or the class name you could exclude them so they will they won't be flag and There's ought you can automate the push to live if there's issues or manual You could manually check using the tool the scrubber thingy tool before pushing to live So the disclaimer I work I work in Pantheon so Yeah, this are my reference for that that Presentation and just to recap I have an intro with automation visual regression tools that is out there How to use the make basic setup of backstop JS and what are the limitations of this kind of tools and Wordpress services that utilizes visual regression testing and Yeah Any questions or feel free to connect or approach me if you have questions so you could find me on Those social media platforms. Oh Thank you you