 Cool, I will start So we keep things on track. I don't want to be blamed for it I'm Robert Mesa Roche, which is a Hungarian name It means butcher in English. I mean my church is I live in Romania, but I'm joining from I been a developer for a few years now and Initially what got me interested in end-to-end testing particularly was some work done on a legacy application on a legacy application where there were no tests and introducing any kind of other tests then end-to-end test is quite complex because that would mean you would have to do some refactoring and with end-to-end testing we can kind of like solve this problem and Give us like a head start and a good baseline For tests so we can do refactoring later Today we will talk about end-to-end testing in the context of WordPress with playwright, which is a tool and especially with the test uterus from WordPress To lay down some expectations first we will Get on the same page We will have a working definition of what testing is what end-to-end testing is particularly then we will take a look how We ended up in the situation that we have two test uterus today in WordPress And how that is going to be solved in the future Then we will dig into playwright and the test uterus based on that then all the things that feed this cost will be put in use in a sample project that I prepared and There we can see everything working together and at the end We will try to draw some conclusions. I will share a few more resources that you can follow and there will be hopefully five minutes for questions That I'm happy to answer then we will say Goodbye, so what end-to-end testing? We can say that end-to-end testing is a type of testing many types of tests testing exists sometimes the types of tests are represented in a pyramid of tests where end-to-end test is at the top and at the bottom we have the unit test and why We have this arrangement Hopefully will be clearer by the end of the talk Testing This is like a definition or a Formulation from the book full stack testing all the things which are like underline are clickable. So you can just like follow the Links find more information there Testing is just a way to validate our expectations in the Software world you are validating the behavior of our application What is important here is that we write the expectations. So the expectations have to be right otherwise or test or worthless and just this practice of Reflecting on the our expectations from the application typically brings a lot of clarity in all kinds of ways We want tests to be repeatable We want to have the test run as often as possible because if we run it often then the chances of introducing bugs is just slimmer and to Do it manually and repeating the same test which quite might be like 100 tests or Pretty boring and time consuming So we can use tools to do that We can never really replace manual testing and the two can coexist easily together and Especially because automated tests typically are derived from the work that was initially done by a manual testing So with all these definitions We can get closer to what end-to-end testing is This is a description from an article from the WordPress make website And here are a few things that are important One is that we are interacting with the application with end-to-end test So we are not testing a function like we do in unit test We are interacting with the application as a whole It's kind of like means that we have to have the entire application booted So we have to have WordPress running and this is why we have the production-like environment also in the description in end-to-end test typically we use the browsers to perform our tests and We are validating different flows This means that we are mostly interested in a journey that the real user takes and The things that happen during a journey journey you can imagine in a e-commerce setting that the user searches for a product it adds it to the cart goes to the checkout and Pays for it. So that could be one flow. Of course like Leaving a comment would be a very different flow That's end-to-end testing is used in this way if you just go back to that pyramid You can see that end-to-end test is a high level test Which means you are not interested in the foundational level of our application, which is the code And that's what unit test does it verifies the code here We are really verifying the integration of the entire components that we have Playwright is a tool to perform these end-to-end tests There are other tools like puppeteer. The puppeteer is one of the other tools used in WordPress for tests playwright is a more modern one It's actually a fork of puppeteer It's considered more reliable these days and it really also allows testing in all kinds of browsers in all kinds of operating systems in continuous integrations we can even test native Mobile emulation and what is more important for others outside the WordPress ecosystem? It also supports other languages like Java or dotnet the WordPress test utils is some kind of helper functions that are On top of the testing tools like playwright or puppeteer these Test utils are practically used by Gutenberg But they are also integrated by core and because they are published like standalone packages We can also like Consume it and use it in our teams or plugins so this this is how a test looks in Playwright and I underlined the part which comes from the test utils Because playwright in essence doesn't have any knowledge of CMSs and also not about WordPress So we need the test utils to perform some actions in a very convenient way We can still do all the things that test utils provide without the test utils itself But it makes it much more convenient the rest selecting an element like heading in this case is Available in playwright and also the source assertion that that heading is visible on the page It's also coming from a playwright To somehow complete that definition from the From the article from the WordPress. This is comes again from that full stack testing book Here what I highlighted is that we can only have just a few tests Which practically activates all components. So if you think about that e-commerce example During the journey that the user takes from searching the product adding it to the cart and going to the checkout it practically activates all or many functionalities of our application and with just a few tests we can confirm that those Parts of our system work and interact Well And in that sense to think about the pyramid We can argue that We can have just a few entrant tests and that might be even enough the other part to complete this book argues that Downstream system should be also included in the e-commerce example a downstream system would be Another service that we integrate may be like QuickBooks for like invoicing The book argues that during the end-to-end test We should also check that we are integration with third-party system is functioning So end-to-end test should cover that for example after a successful checkout Actually the invoice is somehow generated in the QuickBooks or is it available in our system Now I'm going back to the two test utils and how the entire end-to-end testing in WordPress started So the Gutenberg project was kind of like kicked out at the end of 2017 and Even in the early phases Entrant test was integrated in Gutenberg initially they used Puppeter and all the work that was done in Gutenberg regarding testing the foundation work practically resulted in the standalone packages that we use today and so in the middle of 2019 so kind of like two years after Gutenberg was started the after the foundation was laid for the test utils a standalone package was published which is Under the WordPress slash end-to-end test utils and the first testing core was added and also it was integrated into the Continuous integration pipeline and the fact that it was published as a standalone package meant that other plugins Where everybody from the community could start Using it which many did In the next years there were a few more tests added in core But not much happened in core At least not on the surface When the package was published initially it still remained in the Gutenberg repository, so lots of work might not Been in core but in Gutenberg many things happened After a bit after the package was released in 2019 Right after one year already people started discussing about switching to playwright playwright Started become more solid people started noticing it and they were very good arguments for using it But it was decided to wait a bit more until the platform became more stable and In 2021 October after a lot of testing and a lot of work that happened in Gutenberg where they practically Compared the two tools performance It took a look at the reliability of the test was kind of clear that a playwright is a better choice and because There were only a few testing core it kind of like made sense to start migrating at least In the Gutenberg. This is where the first proposal was made available to the public People did not really oppose to it. So practically six months after that The announced announcement was made that Officially the test utils will be switched to playwright Really good clear and solid plan was like formulated and people started following and implementing that plan The migration process is still not finished. You can we can still help with it and if you follow the link under this Text you can see how you can engage with it This was this is the migration steps The first was to integrate it in the Gutenberg which happened practically right before the proposal The tests are currently migrated from popular to playwright in Gutenberg You're kind of like at 70% now. So there's just a few more tests left and Since there were only a few tests in core those were already migrated It's important that the WordPress core is highly tested But it's not highly tested by end to end test and the reason because end to end testing is pretty new to the WordPress ecosystem as a whole but WordPress is very reliable and well tested with some kind of unit test and The last step once the migration happens in Gutenberg will be practically removing the old page Which is the proper term on the Practically the migration of the WordPress core just happened in October so just a Few weeks ago. This is I think a really important milestone because it Signals that playwright is really ready to be used Even it is still on major version zero and we won't see the version 1.0 until all the Tests were migrated in Gutenberg When this will happen in my opinion likely next year, but we can already start using it Now digging into bit playwright Playwright is the foundation for the test users and If we take a look once again of a test and in all tests, we will see at least an assertion We'll see some kind of actions performed and we will see locators Locators are practically you can think of it like selectors Like some kind of query selector that you would do in JavaScript This is for selecting practically element on the page. There are many ways to select Sometimes you want to do it by the label by the role by the text or you can just use plain CSS selectors or something more specific to playwright which are like test IDs Once the selectors or the locators are selected Actions can be performed on them so if you're talking about like a Text input field, then we can just like fill some value in it If you're talking about like checkboxes, we can check them We can even hover over them We can perform all kinds of user like actions that we would do as we would browse website When we perform those actions, the important part is kind of like to validate or expectations of the things that happened and we can do that using the expect functions and We can just sometimes simply check that the checkbox was actually checked or some kind of Classes were added maybe to rebuild an element. We can check that some kind of element became Visible in the viewport with some kind of scrolling happened. We can check Rest API responses if things went well or we can actually check that we arrived to the right page after some kind of form was submitted and just with these three things with Locators actions and assertions we can do quite a lot of things On top of these things that play write offers We can we have the end-to-end test utils as mentioned whatever That the end-to-end test util offers is more like conveniences. We could perform those things also with play write One of the things that the test utils offers are more general Helpers like that we would do on the page like pressing keys This could be very important for example in the editor where like some kind of combination of two keys perform some kind of actions It makes also more convenient to resize the browser or to drag some files from I don't know to the screen Then we have something which is more specific to WordPress like admin utils like Navigating to some kind of admin page or visiting the new site editor or creating posts Using that add page or add post button available Other than that it also offers conveniences to interact with the WordPress rest API So just by calling a few methods we can just create a post delete all the comments or activate the team and when some kind of Endpoint is not supported or we are dealing with Third-party endpoint we can just use the more lower level request utils rest method to perform really any kind of Rest API action either deleting or just like posting something Since the test utils were started in good some work we have a bunch of Methods which deal with the editor inserting blocks publishing it going to the preview page all kinds of things that could be automated or Automated and available as a method Then we have some kind of other utils which are important for doing performance measures like Lighthouse getting the report or just checking how the server responds This kind of like means that we can not just Navigate the website with the end to end and with play write we can actually perform other kinds of tests which are like performance tests and On top of this of course if you have like a bigger projects We can extract something some repeating things to project specific utils and One good example is this the WooCommerce blocks Where they integrated play write and the WordPress test utils, but on top of it They built their own utils And because we are dealing with WooCommerce This offers the possibility to add items to the court go to the checkout clean the court everything that you would expect to do in In a WooCommerce website now that we kind of like have a good understanding of at least a base understanding what's available you can take a look at the sample project and the sample projects tests Community plug-in, which is called WP guest bar We are integrated into github action So we have continuous integration and we are using the official Docker images So there's nothing really Complicated going on if you just like read ever an article how to set up for press for Docker. It's pretty much The similar thing So jumping over to the actions straight to the actions you can see that we have Two branches. We have like a main branch where the tests tests Performed successfully and we have like another branch where we purposefully failed the test So we can actually take a look what happens in those cases Interestingly the test run on two wordpress versions and two different php versions This is kind of like to demo that it's quite easy to Run the same tests in different for press versions or even php versions and if we take a look at the Steps that are performed what we have is just like checking out the repository which is like a Core github action feature then we have the option. We have the step to set up the test environment Here because we are dealing with docker. You see like a bunch of like downloading all the layers that That are downloaded through For press and Once all those things are done. We have the actually the test and We have eight test and what I want here to highlight is that we can actually Without knowing anything about that WP gas ball We can write the tests in a way to really capture the essence of this plugin and just by reading the The test descriptions we can get a really good idea of what the plug-in does so practically We can describe the specification of the plug-in with the test so if you go over it you can see that if The plug-in is not active so practically just after installing wordpress We don't expect to have any admin board So this is the standard wordpress behavior if you are not logged in you don't have an admin board but once The plug-in is activated and we are just a simple visitor Then we expect that we have an admin bar and that admin bar has a login link Which prompts us to log in practically other than that We have the possibility if you choose to to display some kind of custom message in that admin board and We know that custom message is optional because if we don't set it There are no custom message displayed. So this verifies that there's no fallback message display and We know or we expect that that custom message to expect some kind of HTML to and to be able to style it Which means that maybe we have the possibility to I don't know color the text in red and use some kind of a strong text But once we are logged in Maybe as an admin or as a subscriber Then we expect the plug-in to not tamper with the default wordpress behavior This is just showing the default admin bar with the how-the-message and the typical menu items and On the admin side the plug-in exposes a link under the settings menu which allows us to go to the to their option page and On the settings page of the plug-in. We actually have the possibility to set the messages and Practically, that's what the plug-in does. It's a very single-purposed Plug-in, but it does it really well Now how this thing is achieved and how we are testing it If you are like maybe checking it off toward the presentation There's like a recommended order of browsing and you will practically Follow that hopefully this will load It's already loaded. So the first thing is Is the configuration of the playwright? The configuration can be quite complex in some cases But with just a few things we can just get started Can just say that hey every test are in the test folder Which are like it saw that we have three test files We can define to have something run before all the tests are performed And we will take a look what these global setup does and then we have a few settings turned on Which is like in case something goes wrong like one of the tests fails Which we have on or one of the branches Then we want to maintain all kinds of information like screenshots videos traces just to help us to Debug the things that could go wrong and We saw that Playwright offers the possibility to test in all kinds of browsers and all kinds of devices here We are just testing on Firefox, but we could add other browsers here too Now going to the first default test Everything that we see here is we discussed it What we are seeing is some kind of locators in this case. We are just using CSS simply an ID we have an assertion happening which we are saying hey I want to verify that actually the admin bar is not present When the plugin is deactivated and all this should happen on the front page So that's why we are using page go to slash which is Assuming it's our root URL and not the admin page that we saw in the presentation And that's all with this we kind of like confirmed or base use case of WordPress Now before going to check out the front test Front-end tests I want to talk just for a moment about this global setup and why this is needed and Why why you will definitely have it in your test if you will test your team or plug in Because we are testing for this plugin both the front-end functionality as a visitor, but we are also testing the admin functionality Where we are required to be logged in we somehow have to perform that logging in step and When we log into WordPress when we fill out the username and password Actually, WordPress stores a cookie. So for example, if you would log out and you would clear your cookies immediately You would be immediately logged out because the cookie no longer exists and Because we have repeat a test where we are required to be logged in Instead of logging in over and over again We do the login process before any tests are ran. This is the first thing we do We grab those cookies that are set by the browser and we store it We persisted this is what happens here and Whenever we want to be Authenticated later on in one of the tests we just make that cookie available to the browser. This is How we can be authenticated WordPress kind of like hide this complexity of the steps of logging in but practically you can imagine that behind the scenes Actually fills in the username and the password, but just in more convenient way With this information we can go to the front end because here we have the visitor What down we have the logged in group First of all because they are like multiple tests here We extracted the step to activate the plugin in the default test We just had one test So we just like deactivated the plugin inside the test But here we are doing it practically before the test are run in this file and The actual test performed for the visitor are very similar to the ones that we saw in the default test We are just trying to grab the admin bar and we are also trying to grab that login link that we expect expect from the plugin To make it available and we are also grabbing the URL of it And we are just practically checking if the admin bar is there if the login link is there and if the login link actually points to the URL that we expect and If all these requirements are met Then we consider this test Correct We also have a few tests For the custom message that we discussed and to also demonstrate The possibility the option or the way to introduce a custom util We have this WP guest bar util which behind the scenes it hides the complexity of setting up a custom message If you will have time we can just like Go over it later on So before going and navigating to the page we want to set the custom message So we want to have it in the database before going to the front page Otherwise, obviously the message would not show up and you just try to query for that message and you just want to check if it's there Checking the opposite. It's a bit trickier because We cannot simply search for the message or we could but I find it's more More easy or more straightforward to look for some kind of element in which the Message is rendered and to check for that We also have the in the custom util the option to delete the message and It also has the possibility to add that HTML that you're checking with later and here we see some new things like new Methods assertion methods to practically check the existence of the CSS Practically, we are checking if this style is actually rendered on the page And this is where that authenticated step and that storage comes into place To only make that authenticated Context available with this is why we put it inside this task group. So it's practically Not put at the top because that would mean we would be authenticated for all of the tests This is why this is not commented here So in this test because we make available to cookies we act as a logged in user and we practically just check whatever WordPress does by default, which is adding the how the Message and the how the message should practically point to the profile link Then on the WordPress admin tests Because here we want to be authenticated for all of the tests we practically Made the context Available as the first thing For the same reason we also do the activation step Before any other tests and Here we don't we should not see anything new. They're practically using the same old Locators the same old assertions that we saw the more interesting thing is maybe checking the settings page Where we make actually use of multiple actions One is filling the mess filling the input field with the message Then practically clicking the save button, which in fact will trigger the form to be submitted Which reloads the page and then we check for the success message and we check for the Existence of the message that is set This is these are the test to cover all of them now to check The failure because this looks all good, but many times Things failed First of all if things fail we get some kind of error message already in the console We're in these kinds of output from the GitHub action and Sometimes this might be Enough to debug and know what went wrong You can get some kind of message that we expected things to not be visible, but we actually got something Visible this already could tell us what went wrong And later on we also see that we expected the string, but we received something else and If this information is not enough Maybe we have some kind of other test other messages that it's not really clear then we can make use of those screenshots or That video that we are saving when things went wrong in the context of GitHub actions those are Practically saved as artifacts so we can just like download and inspect whatever went Wrong, obviously if you are running the test locally you just have access to the test results immediately So I already downloaded in advance Let's take a look When something fails we get the folder for each Test that have failed and we immediately get the description of what went wrong And here we have the image we have the video and the trace so the things that we call be configured in the Playwright config So if you take a look at the screenshot, we actually see that the admin bar is there Even though we expected it to not be there So this is really a visual way to check what went wrong instead of just reading the message The video in this case it doesn't tell too much because Practically loading the website you see the website loading up, but it doesn't give too much extra information compared to the screenshot But if we take at the other test that went wrong Here we see a bit more in the video Which is like not captured in the screenshot, which is the option to actually filling up We we see things happening real-time so the user or the browser filling up the message and the refreshing going on Still if this is not enough we can use this trace file the trace file Can be viewed If we if it just a zip we can just go to the trace playwright dev Just like drag and drop it here and this will give the steps of the test So we can see that initially we navigated to that option page Then we selected the input and we started typing and in the next step We already have the message filled in Then we selected the save Save changes button and we clicked on it the refreshed we checked the Setting saved message to be there the notification and then we got some kind of error and The error is practically that we are not Getting the right message at least not the message that we are expecting and with this trace It also gives us the option to jump right through to the test and see The exact line where things went wrong You can also see kind of like looks like the net looks like the browsers Inspector where we get access to all the network requests happening You can also see the console messages that might go on and we can actually also check and Debug the Locators Cool, how are we standing with the time kind of lost track? but If you go back here go To the presentation we can draw some conclusions, hopefully one is that Entrant testing is really within reach this means The basics and the fundamentals are not that complicated the locators and the Actions are just really a bit similar to me like the jQuery Well, just gives a good simple language to interact with elements on the page and the assertions might be something new If you never use tests, but I think it's still very understandable And I think it's a very good starting point Because we can cover the the Functionalities of an already existing product product in this case of a plug-in that we saw It's also a good starting point to have like a baseline to then introduce some kind of refactoring speech are not really doable without Solid understanding of the product and I think it's really Immediately valuable because just with a few tests you can actually get a lot of clarity and a lot of value added to our work and There are a few things which we did not have time to cover One is the test generator and the UI mode So in case you are kind of like afraid to start writing this test on your own Playwright offers the possibility to actually generate this test with some kind of clicking Interface so it's Even for very very beginners. It's quite easy to get started It's also possible to do accessibility testing with playwrights. So they are tools The same way for example, we saw some kind of performance utilities from WordPress We have accessibility utilities from other packages It's also possible to do visual comparison So we can check for example if you are working with CSS to check if we introduce some kind of bugs Did that have any impact on certain elements? It's possible with playwright And then two other pages that I want to highlight is the best practices page Just like go over if you ever want to get deeper into writing text tests and obviously the poppeter migration Which is helpful if you already used the Previous test utl from WordPress because this gives you like a kind of like step-by-step process how to migrate your old test to playwright because the syntax is quite similar and It's not that complicated and as I shared this material and this Sample projects is publicly available. You can go there. You can dig a bit more through the files and See how things play together