 Hello everyone, thank you for joining us and in this presentation we will be taking good through how we automated testing for Gauss-CMS. I'm Joseph and this is Tara and we are both developers in Gauss-CMS service operations. As you may or may not know Gauss-CMS is an open-source wide content management system and a hosting service based on Drupal and developed to help government agencies create modern affordable and responsive websites. What is making it easier to collaborate and innovate? Gauss-CMS also help agencies to reduce the technology and compliance burden on government agencies. Gauss-CMS is managed by the Australian government Department of Finance. Gauss-CMS currently hosts over 350 websites. The majority of which runs access using our Gauss-CMS distribution. Our team manages the Gauss-CMS distribution and releases to the platform. We have roughly two weeks release cycle. During this release cycle we update range of things from new module requests, module updates, call updates, security updates, and break fixes. During every release cycle over 900 SaaS websites have to be updated. Around 290 production websites and 580 websites like non-production websites. For security updates we have service level agreements that need to be met which can be as short as 48 hours for highly security issues. Seven days for critical and 28 days for less critical issues. This presents us with a problem in getting testing done within our time frames. If we were to rely totally on human tester or manual testing it could require a testing time of three to four days. After analyzing a few products we decided to go with Cyprus which offered us flexibility and compatibility for our specific needs. As you already know that our challenge is that we need a way to streamline testing and to shorten the length of time which let us to look at the automation. So now my colleague Tara will talk about what Cyprus is and why we are doing it. Thank you, Joseph. So as mentioned before Cyprus is an end-to-end testing framework. It's based on JavaScript and it also runs on a web browser. You can use it to do end-to-end testing. You can use it to integration tests, unit tests and also it fits within the GulfCMS open source ideology as well because Cyprus is open source. Cyprus has a range of features and some of the ones we use in GulfCMS are screenshots and videos, time travel, debugability, automatic rating and network traffic control. We find Cyprus gives us many benefits. For example reading, writing and maintaining test cases is more straightforward and it's written in JavaScript which means that most developers and testers are familiar with the technology. Another example is that Cyprus closely mimics our human tester navigating a site from the browser. One such activity is the login form for Drupal. We can mimic a fair login or a successful login and test out the functionality from an authenticated user's perspective. We can also test search functionality for example and behaving like a typical user who visits the site and does a search. Currently we set up preview sites for all other live sites where we do major upgrades such as a major Drupal Co-Update for a set time period. We use Cyprus to scan and navigate all the preview sites to make sure there are no broken pages. This helps us to troubleshoot any potential issues early in the preview stage. Another benefit is that these test cases can be run from CI if you have CI or you can run it locally independent of the user's technology. Cyprus is also open source and free to use. Now we would like to show you a short video of Cyprus in action. The video will show you a sample of Cyprus test case running over the GavCMS website and afterwards we will show you the results of that test case. So that was the end of that test case. The test run took roughly 11 seconds to test this website and if we compare that to a human tester it would have taken at least five minutes to run through the same test case. So you can imagine this multiplied over 300 plus web sites. We have gone down from two to four days to test all our sites to half a day to complete our testing. This not only frees our testers time but also we have cost-benefit associated with that. We get to know of any potential issues earlier and it's less error prone because it's scripted. These are some of the pros that we have found when using Cyprus within GavCMS. So to start off we have reduction in time to execute. There is low learning curve for our team. Ease of maintenance technology agnostic can integrate with CI CD processors and it can run locally. Some of the cons we found are that you need to have storage if you want screenshots and videos and you need some human effort to validate false positives. Some of the lessons we learned during our journey to implementing Cyprus are you need a dedicated resource to set up the initial environment and test cases. You need buying from your dev team and testing and start small. For example start from the less complicated test cases and move on to the more complex test cases. I'll now hand over to Joseph to show you a demonstration of Cyprus. Let me close my support point. Some examples that we think from our practice, popular examples can give you some ideas of how we can use it. Here we go. So to start our examples we put for simple examples here and put them into two groups. So the first group is PVT. We can assume that PVT happened after a highly critical release or deployment and then you want to quickly test your website. We put some examples here. The first example is a baseline testing and the baseline testing basically let's assume that a human tester want to browse your home page, your user login page and your search page and maybe randomly pick out some pages from your website to see if those pages can be displayed correctly. And the second example is that based on the first example baseline how we can extend these test cases to cover for example in this example. I put two resolutions here. One is desktop, another is a mobile world. And the second group is a demonstration group. I put two typical user cases here, test cases here. The first one is content. For example, as from us as a distribution, we want to test if a content author can create for example create blog events with pages correctly. And then the second example inside this group which is like we want to test if a user can be created or cancer correctly. So let's start with some videos to give you some ideas. But before we start, I may want to show you the basics like setup. So we are using a software called Toilet and this is currently a part of the Docker. We use this to quickly test and give you this demonstration. So from start we have a fresh installed Galaxymus website. As you can see it's a fresh installed website with four like default content tabs, blog events, FOI requests, news and media. And when we click the people, we can see the default user groups. You can picture your own user groups. So this is like the default Galaxymus user groups. We can see content author, approval, set administrator into default for user group. So here the content author is the one we want to test if this content author can create other content. So let's start from the first example and the PVT and which is a baseline test. So as a human tester, for example, you have a website maybe of course will be complex more than this one. We can use Galaxymus.gov. We use it as an example. Galaxymus.gov.edu. As you can see you may have a website like this and after, for example, a security urgent update, you want to quickly test. So this is a home page and then you want to pick some random pages. You may have some interesting pages. For example, this gallery page, if you keep scrolling, you can see it is quite long. It's very long basically. Now let's see how our suppress can do this. Let's go back to suppress. I have a video here, how you want to see a live example. Let's see. Not sure how you can share the live one. So yes, we have a live one. Let's run a lab like a test of baseline test. So as you can see two groups and we want to run the baseline to test galaxymus.gov.edu. So from left side you can see the country suppress is running. From the right side you can see country suppress is checking and auto-scrolling from top to bottom. So it's finishing in 14 seconds. So this is a baseline testing really fast and it will give you a video and screenshots on every page after you finish every page. Then your tester can review those screenshots and developer can also check each step if you found any errors. So now we want to make it a little bit harder to test if those pages can be displayed correctly in both desktop and mobile world, which is the second example. Let's give it a run. So as you can see here, here's a desktop version. Then the mobile version will be later. You can see this is a mobile. And you can see suppress can do auto-scrolling to let, for example, that gallery page. So this is like the second example is that how we can extend our test based on the baseline and make it more complex and more suitable for your business requirements. Then we can see the demonstration examples. The first example is that we want to test. Let's start from test. We can create users. So let's see the test. So we can see some errors, right? Of course. So what we can do is that we can check what happened from left side or we can click a red one because of this script I copied from our current test cases. And so from here you can see suppress can simulate a human tester to type in the username, password, those things and to create the users. Let's give it a red one to see if anything wrong. Yeah, as you can see. And the user is created and this user is deleted. This is quite useful for our distribution release testing for now, but it could be useful for your business cases. So now we can see the second example is that we want to test if a default content author can create users. So as you can see this user login at first and then trying to create the content. I put a very simple example here to only like to have a title here. Of course, you can upload files, put attachments to upload the media, do a lot of things in your own test here. And one good thing about suppress is that it's not only can give you a tester and improve our current process, it's also very useful for the developer. For example, if we go back to check each step, let's use this as an example to create a news and a media. You can see from left side there's some like small numbers there. This is basically a test steps. The first step for example, to visit this page. Second step is like typing the username and password. And it's like a human tester basically. For example, we can type in the title and then do more checks here. The last step as you can see here. So for a good test always remember that to log out to the user and even delete the user to keep your test clean. This is why after all we didn't see any content because the whole process we want to test if a user can create a content. And yeah, that's the goal of each test. So let's go back to see some code here to see how simple it is. I'm not going to show you how complex it can be. Of course, the test cases can be very complex. However, it can be very simple from start. So we can put some very simple, for example, baseline test here. As you can see it's like 15 lines of codes. And it's very simple to rate. So this is very simple. However, you can extend it and then you can check more here. I just put it as a quick example here. So as you can see at first visit home page. You can put the link. You can put something there. And the second part is visit the login page. The third part in search page. I put a side price comment here. Basically you can type in your search keywords which can simulate a tester to type in. So you can extend it and think and picture that, for example, in your website you may have a complicated or complex workflow. For example, like a webform. Then you can use this way to test the webform and to test the feedback. For example, if people type in a wrong password or type in a wrong email, you can test the different messages based on such technology. And so here, based on the first baseline code, you can see that you can extend side price to test the different resolutions. Basically similar codes here. You just extend very quickly. So this is very easy not only for your developer. Maybe your tester, if they know how to write JavaScript, they can write the side price test cases. Very easy. Then here is how you can create your contents. As you can see, it's slightly more codes. But every line of codes is very clear. If you have the experience to write, for example, some other test cases, you may notice that this is more close to a human and more close to like the JavaScript. And years ago, some other tests, for example, BeHat or some other tests, even now it's very, very popular. But the technology gap to use any software like side price, it will be much more easy because this is basically the JavaScript. So this is also a user. It's even simple. But however, here, of course, the line is less than other tests. Here, as we can see, we are using a side like a creative user. This creative user, this is a custom command. This is not native side price command. The cost command in side price means you can extend your likely repeated likely steps and wrap them into a command there. So I can give you a quick example about how this custom command here is. As you can see, we can have some cost command here. So for example, create a user. I'm not sure if you can see it clear. So we are using running a local drash command to create some default user. Of course, you can also test the UI create a user. Basically, it's that you find those class or ID name, then you just follow each step to test them. And those custom commands can be shared across your team or even with the Drupal community. For example, Drupal community may already have those custom codes there and wrap them inside a project or module. You can use that module as a foundation to save your time to write all custom commands here. Yeah. So this is basically that. Those are examples from the codes and from what we can see here. So if you have any questions, comments or feedback, please feel free. Sorry. Do you need to have to be a just developer to write the test cases? No. And there's a user interface? You mean like, oh, so this is a surprise is that you don't have to be a just script developer. So from what I know, Cypress even have some extension from Chrome or a product called Cypress Studio. Basically, it can record your steps. For example, if you have like any tester and there's a Chrome extension, if you click the button to start, all your steps will be converted into a Cypress script. So is this a standalone program or is it a part of the GovCMeth platform? So this currently is that we are using Cypress for two goals. The first one is to do the PVT. As you can understand, currently we have 250 websites and a human tester cannot do it. So this is too suitable for our business case. The second part is that to improve our current release cycle, we want to do more small releases and more releases. So with Cypress help, we can run it in CI. For example, if we have a small module update, we don't have to wait for a human tester to approve it and we can use this to help us to do more frequent and small releases and we will share the code. So like if a GovCMeth customer wanted to use this for testing on their own website, the GovCMeth website, would GovCMeth be able to do that? So currently, I'm not sure the business side, but I think this is definitely a good way that we want to do. And if you're interested, maybe we can talk with our GovCMeth product owner or any like GovCMeth people. Alistair, I'll be happy to help you. And of course, this is our goal, we want to help all the GovCMeth client can use and can share. We can use a similar way to improve our testing. So this is a really good question. So basically, we are testing all the GovCMeth websites and if you are with PES, you may have to do it by yourself. However, for all the sites, after every deployment, we run this similar PVT automation to basically to test all the sites to see if they can be displayed correctly or not. And hopefully, this can help our agencies to like to have a like their website functionally. So this also is like some lessons we learned. After deployment, some websites or small agencies, we do not have people or enough developers to do this checks. So when they realize something went wrong, it may be like days later or even weeks later. So this is also another reason we have to run this after every lecture release or deployment. For example, on Tuesday or Wednesday morning, we may do a GovCMeth deployment. And after the deployment finished, we run this script. And not only by the script or human tester, we also reveal the screenshots. This is why we mentioned here about you may need a cloud storage because the data is huge. Yeah, thanks. Thanks. Thanks. This is also a good question. To be honest, we didn't think that far yet. But we heard a lot like the commit from community saying that can we run our own surprise inside the whole CI process from GitLab. I think Alistair, he will be happy to know that we have such a demand. However, from technology side, I don't think there's any like blockers there. And yeah, I think we are aiming to help. And basically is that if you have such demands that our GovCMeth teams know, we are happy to help. Yeah, currently it's not. Just on the last question. Yeah. So this is actually a GovCMeth task. We have around 300 cycle sets in Git for a particular part of this site. And there are actually going to be around 300 more. It takes around 30 to 40 minutes on local. We are actually so it will actually put it on the CI. So what are the best practices around it? So it will actually put it on the CI. It's going to take 40 minutes, 40 and then another 10 minutes for the GovCMeth CI to perform all of the rest of the operation. So that's actually 40 minutes of lag time for a developer to just see that the changes he has accommodated are possible. So what are the best practices around it to reduce that time? What would be the, what could the solution be? Thanks. This is a very, very good question. As you guys know, it's like although surprise is easy to be like to use, however, if you have like a lot of like test cases, or you have a big website, it can take hours. For example, to test, to do this peer-to-peer test for 350 websites, even we put it into a very quick way where you can order a lot of checks, unnecessary checks, of course. It takes about three to four hours to do this. So I think from my quick thought about how we can improve the CI process is that from what I can think is that at first, to run it in an independent CI. For example, another third-party CI or a separate CI pipeline to, to improve the performance. Otherwise, if, for example, if currently we run on every like the Cypress tests inside our current GitLab CI, it might consume a lot of like the resources and may impact RSS websites. So this is my first thought. From GALCMS, we may, we may provide like an individual pipeline for your CI testing. This is something we can discuss. And the second thing is that you may run your CI, for example, in any other third-party CI, for example, not good example, maybe Circle CI or other CI or your local. The third one is that to think about, to run those tests in parallel, as we know currently Cypress is running line by line. Yeah. So what, what I can think is that you may put your test cases into groups. Then, for example, you can try to use the, the, the Cypress Docker. Then for example, you have three groups, Group A, B, C, then you can spin up three Docker container, then first container run Group A, second like container run Group B, to make it a lecture, tests running in Perla. Yeah, thanks. But I still have actually a follow-up question. Yeah, feel free, feel free to ask us. So, so for the GALCMS test, so it will actually follow an independent CI. So those tests, then those tests needs to be managed separately in a separate Yes. It's not, it's not official recommendation. This is my personal thinking is that for running, personal thinking is that to run any tests, do not test a production site unless you're running a PVT to save the production resources. If you are, for example, develop some new thing, you can test it in your local or other environment. The reason is that GALCMS currently is a shared environment. And if you are like, for example, taking too many resources, could be in trouble to impact other sites. Of course, we can, we can talk more after this presentation. I do have some personal thoughts here, but yeah, thanks. Sure. Thanks. Thank you for the nice presentation. Security question. I noticed that you are using a Drupal login in the Cypress How do you inject the user in the site, you know, like, to be able to login as that user from your test? This, this is a good question. So this is a, of course, you can see here is a fresh install website. There's no TFA protection. In real life in GALCMS, we do have TFA. And I don't think currently we can run a surprise with the TFA module enabled. This is a likely example that show you that how we can test it without TFA module enabled. And the unique example is that I create a default user at first using Drush with the, because it's test, right? So for example, that content author example, I create a user before the test and with a username and a very simple password. Your name is password 123 so that the Cypress can use that preset username password to log into the site. The presentation you mentioned about after you kind of done your test, there's some kind of cleanup where you're deleting user and content and so forth. What about your next step, right? Yeah. The Cypress provides certain templates for, you know, a Drupal site that can easily clean up your tasks afterwards. So Cypress does, we can think it's like hooks, like before hooks, after hooks, similar as Drupal. Cypress itself doesn't have such like Drupal things yet. However, of course, as we showed in the last, likely, from the code, you can have it in your custom code there to wrap up those steps. Right. Yeah. So then you're responsible for it. Exactly. And put them into the hook. Right. Thanks. You mentioned that if the work is going slowly, so just on this, I should hide everything. So the second question is that now it's like we are using Cypress, but we are because Cypress, I don't think you need to pay a license unless you are using their, for example, their more advanced features. And if you want, for example, like dashboard, and if you want to currently, I think, couldn't remember, like one or two developers can visit, access that dashboard. If you want to share, for example, those advanced features with others, you may need to subscribe to Cypress. Regarding the storage, basically the user stories from Gauss messages that, for example, our POT testing, we have to keep those screenshots and the videos there. Yeah. So Cypress itself has like official and also community extensions to help so that you can have those screenshots and upload them, for example, to a three bucket if you have similar requirements from business side. So back to your first question is that why we didn't test before. So it's, for example, it's like we have a highly critical issue. So we have to release it to 48 hours. Basically, we do not have time. And as you know, we do have a really, really small team. We have about four to four developers and only one tester. Before we run this automation, we have to wait until the human tester give us the green light. Thanks. I think it is the last question. Thank you, everyone. If you have any further questions, please go to our gauss and let's both all find anyone with a shirt. Thanks.