 Hello everyone, all attendees of the analysis, testing and automation track of Defcon CFZ in Session Room 1. The session will be three talks. There will be a short break after two of them. And our first speaker is Rob Richardson, whose contribution is gaining confidence with Cyprus tests. And for is your spot. Hi, welcome to gaining confidence with Cyprus tests. I'm great to join you here at devconf.cz. Today we're going to look at testing and in particular how to make browser testing really powerful. Here's the part where I tell you I am definitely going to post the slides on my site tonight. I've been that person chasing the speaker and it hasn't worked out well for me either. You can go to robrich.org. Let's head there now. Click on presentations here at the top. And here are the slides and the code they are online right now. While we're here on robrich.org, let's click on about me and we'll see some of the things that I've done recently. I'm a Microsoft developer advocate. I'm a Microsoft MVP, a Cyro developer advocate, a Docker captain and a friend of Redgate. AZ GiveCamp is really fun. AZ GiveCamp brings volunteer developers together with charities to build free software. We start building software Friday after work. Sunday afternoon, we deliver the completed software to charities. Sleep is optional, caffeine provided. If you're in Phoenix, come join us for the next AZ GiveCamp. Or if you'd like a GiveCamp here or wherever you're connecting from, give me up on email or Twitter and let's get a GiveCamp in your neighborhood too. Some of the other things that I've done, I worked on the Gulp project in version 2 and version 3 as a core contributor. That was a lot of fun. And one of the things I'm most proud of, I replied to a .NET ROCKS podcast episode. They read my comments on the air and they sent me a mug. So there's my claim to fame by coveted .NET ROCKS mug. So I'm not quite sure how this demo is going to go. Let's go take this out for a spin and see if it works. Let's fire up Cyprus. And let's see if we can run some tests. Cyprus is firing up. This is great. I can hear my fans kick up. That's awesome. Okay, so let's take this for a spin and see if the site is working. If it works, then I'm pretty confident that this project will work out really well. Let's fire this up and run some tests. A quick minute for it to compile. And now we'll kick off into our website. Yep, I'm getting a lot of green. It looks like the website is running. This is good. I think this talk will work great. Yes, excellent. Cool. So what did we just see? Cyprus is browser-based functional tests. Or said differently, Cyprus is end-to-end tests that run in a browser. It's an electron app and a browser plugin. It's based on top of Mocha and Chai. So if you're familiar with Mocha and Chai or you've used Jasmine or Jest, then you'll feel right at home inside Cyprus. Like many good stories, we started in the middle. Let's back up to the beginning and start talking about browser testing. When we take a look at browser testing, there's lots of different things that we can test and lots of different tools that we would use to test them. So, for example, we might test from here down, an end-to-end test. We might test this API. Maybe it's a REST or a GraphQL API. We might test this service. And by service, I mean this small unit of work. So no UI interactions in this case. We could call this a unit test. We might test this component. Now we have to mount this component, but we're still not getting user interactivity. We're just kind of clicking buttons and validating this unit works. And we can test that all of this works in a particular browser. So we'll probably use different tools for each of these tests. If we're doing unit tests, we might use Mocha with Chai or Jest. If we're testing that our unit tests work in a particular browser, we'll probably use Karma. If we want to mount components and unit test the components, we'll probably use TestUtils. TestUtils has a version for Angular, React, and View. And so once I plug in TestUtils, I can now run those tests in really interesting ways. If I want to do API tests, API tests, well, I don't need a browser at all. I can just fire requests at an API and validate the response codes that I get. I'd probably use SuperTest. For end-to-end tests, I could use Selenium or Cypress, TestCafe or PlayWrite. Let's double-click into end-to-end tests. Now the cool part is, as you grab the slides from robrich.org, any of these blue links are clickable and you can get to the actual project and learn more about that project. So let's double-click into end-to-end tests. Selenium. Selenium is the big dog here and has kind of invented this end-to-end test mechanism. Selenium is a Java app, and you can connect to it from various languages and various IDEs. There's also some recorders that allow you to record Selenium tests. On the downside, Selenium is kind of hands-off. It just runs browser Win32 events. It doesn't even interact really with the DOM very much. And so the tests are slow and brittle. Did the test time out waiting for the API to finish? Well, let's increase the timeout. Now they're slow. Let's try rerunning the tests. Now they're brittle. Yeah, Selenium is great at establishing this, but it doesn't create the best tests. Next up, we can take a look at Cypress. Cypress is an electron IDE. It has easy debugging tools. It has kind of a jQuery feel, which may not be great. And because it's deeply integrated into the browser, it only supports the browsers that are built, so no Safari. It is great at taking videos and screenshots as your test runs. So if you get a failure, you can watch the video of how it played, even if it ran headless. Test Cafe. If you're using DevExpress, Test Cafe may be right at home. Test Cafe uses CSS selectors, as opposed to jQuery-like syntax. And so it might feel a little bit more at home. But Test Cafe has a really odd assertion syntax. It isn't like Chai or Jest, so that may be another hurdle. With Test Cafe, you also spend a lot of time marshalling content between the browser page and the test harness. So you'll end up creating these methods that go grab content and send it back. Next up, Playwright. Playwright was built by the people who built Puppeteer, and so it feels a lot like Puppeteer. If you're familiar with Puppeteer, you'll feel right at home with Playwright. On the upside, there are lots of different languages supported, like JavaScript, TypeScript, Python, C-Sharp, Go, and others. On the downside, it feels like the assertion syntax was kind of bolted on to Puppeteer. So it's a little weird. You also spend a lot of time marshalling content between the browser and the test, much like you would with a Puppeteer experience. So any tool here will accomplish a particular job, and ultimately which browser automation tool you choose is definitely up to your field. But let's take a look at Cypress and how that works. So we start off by doing an npm install. Once we've got Cypress installed, we can say npx-cypress-open, and the first time it opens, it will actually scaffold out a bunch of example tests, which is excellent. Let's go dig into that IDE. Here's the Cypress IDE, and I can see the example test that it created for me. Now that's great. I flip them over to TypeScript, but by default it will create them as JavaScript. Here in the IDE, I can pick a particular test that I want to run, or I can run all tests. I can also choose which browser I want. It will discover all the browsers that I have installed on my machine, and Electron is the one that is built into it. It's a WebKit-based browser. So if I want to use, if I don't have any browsers installed on this machine, I can definitely test with Electron. So once I've chosen my browser, I can just then pop open a test and away it goes. Now in this case, it's going to run through each of the tests, but what we can see is that it's running in a different version of Chrome. Here's the version of Chrome that we had doing this part. Here's the version of Chrome that is running our tests. And the interesting thing that we can see is that the window doesn't need to be on top to be able to run the tests. So where Selenium is actually moving the mouse and clicking on things, Cypress is instead controlling DOM events. It's firing up JavaScript. So that's really cool. It can listen into the JavaScript events happening in the page, the DOM ready event, for example. And so whenever the API finishes, it can continue on with the test. It doesn't need to wait a second or two seconds. It waits until the server is done or until the page is loaded. Once we've then got all of our content, all of our tests run, we can now take a look at the details of each test and see the page at that particular spot. So for example, if I'm going to type in this box, let's actually push a button. Where's a button? There's a button. We can see that red dot saying where it clicked. It just happened to pick the middle in this case. And the cool part is that this is just a browser. So if we open up the browser developer tools, we can see in the console all of the results at that particular step. So as we cruise up and down through our results, we can see different results inside of our console. And even better, because it's a browser developer tools, we can pop open here and we can take a look at our tests. Not there. My app. It's not going to help me find them. I can set breakpoints inside my tests and just watch my tests run. Maybe I need to start off with my default page. Nope. It's not going to help me find the test today. So I can just click on a particular script and I can set breakpoints and then walk through my test. Also in the IDE, we have this selector. So I can open the selector and go pick something and it will help me build the CSS selector that I need to be able to get there. Now we'll talk about best practices in building selectors, but it's nice that Cypress has this tool to help me pick it. I can choose to rerun my tests and I see the number of successes and failures. That's cool. So the Cypress IDE is able to run really elegant tests. Now let's take a look at the code that we used to build this. We'll start with a hello world example and kind of build up through it. We start with a Cypress.JSON. This configuration file gets us set up about how we can interact with all of the other pieces. So here's Cypress.JSON at the root of my project and this code is available up on GitHub. So there's that Cypress.JSON. The Cypress.JSON explains where the plugins file is and that's where the rest of my configuration is. In this case I chose to put it inside the test folder, inside the Cypress folder, inside the plugins folder. Here's that index.ts. Now this then explains where all of the rest of the details of my tests are. Where is my fixtures folder? Where is my integration folder? That's where all my tests are. Here they are. Where's my screenshots folder and videos folder? In this case I chose them to put them in the results folder so that I can exclude the results folder from my repository and now all of these results will end up in my build output. But it's cool to see that it will also record videos of each of these tests the last time it ran and so I can watch that video to see how the test turned out. Next up the support file and we'll come back to that one but here's that file right here. Now in this case I chose to make them typescript files and so I have a tsconfig associated with my application. Now this tsconfig knows nothing of Cypress. You can see there's no Cypress content in here but I do have a tsconfig inside the test slash Cypress folder that builds on top of that so it extends that previous tsconfig and it adds the hooks that I need to be able to run Cypress. Perfect. Now once I've got typescript configured everything from here on can be typescript. Now I went into package.json and I rigged up some scripts so instead of saying npxcypress open I can just say npmrun cy.open and npmrun cy.run which will run Cypress run. Open pops open the IDE like we see here and run runs headless for use in a CI environment. I can choose to specifically call out that it's headless and I can specify a particular browser in this case Chrome Firefox and Edge so now I have three different mechanisms for running three different tests running all of my tests in three different browsers. So npm test then we'll run this one and it will run this one and it will run this one and if all the tests pass in all of the browsers then I'll get test results inside that results folder and I can pull those in with JUnit. Now I've specifically called out that I'm configuring my results to be in JUnit format and here's the XML the file name for that format so that now I can pull those in as build assets. So now that I've got everything configured in my project let's take a look at the tests that we just ran. This is to do MBC and to do MBC is built on a project where it's a bit dated now where we can see various implementations in various client side frameworks. So a standard HTML file, a standard CSS file and then go build the JavaScript in your framework of choice. Well if they all pretty much work the same we ought to be able to write tests that work the same as well. So I've got one in Angular, one in Backbone and View and I can just flip the site URL. Now I'm going to start off by visiting the site. I could do this at the top of each test but I'll do it in one place just so I know that by the time my test loads then I've got that content ready to go. And our Hello World test where I just go validate that that site is as expected. When I was first writing this talk it was an HTTP URL and of course it flipped it over to HTTPS once I started building it so this test helped me discover that they had changed the URL of this external site to HTTPS and so that was great. Now let's go to my.URL. You notice that I'm not awaiting for the page to load it's loaded by the time I get to the next line. That's great. So let's start interacting with the page. I'm going to use the CSS selector to go look for LIs in this to-do list. Now I haven't added any so it should not exist. We can see the jQuery syntax here where all else being equal I'd rather have kind of a Karma or Jest syntax should not dot exist but yeah it's jQuery field. That's fine. We can now see the arrange act assert mechanism here. We'll write a test and then we'll type that and then push enter. We can see that this is a replacement expression but this enter is not. That's a specific keyword that Cypress understands to be able to type that enter key. We could type other keys like function keys, shifts, num lock and be able to interact with that so I'll go select this to-do and I'll type that. Now let's go validate that we've got that in the to-do list and so we'll go validate that the box is empty and that the list contains this to-do. Leveling up again, let's go finish the test. So we'll create an irrelevant to-do. We'll create a new to-do and then let's go find that new to-do and click the toggle button. That will mark it as completed. Then because it's completed it will have this completed CSS class and so we can go check for that to validate that it is completed. Let's delete it to-do. Now in this case the delete button only shows up when we hover over the to-do item. We're just running JavaScript events. We're not actually moving the mouse so CSS selectors like hover don't actually work in this case. In this case we have to say click force is true. We're going to say force is true so that we can click that button when it's not visible. And now once we've created these to-dos we click it and it should have length one and we'll validate that it's only the irrelevant to-do that is left. Now we've been doing a whole lot of get a new to-do and then type. That's not working out great. It's not very descriptive. It's also kind of leaking the implementation details. If you've ever used the page object in Selenium we're about to do something similar. I want to create this command this to-do add. So here in support we have commands and here's this command where I created this to-do add that we'll do that. And we're right here. Let's just go validate that after I entered it its value is blank. Now we do need to do a little bit to get this to work in TypeScript. So here's the Cypress docs on how we do it in TypeScript. Not only do we do the command but we also need to create an interface. So I've done that here. Here's the commands.d.ts and the interface for how to pull off the command. So now I can just say .to-do add. That's much more descriptive. Great! So I'll create a bunch of to-dos. I have a complete method as well and then I'll go click the active button to validate that I now only have the two irrelevant to-dos. Not the one that I completed. We'll show only completed tasks in a similar way and then clear completed tasks to finish out that list. That was great! We were able to level up from starting off with Cypress all the way through some pretty interactive things in the page. Now let's level up again and take a look at another set of tests. Let's look at Hacker News PWA. Now Hacker News PWA is based on Hacker News and we can see here this Hacker News site that allows us to build a progressive web app in various things. It's kind of the spiritual successor to to-do MVC. Now in this case it's actually pulling data from Hacker News. Well how would I write a unit test that validated that I rendered this part correctly if I'm just getting random data from an API? Well in this case we can see that our tests are loaded and we'll go pull data from Hacker News but we're also going to mock the data in Hacker News so that we can create some artificial results as well. It looks like these tests passed so let's go look at the code. Here in the code we can see that there we go. We have a similar mechanism where we can flip between various implementations. We'll load the site and validate that we're on the correct site. Let's go validate that we have a specific top story. Now if we're getting raw data from Hacker News we don't know what today's top story is so instead let's intercept this web request. We could use a string in this case I used a regular expression and replace it with a fixture. Here's that fixture the Hacker News fixture and I know that this is the top story so the title is this is the first top story. So now I can go intercept that web request and whenever my app calls out to that API it will send it with this result. I can go visit that site and now I know that my title is a deterministic result. I've mocked out that web request. Now it would be easy for us to lie to ourselves and believe that the API will always reply that way. So here's one where we don't intercept it. Instead we load the real Hacker News I don't know what any of the stories are but I can assume that there are at least 30 stories and that's how my app pages so I'll validate that I have 30 stories in my list. Now that was great. We saw an elegant way to be able to validate not only local apps but also calling into other web requests. Let's quickly run through a few best practices in our last two minutes. When we take a look at selectors we want to focus in on an ID or a class. If we have for example a really long selection path then if we modify our DOM at all maybe wrap things in visual elements then our tests will break. So instead focus on an ID or a class to a particular element or even better create a data-cy attribute on your element and then in your test you can say get the thing that has this data-cy attribute and validate it's here. This also really elegantly documents that this is covered by a test so if I'm modifying some HTML that has a data-cy element attribute I know that I need to modify a test too. Next up we want to create commands for frequent tasks. This makes our tests more descriptive because it's describing the business processes not the technical implementation. Click here on the slides from robrich.org you can learn more about commands and typescript. Next up don't log in to your UI every time. Instead log in once to validate that your works as expected and then do a before all that will go grab tokens for each of the users that you need to run your test. We do need to validate that the login works but not before every test. It'll make our tests run a lot faster. Next we want to use fixtures to be able to mock out data. Now if we have a big block of JSON in our test that's a great example of something we should move to a fixture to make our tests more terse and legible. In this case we used a fixture to be able to replace a network request. So here's a network request and we can replace that content with a fixture to be able to validate that our UI renders the way we expect. But don't lie to ourselves and assume the API will definitely reply that way so also do some tests where we don't mock out the API results. Cypress is a great place for us to build browser automation tests. You can grab the code for this session at robrich.org and these slides are available on robrich.org as well. If you're watching this video later hit me up on Twitter at robunderscorerich. For those who are here live in the conference what are your questions? What are your thoughts? Hey Rob, thanks for the talk. Unfortunately you ended just right on top of the end of the slot allocated so we don't have space for questions and answers. Are you planning to hang out on the WorkAdventure for people to reach out to you maybe? Definitely. And I'm on Twitter as well. Perfect. So if anyone has questions on the talk reach out to Rob on Twitter or on WorkAdventure. Thanks a lot Rob. It was a very interesting talk. I spent some time developing Selenium test fees as well so it was enlightening for me and yeah that's it. Thanks a lot. Sounds great. Thanks everyone.