 Oh, everyone, welcome to Automated end-to-end testing with Nightwitch. Thanks for coming for the last session of the conference. It's really great to run up. If you're in the wrong session, yeah. We're going to be talking about testing. So how many people do like testing? That's not a lot. There are a few explanations to that. So we're going to go through and I'm going to see what we can do about it. Maybe after that we're going to have a few more hands up. So just first things first, Friday Contribution Sprints. Tomorrow, please come if you never contributed. It's a lot of fun. Sometimes, sometimes not. But it's a very good process to learn and help others to contribute. Please evaluate the sessions. And that doesn't mean only this one, that all you've seen, even online, that helps a lot to build the sessions for the next Drupal console. Please do that. All right. So my name is Vladimir. I am team lead and backend developer most of my time. Many other things. This is a link to a Night Watch where I put all the slides and resources. And you can tweet about, use this handle to tweet about me. Came all the way to Australia to present. And pretty much right there on the bottom, that's where I spent most of the time doing Night Watch in the last year. Oh, let's talk about test. And I'll just talk about this session. I put it as a beginner session, because I want to basically give you a basis why do we need to test, why the testing is not something that you can exclude, put it in the box, and forget about it, and why everyone should care about it. So let's look at software development lifecycle, everyone's life diagrams. So we have requirements, coming from the client, coming internally. You want to build your startup. You have implementation. Let's say it's Drupal for no particular reason. And we actually produce the test based on requirements. So we have a set of tests. And we test our implementation against the tests. If the test pass it means the requirements are, basically we build the implementation according to the requirements. So let's create a simple requirement, right? We want to search for a surname. And we want to display a person's information, right? So to write manual tests, it's quite easy. So we go to a search engine, no particular one. We type surname into a search box. We check for the results in the right-hand side area, just because it's no particular search engine. Type particular last name. And we see on the right-hand side, you can see the details. You can do that with movies as well, and music. So let's write another method test. What are we actually going to test? It's a bit different from what we did in the requirements. So first of all, making sure that the actual page is visible, then we make sure that the search button is visible. So these are our two assertions. Next, we enter our surname, click the button. Our third assertion is that our right-hand side area is visible after we click the button. And it actually contains what we want. So that's our test. As a night watch, we're going to go through it later on. But let's run it. So basically, type it, and you can see our four assertions we just wrote. They all pass. So all happy campers, right? So what is night watch and why we're actually using it? So night watch is a common line test runner. So you run it from your common line, and it runs tests. Tests are written in JavaScript. How many people did anything in JavaScript ever? That's a bit more, so it's quite good. I think we're going to be all right. It uses CSS Selector. So if you work with a jQuery, it's pretty easy to understand that type of stuff. If not, it's not a rocket science. It actually also uses XPath for the lovers of very complicated things. It has continuous integration support, so whatever your continuous integration processes. So on the left-hand side, you can see there is code repositories. In the middle, there is all sorts of different atomic building tools. And on the right-hand side, you can see some reincarnation of containers, virtual machines, whatever you do there. And you can pretty much implement night watch anywhere there. And it also has cloud services support. So people using browser stack to check out looks on their browser. People using source labs. If not, we're going to have a look at the one of the example with the source labs in the end. And it's easy to extend because it's written in JavaScript and Node.js. So you can actually extend and create custom commands. So for example, for Drupal, there can be specific commands you can test. For example, make sure that on every page, there is no error message, something like that. That also corresponds to assertion and reporters. In terms of reporters, it gives you reports in a particular format. If you want to actually want to see them in a different format, you just write a custom reporter. So for example, it produces UXML. If you want to see it in JSON, you just write. So let's talk about reports. Who needs testing? And who likes reports? So we have our developer. Our developer created a particular functionality. On top of that, he created another particular functionality and broke the first one. But on his machine, it's running. So we have our project manager coming in, saying, is functionality too ready? What our developer says? What are you thinking about something else? Business analyst comes asking, great. Good production, because that's how we're all. And of course, he runs to the client and says, sold done. Client says, great. And then says this, your software testing is bad and you should feel bad. But let's see it from a different perspective. Let's say we have a coverage of Nightwatch for those particular two functions. Once the function two pushed, Nightwatch runs test against function one and function two, produces reports, and reports it to, let's say, everyone in the company. Just for no particular reason. It might be just developer, but let's say it reports for everyone. Of course, Client might ask is my functionality ready, NBA would ask is, but F1 is broken. So what Client would tell after that? Because we all want clients like that, right? So to summarize the reporting, why developers like reporting? So of course, it's a technical detail to report on the features and functionality that they put in. And also making sure that the integration with any other systems actually put together properly. Developers can use it Nightwatch in a sprint retrospective. And I had practice of that, whereas instead of actually going through the features of the last sprint, they implement that they just set down, click play. And the automated testing tool would just play for them. So the boss can see. So internal team, managers, PMs, BAs, we can report them less technical details. It depends on our testing structure and the issue structure, but still, for example, we don't have to send them technical details. We just send them. Here's number of tests. We've failed five. We succeeded 100. Here we go. So they already know what to say to the client, and they already know what to go to next morning meeting. You can measure velocity on that, so how your team is delivering or are they delivering the actual functionality they promised. That's a very good tool to have. And you can also integrate it with your internal tools. So Slack, I have a notification every morning coming in, test run at 5 AM and 6 AM, the test run. Have a notification to Slack, come to work, or look on my phone. So good, oh, not so good, so someone must be responsible. Clients, clients, doesn't even have to have the as detailed report as internal team, but it's always good to have, here's a piece of the software, and here's a warranty to it. We tested all those features, and they do work. And trust me, when you start the project and you produce the report on the first day, that's very, very impressive. On the other hand, client can see if something is not being properly defined. They can actually go and do a review and say, yeah, we actually need to redefine particular items and redo them later on. So our report output can be visual, so we can actually see what's going on. It can be in the command line, so we saw that thing happening with ticks and stuff. And Night Watcher also produces the report in JUnit XML. And as I said before, you can produce any sort of report. At one project, we actually produced the JSON, which was wrapped in an Angular small app. And anyone can go and tick say, show me only file tests, show me details of the file test, show me all tests and just basically quickly adjust to what you're asking for. Let's have a look at the demo. Can everyone see that, actually? What should I? Just to show you what I'm actually running. So first, I'm going to run test name. So it's basically a simple Night Watch command running in Safari and just running a single script, which basically tests what we saw before. So you can see the browser opening on the back, runs through checks. I put a bit of pause so it won't close straight away. You can see the second visual result is actually in our CLI. So four assertions passed, as we said. And for developers who really like to read the XML, we go to a reports folder. And we can see a particular test running on a different machine. So we actually run Safari. The Chrome was from previous one. So yeah, so it says errors, error, and so on and so forth. Any questions so far? So let's talk about the actual tests, how we can run them. So Night Watch test written in JavaScript. Most of you can write a JavaScript, so it shouldn't be a problem. Each test can have multiple assertions, as we saw before with that. We need to check for assertions for a particular functionality down. So here's our simple test that we wrote. So the title of the test can be anything. Don't treat it as a user story. I just put it there as an example. That's what I use. You can put test in Google. So it's a standard Node.js functionality. You define the expert function, so you can run later on. The old magic happens actually inside the function. You pass the browser object, and the browser object says go to this URL and wait for this element is visible. And time out for one second. Set value in this particular input field. This particular value. Wait for element to be visible, and then click. So this is our third assertion. We're waiting for element right-hand side block is visible. And then we just make sure that the text that we put in is there. And then we end our test. You can have a multiple functions. So you can define multiple user stories, for example. And we're gonna have a look at the example of that. So each file can contain multiple tests. So as an example, I would go to home page, and here we see the first test. Here's the second test. If you notice, there is no end on the end of the first test. So it just keeps going. If it fails, I think it skips most of them. The following one, but it will go on the next file. And each folder can have multiple files, and there can be multiple folders inside it. The folders are more like groups. So we can group tests together, for example, authenticated user functionality and authenticated user functionality. So we can group functionality accordingly. So if you look back at our example, so we can see inside our tests, we have folder Drupalcon, and inside there we have folder Dublin. So if we'll go and look at our examples, how we can group stuff. So if we look at our NPM run, there is, actually removed it, so good. So as you see, the standard example is night watch, then you specify the environment and then you specify the tests. So what if we just wanna run all skip the name and run all the Drupalcon stuff? It's pretty easy, like we just copy that. We actually have executable. And then instead of the actual file, we put .g and .drupalcon. And you can see it goes Drupalcon homepage and it tries to run a particular test and it would fail it. So you can see here is a fail report. So again, first test, three assertions run past. Second one, the second assertion failed. So it just went and it would jump on another file if there would be one. So I created the folder Dublin inside it, which is empty one. So if we'll go and say Drupalcon slash Dublin, it would say there is nothing there. As you can see, there is a Selenium running on the back and we're gonna talk about environments quite soon. So and there is ability to tag the tests. So as you saw before, I named my test as a user stories. So if, for example, manager wants to see which user story is passed for a particular sprint, I can actually tag all the issues for particular sprint as sprint four, let's say. So here name was a sprint four or project, particular project or even user. So you can put as many things as you want. And so let's say sprint four as a tag. Then we go to a common line. And instead of group, we'll put dash dash tag sprint four. So theoretically it should run all of them, but it would only run name because name has a tag sprint four and it would skip Drupalcon fingers crossed. Yes, live demos that always. So and yeah, that's what it just went through. So there's examples you can actually have multiple as many tags as you want to just make sure you work with the people and make sure the tags are sensible. And if you give developers to do the testing, make sure they put whatever you want them to put in. Otherwise it's just gonna be bunch of tags that no one uses. In fact, I would encourage you not to use tags in the first place and then see what you actually, after you probably grow out of 100 tests, then you probably would start using them. Any questions about the actual writing test and running tests? So let's talk about environment. So the first is my box. At the moment what you've seen, it's around here. I can run it on any box, Windows, Linux, and here's a stack for it. So we have our scripts. The scripts run in night watch and night watch is based on Node.js, so you need to have Node.js installed in your box. So you have a Selenium. Selenium server, it's a set of tools to actually help you automate browser testing that are quite popular and they're downloaded for free, but they're based on Java, so you need to install Java. And then we have a Selenium web driver. Selenium web driver, it's a set of executables that actually allows you to emulate in the real time the browser. So you can have any particular browser running there. So it's not really a browser, but it's basically, it's like a language to talk to your browser. So for Selenium you can even do a phantom.js. So if you did stuff before just with phantom.js, it's actually, you can include it as a part of the Selenium suite. And Selenium talks to the drivers and driver talks to the browser. So if you'll, so if I run my script, you can see the Selenium would actually respond. And once you install Selenium, it's just one JAR file. There are instructions on the Selenium website and all you need to do is just run Selenium server, dash p and then port, the standard port for Selenium was double four, double four. Here we go. So this is a local stack. So you can run, you can run it from your local box. So as I said, Selenium is a set of tools downloadable as a one JAR file. It supports many operating systems. So I runs on Java. And if you actually try to write tests directly into the driver, not through Nightwatch, it's pretty complicated. It's a big time waste. And for this particular environment in the Nightwatch.json file, you just put the configuration of your local host and the particular port as I showed before. Here's how you run the Selenium server. So Selenium web driver, it allows Selenium to run native browser agents. Firefox used to be by default because Firefox didn't require any configuration changes until last month. They decided to go with a new driver called Gecko. They didn't include it in the master yet. So if you'll run example on a Nightwatch test and your Firefox would open blank, that's the reason. In the next couple of versions, they're gonna put it inside and you need to provide a couple of more parameters in the configuration or there are instructions on Stack Overflow how to actually enable it at the moment. So they basically took over and said, we wanna run in a marionette format. I'm not gonna go deeper than that, but if you'll see the blank Firefox when you start running a Nightwatch test, that's the reason why. Safari requires, and I didn't say what it requires, it actually requires the extension. It's downloaded from Selenium website. They stopped doing it at version 248, they now have 253, but now it still works. So Chrome, again, you need to download the web driver for that, i.e. OH browsers. You can actually test them on Linux machine and phantom.js web driver. So here's how you install the Safari one and it's the easiest way at the moment to actually run the Nightwatch straight out of the box. So you download this extension from Selenium website and you put it in your Safari browser. And you basically define the path to your web drivers like that in CLI arguments of the Nightwatch.json. All right, so a lot of people would ask if I wanna put it as a part of my continuous integration, I usually would run it on a server. So no test stop, what should I do then? You still can test it. There is a tool called xvfb and all it does, it basically creates a virtual screen and you can still install, for example, in your Linux you install Firefox or you install Chrome and then you run it through xvfb. It's basically runs like a service and it's a virtual screen. You can define multiple virtual screens for different purposes. So again, just runs natively, again, any browser, just outputs the HTML and test the HTML. And my favorite one is using cloud services. So if you don't like doing all of that and install Selenium and Java and Driver and try to configure it, there is a way to do it. You go in a cloud and using Source Labs, today's example. So Source Labs is basically a service that gives you a Selenium and bunch of other stuff that you can leverage in your environment and it gives you an output back. On top of that, it also gives you the ability to test in multiple operating systems, including mobile phones. So the changes to our configuration file is we basically say instead of a local host, now it's on demand Source Labs with port 80 for test settings and Selenium settings and then we just define the variables for our username. So when you register with Source Labs, they give you username and they give you password. This is a paid service, the Source Labs. They give you the two weeks free. So let's have a look at how we run the actual Source Labs. So the actual website is here. So once you log in, you have a dashboard, you have a list of the tests that's run before. I was trying to integrate by default, you see it would give you a question mark because you have to communicate because Source Labs doesn't really know if your test pass or fail. All it does, it takes your test, it runs them through Selenium and sends you back to you. So it's up to you to write a hook and actually to update Source Labs. As you can see, I did that before. So if it failed, it would go like this and if it didn't, it'd give you a nice little check. So I have a different configuration here with Source Labs username and access key already enabled. The stuff I didn't go through is actual configuration so you can configure different things and Source Labs gives you also ability to configure the operating system in a version. So here's a good example of Chrome desktop MacOS. So you go and say, I wanna run Chrome on a MacOS 10.11 and using version of Chrome 53 and using particular resolution. Most of my tests failed because it actually started testing them on 10.24 by default. So a lot of them kinda hit the menus and did a bunch of stuff because it was a tablet. You can see other examples like A11 Safari Edge. iOS, here's the iOS example. So because it's a different Nightwatch.json, it's actually Nightwatch.json remote. What we're gonna do, we're gonna do the following commands. So a Nightwatch, we're gonna test the Chrome desktop MacOS and we use this particular configuration. We're gonna send all tests and let's see what's gonna happen. All right, so it talks to Source Labs at the moment so you can see it takes a bit more time to actually communicate, but you'll see the tests popping up once they actually went through Source Labs. At the same time, if we look at the Source Labs, we can see the test is running. You can see which operating system, which browser. You can pretty much run it overnight and have all your tests covered. It's a great little service. It has a bunch of stuff, so once the tests run, so it's still slipping. So let's give it a couple more seconds. So one test run. Oh no, it did, sorry, just missed it. So it's on top. I did exactly the same test as you can see and now it's trying to go through a name. So the good thing about Nightwatch, it allows you also to take screenshots and Source Labs also records a video for you so you can go and see what actually happened if you're not sure why that particular test fell. So our test fell and you can see here, but as I said before, Source Labs doesn't know anything about it. So if we'll go to our home page test here, so as I said, you got a video, you can run the video. Let's see what actually happened. So it actually checks for register in the L button if you didn't see the test and that's it. It actually couldn't find it, that's why it fails. And then it takes default screenshots. You can also set up which screenshots you want. Some people use in software which compares screenshots to a previous version of screenshots making sure that CSS isn't broken. So you can leverage that as well, but it's not part of the service. This allows you just to take a screenshot. You can see the logs and you can see the metadata. Any questions about Source Labs? Yeah, wanna grab the mic? Just quickly, so I mean, does it have the capability to kind of have teams so the clients could like go on and see reports without being able to kind of run tests and stuff? So team management, you can actually see per team, you can see the reports per team, who is running the particular test and so on and so forth. So yes, it does. All right, let's keep going. So one of the tests I have, whatever runner are you using for tests? For example, in one of the examples, we use code chip or any other CI tool you use. You can basically communicate with the Source Labs. They have pretty good API. Okay, so again, why not watch? Quickly, it's a common light test runner. Yesterday at one of the JavaScript testing, someone asked why do we need another test runner? I think the more the mirror, it's actually, we have a choice. It uses JavaScript, so pretty easy to write the script as you saw. Can get quite complex, but in the end, you just check in basically for particular values that try to interact with a particular functionality. So it uses CSS selectors and XPath. It has continuous integration support and it has cloud services support. Advantages, you can test any website. Doesn't have to be Drupal. So from that perspective, it's pretty good. You actually, yeah, can run any project. Node.js becomes popular. Your company does both. Fine, Django, sure. Compliments unit testing. So if you're using any particular unit testing, it's very good. D2E, it's something you can show to your boss, it's something you can show to your client in the reports that we've actually seen. It has visual artifacts, screenshots, and if you're using source labs, videos. Always handy, especially if you really rely on making sure that the brand stays visually the same. And continuous integration and services support. So disadvantages, takes time for initial setup. Especially when the Firefox fail. There was no note, so people started figuring out what's actually going on. And it's because they just remove particular functionality from Firefox. And Selenium can give you grief and I can go on and on and on and on. But at the same time, again, it's like any other tool. Basic coding, knowledge required. So, but as it uses JavaScript, that's probably one of the most popular languages there. So here are some resources. So Nightwatch, GitHub, and the website. Website has a quite sophisticated API. Selenium website and source labs website, check them out. If you wanna know more, check out Matthew's yesterday session on JavaScript unit testing. He actually goes, they're in a query using nightmare, which I found out about yesterday. So probably should have like a deathmatch or something. So they use a nightmare, but he talks more about unit testing and actual, he goes deeper into testing for front end. So, but he also covers a nightmare, which is alternative to Nightwatch. So you can check them out and see which one actually works better for you. Again, Friday contributions, Prince. If you're here, please come and please review the sessions. So I can take quite a few questions or if you wanted me to demo something, I can do that. Anyone? Yeah. Yep. So a good example was I'm running a conference in Australia and I put the Drupal 8 website together and a bootstrap theme was updated from beta to RC. They changed something to their manufacturer. So visually it looked fine. So when I updated the bootstrap to RC, compiled SAS, yeah, compiled SAS all looked fine. Until actually a week later, I started clicking on the menus and the background of the menus went gray instead of blue. So I just put it as a test. So next time I'm gonna upgrade bootstrap. It's gonna find out if actually there was a change. You can actually see on the Nightwatch website, there is other ways you can test stuff and one of them is behavior-driven development and mocha style. So BDD assertion, so you can say, for example, I want to make sure that the element body is present before one second. The element have particular CSS. The element has an attribute that has a particular value. The element is an actual particular type or element is visible, stuff like that. So yeah, you can test the CSS values and it's actually quite handy. Some people think it's an overkill, but in my case scenario they really wanna go and after each bootstrap update, check my menu background color, anything else? Yep, there is a driver for IE8. I'm not sure what sort of support if they dropped it or not. Selenium wrote most of the drivers except the Gecko one for Firefox. I think some of the companies just pick up their own support for the drivers, so you better check for a particular one. But yeah, there was IE8. I think it started with IE8. I'm not sure if there is a IE7. Okay, cool. That's a good tip. I'm actually not sure if I can check if Source Labs actually have a list of browsers. There is a way to put together the test actually saying which particular operating system you... Yeah, I need to check the docs. I guess I'm not gonna spend a bunch of time, but I will check for next time if Source Labs has IE8 support. Anyone else? Not necessarily, no, you can run Phantom with... Yeah, it just have an option. It's usually for continuous integration, so for example, if you wanna run Phantom as well as other browsers, I haven't seen that's like the case in a live situation. Some developers do because they wanna compare how Phantom natively I guess compares to Phantom and Selenium, but just I guess if you really wanna do that. Usually use Phantom when you can't access the browser, but if you have access to browsers, why would you do that? So there is an option to run that, so why not? All done? Well, thank you very much for coming. Again, please review the session. You can tweet this session as well and I put the slides up to Drupal.org and please test. Thank you very much.