 All right, sweet. Hi, I'm Dan Gurren. You can tweet me at D. Gurren and just about everywhere else on the internet. I'm Dan Gurren. I'm a Drupal Camp Asheville organizer. That's Asheville with an A. It's in western North Carolina, just over the mountains. And you can join us this summer, July 13 through 15. We're doing a science summit this year and we'd love for you to join us if you'd like. By day, I'm an engineer at Civic Actions. It's a digital agency that focuses on government, open and agile digital services, all while maintaining balance. Yeah, so I don't want to have to come in on Saturday and I don't want to have to come in on Sunday either. No, it's not good for our balance. When I began working in Agile, it occurred to me that we weren't so much agreeing upon what to work on as what not to work on. Test-driven development helps us clarify that definition of done with a simple yes or no answer. Throughout our development process, tests will initially fail up until the point where they pass and then we know that we're done. So we make only what is important without unnecessary bells, whistles and other stuff. Our tendency is to test less, if at all. While it's difficult sometimes to conceive a test for something that does not yet exist, it's a great exercise and once you get used to it, you might not be able to develop without it. When we test first, we can achieve greater balance. With all the demands on our time, energy and focus, it's essential that we maintain this balance as it allows us to deliver consistent quality over time. When we know that our automated tests are passing, we have the confidence that the job is done and we can get on with the rest of the things in our life, or maybe just more programming. We know we're ready to move on when we see green on our screen. So how do we get there? Let's look at a few common tools used to test Drupal websites and applications. Visual, behavioral and unit. Visual is a great way to quickly get wide test coverage. Behavioral works well for site building activities and unit tests help ensure that our custom code is doing what we think it's doing. Let's say we've got a new site, we have some design comps or maybe an existing site and want to establish a baseline for how that site should look. We can use these images as starting points. Visual regression testing makes it easy to implement wide coverage without much thought. When we update core and contrived, we don't want to see any changes. With custom development, we might expect to see changes for some or all pages. Visual regression testing works by taking snapshots before and after changes and comparing them. It can be hard to see these differences with the human eye, but the comparison makes it easy to see even small differences which can have cascading effects as we progress down the page. So how does it work? Magic or science? The intersection over union or IOU allows us to set thresholds for how different we want to allow the before and after images to be. The IOU score is a scale that goes from one down to zero with one being equivalent and zero having the least possible amount of intersection. We can target our comparisons to Drupal theme regions and other parts of the page using CSS selectors. Behavior driven development is a great way to test our site building activities and business expectations. The B-HAT Drupal extension is the way to go if you want to use B-HAT with Drupal. B-HAT uses Gherkins syntax, which serves as both documentation and automated tests. Each line in B-HAT is a step. The Drupal extension helps us define functionality by providing out-of-the-box steps that we can use all the time, like logging into Drupal. This extension can itself be extended with contrib and custom modules and themes. Features in user story format give a high-level overview of what will be contained in each dot feature file. Scenarios are collections of steps, like user acceptance tests, but can also be automated. There can be multiple scenarios per feature. A keyword begins each line. Keywords include given, when, then, but, and, and. All keywords are technically interchangeable, but for human purposes, should be used in a sensible order. When you go to write your first scenario, and it doesn't work, try adding the at API on the line preceding the scenario to tag in the Drupal or Drush driver. You can also add the at JavaScript tag to have a scenario run with JavaScript. Groups are a great way to isolate types of functionality and run only those tests if you're looking to isolate application features, infrastructure, or some other subset of the entire test suite. This can save both time and energy when working on certain areas of a code base. Books can be used to run at certain times like before or after Drupal processes, scenarios, or even be have features. Black box is the default driver. It's for tests that require no authentication or access, such as with anonymous users. No tag is needed to use the black box driver. The Drupal driver contains the bulk of step definitions we regularly use, like logging in and operating on entities. The Drush driver, in addition to the ability to run Drush commands, of course, has the advantage of being able to run remotely. JavaScript facilitates tests in a browser similar to what we typically use on our mobile and desktop devices. Now before we get too far, it's good to know a little bit about the BHAD configuration file. It's a YAML file that defines profiles, which define extensions like our BHAD extension. And here we see a default profile where we define our driver using the API driver key. To use the Drush driver, it's best to set up SSH with keys to avoid needing to type a password. Configure Drush aliases in a profile within the Drupal extension to use them remotely. The step definition is, I run Drush followed by the command. To run BHAD tests with different profiles, use the P flag to specify which profile you'd like to use. One thing to look out for is that the machine name for node titles is automatically filled in by JavaScript in Drupal 8. If you hit the enter key too fast after typing a node title, you'll see in your browser that JavaScript doesn't have time to create the machine name. When this happens, the browser validation fails and shows the required machine name field is not complete. When BHAD runs without JavaScript, however, this fails silently. It took me a while to figure this one out, so hopefully this can save you some time if you run into this situation. If you want to discover similar situations, you can use Google Chrome dev tools to toggle JavaScript on and off and simulate in your browser how a Drupal site or app will function during a BHAD test that does not have JavaScript running. In Google Chrome dev tools, you can just click the contextual menu and then settings and then check the disabled JavaScript checkbox at the bottom of the settings. You might have to scroll down. There's a lot of settings. They're adding them all the time. There are no multi-line comments, just the hash sign to comment out anything following it on a given line. If you have a fancy editor, you'll likely be able to comment out multiple lines by selecting them, and then you can use the keyboard shortcut combination of control or command on Apple computers plus slash. All right. Let's say we have a handful of scenarios that all have the same entities. Tables are a great way to create multiple entities in a background. Now, these are two different concepts, but we'll start with our feature out with a background, which will be run before every scenario to prepare the content. We use piped to limited tables to add field values. The first row in each table is going to be the machine name, and all subsequent rows will create entities with the specified field values. We can also use tables for multiple assertions, like the example here tests for multiple roles names. The Drupal context is included with the behat Drupal extension and the raw context is more the backend glue that allows Drupal context to easily provide these step definitions for core entity types. Mink helps us mostly with the theming layer and markup works with HTML tags, not to be confused with Drupal or behat tags, as well as classes and attributes. The message context refers to Drupal messages, and the drush context allows us to use the drush command line interface. Sometimes we'll need to test something that isn't already defined for us, and we can create our own step definitions in the custom context class, which is in the feature context PHP file. We can get started by writing a step we'd like to see in our behat test. When we run the tests, it will fail and generate a snippet with an exception that we can copy. We'll replace that exception with some PHP code of our own. If we don't replace the exception and we just copy and paste it in, PHP will throw that exception when we run the test. It's a good way to see if the custom context is even being run at all. But in normal practice, we'll delete the exception and add our code, which is a combination of PHP, behat, mink, and Drupal programming conventions. We'll then test it again to see if it works. If we need to add a variable into our step definition, like a web form ID, we'll put a colon before the argument variable in the annotation to the function, and then we'll use the variable as the argument to and within the function. In step definitions, we always put variables in quotes to distinguish them from the step definition, and quotes can be single or double, depending on your needs. Here's how to make behat get and visit pages. Use the get session container object from mink to capture and visit paths within a site. In this example, we get the URL we're on, then append slash edit to visit the edit form for a typical entity. Sometimes we'll create nodes or users and won't know which entity ID they will be assigned. Instead of using path auto, which we may not want, if we need to go to a previously visited URL, we can create, save, and visit URL functions using this to reference the current object. This allows us to store URL as a property of the object. So it can later be used in a different function that is called in a subsequent step. This works great if an entity is created and then needs to be visited later, perhaps when logged in as a different user, when path auto isn't installed. Otherwise, it can be difficult or impossible to know which entity ID will be in the path. If you're tested and passed it first, and it probably didn't, here are some things you can do to figure out why or why not. There are steps that can output the URL or response. Print R can be placed within a context to reveal the actual value of variable, not necessarily what we think the value of that variable is. Behat can be configured to take screenshots and output source code for pages that fail, which is when we most need to know what's happening. It can be a real time saver to run a feature starting from a certain line in a file. This can be accomplished with a colon followed by the line number where we want to start our test. The background steps will still run. Behat starts from a defined line. It's possible to add sub contexts in contrib projects. There's good documentation for this in the Behat Drupal extension docs for modules and themes if you're contributing. And if you write reusable step definitions, please feel free to share them on GitHub. Alternatively, you can create an issue in the Drupal extension project in Behat or even Mink. And not surprisingly, these projects do require that you write automated tests for their features. It makes more sense to use unit tests when custom code is being written because core and contrib usually already have these tests. So unit tests are typically faster to run and more targeted than behavior driven tests. Drupal 8 ships with four PHP test suites. Give you a little time to soak this page in. There's a lot on there. So the suites are unit, kernel, functional, and JavaScript. They progressively increase in scope. Starting with the unit test, it's fast and the most granular. You actually don't even need to run Drupal to run unit tests. The kernel suite runs an environment similar to the Drupal install screen. No modules are installed and there's no configuration. The kernel suite is great for testing APIs and decoupled backends. That's kind of a big thing now. The functional suite is similar to what you would experience in your browser except without JavaScript on. The functional suite is fully booted. You can visit Drupal pass. It's web-based and actually somewhat similar to the behavioral B-hat testing. Now the JavaScript suite uses phantom.js to simulate JavaScript in the browser. When you're deciding which suite to use, less is really more here. So don't use JavaScript boot or bootstrap if you don't have to. Here's a very simple unit test. What it looks like. We'll extend the unit test case class with a class that must end in test. And any public function name that starts with test will be tested. So the test we see here should pass as long as 1 equals 1. Some of the common assertions include equals which compare different instances of objects to see if they contain similar values. Same as like comparing with two equal signs so the objects would need to be the same instance in addition to having the same value to assert true with same. With equals the objects could be different instances and still assert true. An exhaustive list of available assertions is available in the PHP unit documentation which is linked here and I'll share some updated slides when the presentation is over here. The kernel secret recipe for testing an API or decoupled backend is a kernel test. And since this suite operates in a bootstrapped environment much like the Drupal install screen, we need to recreate any environmental conditions first that will be necessary to run our test. Before we do anything we probably need to enable some modules and this holds true for kernel tests and functional tests as well both with and without JavaScript. The modules variable is an array of module machine names and it handles installing those modules in the PHP unit test environment. The setup function comes next in the class. It'll run before the test functions similar to the background that we had in B hat tests. Setting up the parent inherits all of the configuration from the kernel test base. Once that parent is set up we can install any additional database schema and configuration necessary for the subsequent testing. Here's what a setup might look like. We can see the parent setup first and then the database schema and then the configuration. So the install schema allows us to install a select list of specific database tables for a module whereas the install entity schema installs all of the database tables for a module and takes an entity type ID. Install config installs the configuration management YAML files for an array of modules. We might need additional configuration beyond what is in the module files. Maybe we need to install some fields it's a typical scenario. So to do this we'll get and parse the YAML from configuration then create and save the config. We need to install the storage first so we have somewhere to put the field and note that for the field config we need to reference the bundle and configuration management to get that specific instance of the field. Field installation is a great opportunity for a reusable trait we'll talk about a little bit later. Often we need to go beyond just APIs and decoupled backends. We need an end-to-end test for a website with actual pages. For this we can use the functional test suite and while it is usually slower than the kernel suite we usually don't need to do as much config and we can visit paths. For functional browser tests we'll typically use methods within the web assert object class and that's a link to the documentation there. Do you want to check out the slides afterwards? I'm not going to click it now. A couple of those methods are status code equals and link exists. Here's an example of a little functional test that extends the browser test base class. It basically says we're going to visit the home page since the path is empty and the HTTP status code equals 200 also known as OK. We also see that the link to the blog exists and note that we're looking at the human readable label for the menu not the machine name since the functional test is looking at the browser. Sometimes we need to test JavaScript or just have that functionality there to support what we're testing. We don't want to have that situation like when we only entered the entity label and JavaScript didn't auto complete that machine name for us. An extension of the browser test base the JavaScript suite uses phantom js driver to simulate a functional environment with JavaScript and here are some typically used functions. Execute script returns a value which is usually jQuery. Evaluate scripts does not return a value. Key down is when a key is pressed on the keyboard when we need to test conditions with a specific browser with we can use the resize window method. Now if a picture is worth a thousand words then create screenshot method is worth a lot of words. Give it a file name and it will save a JPEG for us so we can see what's happening from the test perspective. In JavaScript test base get Drupal settings still parses configuration values into an array just like in functional tests but here it is after JavaScript changes to those settings have been made. The JavaScript test base also includes some special assertions. Some common ones are assert js condition which will wait for 10 seconds for the js condition to become true. So you might not get the results you expect if your script takes longer than 10 seconds to complete but that's kind of too long so you should make it complete faster. There are a lot of assert element visible and not visible even though those are deprecated so those will be moving to mink as node element is visible and not visible. That could be a good opportunity for contribution maybe if you go to some sprints on Friday and you need something easy to tackle just kind of changing some of that deprecation stuff can be a good way to break into doing contrib work and there'll be mentors around to help out with that. Assert session can be used to do more with JavaScript can wait for events such as Ajax button an element to be visible field or autocomplete. JavaScript can also assert that a response contains a value or the visibility of an element on a page like this picture. When we need to test something that isn't really there we don't necessarily need to recreate the whole thing. We can create a stub to return a simple value. For an object test stubble we can use a mock. This will of course have expectations about how it is called. You can use get mock builder to specify a class such as the web form entity in this example. We'll disable the original constructor first so that the full object is not created. We then will specify only those methods that we need and then get the mock. The matcher in this example is once. The matcher can also be any, zero or more times. It can also be at least once or a specific number of times. This example will return a mock object with a web form ID of a test web form. To better organize our tests we can create base classes and extend them so we don't duplicate code in many classes. Base classes are fantastic for common setup items such as enabling modules, installing database schemas, configurations and fields. Mox can be placed in base classes as well. There are some outstanding examples of base classes in the JSON API. Contrib module if you want to check that out. Traits aren't really specific to unit testing. It's just object-oriented PHP. They can be used in situations like we had with the installing of fields example that we saw earlier. We might want to install many fields in our tests and just passing the field name to a trait in any of our classes significantly simplifies that process. Another trait is a reflection trait. This allows us to test otherwise untestable protected and private methods. This is done by invoking a method and passing an object, a method name and an array of parameters. We set the method to accessible then return the result of that method when the object and parameters are passed to it. Here's an example where we invoke method with an object, method name and the parameters. Data providers. These allow us to run a single test with multiple sets of data. In this example the first data set would pass as one plus two equals three. The second would also pass because four plus five equals nine. The third and final set however would fail because seven plus eight is not equal to nine. We can use associative arrays as data providers and this will make failure messages be included. The array key will be included in the failure message. So in this case the failure message will indicate that the third and final data set failed. False is the key to that array that has to seven plus eight equals nine, which of course it does not. So debugging in PHP unit, print R is your friend except when there's nothing in the variable and it just doesn't print anything when it's empty. So sometimes it might suggest print R in a string in front of the variable to show something just so you can see where it is or isn't or if you have multiple print R's then you can see which one is which and you get to a certain number it starts to get kind of confusing. You can also do screen shots and HTML output and those reports can be configured in your PHP unit XML file. The messages will give you a line number and often great clues as to what went wrong in your test. All right, so we've got our tests all written and we want to run them. There's a PHP unit executable with a lot of options and arguments. If you use aquia build and launch tool that has a command as well, continuous integration can ideally be configured to run your PHP unit tests and when we're testing on a local we can use the at group annotation to only test what we're working on currently. One way to do that. Note that if you do choose to use the print R function for debugging purposes that it will report as an error so if you're just seeing that one error when you take out your debugging you'll actually be fine. Other results other than pass, error and fail are risky, skipped and incomplete but those are less common. You'll usually see pass, error or fail which will be the dot, the E or the F. So our tendency is only to test success but we should also test for failure where appropriate and so we should envision not only what should happen but also what should not happen and test for those scenarios. So when we expect a failure in PHP we expect an exception and the expect exception method will have a parentheses after it and the annotation will have an at symbol in front of it. We can expect code, a message or a regular expression to match a message with a pattern. All right so now that we have our test suite and we're feeling balanced can open it up for discussion just please step up to the microphone if you'd like to participate. Hi thanks and sorry to take you all the way back to the beginning of the presentation but can you talk a little bit about how to get started with visual regression testing now that WebDriver CSS is no longer supported? I mean there are some other tools out there. WebDriver CSS isn't supported but you can still use it. There are some other options out there. We prefer to use open source when possible so we tend to shy away from those commercial products but they certainly are out there. I'm not really sure if anyone knows of some of the other follow-up if there's like a some other open source options that are out there other than WebDriver CSS. But you're still using WebDriver CSS? I mean does that mean you like my understanding of this is based on googling it from my seat after you moved on right but it sounded like WebDriver IO has like progressed and the most and the WebDriver CSS isn't compatible with those versions so do you have to just use an older version of WebDriver IO underneath? Yeah I mean the version that we have is on a slightly ancient site you know from like a year or two ago so I think it was like 2016 was kind of the heyday of WebDriver CSS but yeah if people have other recommendations or you know you can step up to the mic or feel free to comment on the on the node in the for the session on the Drupalcon site. I would like to know also. Okay thanks. Yeah any other questions or comments? Hi thanks for the talk. So if you are building a distribution how would you group B-head tasks like how would you organize them? Would you put them all in a single module or do you have a B-head task in each of the module in distribution? Pretty good question there. I mean there are a number of ways to organize it so typically as far as just the files themselves the way I've seen it done is it'll be there'll be like a separate file for B-hat and within there will be a features directory and you put all of your features in there and then you can name different features for your different features so you know you could you could break this out by sometimes we've done it like by role so you know like administrator or you know editor would have a feature you know it could be broken out by content type there's you know lots of ways you can slice it right you can have like a general just for like you know system-wide config stuff like that do you have do you have a way that you like to split it up or not really we're still trying to figure out where we should put it though like um do we put in a single place or do we want to put in multiple place depends on the functionality of features oh right I see what you're saying like to like if you wanted to break them out into the modules right okay and you and it's a distribution right so you hmm that's a tough question I would say you know if there are separate modules and you were able to to separate the functionality of those modules that would be better because you would possibly use those modules not in the distribution right correct so I mean that would be like the best way to do it if I might be so bold as to suggest that okay right thanks one more question now how do you normally group your tasks like what are your recommendations do you use in tag or do you use in suite I think I've seen tag used more okay yeah just because you know the suite is going to be more just kind of on what level you are if you're on like unit or kernel right okay and then yeah I think we we tend to use the the tags more so okay thank you yeah sorry one last question so when you showed the screenshot of the one test suite and eight tests pasts what was defining that what was containing the suite what defined the suite of of eight tests that were in one suite as right oh I think I think that might have actually been a javascript screenshot that I so I didn't actually have that in the talk but you know most of the most of the tests when they pass they'll have like you know some sort of so it could be just like a b-hat suite of okay but right yeah but is there anything that would encapsulate visual b-hat functional testing in one suite or is that just something like travis would just return past do you want to step up to the mic just for the recording sure we get some snapshots in circle so we're actually using night watch there's a patch that selly and daniel wrote for eight five and it's going to go into eight six and circle will actually give you artifacts that you can look at if a test fails you can see what what broke yeah yeah yeah it takes a screenshot for you it's just doc yeah night watch is going to go into core it's faster than b-hat um but yeah there's a patch for it and if you integrate with circle you can usually have one of those ci tools take screenshots for you yeah it's worth a look I was wondering if that would actually be a reasonable replacement given the last answer and it may be night night watch is the testing framework night watch js um and you can find uh stuff in the core issue q around it too but circle and travis may do the same thing yeah I actually presented today on uh basics of night watch as well okay and my my question is today did you find that the test of set of tools that comes with Drupal is actually pretty redundant in terms of writing test takes long long time and I'm talking about like more than 30 percent and when we're talking actually selling the test during development to a client never works so uh the actual question is did you find any other tools that are actually more faster or more you know straightforward like for example night watch I found I can I can get a front end developers to put them on writing tests for it whereas I can't put front end developer to writing tests for a minko behalf right yeah I mean it's definitely like the fastest easiest way to get the most coverage is to go visual right and and so it sounds like night watch is the way to do that now so um yeah okay okay so we're still working on that um yeah I mean to ask a front end developer to I mean the PHP test is probably absurd but I mean I just even in regards to end to end testing so if we'll just take for example end to end so something that follows the user pass and not talking about unit tests but some end to end do do can you recommend any other tools outside of Drupal box that you found along the way um I mean these are pretty much the tools that we've been using um you know I I think there is some redundancy with uh like B-hat tests and then the functional PHP tests right so it's like yeah yeah so I mean I think it's I mean it just kind of gives you some options right so if you if you want to do you know the super fast uh visual regression you can do that or or maybe even just the type like it's because it's a little bit of a different test there right um but yeah and then I guess it's kind of like just pick your preference if you'd rather write in PHP unit or gherkin you know maybe maybe you'd prefer gherkin so you would also have the user acceptance criteria um you know I'm sure a project manager would hate to read a PHP unit test thanks yeah thank you all right anything else okay okay thanks um I mean we use all the testing environments but but I mean all the different hosting environments out there um we'll do um I mean sometimes we'll have circles CI running it um yeah I mean we have some sites on Pantheon some on Aquio um I think that's on Pantheon yeah right like yeah so what I would recommend in that situation is starting with the things that you're working on so whatever you like you work in agile so like you pick up a ticket and just start with like that right so yeah I mean because like anything yeah anything that you need to work on you'll work on right so like even if something breaks like then you'll have a test for that right so yeah because I mean you can't go back and do everything right and you never you've never to get everything yeah I mean that would be probably be overcome right I mean unless it's like you know if you have like a very critical type of an application then you then you might want to have like everything right I mean so but even then there probably only certain parts of it that are right so yeah yep hi okay okay so I'm trying to write me that just for And it's driving the nuts. Well, I am getting different results in problem. I am using Gin there. Well, let me start over. My test is working fine. The one I'm trying to use. This is built-in. I see all the information now. It's seeing things in Chrome, and it's not finding it. And I'm looking for like, behind an angle. Nope, sorry. So it's just like completely failing? Yeah. Like some things are passing. Okay, cool. And a lot of things are. I was just wondering if you like heard of that. I think the only person this has ever happened to, is there... I think there's a slack channel for testing, too, so if you want to share that. There is... I mean, you... I mean, it could be about four pages. Do you think there are more than I watched? I don't know. I mean... I don't know. It sounds to me like... It was working great. There's going to be some CSS or JavaScript that is compliant with Firefox, and not Chrome. I'm just telling you, now somebody needs to... It's working in Chrome, and not Firefox. Okay. Okay, so sorry. So it's compliant with Chrome, but not with Firefox. So like, you know how there would be like, the numbers, whatever. Or like, something that would be like... Okay, but I mean, if I'm looking for... I don't see there is a path to make it work, but I'm looking in the source, and I can see... Oh, that's great. Did you take a screenshot of the failure? I would... You could take a screenshot of the failure. You could take the HTML source code of the failure. You could... Well, the main thing... I mean, that's the tool that they already recommended and have integrated. I mean, they're doing all kinds of things. Right. Okay, yes. I can... I mean, like... Yeah, thank you. Is that Mike? Yeah? No? No. No. Okay, sorry. Oh, sorry. You look like someone else. So... Yeah, I mean, now, you know, you're talking about fire, which is... So... You're getting in a lot of patients. Yeah, I mean, I saw it. I just walked around. It looked like there were a couple companies that are offering some pretty cool tools. Like Lullabot has a tugboat and ZivTech has Provo. Well, they... I mean, they are digital agencies and they use those tools themselves. So... And they open source them. Right. So, I think tugboat's a little bit faster and it's newer. But... So, is that a product that opens source me? Yeah. But they're using the email, right? I mean, that's just very... Okay, you gotta be... Yeah, I'm just... Very insightful.