 Okay. Cool. Okay. Hello, everyone. Welcome to Human vs Machine, a love story. My name is Simon Chisnell, and I've come across from Auckland, where I work for a small company by the name of Whipscope. I've been around for a while. I've been developing Drupal for six or seven years. So quick introduction about this presentation, so at Whipscope we spent years refining and adding to a developer workflow, and now we think the structure will go. This is not written by me, this is written by my boss who's not here. But we love automated tests as they allow us to spend more time doing what we do best, writing code, or letting machines do what they do best, automation. But this is more my kind of concept, is automation isn't easy, and in this presentation we'll look at some different approaches we've taken personally, as a company, and share experiences with each of these. So an overview of what we'll be looking at. Three different testing methods, mythologies, CII continuous integration, which probably everyone knows, continuous testing, which is something I made up, and visual regression testing. And quick discussion around the technologies we use in the tech stack area. Pantheon for hosting, I'm sorry for platform SH and Acrea, but we use Pantheon. Bitbucket for a repo which we need to coach it, and then Behat, I don't know if you were in the sand beckers, well I'm but he didn't seem to like it, but we use it. And then Vagant for a VM look at development, but maybe Beatbox now, because that sounded pretty cool. So each of these testing methods, we'll have a look at what it is, how we implement them, and then pros and cons for each. Okay, continuous integration, what is it? People are probably familiar with it, the concepts, basically a development practice where developers constantly check in and integrate code into a shared repository, and then each check-in is verified by an automated test before being deployed. So how do we implement CI, in particular around Pantheon, which is not so CI friendly? So a little diagram here, developer does some work, commits, develops some work in the Bitbucket repo. So I'm not talking about branches here, I'm just kind of generally talking about the flow. Doesn't work, commits, Bitbucket then fires code chip, or I should say code chip watches Bitbucket, and then a test is run, or a project build is run in code chip. Now code chip basically builds the complete website that provides, I think a base kind of VM or environment that you can then do what you want with. And in this case, we run a script there in the corner, so you can see it, which basically says use a code base from a Bitbucket push, use terminus, which is a Pantheon thing to get a database. In this case, we use a development database. And then using a custom settings, pull in that database, start the server, and then run some B-hat tests. And then if the test do not pass, we get notified and develop and fix them. If they do, then that should be automatically deployed. And in this case, deployment basically involves cloning down the Pantheon repo, copying over a Bitbucket code, and then committing those changes. So then that pushes you or your commit all the way through from Bitbucket into Pantheon. So pros and cons, pros, it's useful to stop multiple developers committing code and affairs with other developer codes. So that's a CI concept in general. Again gives the developer confidence to commit and release code and forces developers to fix code, so they won't, or won't get deployed. Or my bug bear with it is they might just bypass the tests because they're just in a hurry. So the cons can be complex to set up, especially for small web shops or personal people with a lot of overhead involved in that. There you go, large set up overhead. And then minor issues can hold out development and deployment, I should say really in there. If it's a small thing and you just want to try to get it into your development environment to show people, then you've got to fix that test. Or in the end, you might just bypass the test. There are some resources if you wanted to try that. We've got a repo at the top, and then that basically is based on that particular Pantheon repository, which I found very useful and they've got some other useful ones as well. I think through to CodeChip and BeHat, that's the Drupal extension which BeHat runs on. Okay, so another testing method which started implementing more. It's the one that I've made up, Continuous Testing. It's basically CI without the automated deployment. So it's less likely to interfere with day-to-day development. So the concept is more, it'll run your tests, but you don't have to worry about it. You don't have to worry about if it's going to do a deployment. You don't have to worry about running a test locally. It's just constantly running in the background. So yeah, so basically it follows a similar process to CI, except there's no deployment at the end of the cycle, and we only use, another advantage, only use the Bitbucket repo to hold files required for the actual test build as fired by CodeChip API. So I won't demo that, I'll demo it shortly, but... So a little diagram of that. So this time we're working directly on the Pantheon branch, whatever it may be, develop, or if you've got a multi-dev branch, and you make a commit and we use Pantheon's Quicksilver hook, which basically allows you to run anything you want on certain actions in the Pantheon repo. So in this case, we run a command which calls in some PHP script to run, which then triggers a CodeChip build. So previously prior to this, we've set up CodeChip, a CodeChip project and everything ready to go, and then so basically the Pantheon commit fires off the command to run that CodeChip build. And then test pass, nothing happens. Deployment will still be handled by the developer, the usual kind of concept with Pantheon. However, if tests fail, then they get notified by whatever means you want, Slack or email or anything you desire. OK, so pros and cons to continuous testing. Pros only requires code to level my repository, so this is a big one for us. Me personally, especially, I didn't like code living in Bitbucket and Pantheon. Uninterrupted deployment, so if you're in a hurry, you just want to get things into branch or into Pantheon to show a client or something that goes straight through. And then because of this test, you never need to be bypassed. Cons, failed tests won't stop deployment. And then we basically as a side thing, bad code can make it through more easily. So it's not a foolproof gate. It's more of a notification that your tests are running OK, that your code base is going OK. What have we got there? So it's not really using testing platforms as they are intended. It's kind of a hack that we've come up with to be useful for Pantheon. OK, so resources for that. That's top one of another repo we created. It's got everything you need to basically set up and try to run this kind of cycle, this method, and then the same code chip and dribble there. OK, so I'll do a demo, which is going to be exciting. So this is a B-hat test. Well, first of all, I should say that I've just installed default Drupal site straight out of the box, standard profile. I've created one article there just to show that it's got content. I'm not actually going to use that content. So going back to our feature, we can see our scenario of your home page is pretty irrelevant, but reading it through, using B-hat saying, given I am an anonymous user and then and article content. So this is using the Drupal extension to create content quickly and easily. So to create those three nodes of article content, the title status and body, and then go to the home page and then telling us what we should see. So we should see those three articles. We can probably clear that. That's really small, too. OK, so running the test, creating the content, and it's passing, which makes sense, because we've just created it. You'll notice that it's created that content, but it's not in the database. That's because Drupal extension with B-hat will delete any nodes that are created in a test, which is quite handy. OK, so we'll go in, we will break the test. We're doing the hardest scientific way of commenting out a title and rerun the test. And we should see it fails. OK, so say I'm a developer. I'm a notes to me, me commenting out a title, broke something. I commit it anyway. I'd commit titles. OK, so if we're going back to a slide previously, that commit is going to fire off a Quicksilver hook, and we can jump into that quickly. Quicksilver hook, that sits within our actual Pantheon code base, calls a code chip integration PHP file, and in particular here does a call request to an API endpoint on code chip. So basically it says code chip, run that project build, and if you go into code chip, OK, so this is the actual build. We can see that it's running, it's been triggered, and so it started, and that takes a little bit. So as you can see, this is, I mean, I think this one takes three minutes, and that's one test. So it's not the most efficient way of testing, but I guess that's functional testing. But the concept being that every commit this will run kind of in the background, you don't have to really be aware of it until something comes up you weren't expecting. This is particularly good for front-end developers if they don't like to run tests locally. They won't even be aware that it's being run on every commit. OK, so if we look into the setup of the code chip build or project, you can basically tell code chip to run whatever commands you want when the project starts. In this case, we run a script and with a couple of arguments, if we go into that script, big enough. That's kind of big enough. OK, now this is where it gets kind of complicated, but it's quite a bit of work in setting up the environment that we need, starting off with removing some guff, XD bug that comes with the composer, installing Imimagic, then basically using composer to, I think we're adding in Drash Terminus, where you're at in there, connecting to Terminus, whoops, OK, and then we can see in here, so we're running Git clone to pull in Pantheon repo and then using that code base, using Terminus, which is another Pantheon tool, to basically grab the database down from development, and we're going to test against and install the site, do a bit of cache refreshing, and install Selenium and start PHP. So if we go back to our test now, it's a little bit variable. OK, so it's running, but you can always jump into any test. So we just chose co-chip, because I think it had free tests, so it was easy to start with. Oh, there we go. So you can see it's come through, it's run our test, and it's failed. Now, this is where notification will come through. It's important if you use a system to have good notifications, an email, or I think there's a Chrome plugin and something like Slack, will pop up saying that that particular commit has caused a fail. So the developer can go and fix it at their own leisure, good concept. OK, last one we're going to look at is visual regressing test. So I've only just started playing with this, slightly not through to the introduction. The concept is using screenshot comparisons to automatically detect changes and then report them back to you. So the usual process is a set of baseline screenshots of your application or website that's generated by creating node content, or nodes of set content, and then any point after that you can compare against that baseline to see if any changes have come through. Only really useful for after-base development or after the website's been developed, but it can be very powerful. So the usual process is two-prong. The developer's either content type or the entire project's complete. Comes up with some baseline images or baseline tests. So I'll give a quick demo of that. And then runs this BHAT command or a set of tests essentially to create those baselines. And then maybe a month later he comes back, decides they want to work on something else in the project, makes code changes, etc. And then runs a different suite of tests to test against that baseline. Okay, maybe I'll just go straight into a demo of that. All right, so this is a test to set up to... Just have to do this locally because we've only just got it going really. This is a test to create that baseline. So advantage of BHAT for regression testing, BHAT and Drupal extension is you can easily create nodes of set content. So everything being the same, they should always look exactly the same. So walking through it, starting off giving an anonymous user a set of browser width and then create, or in this case I'm using a slightly different one, saying I'm viewing an article or a note of article content type, fill in some fields. And then this is a custom trait or custom step that I created. Then I take a regression screenshot. So if I run that, we're down there again, sorry. Okay, so we can see we've run through, created a node, and then the output of that step saying, can I take a regression screenshot? We'll actually create a PNG and tell you where we put it. So we can go over and look at it. Okay, there's a lovely screenshot with a rabbit. Okay, now if we were to make a change, we're above the developer, we've done a CSS change across all nodes without thinking. Make a change. So we can see that change has lost that margin. The article's pushed up, the content's pushed up against it. And a test should spot that change. If we look at what this test is going to be, it's exactly the same setup. The exact same node. The exact same node, everything is exactly the same, except the last step, the screenshot should be equal to article. Is there anyone that's different? Run that, set it up, and it passes, which it's not supposed to do. That's because... We'll just try it again. Maybe because I haven't set up the settings correctly. Okay, and you'll actually see, we get some output from that saying, file's not equal and it'll output a comparison file of the differences. We can have a look at that again. Okay, it's not ideal. But it'll give the developer a chance to find something to start with or a starting point. Okay, the pros and cons of this one. Pros can test entire pages and nodes for bugs. Obviously, because it's the entire look of a node. So it'll catch unintended visual changes. Styles might change for one thing that you didn't realize impacted another. It can be used for cross-browser testing as well. You could either do locally or with browser stack with a bit more setup and time. It's really great for testing module and core updates because obviously you never know how they might impact a site. That along with standard be-hat testing. Cons doesn't work well with dynamic content. So you notice I've used a set node there because we can then maintain exactly what's in the content. But even things like a time might change if you've got a time output on a node or something. Or in particular, views. As a developer, you'll be developing a way. You'll have some rogue nodes in there. Like in the home page, for instance. We tested that. That dummy-article is going to sit there. So it's always going to have... It's always going to break. Other cons, it can be difficult to set up and maintain. I think we're going to start running into that soon. And then the baseline screenshots require constant updating. So that's something we've got to be aware of. Same resources, really. Again, put everything we've done into a repo there. And then the bottom one there is some base B-hat steps that I've used for the actual screenshot comparison. Okay, so summary. Continuous integration gives you a complete test suite. But it's a lot of work. So yes, we'll hopefully get you going quicker and works better, in my opinion. Our team for smaller teams. Visual integration is great for testing completed sites. Oh, I haven't actually changed that text. Testing is a pain in the ass, but it's better to test something than nothing and get something in there. Thank you. All right, questions? All right, cool. Thank you very much, Simon. Do we have any questions over here? Yes. With the regression testing, is it possible to sort of, like, ignore parts? So, like, you were saying that it was tough with dynamic content. How do you get it to work, then, if something, like with views or something like that? Yeah, I think... Does it just not work at all? There are... There's some ones, like, Shove and things, some kind of completed ones. So that kind of one I showed you was something I've basically pulled together really in the last week. I want to start using it as something I wanted to do. But the deeper you delve into that, the more complex it is. And there are, I mean, a few Google visual regression testing. There are ones out there that do it, but they're kind of complete systems. And sort of what I was trying to get at in here was it's better to start off with a baseline. So maybe to start off with your base content types. But maybe, yeah, if you've got more time and passion, then you could delve into that more. In my experience, on those things like timestamps, you can stub out content with the same timestamps so that your views are always going to look the same. It's just a lot more... Yeah. Yeah, I'm just wondering about the BeHat feature for actually doing the comparison between the images. Is that something you've open sourced? Yeah. Let me jump into it. I wouldn't have open sourced it because I don't create too much. I'm just consume. So what have we got? Okay, so this one, I take a regression screenshot. Okay. Yeah, so this is one of the steps I've created myself. But then you can see this one here, this... Create screenshot. That I'm pretty sure is coming from... I'm pretty sure that's coming from that other guy, that Cevue. Yeah, but I mean, to get that running, you just have to grab this web scope, Mint Context, and throw that into your BeHat test and it'll be there. And that should be in that repo. I'll refer to Sam Becker's presentation where at one point, he was putting creating screenshots, not with BeHat, but with Mink, I believe it was. I think I'm pretty sure has users of Mink to actually do that. And so to be using the same feature. In his case, he was just creating, like, it does this and then create a screenshot just so you can review it and see what was actually happening. Maybe. I'd just say if you... I would have to refer back to the video to find out what it was. Cross-referencing for future internet users. You mentioned in the install scripts that you're installing Selenium on the servers, but are you running the tests? I thought about that before. I mean, are the tests running through Gaunt from the server or through Selenium? It doesn't work on a server. So with BeHat, you can see we've got some tags in there. So API is one of the requirements for the Drupal extension, so that's needed basically to pull on the Drupal extension stuff. JavaScript then wouldn't BeHat decide that you're going to use... OK. So locally out here, I'm running PhantomJS. Yeah. So that's obviously taking the screenshots locally. And then I should be able to see what I'm using on the... I see Selenium in the scripts there. Yeah. That's right. I'm wondering what they could possibly be doing on the servers. Yeah. So this is the BeHat file that's used on the... on CodeChip. Yeah. So I guess it's just running... it's using Selenium. For anything that's JavaScript. Anything tag JavaScript, yeah. Which takes longer, so you've got to be a bit more careful. Any other questions? Yes. Do you find CodeChip flaky at all? We were actually using it with Docker, using their Docker integration stuff. I mean, just we've got a... I mean, it was great, but we did get a few false positives. False negatives. Yeah, I think... I mean, in parts and other projects, we've got more and more complex in our testing, and we see as conception, and we found that more flaky. I don't know if it's just conception, or because it's putting more on the system, or just because they're more complex. But again, it's kind of like... I prefer to start off baseline, get things going, and then, yeah, start with that. Move into the more complex that you can, but... Yeah. Tom? I'm just wondering about CodeChip. Do they keep the environments up after the build so you can do any debugging? Yes, yep. Yeah, so you can jump in. So this is a failed one. You can see a history of them. So that's one thing. There's always been a lot more, but because I'm running the same project over and over again with the CS kind of concept, it's just re-running the top one. But any particular one you can jump into, and then... Can you actually access the sites? No. So what you can do is run like a link server or something, or a link browser. There's a really good tool called Probo that we just started using. It actually builds out your pipeline, your build. It shuts down the container afterwards and you can still access the site at that point in time. Okay, it could be good. Yeah, good. The all-popular UAT testing. Let's sign off. Have we got any more questions? Okay. Thank you so much, Simon. You've come over from New Zealand. Yeah. Big round of applause for Simon for coming this way. Thank you.