 Today we're going to talk about implementing full-stack test automation for Drupal 8. It's all applied to Drupal 7 or basically any other system you're using. The whole point is to just do the test automation part, but I'll talk about myself briefly. My name is Anastasios Lascalopoulos, and yes, I am from Kansas, despite the name. Yeah, we've been in Kansas for over a hundred years. We've moved to South Texas, basically, but yes, I am American. And if you want to reach me, this is an e-mail, DOSCO. This is how I go by DOSCO. It was pinned on me when I was like four or five or something at exo.fi. Or on Twitter, on An DOSCO, just to tell about myself briefly. I've had several careers, which is from university teaching to the military. And I've actually learned to apply a lot of what I learned in the military to testing, to bug finding. There's a lot of overlap. You're looking for problems. It's threat elimination. We can go into this in more detail, but my training worked out in testing very well. My original education in IT was as a Java developer, which I've never done. I've actually never done any Java developing professionally. But nothing, I ended up having a very parapetetic life. I ended up briefly in education in Finland. And as a lark, I applied as a software tester at Nokia. And I just applied and got the job. It was basically that easy. And at the time, in 2003, during Nokia's big heyday, their big power in cellular technology, mobile development, all that, to get a job at Nokia back then, basically all you had to do was be able to recognize what a computer is and prove your bipedal and basically that quality. They just needed people. But I stuck with it. I stuck with the software testing. I found it very interesting, very fascinating. A lot of us like it better than development, but as we'll see later, there's no real divide between testing and development. We're both doing the same things. But again, test automation, I started at 2006 with QuickTest Professional. Nokia used to use the Mercury suite of tools, Quality Center, QuickTest Professional, LoadRunner. And they did not allow open source at all. To them, back then, open source was unreliable. It was just made by stoners in Eugene, Oregon, just trying to be rebellious. Serious software development meant you paid Mercury a lot of money. And after that, I've been using Robot Framework, Nightwatch.js and Phantom.js. I'm currently on codeception. Now, you'll find out that there's no one size fits all tool that will work with anything. You have to be able to adjust from one tool to the next. And it's not like one cookie cutter. You have to find the right kind of cookie cutter to implement the right kind of test automation. But moving on, I like to start with this one. Anybody have a Galaxy Note 7? Yeah, I often wonder what happened here. Was there a tester who just didn't notice this problem? Was there a problem with the requirements? There wasn't a requirement that said the phone will not burst into floors eventually. Actually, this morning, I found out that for, what is it, the Galaxy S8? There's a patch already to fix a screen. When did they release it like Saturday or something? They already have a patch for a problem. And the funny thing is I've been seeing all kinds of commercials about how important quality assurance is to the Sansung Corporation. And all of this happens. The important thing is like, they often say in the Midwest, probably here too, you have to lock the barn door before someone tries to steal the horse. You can't quality assurance, so you can't get an afterthought. Going on, let's talk about the real cost of bugs. Well, the funny thing about the word bug is there's no one definition of the word bug that satisfies everybody. Basically, the happiest term for bug is any unexpected or undesired behavior. So any undesired behavior can be a bug. That's the most prevalent definition everybody can agree on. And as I say, buggy software is very expensive, much more expensive than we think. An article I read said the National Institute of Standards and Technology. It's a part of the Commerce Department. It says that software bugs cost $59.5 billion, yeah, billion with a B every year just in the US. So how can we reduce this cost? The thing is, as I said, bugs are inevitable. You can't get rid of all the bugs, but you can reduce as much as possible. You can make sure your critical features are operating as expected, but you just have to reduce the problem. Let me say losses from buggy software will be reduced by continuous testing as part of the development process. I've been talking with people around here. A lot of people have never heard of the idea of continuous testing. People, yes, heard of continuous integration, continuous development, but there is a continuous testing. And we'll talk about that. But yeah, here's a graph about the cost of bugs. The cost of bugs is to be defined in two ways, two things. There's a cost of bugs identified. In other words, the cost of paying people to find bugs, the cost of finding bugs. And then there's a cost of fixing bugs. As you can see, it's a lot cheaper and easier to fix the bugs here in the documentation stages than it is to fix on the right way over there. And this is kind of an old slide now, because with the whole idea of DevOps, this development and testing slash QA is going to look a little different. But the thing is, we want to start testing like right around here. The QA part will be around here. And we want to avoid the post-release bug fixes as much as possible. Now again, the two costs, remember to keep in mind the cost of identifying bugs and the cost of fixing bugs. Unfortunately, by experience, a lot of projects try to eliminate this part, the cost of identifying bugs. What that does, it makes the cost of fixing bugs a lot more expensive. If the thought is, if you just take away bug finding, you'll also take away bug fix. Projects manage your logic. But as you can see, the latter becomes much more expensive. The cost of fixing bugs becomes much more expensive, much more expensive for the project overall. The best thing to do is to invest in finding the bugs early. And the takeaway, as I say, identifying fixed bugs as early in the process as possible. Even in the paperwork process, you know, you're also testing. And again, don't assume that by removing the cost of identifying bugs, you'll also remove the cost of fixing bugs. And I would think about what causes bugs other than actual coding. I'm starting to think that a lot of the big bugs like you saw with the Galaxy phones, I think it's more of a political problem than a coding problem. We get a bunch of people talking about dividing work, expenses when there's money involved, when there's a budget involved, things get weird, things get tricky. And so what other ways can we see that bugs come up in software, any kind of software? I found very often a keyway process doesn't exist, meaning that there are no professional testers or just no one dedicated to really looking for bugs. You have people dedicated, of course, to coding but no one dedicated to finding the bugs. And often there's no plan. There has to be a plan. There has to be some way and in the other one there's no procedure. There is no process. You have to have some kind of process. Everything can't be just ad hoc. And too often there's a reliance on developer-driven ad hoc testing. As I heard someone once say, this is a my quote, developer testing is basically no kind of testing. And I will say a project can't have all or most of the above and still incest that their application has been tested. Now let's talk about mindsets. Developers, as I say, need to show that the application works. When I've been a test manager, I didn't allow the testers to say that an application works. The best thing you can say is you didn't find any bugs. It's not up to us to say if something works or not. And often when developers test their own software, they're not really looking to reveal problems and issues. The thing is testers really are looking to find problems. We like finding bugs. Dust bugs are good. And we need to show not just bugs as this, but how many exist, what kind of bugs are they, and where are they, and what happens. It's not just looking for a bug. You're looking for what is the problem, how do you get it? You record what happens, you record how to find it, and you record how to fix it. And the thing here is it's an issue of critical distance. I think it's a psychology term. Developers are too close to their own creation to search actively and closely for problems in their own creation. I kind of call this the meatloaf effect. You know how people say, if somebody says, oh, I hate meatloaf, the other person says, well, you just haven't had my meatloaf? A lot of developers are, yeah, other developers can't search for bugs in their own software, but I can find bugs in my own software. It's never really the case. Testers, however, can stand back and look objectively for unexpected or undesired software and user expectation issues that the developer himself may not think. And there's also the idea of fresh eyes on something. The more eyes on something, the better. And as I say here, just in my own way, testing is a complex and creative skill and not just the last of many tasks development needs to just move the project along. This is one thing that I really like about testing. To me, it is creative. I know when I first started, testers were often regarded as the people who just weren't, like, talented enough to really become developers. But luckily, you know, at times change, the field has changed, situations have changed. And now testers need to be able to develop in some way, in some language, scripting language. You have to develop too. So the walls are kind of breaking down. And by doing this, you know, I'm trying to break the walls down even more. As I say, testing should never be an afterthought. Solution or automated tools. Automated tools now make it possible to create tests during development. That's the whole idea I'm trying to stress for the whole DevOps thing. You can code to develop the features and also somebody else codes to find bugs. And the way I try to sell it is test automation is a development project. Don't think of it necessarily as testing. Think of it as development where you write code that finds bugs in a specific feature. Now, again, don't just pass tests. You have to write tests to find bugs. And very often, people will write, you know, like 100-something test suite of, you know, unit tests or functional tests or whatever. That all pass. And there's a big problem if all your tests are passing. You want bugs. You want to find bugs, not just pass the tests. Again, and this is where creativity comes in. You have to write the kind of tests that find bugs. A good test finds bugs and not just shows pass. If you have too much passing, there's a problem. If there's too many fails, there's a big problem too. You don't want everything to fail. You want to make sure that there aren't any script problems, but you have to make sure that there are bugs being found. In other words, you have to make sure that undesired behavior is being found. Okay, so what can we do? I've seen the whole idea of test automation in the past 10 to 12 years change. It used to be just regression tests. Test automation was done at the end, sort of at the regression test stage where you take all your past manual tests and turn them into automated tests. And you keep executing those. And you also use test automation for features that have, looks at a lot of repetitive actions. Like you have to enter a lot of fields, drop down making use push buttons, radio buttons and everything. But test automation is changing. And we're going to concentrate on the first three things. Actually the second and third, we're going to see what happens when we build and test. And we'll also see how test automation impacts these aspects of DevOps. Okay, this is a V model. The only thing that's important for me are these ideas here, verification and validation. Verification means you have the system is working correctly, the system is right. Validation means you have the right system to begin with. You also have to test that this is what the end user wants. Because I have seen things in user expectance test failed. When the client basically says the user says we can't use this, this isn't what we wanted. So you have to make sure it's what the user wanted that's testing also. Okay, but going on quickly, there's a new idea of test automation. Continuous testing as a fundamental part of continuous integration. So you create and execute tests as part of development. And we'll talk about test driven development briefly. Before I even, I should have mentioned this a while ago, but what do we mean by full stack test automation? By full stack we mean unit testing, functional testing and acceptance testing that's all automated. In other words, the whole process is automated. That's a full stack automation set. And starting off, we have acceptance test driven development. So the acceptance criteria is put into code in several different kind of tools. I'll give recommendations later. And run until the final application meets the acceptance criteria and passes. Okay, so the test, the acceptance test, which traditionally in the waterfall method was saved to the end, is actually run at the beginning. And development has to meet the criteria of the automated tests. And here, this is important rather than like a kind of tool set. Acceptance test driven development is more of a mentality, an approach to software development instead of just, you know, two specific tools and processes. What do we do? This is the role of testing. The role of testing is kind of elevated here. So acceptance tests are created first by QA and basically used as communication tools with developers. Tests are customer oriented from the user's point of view. Any development and test executions run integrated until the tests finally pass. Okay, this may not be new to anybody, but I want to start, you have to start with unit testing. The first level of testing, theoretically test every object in the code. Okay, and test driven development is you create the tests first, the unit tests are run first and then the code is written. An approach here, I hope everybody can see this. I'm not a PHP developer. This is the most advanced I've ever done with PHP development. But this is a simple and I hope correct. You can do your testing here if there's a bug, please let me know. But with this, basically we have a simple application that just returns hello and then adds two integers together. The point here is that this, the test part was written first and then the code added to make sure that everything runs. Okay, since I'm kind of running out of time, we go quickly. Now let's do test automation specifically for Drupal 8. We're finally getting around to Drupal 8. The cool thing about Drupal 8 is that Drupal 8 can create unit tests for you. You have to add and expand the kind of testing, but you can do it with the Drupal console, add controller to module with Drupal console. And there's an option, it'll even tell you, I'm sure a lot of you know what I'm talking about. It'll say, do you want to add a unit test class to this? And of course you say yes and it'll create the class. And PHP unit can be run, I think it's already automatically integrated with PHP unit and Drupal 8 are already run together. And you can use numerous tools. Our company does a lot of PHP unit tests running Jenkins daily. Okay, that is white box testing. Now for functional tests we do black box testing. So you basically examine what the system actually does from the end user's point of view. So input is repeatedly sent through the system using a number of different tools. Like I see here from Night Watch, Nightmare Caps for JS, Smoka. There are just more and more and more of these. So take your pick, whatever is good for you and what's good for the project. And end-to-end tests can also be run. Again, at the end we get to the acceptance test. So the acceptance test is finally when you can really say to management, to the client, that the test is ready for them to look at. You know, the application is ready for them to look at. Cucumber Be Hot has acceptance test automation, robot framework. Codeception has unit testing tools. And basically the tools test the developed software at a higher level. So at this point as opposed to the unit test and especially the functional test, for the acceptance test you don't really want bugs anymore. All the bugs should have been found during the unit testing and functional testing phase. So by the time you get here everything should pass. If things haven't passed then there's a problem. Okay? So somebody asked me, you know, who should do the testing? I say, you know, everybody, everybody can find problems. Developers and testers, anybody can find a problem all throughout the process. And like I said, always finding bugs is a good thing. Well, and often you have to get the whole project into the mindset that finding bugs is always good. And again, it's difficult to do because often people's mentality has to change more than anything. So I remember one time at Nokia, a project manager when we were really getting close to the finish time, to the release date. The project manager asked me, was there a successful test run? And I said yes. And he said, great, then we can release. And I said no, because in my opinion a successful test run meant bugs were found. So he got angry. That's fine. But that's fine because that's successful testing. To me it's like fishing. You know, no one goes fishing and they had a great successful time fishing if they didn't catch anything. And they showed that I've proven there are no more fish in that lake. So a different attitude. You know, successful fishing means you find fish. You catch fish. Same thing with successful testing means you find bugs. So who should do the testing? Really everybody. Everybody should find problems all the time because our bugs are ubiquitous. Functional test automation is for QA. There's a movement now just to have testers, to have QA doing unit testing. Because unit testing should be, is supposed to be. It's a great idea if everybody does unit testing, but very often unit testing's kind of slipped aside. So having QA doing unit testing is becoming more common. Functional testing QA should do that because you need the critical distance. You need to stand back. So I'm going to need to stand back. Acceptance testing is done with quality assurance. Yeah, with heavy involvement from management clients, developers. So everybody look for bugs all the time at all levels. And of course finding bugs is good. Find as many as possible all the time. So just to conclude briefly, testing should be planned early and started along with development. Keep in mind the critical distance. I think developers can be very good at developing test automation. But it shouldn't be your own code you're writing the test automation for. Assign a developer and think of it as your writing code that will find bugs. Like I said, it's easy test automation. It's easy development. All the features have been approved. There's no paperwork. The tools are easy to use. And like I said here, they're available for every stage of development. You have to find the one that's best for your own project and also best for your own mentality. Like I said, test automation should be regarded and planned as quickly and easy development project. And to end up, the main point of test automation is to find bugs and not just to confirm that the code works well. That's all I've got on test automation. So start automating. You'll find it's fun.