 So I think we can start. Hello, everybody. Get in, take your seats. We're going to talk about automated testing today, especially about the PHP unit framework, which we are using in Drupal Core to run tests. My name is Klaus Burra. I work as a software engineer in Vienna for Epico, a small company doing job boards based on Drupal. I'm on the Drupal security team, do a couple of other stuff on Drupal.org, maintain a couple of modules like the Coder modules, the ROOS module. And recently I've been helping out with the PHP unit initiative. It's a core initiative that we just launched yesterday. Yeah, I'm Daniel Wiener. I'm working for Chapter 3 in San Francisco. I'm maintaining a couple of subsystems in core, of which we use the menu system as one of them. I'm involved in the PHP unit initiative as well, and I'm also part of the API-first initiative. Okay, so we get started quickly because we have a lot of ground to cover. First, we are going to talk a bit about the theoretical background of automated testing. What is it actually? So if you consider you have a website and you want to make sure it actually works, then what you can do is build a checklist of things that should be there and should be working. For example, that there is a login link on your page. So you write it down, check that there is a login link on the front page. Then you do changes to your sites. It's also known as development. Deploy them to your site, and of course you want to check your site again to make sure that it has not been malformed, that it still works, that the login link is still there. And you can do this with manual testing, like a human, then go through the checklist and then make sure that everything is still working again. This is how we do it when there are no automated tests. Of course, this is a lot of work because a human has to go through many things. This is tedious because humans easily get bored and they make mistakes when they go through the checklist. So it's not ideal. What we can do instead is automated testing. So you have this checklist, what you want to make sure that your site delivers, what the features of your site are, and you write them down as executable code, and that is automated testing then. For example, instead of a human checking that there is a login link on the front page, your code does it for you, your test code that you have written. Then you do the changes, also known as development, your site changes. Then you can execute the tests again. And when they are green, you know at least the login link on the front page still works. I'm saying at least here because with automated testing, you never get 100% coverage, right? You can try to get as much coverage for your application as you can, for as much features as you can, but there's only so much that you can do to make sure that something works. So you can even go further and run your test code in regular intervals, which means you automate your test runs. You can, for example, run them once per day and then see everything is still working. Or you can run them whenever somebody pushes to the branch, run them up. That change that Git commit didn't break anything, we are still good, everything is still green. And you can even do continuous integration, which means somebody pushes stuff to the master branch, it's get tested automatically, when everything is green, it gets deployed automatically. You can do this really to a high extent. Here's an example of what the output of PHP unit looks like, you see the dots that are the test cases that run and then in the end you get a final report that everything is okay or not. So this is kind of a bit of inception, so you write a program and then you write another program and you use a program to test the program. So it's really a lot of programming going on and what I want to say with this slide is you shouldn't underestimate this. Writing automated test is a bit of work and it's a bit of effort. I mean, it pays off for your most important features but you should keep in mind that this doesn't come for free, you need to be able to sell this to your client included in your price so that you have actually time to work on automated tests. It's not for free. There are different testing levels that we have to consider when we write automated tests. The most important three I have listed here, those are unit tests, integration tests and system level or functional level tests. And unit tests, they try to test the smallest possible unit in isolations, for example, PHP functions or PHP methods or even whole classes. The advantages of that is that they verify that individual parts actually work which also means I can quickly track down the problem if the tests fail, PHP unit will tell me this function didn't deliver the output as we have expected in the test case so I know this function is at fault and can fix the problem there. I don't have to do any system setup because I'm only testing one part of the system which has been isolated so I don't need to set up a database, I don't need to set up a browser or anything else. I can just run the unit test. That's also why they are very fast. There are also, of course, a couple of disadvantages so whenever you change your code very heavily which is called the refactoring so maybe that function doesn't even exist anymore. Of course, your tests testing that function will also fail because the function is not there so you also have to adapt your test cases. It can be complicated to provide fake objects which is called mocking to your class that you want to test so your class of course calls out to our subsystem, it wants to send an email or it wants to do some logging so it needs those dependencies mocked into the class that you can actually call it so in some cases if a class has many dependencies you will get into complicated mocking which can be a bit annoying. And of course in the end that's the biggest downside of unit tests you have no actual guarantee that your whole system actually works. You've verified that single components work but who knows if the whole system works. That's why you should also have integration level tests. They do tests on components so not a single class but how components interact with each other. For example, if you think of Drupal 8 you have a plugin, you test with an integration test that the plugin is actually found by the plugin manager that it can be instantiated and that the plugin actually does something. This is more than a unit test. This tests integration with the Drupal plugin system itself. So the advantages of that that you verify a bigger part of your application which means not only some small parts with unit tests but even more is verified to be working. And it's also somewhat easy to locate bugs. I still know it's in this component. It's in a mail component for example if I test that component. So that's also pretty good. Of course there are some disadvantages. It's a bit slower than unit tests. Depending on what you need your integration test to do you probably need a database or some discovery of plugins or whatever. So there is some setups required which also makes the test executions slower. And we still have the downside that it doesn't actually guarantee that the end user facing feature of your Drupal project of a website actually work. So what can we do about that? There's the level of functional level testing and this tests the complete system. The pros for that we can verify that what the user sees we replicated in a test environment and verify that this actually works. So this is really good. And it also works that we refactor code so every implementation details are changed. We can still execute the tests most of the time and they still confirm. Yes, when I go to this page and requested it still returns the same output. This also helped us to make the big migration of Drupal 7 to Drupal 8. We had a lot of those functional tests that verify that nodes still work while we exchange the complete architecture of Drupal 7 and swapped in the Drupal 8 code base that we have right now. We still could verify with this test that Drupal still delivers its features. Of course, there are a lot of cons to functional level tests. They are very slow because you have to set up a web server. You need a database and you need a browser and you need to install Drupal for each test run. So this is a lot of work. It's a heavy system setup which means you have a lot of moving parts which means you can have random test failures, right? Suddenly on your test system the disk is full. This happens on the test bot all the time or my SQL goes down or the browser has a timeout or whatever. There are several moving parts that can go wrong and it's also hard to locate the origins of bug, right? If some page doesn't deliver this which model is responsible? What is actually broken in my code? I need to then debug this to actually find out where something is broken. Contrary to that, unit tests are much better because they tell you where the problem is in which function. And yeah, as I said, prone to random test fails or kind of things can go wrong and they are also hard to change because they are so complicated to set up. It's not an easy task to maintain functional level tests. That being said, it's important that you have all three. So if you want to have really good coverage for your application, it's important that you have these three levels so that you can track errors down fast but also have a complete test of the user-facing features. But there's a whole testing universe out there, right? So we distinguish between white box and black box testing. You would say to unit tests are mostly white box testing because they know about the code flow, they know about functions, they know which code paths can be taken and the explicitly test for that versus black box testing which more is like the functional testing category where you test just the input and you provide an input and just test the output that is actually there that the HTML page is there. There are also other testing types for performance, for security, for sanity testing to make sure that Drupal even bootstraps so to make sure that PHP doesn't throw fatal errors, regression testing, usability testing, all kinds of stuff that you can get into. And there's also the discipline of test-driven development which means when you implement a feature, you actually write the test first and then see it fail, something is read. So because you haven't written the actual implementation yet, then you write the implementation, make the test green again and level up your implementation one step after the other. And then I already mentioned continuous testing and continuous delivery. So automating your test runs, give developers feedback early when they work in their feature branches or when they deploy stuff. There's also property-based testing where you analyze the data types that some function, for example, in your code accepts and try to auto-generate test sets that you pass in. I'm just mentioning those here so that you can look them up if you find them interesting. We will mostly talk about these three-level things I explained earlier. All right, now that we have seen like a little bit of history and especially like theory around testing, let's have a look how you can actually test stuff in Drupal and how we do it in core and in control. So historically, we used to use the testing framework called SimpleTest. Back in 2008, I guess it was like a good thing. People used it in the entire PHP community. You know, we started to leverage it. We started to write our tests. As Klausie mentioned, like the entire code base of Drupal 7 was tested with it. We wrote a good bunch of tests. These 405 tests are actually not the right number. We are actually having much, much more tests but 405 tests are still SimpleTest. And all those tests are those functional system-level tests. Those allows, as Klausie mentioned, those allowed us to rewrite Drupal in a sense for Drupal 8. So it was really a great tool back then. But, you know, life changes, the world changes. So what happened is that the entire PHP community moved to a different testing framework called PHP Unit. It's way more mature and advanced compared to SimpleTest because, like, it has way more features. And especially, it has way more integration with other systems. For example, if you use Jenkins for your continuous delivery, it is just there. You have a plugin for PHP Unit and you don't have to think about it. The same with your integrated development environment. Like PHPStorm just have a PHP Unit plugin and you can just use it there. So Drupal started in 2013 to think, yeah, that's the future. So let's go with it. So it's still 2016 and we haven't finished the process yet. But we are in the process of doing so and that will be part of the talk later. As of now, we have 770 tests. I guess the numbers are wrong anyway. But it is kind of like a comparison already. And those levels, those tests what we have in PHP Unit now are unit tests and integration tests. And most of the functional tests are still in SimpleTest. But yeah, let's have a look how we actually do something. So let's talk about unit tests first. It's the foundation of testing. I'm a huge fan of thinking of testing as a pyro-mit. So you have a base layer of unit tests, which cover a lot of your logic. Then you have a smaller layer on top of it for the integration tests. And you have a final step for the functional tests at the top, which so you have way more unit tests ideally than functional tests. Anyway, unit tests are tests in isolation. You test just the component for itself and just how the component works without thinking about the world. Otherwise, this is really good for testing logic. So in case you have like a complex logic, if A and B or C or D, then you can test all the cases which are possible and ensure that things are if you work solid. As Klausi started to mention, dependencies in your code are problematic. So if you, for example, send out an email, you need something which sends out the email. If you talk to the database, you need somehow the database in your code. That makes it harder to test because you need to fake those real things. In order to avoid that, there is like an entire thing called functional testing, which is like a totally different programming way, but we can leverage that partly in PHP by writing pure functions. Pure functions are functions which just have input and output and no side effects outside of that. So in there happens something, so you give it some input and it returns some output and that makes it really easy to test. Here's some nice comparison between unit tests and integration tests. Here's a unit test which perfectly works. You know the lock works, SCC? Yeah, it totally works exactly like the specification says so, but SCC, yeah, in real life, things are different. So unit tests don't cut it completely, but we need some kind of integration test, but it's important to get that part of the lock done properly. All right, so let's have a look at an actual example. So if you want to write a unit test, you need to write a new PHP class. In our case, it's called HTML escape text test and this class has to extend a certain base class called unit test case. In there, you need to define the things you actually want to test. These are a couple of test methods. So a test method is a function which starts with the string test and then some name. So in this case, it's test count. This indicates that we are testing the count method. This ad covers here at the top, also indicates that we are testing the count method and what we're doing in this test function is we're getting a string, we're doing something here, we create a new HTML escape text object. Doesn't matter what it's doing here, it's just general idea. We're doing something, you see here this count method so we have this object, we call it the count method. We expect it's doing something, it has some result and we ensure that the result which is the right side of the equation in quotes is exactly what we expect it to be. So as sort equals expects like an expected value and an actual value. And if you specify it like that PHP unit in case you have a bug will tell you this is the expected value but this was the actual value you have a bug in your code. That's a little bit more. You can define in those tests here at the top you see the ad covers default class annotation that allows you to specify which is the code you are actually testing. By doing so PHP unit can provide your code coverage. A code coverage basically says, okay these are all code paths you have tested and these are the code paths you haven't tested. So if you want to have a work-solid test suit you basically check which parts of the code aren't tested and then you can write additional tests for those cases. That makes it really handy potentially for reviewers to see how well is your test coverage. Sadly you don't leverage that into a little bit just provided. Yeah once you have defined the test function you need to probably also run the test. So how do you run the test? It's interesting because so just to be clear this is green, it's not yellow. It is kind of yellowish for me. So how do you run a test? You first copy this PHP unit.xml.disk file to phpunit.xml that allows you to provide some custom configuration. Like for example the database connection details like username, password, host and database name and the URL to your site. Once you have that set up double or no the testing framework can run those tests in this case. We are running this hml.escape.textTest and you know to do so you like specify the path vendors in PHP unit. You give it like dashdcore that picks up this configuration file you specified down there and then the entire path to your test file. And by doing so it's loading the file, executes it, checks the result and in case everything is all right it returns okay. Here let's go to the next level of testing. Klausie explained the next level is integration testing. Our integration testing is called kernel tests. It's called kernel because the main component of tuple which boots up tuple is the kernel. And in that kernel it ensures that you have the database. In this kernel test it ensures that you have basically every API available. So with that you can test your API. For example you can test that if you save an entity and you validate it that the actual validation is one and you save it and then you can ensure that it actually got stored in the database and so on and so forth. Compared to unit test this is much, much slower. It's probably like hundred times or so. I would guess so. But it allows you to cover much more. You know database IO it's over slow avoided side effects. But still it's pretty nice. Let's have a look how an actual example looks like. So this example is testing the locking system in tuple. So the system which allows you to create a lock and ensure that not multiple processes hate each other. So in order to test that we need to write another test class. In this case lock test, the base class you need to extend this kernel test base which does magic for you. So this sets up the database for you and shows that everything is loaded properly. In the test we create some object and call some methods on there. In this case we acquire a lock check whether this acquiring worked. Then we check whether a lock is still available. No it's not because it's already acquired and we check that. On here you see another test assertion method called as a true which is expecting a boolean. So if it's true everything's fine. If it's false something is wrong. On top of that here you see like a message you can give those these assertions and this method helps you to understand what went wrong when you actually have a failing test. So this message on the right side would be printed out when an actual bug appears somehow in your code. So yeah but it is conceptually really similar to a unit test. Let's go to the next level. This is the functional test level. We call it browser test because in our domain the web the system we are dealing with is mostly a browser. So a browser test sets up an entire Drupal so it creates the database, creates all the tables, even creates some users installs the configuration on your site and then you can just use a browser and go to your site do some stuff like login, create a note, save it, check whether the title appears on the top of your page, something like that. In order to do that we are leveraging a thing called mink. Mink is a browser abstraction layer. So this talks with some browser by default we are having like a PHP based browser but we can also plug in other kind of browsers. For example we now plug in phantom.js which is a headless browser based upon Chrome. So that allows us to actually test for example JavaScript. We will see that later. Browser tests are even way slower. It is just another order of magnitude slower. So if your kernel test takes a second for example browser test takes like 10 seconds to set up in one. So let's have a look at the browser test example. This is coming from rules because they adopted browser tests really early. So in this case we're actually extending a rules browser test base which according to Klausie is much, much better than the default browser test base. But maybe we should get something from that in the car. Anyway, you are extending browser test base in this case and you see here are actual like written down definitions of kind of what the user should do. So we create a user, we log in, we go to a certain page this to get goes to a specific page. We click some link, we fill in some form fields we press a button and so on and so forth. At the end here, we then actual ensure that like the browser did or like the system did what we expected. This is a little bit special down here because it's not longer just SO2 or SO2 equals but it's rather SO2 session which gives us a helper object to deal with assertions for browsers. So in this case, we are ensuring that the HTTP status code is like 200 and we see some certain message on the page. With that kind of pattern, you can then write the functionality or you can write tests for the actual functionality on your site, pretty easily depends but it is a lot of code. So in case you add test coverage, it adds maintenance cost. So we have to get to think about what you're doing. The next level is JavaScript testing. We are using PHP to test JavaScript. Let's frame it like that. Basically, instead of the PHP based browser as we had before, we are now plug in phantom.js at least at the moment which allows us to actually do screenshots. Oh, sorry. There we go. This is an actual screenshot from a running test. It is really minimal because by default we are installing no theme or some basic theme. JavaScript tests are pretty tricky because browsers or JavaScript do things asynchronously. So in this case, this is the toolbar. So we want to ensure that clicking on this link, the toolbar pops up and if you click it again, the toolbar is hidden again. This requires in Drupal, for example, an actual HTTP request to fetch the data of the toolbar. So it takes time but as PHP is running synchronously. So like just imperative style just next to each other. We need to ensure that the browser which is in another process actually did its thing. So you need to basically check continuously until some interaction finished. We have a look at that now. So this is another example which is click sorting. So you can on a table click on the header and then it sorts the rows according to this column. So in this case, you need to extend the JavaScript test base class and then it ensures that some phantom.js works for you. Stuff is going on, you click on the link. This is the actual table header and then in the background an HTTP request is fired in the browser and you need to ensure that the HTTP request finished and everything and therefore we have developed an actual helper function called asset rate on HX request. But on top of that, there are also helper functions to ensure that a certain bit of interaction on the page happen. So you need to write some jQuery to detect that a certain DOM element for example exists. That's kind of tricky to figure out potentially. So yeah, anyway, at the end, again we test some stuff and ensure that things are working as expected. So now talking a bit about test ingredients, we saw some examples already going a bit deeper into that. We already saw the setup methods in this test classes and we do that to create stuff that you need in every test method beforehand before the test is executed. So what the base class do for you is set up the database, set up the configuration, set up even test data and what you can do in addition, for example, in the unit test create some marks that you will need in all your test cases and there's also the tier down function which you probably will not need but set up is something that you do. So what you do in a setup method, you call the parent which does some parent work like installing databases or whatever, so always call the parent method in the beginning and then you do, for example, create a content type that you will use for testing and then a node for it so that you have two nodes and then do some stuff with them. This is pretty straightforward, not a big deal. Then the most crucial part of a test is of course to assert something. So what we do with assertions is we compare something expected that should happen with the result that actually happens. So we do some testing and in the end, there should always be an assertion which ensures that the executed stuff has the result that we want to have. So if such an assertion is not fulfilled, if the expectation is not fulfilled, then the test is failed automatically and what PHP unit does in this case, it throws an exception. Which means PHP unit will walk down the test case until it reaches the first assertion that fails and then the exception is thrown and your test case is over and you will get an exception and PHP unit will be read and will print you out. Oh, I expected the boolean true but I got a false at this line and it points you to the exact line in the test case and then you can start debugging and see what's going on there. This is a bit different to simple test which we know from Drupal already because simple test, if an assertion fails, it will still continue to execute the test. So this has advantages and disadvantages but it's just something useful to know when you work with PHP unit that as soon as the first assertion fails, it stops this particular test case then either moves on to the next test case if you run multiple ones or if you just run that single one then it stops immediately. What PHP unit also does for you, it automatically asserts that there are no PHP warnings, no fatal errors or anything. Whenever that happens PHP unit stops immediately and says, oh, a notice has occurred. You shouldn't have notices in your code base and also the test has failed. PHP unit also gives you a tool to have data providers versus multiple assertions so you can have a set of test fixtures that you can pass to your tests and then assert stuff on them. I will show you some example later. Some assertion examples here so we have a wide variety of assertion methods in PHP unit core. You can of course compare stuff if it's actually the same. This works on primitives like integers. This works also on objects if it's actually the same object or on arrays. There are helpers to assert that something is contained in something else. For example, in arrays or it works on strings. You can count stuff. The reason why there are this assert helpers you could always do all of this with assert same, right? But it's much more readable if you use these assert helpers then the line gets shorter and it's immediately clear what the assertion should do. So there's also stuff to assert regular expressions or that the string starts with something, yeah. And then we have the browser tests which are on a functional level that we mentioned. There we have some special session assert object where you can do stuff with the current page you are on. You can verify that the response that the browser got actually delivered the status code 200. You can assert that you are in a certain URL. You can assert that you have a button on the page with a certain label. Yes, there's a lot of stuff that you can do. Checkboxes, whatever. There's also a link to the PHP unit documentation which has also some great info about assertion messages. We already mentioned mocking for unit tests. This is where you have to fake dependencies. So you want to test your function, your class in isolation, but it tries to call out to the database so we need to fake the database. And what we do nowadays when we write PHP unit tests is use the mocking framework called prophecy. And prophecies give you a lot of nice helper functions and classes that you can use. So we have four different types of test doubles or test fakes that we are going to use. The first one is doubled that aren't really used. So you just get an object and whenever you call a method on it, it will just return null. When you test code and the database is not actually used, you can just give it a dummy the class in a constructor because in a constructor it might need the database. You give it some dummy and then you test a method which doesn't even use the database then such a dummy is fine. Then you can also have stops. Stops then there you can put arbitrary primitive functionality on them. You can say when this method is called then return a certain predefined value. You can go even further and make your fake to a mock. Then you say, I specify that a certain method should be called. So for example, I'm faking the mail system and now I'm testing something with the mail system and this mail method on this fake object should be called. So you can assert that and when it's not called then PHP unit will fail your test. It works together with prophecy to ensure that all the expectations are matched. And the last one is spies. It's basically the same as mocks but you can run your test code with your mocks and in the end you can question prophecy. Has this method been called and how many times? So there's some inspection features with mocking where you can say which method has been called and this fake object exactly. So here is a short collection of prophecy examples to see this in practice. So at first we set up a dummy, our mail manager object and we call the prophesize method. This gives you a so-called profit in prophecy which means you can get a fake object that will implement the mail manager interface class that we have here. To get our dummy out we just call the reveal method. So we don't do any further setup of our profit we just say okay whatever just some random dummy which returns null every time. I don't care that it does actually anything then we just call the real function and we are done with the dummy. If we want to do more advanced stuff we can turn it into a stop by returning something. So I can say if the mail method on the mail manager is called with the parameters of this email address admin.example.com and the second parameter which might be a subject test then this fake object should return some array which says result true. This exactly fulfills the stuff that we have defined in mail manager class. And then we can also make it a mock at the same time by saying this mail method should be called exactly one time. So the test will fail if it hasn't been called at all then the test will fail or if it has been called more than once then the test will also fail. So the next part is where we actually set up the object that we want to test in this case it's a rule action so we instantiate it. This is the actual class under test so this is not a fake. This is the real class we actually want to test. We pass in the fake object we call reveal again then the profit is turned into the actual fake thing that we want to test. Then we do some testing we set some context values here and execute the action and in the end we can spy on stuff we can say oh if there has been some other email address defined this should never be called then the test should fail and then we can also specify this expectation here. So PHP unit runs through all of this and in the end it verifies that the correct methods have been called and either gives you a correct result or not. So this are the basics of mocking. It will be a bit might not be really intuitive at the beginning but if you look at examples it really makes sense. And then yeah this is the output when you run this implicit assertions so in this case PHP unit and prophecies are telling me oh I expected a mail call with the exact admin at example.com address but what you gave me was the other at example.com address that doesn't really match I expected something different I'm failing the test and that's why you then get one errors and the red result. So that's all there is to it it also gives me a nice back trace uh-huh this was in the test case on line 94 I know where to start. Test fixtures is when you test the input and the output of a method or a class whatever it reads and produces and we want to do it with well known data sets because they have the advantage that every time you run those tests they run on the same data it's very predictable what the output should be and very reliable. PHP unit has the concept of data providers for this and it uses the data provider annotation. So we also saw a couple of annotations this is very common in PHP unit test that you see this. And what a data provider looks like is you have the actual test function which is the first function here it says test to string so we want to test that some class correctly converts stuff to a plain string. And what's different about this test method here it suddenly has three parameters because usually our test methods don't have any parameters PHP unit just starts them and goes through them and we specified here in the doc block that this test method has a data provider attached to it and this is the function name the exact function name being used here says this is the provider to the two string method and as you might guess already this is an array of test cases so we have one to three test cases here and what we specify in the arrays is then passed up to the text parameters to the first parameter and the expected result that should come out of it and even in assertion measures that we will use above there and so we build up this array, return it and then PHP unit will take each element of this array and run this test method about here three times and in there we have the assertion messages again make sure that if we convert this object to string they expect the stuff is coming out same for the JSON serialized here oops and then the test is green or not so that's basically how data providers work we used that a couple of times in Drupal Core so you can just grab for the add data provider annotation and you will find some examples testing exceptions, what can we do about that so the very naive solution is the ugly thing I wrote up here you have my example function that might throw an exception so how do I check that it really throws that exception when I pass it the test parameter I can call the function and if there is no exception I can just fail the test, that would work and I can catch the exception oh great, the exception has been thrown so we can pass the test so I have some fake assertion here which is not really nice I have to assert that true is really true which is of course always true just to get in this branch this is not really the ideal solution PHP unit helps you out with that you can say set an expected exception for these tests and whatever class the exception should be so this basically corresponds to this class and when I then invoke the function it will throw the exception and PHP unit will catch that and though this is not a real exception this is actually expected in this test case so we are agreeing this is exactly the thing that should be working there's also the ad-expected exception annotation in PHP unit but it's deprecated and the authors of PHP unit advise you to not use it because you cannot specify exactly where in your test case an exception is expected so the set expects this exception call should always be the almost last element in your test before the actual function code so when this is the last line in your test method this should be the line before the last line because with the annotation you don't know where this exception is thrown in the testing and by using this method it's much more precise that's why we should use that okay Daniel talk a bit about the tests yeah so we have seen already some good examples of tests and assertions and stuff and I will just continue with a couple of more best practices we actually observed or found out while writing unit tests into the call so the first one is about assertions you could write as Klausie said everything with asset equals there's some expected value some actual value but it is really hard to read in this example it's still okay asset equals true giraffe is bigger than 100 whatever unit but if you really like maintain a really big test suit you want to have as few lines as possible and as precise lines as possible in this line you would for example use assert true because that's exactly semantically saying and telling you this is what this line should expect another example where this comes up is the method assert instance of this is checking whether the object on the right side is an instance of a specific class alternatively you could for example use asset true of giraffe or object instance of example class but this kind of syntax with as an instance true is much easier to read like I mean of course in a single test you think oh so what but it sums up over time there are a lot of really custom assertions for example this one is a great example asset chasing string equals chasing string so for example you could use that in order to test your West API so you test that the West API response returns like a certain bit of chasing so sure you could use asset equals and check two strings together but what this one is doing is for example it decodes the value and compares the values decodes the string coming from your ACP response and by doing so it can actually compare the values probably and then show you exactly like which key of the object exactly is different so by doing so it just provides you a better time afterwards if something fails another best practice is to use less assertions per test you could of course write a single test and do everything you would ever want in there but this makes it really hard for example to figure out which kind of code is a problem when something fails instead you rather write additional test functions for each logical thing you do for example you have a method which tests the length of a string and you would then have like a test length of string method and then there's a special case for example the empty string you would have a test method test length of a string with empty string and for the infinity case or something you also want to write another test case for that another best practice or not we have solved is the entire problem space of random data so one thing of testing one way of testing is you specify exactly the values you want so for example I create a note with in quote my title and the body is my text body and so on and so forth which is nice it works but it doesn't help you to find the edge cases for example there's the edge case of some weird random characters which play a role in HTML like the ampersand and the bigger sign I don't know exclamation mark everything could cause an edge case so one way to deal with that is to use random data instead of actual static strings in this case we provide the random string method in Drupal which generates a random string but it not only generates a random string it actually provides a random string with special characters always so these are the two characters which are always included when we generate a random string on the other hand that has downsides for example it makes it harder to debug from you run it once you get a failure you run it again and then the entire output looks different because you have different random strings so think about whether you really want random data but I would recommend for user input related stuff it's probably better to have random data because that provides potentially a better security yeah one thing we should experiment with is actually the idea to somehow be able to reset the random seed because then you could for example figure out okay this was the random seed we used to generate the random data and then we could set the same random seed again on the next test one and then it would produce the same random data but that's not it so another best practice is the idea of risky tests PHP unit has a way to detect common problems with tests what are common problems with tests one is the idea of having no assertions when you have no assertions your test is pointless because it's testing nothing another common thing is unintentional code coverage that happens if you test a specific thing with a unit test but you actually call out to some other code but a unit test by definition should just test that unit and not something else so by having those risky test detections PHP unit can tell you oh you're actually calling out to some other code you probably don't want to call to another common thing is like output here you see print here so you don't want to do that it is bad PHP code to print out stuff because you want to collect the stuff together and then print out at the end of the request another common thing is uh... really slow code so in this example I just came up with like an example sleep 20 you know sleep function is probably working exactly the same as in JavaScript as a time of function uh... but actually you know stuff like that can happen right and you want to detect that as early as possible in this case uh... JavaScript timeout takes milliseconds so uh... maybe there was a bug here another common thing is to uh... change some global state for example the global user class uh... user variable and that's also bad practice so risky tests allow us to detect common problems and you should totally enable them we have enabled some of them by default but not all of them I think for example we have not enabled the slow test detection because browser tests as we talked about a much slower so they would always fail basically because browser test uh... another thing we found out is that we don't need t functions in our tests so a t function translates our uh... some arbitrary string you could do that in your tests so for example you press a button and then you specify okay I want to press t safe so I want to press it safe button and the t function is wrapped around that string but actually it is pointless because the t function is basically nothing if you have no locale module enabled therefore you just you don't test anything at all my screen is working maybe the matrix is coming what? perfect to readable oh they did maybe they restarted it should I yeah I did hmm there we go thank you uh... yeah so anyway best practice is don't use t functions in your tests unlike unless you actually test translation related stuff and you should do that of course uh... another best practice is to write simple tests so here at the top we see an example of forward module so in there we have some set of code and then we get first the amount of unread topics um to provide a string called uh... like one new test one new post or an uh... account new posts um... then we do an x path tries to find a certain link and then we ensure that there are n amount of unread topics given that your setup code is always deterministic it always returns in this example six new unread topics okay so this function always return six this function always returns six new posts uh... and then you do and then the assertion down here is always doing the same so we could basically collapse it down to the single line uh... link exists six new posts uh... so yeah what i want to say with that try to write readable and small tests which kind of tests stuff another thing what this example uh... shows you is that if you write those kind of lines uh... those are those enable you to refactor the underlying application much easier because that one really implement or really need some implementation details for example it requires that your h.m.l. has like a table you know at some point people thought maybe tables are not a good way anymore maybe that will happen in the future um... so uh... you know some kind of that kind of test code is much better to maintain and read everything so another thing which is a little bit of an abstract topic is the idea of test abstractions uh... you have abstract code like code which does something for every entity type uh... and then you want to test that one thing you could do is you could iterate over every entity type try to come up with the expected values for every entity type and then test that so the problem is that you basically introduce the same kind of abstraction as you do in your one-time code if you do abstractions on your test code that has the problem that if your one-time code has a bug in their abstraction layer your test probably has the same bug in your abstraction layer so the alternative is to always use concrete examples instead of testing every entity type just test node user and the entity test entity type for example on the other hand it is good to provide test abstractions but don't use the same abstractions in your test as in your one-time code because tests are not your one-time code tests are totally different and code style or like best practices for one-time code doesn't necessarily apply to test code anyway uh... we're seeing PHP unit now for a while and uh... we want to bring it into core and really leverage it for everything and this is the battle plan we came up with during discussions in the recent months uh... what we agreed on is we create a sandbox where we convert as much as possible uh... we get that working and proper working like uh... resilient against random test failures so for example the plan is to convert everything and then run it against the test board really often so we can detect random test failures like if every tenth one causes a failure because of some bugs somewhere uh... we want to detect that as early as possible because we don't want to block like other work which is going on like I don't know, media or stuff like that so by having like a sandbox in which we experiment uh... we can ensure that and then like when we are done hopefully until february 2017 which is the 8.3 alpha the plan is then to commit this big big bang patch into core hopefully most of the conversions uh... are done uh... at the end I want to invite you to discuss a little bit about it there are a lot of questions you could ask because uh... the testing framework is not just uh... that for example simple test provided a UI what should we do with it like are you using the UI is it useful what do we do with it like do we replace it with another UI do we remove it, do we just keep it uh... do we improve it for PHP unit yeah open questions another one is what do we do with the one test.sh command this allows you to run tests using simple tests uh... for simple tests but you know we have PHP unit and as we saw before you can just run them with PHP unit so do we need it a couple of more questions is uh... for example do we use so we are using phantom.js and some desktop.js which is weird custom stuff in a way uh... should we instead use selenium and webdriver which are actual browsers uh... so we can't like in like Firefox or something like that which is much better potentially uh... another question is when do we deprecate and remove simple tests finally or should we ever do that or something so yeah these are questions and uh... let's discuss yeah so there's a microphone over there you can if you speak in a microphone then it's in the recording and everybody can hear it so in the meantime what we discussed yesterday for deprecating and removing simple tests if you can make this happen to have this in two thousand seventeen in february committing a big bang patch where we do a lot of conversions then we might make simple test deprecated one release after that so for 2.8.4 so we'd have a gradual process as soon as the uh... conversion is done we uh... commit the deprecated attack to the 8.4.x. branch so with the next release when that comes out in a in about a year really simple test is deprecated and we should start every new test that we write should really be a browser test base which uses php on it um... yeah for the simple test ui i think i opened an issue a couple of months ago and it it got some some feedback that people are actually using it the simple test ui so what we discussed yesterday is that we would try to pump as much output that php unit gives us into this legacy simple test ui so that even if you execute a php unit test uh... from the simple test ui which is currently possible to some extent so that you at least see some uh... debug output for example which urs the browser has uh... actually visited and also which assertion is filled in at which line this should be possible with not too much effort the reason we want to get get rid of simple test ui because we don't want to have to maintenance burden right uh... we are switching to php unit because we don't want to be in the business of maintaining our own test runner because basically simple test has been committed to core there is no external maintenance we have to all to do this all ourselves and we actually want to leverage php unit which is supported by a wider audience by several frameworks and other projects nobody's asking questions here just answering my own stuff continue to discuss yeah sure be quick so um... the first time they I approached the new testing way I got some I got stuck because uh... one example that will probably have it is the data provider am I right saying that we can use data provider uh... on on unit testing and using the um... php unit test case and not with the web tests yeah usually the use case for data providers mostly unit tests and the kernel test integration test they might make sense there as well for browser tests they should still work so the problem with browser tests is that they are really slow and each data provider entry starts up a new triple instance so it's really a trade-off if you use a data provider with browser tests that you have just few test cases in your data provider then triple we install um... the site four times the the test site four times and run it for each set for each entry in this in this set and it makes it really really slow yeah yeah it's good to have to be at this presentation because I now I do understand the same problem that I had is with the cover uh... and I was trying to test a uh... uh... a Drupal internal function and uh... probably on on on on defining the cover tag probably the module is not loaded at that time yeah and I get an exception yes uh... so uh... it would be good to it's good now to understand it would be good to have these documentation somewhere which tags you can use with this tag we started to write phpunit documentation at drupal.org slash phpunit so there's a no browser test tutorial which you can look at there's some explanations how to run phpunit tests that's a bit about kernel test case and and how to use that um... what do you want to say but yeah the phpunit documentation itself on phpunit.de for whatever reason is pretty good like if you want to learn about unit testing itself go to that site it has a lot of documentation and like it explains every assertion and uh... providers and what not yeah and for you both for historic reasons uh... some Drupal stuff is really hard to test because we have we had a different code base in Drupal 7 and it's also the reason why find almost zero unit test coverage in Drupal 7 because it was so hard to unit test Drupal 7 because it was very hard to swap out dependencies our code uh... dependent on so functions were just calling other functions how do you swap out a function call which is just hard coded it doesn't work so that's the reason why we have so many web tests and had to basically writes everything in phpunit tests from scratch for Drupal 8 and even Drupal 8 is not that mature yet that we have to do sometimes some weird module includes so that constants are there this is really a problem yes unfortunately Drupal 8 is not the perfect system that it should be beside the five or four years it's still pretty good I just wanted to make a quick note it's sometimes nice to test the zero case and empty array, the empty string or something with just one item so it's number zero, number one because often you have some special behavior for that and then it works for all the other numbers and just for zero it suddenly just something, oh instead of returning false it returns null or something and so you sometimes want to have a special test for that one yeah I would advise not to use a certain empty or something so you should always make sure that the data types that you are asserting with are exactly the same 3 equals sign, it's identical with sign yeah like use assert null or assert false yes then you get the exact match yeah yeah I was mostly talking about empty string as a parameter of some function that you want to test now that's a return value but of course it's also important yeah true you touched on it a little bit with the random data testing how does phpunit integrate with fuzzing database or fuzz testing I think not out of the box yeah as far as I know I haven't seen anything but I would bet there is some kind of phpunit plugin out there which does that kind of stuff so what we have in Google Core is a trade which provides the random data I'm not a big fan of it so my stance on this issue is that you shouldn't use random data at all in your test because it makes the test unpredictable I'm a fan of predictable tests so always the same string Daniel made a point that it's important to test special characters and the problem is if you test special characters only sometimes you will only get sometimes fails so you have random test fails this makes dealing with with random data really really annoying to me I don't like it when tests randomly fail and I have to track it down but what was the cause well which character in the random string wasn't which what yeah right well but on the other hand you don't find the random test failures if you don't have random test failures yes that's true then you have no coverage at all right because yeah right those are actual bugs in your code not bugs in your tests most of the time yeah I mean if you find something if you know something just maybe open up an issue I mean for XSS for example it could be pretty useful actually for XSS tests and that kind of security related stuff yeah yeah thank you excellent you've talked about mocking subsystems and things like that I wonder if there is a way to mock external web services for example you want to talk to a different application and you want to mock that in so you don't have that components interfering with your with your test or with your code so yeah there are a couple of possibilities they go from very primitive to very high-level so in Drupal 7 for example I remember you could use Drupal HTTP request and it actually had a variable in there which function you should use to call so you could swap that out and provide a static function which returns a static JSON instead of going out some to some web request and doing something so you can do stuff like that or you if you use proper dependency injection and you have some HTTP client injected into your your class that you want to test that then makes the request you can say I've given the fake HTTP client and then it calls a method on that fake and of course the fake is totally under your control and you can totally mock it so if you do object-oriented programming properly with dependency injection it's not such a big of a deal yeah thank you sure one addition to the last question you can also use tools like SOPUI for mocking your actual web services you can completely program every response every operation every chain you want so you have actually two sets on the one hand you have your code on the test and on the other hand you have a programmed set of responses in the SOPUI environment that's the way we use to test and develop against mock services what was the name again? SOPUI SOPUI yeah it's you can use it for SOPE interfaces and REST interfaces nowadays that's good to know I think there was also some PHP projects which used the metaphor of a video recorder so I think you can say record and you have some API cores and it would record and write those responses down and then you can replay those responses in your test that's possible and with SOPUI if you have a WSDL you can have SOPUI create a mock service automatically create mock responses things like that it's actually really handy for testing web services yeah it's good to know thank you so I think with that I think we're finished thanks for your attention one general announcement join us on Friday for the contribution sprints even if you haven't contributed to Drupal core or any contrib module there's a good first time sprinter workshop where you can get started with Drupal development also meant the core sprints to work on Drupal core it's a great opportunity to join the community and don't forget to evaluate the sessions on the event site so that we can nice reviews and get to do this again thank you