 I guess we can start. OK. Welcome, everyone. This session is about PHP Unit Initiative yearly report. My name is Daniel, and he's Jibran. So yeah, so let's start now. First of all, thank you for coming in. We will discuss brief history, how initiative started, what's the progress we've made, and then we go from there. Discuss future possibilities and how we can improve things and how we can deprecate and remove old stuff. So let's get started then. OK. First of all, we have to discuss the history, like how testing started. We started doing testing in Drupal. In Drupal 6, we have a contributed module, SimpleTest, which was using open source library at that time. In the 2x branch of that module in Drupal 6, we moved that contributed open source solution to that module and started doing our starting, updating that, and moving stuff to the module itself to that module. And after a little while, in Drupal 7, we moved SimpleTest to core, and that module was also brought forward in Drupal 8 as well. But at that time, we have PHP unit evolving in the open source community, and we wanted to use that. Yeah, right. So the main reason why we want to use PHP unit, or there are probably several reasons, is we don't have to maintain that code. It turns out, writing code is super easy, compared to maintaining it. Like, another huge advantage is that PHP unit has a huge ecosystem. Like, if you use PHPStorm, you can run tests from there. There's like a PHP unit UI. There's a parallel test runner. There's code coverage, all these things. And on top of that, there are just so many people out there which know PHP unit compared to SimpleTest. So it's the same argument as with JS Frameworks as in, it's good to teach people general tools so they can use it in other places and not just Drupal. Yeah, and we are moving in a kind of similar direction as a Drupal community, where we have test cases based on SimpleTest, which was unit test cases, which was ridiculous. You are creating unit test based on whole functional and installing Drupal from scratch, where you only want to test one class for that matter. So we, first of all, introduced PHP unit test case in Drupal. And we converted all the unit test cases from SimpleTest to PHP unit. And after that, this whole thing speed up, kind of, because we saw the benefit of that. And we already have kind of a system, kernel test base, another base class, which was installing partial Drupal and using more functionality as per the whole installation and functionality of Drupal. So you can test one subsystem in kernel test, and we started converting that to PHP unit. And that got hold up. And then we introduced to symphony functional test base. So we introduced Mink there. And we, I think, round about Drupal Amsterdam that came into being a proper patch, and it was a big patch. And I think after Drupal Amsterdam that got committed, but it was no way or form was a replacement of web test base in features or in compatibility as well. So we, at that point, kernel test base was already using PHP unit base. And we were converting stuff. And we learned some important stuff there. And that thing kick into a PHP initiative which we think after like last year, which we formally presented to core committers during a core conversation that we should introduce PHP initiative. And what was the purpose of that? Yeah, let me quickly point out that it's not us doing the work. There was Clousey doing a lot of work. There's Mikko and Lendu doing a lot of work. There are a lot of people involved, which is really nice. It's an actual community initiative coming from the community, which is kind of nice, I think. So the lesson we learned from kernel test base that the compatibility should be there. The conversion should be seamless. So to speed up the conversion process. And you have to give good documentation. And meanwhile, you have to improve API in such a way that you are not breaking existing tests and you are adding new feature as well. So that's all we had at that point. During that, we also wanted to extract all the things we have been doing in separate test bases, which are kind of similar things, like installing the Drupal, setting up subsites for simple tests to run functional tests. So we did that and extract the similar code separately. And other than that, I think we also want to use PHP to our advantage, like code coverage, which is, like in my had PHP unit test, who is specifically testing a class, it makes sense. But for functional and kernel test, it doesn't board well for me. Yeah, I mean, code coverage for functional tests is weird because you actually like functional tests you really care about like the functionality of your website. So you simply shouldn't care which code is executed. Yeah, yeah. So now let's talk about some progress we made during last year, especially. Thank you for all these contributors, which Daniel mentioned. So in core, we have converted unit tests to already PHP unit test. Kernel tests are already converted and all classes deprecated. It's still there till Drupal 9 and once we'll open Drupal 9, we can just remove that class and core is, there is no other thing we have to do with core. So kudos for that. With web test, this is still in progress. We have a bunch of tests converted. We are still figuring out some backward compatibility issues. We have made a lot of progress during past year in which we have improved our backwards compatibility. We made feature parity as well. Another thing we have done is improve the documentation, how to convert the documentation around that. We had made a lot of progress to help people understand how they can convert their existing bench test to functional test. Similarly, we have converted some upgrade part test as well, which was a big functionality because testing upgrade parts were all this difficult. And now at this point, we have 262 web tests in head and there are issues they have been constantly worked on. And I think it's quite real to say that we will have next release, before next release, we'll have all the web test removed from converted. There's some interesting thing we can see in the graph which is you see this at around, I don't know, March or February of 2017. There's this spike, like Clousy and I and others came up with this idea to have the most simplest conversion as possible, which is like moving the file, renaming the file, no, moving the file, changing the namespace. Yeah, you know, so probably everything. We're kind of doing that in kernel test base as well, right, so. Yeah, and we did that with like as many tests as possible and we basically excluded the ones which didn't work. So we managed to convert like a third, I think, by just doing that. And then we continuously like iterated here. I think though that was the easiest way to do it. And then over time, we convert more and more things, but I think it will be really hard to convert really special tests like this, this module installed, uninstalled test, installs every module and uninstalls every module. There's migrate, import, migrate, something all, it migrates an entire site from Drupal 6 to Drupal 8. And like these tests that cover a lot of edge cases, you otherwise wouldn't be able to like find, but that also means that like it uncovers like a lot of edge cases inside the testing infrastructure. So I think it will be more exponentially harder to convert the tests over time. But yeah, maybe we'll get there, hopefully. So let's talk about some stuff which is already in pipeline and been worked on. So as we discussed, we have been converting all the unit work test base to functional test or where it necessary to functional JavaScript test. Then after once that's done, and the aim is to remove all the usage of deprecated code and remove the deprecated code in the appropriate code version, maybe Drupal 9 or in some minor releases, then there is another thing we want to do which is really important to update PSP unit test and its dependencies because PSP unit has its own release cycle and it's quite different from Drupal core and we always run into some edge cases and backward compatibility issues with that. Yeah, so we are currently using PSP unit four and the current state version is six. And the problem is we can't update to five right now because five requires PSP 5.6 and PSP unit six requires PSP seven. Maybe it's not, yeah, I'm not sure 100% correct, but basically that's a problem. There's also an API change inside PSP unit five point something or deprecation bother. So we have this small, simple idea that we provide forward compatibility as in we try to back part the APIs from PSP unit five to PSP unit four so we can already use them and then if we convert there was no problem with that. So generically it's a big problem with all the components we are depending on but with testing especially and we have to make sure that we were testing our all, you know, test suites on all, you know, minimum PSP version because that's the only way we can find out those edge cases which has been happening and I think lately in 5.59, PSP 5.5.9 we had discovered some really weird edge cases and this is only possible because we are relying on that PSP unit version. So it's important to make a decision how to move forward with this upgrade part. I wanna give a quick shot out to the people behind Tupacier, it's mostly like wine from the DA that they are able, they are allowing us for example to one test in multiple PSP versions in parallel and stuff like that, it's just a huge saver for everyone who is contributing to Tupacier, I think. And last thing we want to drop simple test as well from core but as we know we can't remove that before Tupacier 9 but we want to go to find that sweet spot where we can just as soon as 9x branch is open we can just, you know, get RM simple test. So that's the goal about dropping simple test here. So let's hope that will happen soon. Let's discuss about a wish list, things we want to become a part of Drupal. They have been, like people have been working on these things in the background. Some of the things are already in issue queue, some are still table discussions. We need a lot of opinions from your people to understand and how to move forward with those. One thing is, we had like testing. So Daniel? Yeah, we had in this issue, you see there's a lot of, there was a discussion started like around Drupal DevTaste Dublin, so like 2014 or even earlier probably around the idea to use Behat for our testing. And I think we all agreed on that like Behat is not the right tool for that. Yeah, and so right now what we are doing in that issue specifically that we are not installing new site from scratch. We are running tests against existing site which Behat actually doing in Contrape for us in Drupal 7. So that's why we use Behat like testing, it's functional test as a core, but instead of installing a new site from scratch, we are using the existing site. It has like core contributors has some reservations around that at my workplace at previous next we have been using that for a year and a half now. What we do there is we spin up a new instance for every pull request and test existing DB and settings against, you know, functional test class which is really handy like we have reduced our test run time for 20 minutes, 30 minutes to just like five, 10 minutes, which is only beneficial because installing Drupal takes like eight to 10 minutes, eight to 10s, how much time? Depend, I don't know, depends on the computer and stuff. Yeah, so installing Drupal takes a lot of time and running test is quite easier. Another thing is we want to deprecate or remove run test.sh, it's a custom script, it's doing a lot of things and it's untestable at this point. We can do kind of a couple of things with that. First we can convert it to a symphony command because symphony console is part of core now so that it's unit testable, it's maintainable, but in the long run we just want to remove the dependency of running test because we can essentially use a PHP unit binary and a PHP unit configuration to set up all the Drupal variable and environment set and environment variables and settings. Yeah, it's tricky because one test of this age also has like all kind of stuff like parallel test running involved, like error checking, it's like Web's PHP unit to catch some errors PHP unit doesn't catch or it didn't catch in the past. It has some integration with the test port. It's like, yeah, it's tricky. So it's tricky and in next slide we'll discuss it in more detail. Another thing we want to do is front end regression testing. That is something really important for the front end community to automate this because sometimes we'll have visual regression after committing the patch, someone report those issues and the reason behind that is those front end changes are really untestable and only visual regression can be tested and there are still some tools around the PHP unit which we can leverage in core as well. So that's an open discussion. We want your ideas around this and we want to discuss it in more detail. And then finally this is Daniel's favorite zero setup test. Yeah, I mean it's just, I would like Drupal to be like one command thing that you can just download it and like run a command and be done and have a Drupal installation but also have like no setup test, no setup needed for running tests on your local machine as in like it somehow is it uses SQLite automatically, it sets up a web server so you just don't need to do anything. I think it's a small thing but like small things met as well. So let's discuss about the improvements. We want to improve this system because there's a lot, there's still a gap between simple test and test running on PHP unit. So first of all, the UI is important thing. Now this is some discussion which is, which people have really strong opinion about because as a developer working on these conversions we don't really think we need a elaborative simple test UI kind of thing because it's difficult to maintain the whole subsystem on top, built on top of testing system and we have test, we UI test with that separately. Yeah, there's a proposed load there which is like keeping the existing UI but instead of like running the test inside a batch process inside Drupal which calls out to PHP unit which does the testing to basically provide like a UI which tells you which command line you should use to run a specific test, which a command line tool you should like copy and paste. Yeah, maybe that's a solution but people care. So yeah, maybe we need to keep it around or switch to some external solution or I don't know. So it's a hard blocker for removing simple test because every time we discuss this with core committers we don't have particular replacement for simple test UI. Yeah, and another thing we would like to have on the test board is code coverage. I don't think, I mean it is tricky because like if you want any kind of code coverage detection like your tests low down but it's just for the unit tests so maybe it's a price which is okay to pay but then you need to communicate that somehow back to like Drupal.org or like have some page in between which shows you all the reports. Yeah, and what to do with that? Like do we just show the number or actually like warn people if the number goes down too much or I don't know. So another thing is that with PHP unit you can run most of the tests parallel because they are not independent for a lot of, if you are not running functional test you don't need to set up a new website and even if for functional test you can run them in parallel because you can install multiple subsides during test run but that's whole new set of kind of worms right there. So it's tricky but we do want to leverage that and bring down the test run time for core which is almost near about 30, 40 minutes, right? And a final thing we would like to do is better JavaScript testing. Right now we have some JavaScript testing. I mean, Duice talked about JavaScript and it's Duice node and many people are super aware of JavaScript in one, two things. But like our JavaScript test coverage tool is currently based upon like PHP unit which calls out to PhantomJS to do a testing. There are several problems. A, it's PhantomJS which is unmaintained software at this point and we want to replace that with Chrome headless or Firefox headless or something. Selenium or web driver, yeah, that's the same thing. Then there's the other problem that is we're using PHP for testing and I've heard from many people which want to get involved which are already involved in core in the JS side that they don't want to write their tests in PHP. But they want to write it in JavaScript which is I think a fair point. There are a lot of tools out there which provide support for that. There's Mocha, Mocha is a JS testing framework. What is that? Chest, that's chest and the one is night watch because yeah, that makes sense. So right now in core we have several issue, one issue worked on by JS maintainer is using Mocha, one issue created by Daniel and he worked on that is using night watch. So we definitely need a consensus on that from JS community as well. It's quite difficult honestly to write a PHP code for drag and drop in functional test, functional JavaScript test. It would be easier to write that in JS. Even as a PHP developer I think it's hard to write that drag and drop stuff. And like as always same argument we want to teach people stuff. And I don't think that the industry is using PHP to test JavaScript. Yeah, so at this point we want to open the floor for questioning and suggestions like at your workplace which different framework are you using and how do you think we can move them to core and improve that setup. One thing which we want to do in JavaScript testing is to run JavaScript test against the installed site. We want to have kind of analogous unit test suites in JS where we can test unit test JS as well and functional test JavaScript as well. So if you have any suggestions, any opinions around that feel free to jump to the mic and share with us. Yeah, yeah, yeah, please get involved, talk with us, anyone here that's like to contribute in ISE, Slack channel, probably something PHP unit maybe. And there are the sprints on Friday where I've heard that Sebastian Bergman, the inventor of PHP unit will also be there, which is nice, hopefully. But yeah, yeah, get involved. Here are some questions you could like think about. Some, if you want to help on a conversion, there are like two issues you can jump on and like help out maybe. But yeah, yeah, that's it, yes. Brian. Awesome stuff. I think it's one thing to have tests that work really, really well in core. But it's another thing to really sort of tell people about this excellent framework that we have and make it easy for sites in the wild and organizations who deploy sites all the time to make it really easy to write tests for them. Because we have such a good testing framework in core. But when I look at sites that are being deployed out in the wild, test coverage is so bad. And documentation, like we need to make it super easy and super clear for site builders and developers to write tests for sites out in the wild. Like there's a lot of good stuff we can leverage here, right? So very good question and suggestions. I think there's a session at 345, Sam Baker is presenting a session all about testing. You will know all four test suites, how you can use them in the live site. I think we need to have more sessions like that and we need a proper place for documentation. Yeah, we need some kind of resource place that we can point vendors and developers to, like... Definitely. Yeah, and that's where the zero setup thing will come handy, right? If you can run your test just by installing Drupal and you don't have to configure so many things right now, you are making changes to PHP unit configuration, changing global variables. So these things will help in future as well. So very good question. So you can get involved and help us by converting your contrib modules. You can help out and issue queue and you can also jump in Slack channel, ask for help, IRC channel, that's there. Do you have any idea regarding JavaScript functional tests? You cannot install Drupal from JavaScript, right? So you need to run some kind of PHP which installs Drupal for you. So the current patch which is out there, which is like really just an experiment, what it provides, command line utility to install, like to install a testing instance of Drupal, just like browser test base installs Drupal. Like allows you to basically do that with a command line and because the tests like Nightwatch, for example, once inside node, you can execute like any kind of binary or any kind of executable and then we can install via that in the setup. Yeah, make sense. Yes. Hi, it's Ali. You could also install through JavaScript Nightwatch, let's see you abstract away things into commands so you could just like have dot install Drupal, the problem with it, it would be really slow. But yeah, but it still wouldn't be awful to have some kind of test for that kind of install. I think the other thing for things like Nightwatch tests is they're really well used, they're like any of the Selenium and web driver testing stuff. So I think site builders might be more comfortable writing tests in that format personally. I don't know what you think about that. Yeah, I think, yeah, I just agree. Let's use tools people can also use in the day to day. So be brave about it, nobody get fired writing test. So thank you, everyone. Thank you. Thank you.