 So, welcome to the future of Drupal Functional Testing. Yeah, let's sort of discuss a bit about us, who we are first, before we get into the meat of it. So, myself, my name's Nick Xu, or at Awesome1989. As you can see, I sort of touch a lot of technologies. I play with a lot of stuff. I'm very passionate about all these technologies. I'm a guy from Australia, Queensland in Bundaberg, which is 36 hours away by plane. So, I'm very, very passionate about this topic too, to come here and talk to you about it. I work for an agency in Australia called PreviousNext. And, yeah, I sort of deal with a lot of developer and sysadmin tasks and sort of bridge that gap and work with really sort of interesting problems and build out tools. So, this kind of stuff's really close to my heart because I also maintain our internal test infrastructure. So, this is the kind of things that I really like and I like to improve in core. My name's Karen Zamek. I've been PHP developer for over a decade now. I've only been using Drupal Sense version 4.5. Only this year, I started working at PreviousNext and got into contributing to core. And, besides what we're going to be talking here today, I'm also involved in Drupal Module Upgrader. I'm the developer of the library that it's using in the back end called Faberist, which I'll be talking about later in a bof today with WebChick. And, we're here to talk about functional testing, so I'm going to pass it back. Hi, everyone. I'm Konstantin, mostly known as Everset. I think I'm here today mostly because I created tools like Behart Mink PHP spec prophecy. I'll talk about Mink a bit. So, yeah, I'm here just as a voice of wisdom for those libraries, I guess. Cool. So, let's sort of get a little bit of show of hands. So, who's written a simple test? Cool. So, in Drupal 7, who's written a test in Drupal 7? And then, Drupal 8. Okay. So, very, very, very similar. Who's used any other sort of crazy testing frameworks? Yeah. There's some pretty cool ones out there, right? Yeah. Who likes to write tests? Okay. So, first I want to say thanks to the community. This is purely from the fact that we've had a testing suite for six years now, and it served us pretty well through that. I feel like today we released beta and I don't think we would have hit that milestone if we didn't have a test suite. We couldn't put our hands on our hearts and actually say this thing works. So, and we've definitely ramped up the amount of test coverage that we have in Drupal 8 to do that. So, yeah, like a big thank you to what we have now. I also want to thank you guys for coming here because I feel like this is an area that needs a lot of improvement, so it's good to have everyone here to provide a bit of feedback and start a conversation around this as well. So, first off, so what is functional testing? So, put very simply, it's full stack testing, right? So, in the case of this reveal.js thing, if I were to test it, I would be testing from top to bottom, I'd be testing like the node server. I'd be pointing and clicking around and checking its functionality, you know, flicking right, flicking left. Is that on the screen? Do I see that? So, important output, just things that you would do every day to test if a feature that you've written actually works. Functional tests come in many different types. So, you could have smoke tests where you're sort of making sure that your product as a whole works for the most part. So, you've got a little bit of reassurance around there. There's also like regression tests where you find a bug. You're like, oh, yeah, I'm going to write that test that does X, Y and Z and, like, before and then I'm going to fix the problem and it's going to go green and that's awesome. So, that's sort of functional tests in a nutshell, right? So, point and click stuff that we do every day to make sure that the stuff that we write is good. So, the state of Drupal testing right now. As I mentioned before, we've had a test suite for about six years now. We brought the simple test project on in Drupal 7 and started using that which is great. It's awesome. The only issue with that is over time it started to the maintainer of simple tests started to or the team around there, there wasn't really much action going on there and we took it and then we started to maintain it and now it's forked. We've morphed it into what we want and how we use it. So, it uses and it uses curl and simple XML to do the sort of browser operations of it and we maintain all that code. So, I know that Sun has put the maintainer, one of the maintainers of simple tests has put a lot of hours into maintenance just of that library itself and just think about and I'm sure everyone else has sort of debugged through it and imagine if we could rely on something a little bit more solid and upstream and yeah, imagine what you could be doing. So, I'm just going to quickly take you through the stages of a test so what actually happens in simple tests as it stands. So, first of all, we bootstrap the parent site. So, that could be like the simple test UI that you're in on the Drupal site that's you bootstrapping it to perform operations to set up a child site. That's also you running run test.sh on the command line which our test bot infrastructure runs. So, that's bootstrapping the parent site. We then set up a child site under the simple test directory which leverages multi-site to make that happen. So, then you can get concurrency. So, you could spin up like eight of these and then start testing around there. So, bootstrapping a full Drupal instance there which is pretty costly. We then run the tests through our sort of custom browser I guess you could say and yeah, curl simple XML, point and click around, test what you want. You know, go to the home page is the login form there and then result back. And then after that we then clean up the site and sort of put everything back to the way it was. So, that's kind of the full process. And I sort of want to point out that those first two there, sorry, the first three is more like the framework, the actual simple test like set up and tear down and that one in the middle is the browser. So, I feel like it's time to, I don't know about you but I feel like it's time to get off the island. Just looking through the code and all that maintenance, it's time to sort of leverage things out there that everyone's already using and they're battle tested, so we should be doing the same, right? It's just like Dries was saying in his keynote about sort of getting rid of all that, you know, getting rid of, you know, like leveraging other libraries so we can focus on the awesome and cool stuff and maybe bring in extra features. So, maybe like JavaScript testing or something along those lines focus on the cool stuff that we really want. So, yeah, time to get off the island, right? So, this leads me to my first sort of technology that I feel that we should adopt and that's PHP Unit. I like to think of it as a runner, that's why I got my little Lego runner up there but really it's a framework, it's a testing framework. Don't be disheartened by, it's called PHP Unit, it's not just for unit tests, it's a really solid extendable runner that we can leverage for this. I also want to point out that we already have PHP Unit in core doing our unit tests but we sort of wrap it with simple tests. So, for us to adopt this wouldn't really be that much work given we've, you know, we've already adopted it for one layer of our testing. So, this is just an example of a very basic PHP Unit test, just modifying an array and asserting that there's nothing in the array, just adding food to it. Do we have a count of one? You know, ditching it. Do we have a count of zero now? Like, that's, but if you look at that a little bit closely, sorry, a little bit closer, you can actually see that that's in like a very similar format as to what we already run in simple tests, right? Prefixed for test. There's a cert equals, like, we've got all those functions there and we maintain those. There's a slight little bit of a name difference between I think it's a cert equal instead of a cert equals, but that's, you know, that's fine. We can refactor that. So, very, very similar. It also means we can rely on some contrib. We can bring in some contrib libraries. So, like, right now I can get my nine cat when I'm running a full test suite. I can just watch that for about 45 minutes. That'll pass the time for me. So, but in all seriousness, like, very extendable. We can do whatever we want with it. And here's a few libraries that sort of catch my eye, right? So I see that there's a PHP unit UI which could maybe replace simple test UI. We could ditch that code as well. We could get rid of that code as well and we could contribute to these other libraries or even make our own, right? And build a little bit of an ecosystem around that, that small component. There's a PHP timer library and there's also a parallel testing library. All doing their own separate thing, right? Their own little core. And I think that's really, really cool because right now our run test.sh script does a lot and it's very untested. I mean, the way that we test our run test script is by running it through the bot but it'd be nice to actually have, you know, some kind of test coverage on that as well. So let's remove it, replace it with a runner or a framework, like PHP unit. Cool. I'll hand it over to camera. Yep. So who's here? Use B hat. So you're probably going to recognize this syntax. We're going to a web page checking the status of that page and filling in a form and submitting it. Well, the power of B hat, in this case, is actually powered by mink. It does the browser interaction without website. So that's what I'm going to talk about briefly now. So what is mink? It's a browser emulator. It gives us control of a browser and interact with our website. We've got two types of emulation. We've got headless, which is done by the Goot driver. So that's similar to what we've got now with simple tests. We've got all this code with curl and simple XML that's trying to emulate a browser. So instead of having all that code ourselves and maintaining that with mink with the Goot driver, it'll do that for us so we could replace what we've got now with the Goot driver for the same functionality. But the beauty of mink is we can also plug in a real browser. So it lets us control different browser backends. And each has their own advantages and disadvantages. So with the headless browser, it's nice and fast, but we don't have any JavaScript support. So at the moment, there's a lot of code in simple tests that sort of half fakes the Ajax part when it comes to Drupal Core. So instead of doing that and having all this code that's just trying to mimic Ajax, we could plug in a real browser with mink and have proper Ajax support. So this is just a quick example similar to what we were just doing there before with BHAT, but with just straight mink code. So this is NPHP. And we can... We start off a pick what driver we want. We start a mink session. We visit a page. We can find elements in the page using CSS selectors or XPASS selectors. And then we can click links, fill in fields, submit the page, press buttons. But as the same before, with mink, we can actually control a real browser and have JavaScript support. So we can test dragging, dropping, double clicking, like clicking, hovering over something and seeing if an element pops up. These are all things we can't do now with simple tests. So if we were to use mink, we could get all these extra functionality that we could test for in our functional testing. So that's what we're here today to talk about what we've been working on on getting it so we can use mink and PHP unit in place of simple tests. So we're not talking when we say JavaScript testing. You probably still want to do JavaScript unit testing with something like Q unit. We're just more talking about having it so in our functional testing, we can support rich web UIs with... We've got full support for Ajax and JavaScript through the use of using mink. So one of the problems with... when you're controlling a real browser with mink is the browser is slower than the Goot driver. So I'm proposing, for majority of cases, we could use PhantomJS browser backhand through the Selenium driver. It has a pretty good feature set. It's based on WebKit. It's a headless version of WebKit. So it's pretty close to what browsers out there are using now with Firefox. Sorry, not Firefox, but Chrome and Safari use WebKit backends. So it's very similar in the way it renders web pages. And it's got JavaScript support. And it's in a single standalone binary. So it's pretty easy. It's pretty easy to set up. You've got the PhantomJS binary. You start it up. And then the Selenium driver will, by default, connect to it running on localhost. So how this started was... Lee Rollins started the issue to replace simple tests to use mink back-end. So he started... We started porting web test-based as-is because there's a lot of tests in simple tests that use web test-based. And we got pretty far with that. We got down to under 200 fails and 65 exceptions. And we were doing this, making it backwards-compatible, had decorated classes and all sorts of things, trying to get the existing tests to work with it. But it's a bit of a rabbit hole trying to get that going. So then we decided to start fresh. And we also decided, let's throw out the simple test runner as well and use PHP Unit. So that way, people, as I was saying before, there's some slight differences between the search and methods with PHP Unit and simple tests. And more people, when they come to testing, are more familiar with PHP Unit. So by doing this, people that are used to PHP Unit were able to do Drupal functional testing in the same way because it's using the same framework that they're already familiar with. And then we're adding this mink layer for the browser emulation. So when you're debugging your tests, you're not using all this 1,000 plus lines of custom code, having to go through curl and simple XML and try to figure out what it's doing. You've got mink, and we've got that with BHAT. So it's way more battle tested than what we've got now with simple tests. Also, that was... Core has... I was just in the next generation test bot meeting before. And right now, Core has roughly 75,000 assertions and quite a few edge cases. We hit a lot of edge cases when porting across. So it was... after a meeting with Sun, the containers of simple tests that the easiest way for this to get into Core was to commit it with its own test suite. So it tests itself and then we start porting tests over, mainly targeting the AJAX ones to begin with with JavaScript. Yeah, so as Nick was saying, eventually, like it's not going to happen straight away to get rid of simple tests, we're just here today to talk about how it can move forward to make that happen. So this is on the left before, this is what we've got now. We've got this forked simple test code. I think we forked in 2008 and it hasn't had a release since 2012. So it's pretty unmaintained. So instead of having all this forked code that is only specific to our project, we moved to using PHP unit which is used by pretty much every PHP framework. And then instead of having all that curl and simple XML and 1000 plus lines of code doing browser emulation, let's use Mink. It's way more solid than what we've got and it will open up doors for having JavaScript support in our functional test so we can do full end-to-end stack testing with JavaScript, finance. So this is just a simple code, can't fit much code on the slides of where we're using this new class that we've made, browser test base. So it's using PHP unit and Mink and it just we've just got a simple wrapper over the visit function for Mink that just takes a Drupal path and converts the URL into a URL and then visits a page and just checks that the page responded and with simple tests the child site is sort of protected so it handles being able to actually access the child site and then there's a bunch of assert methods we've added data Pacific to doing web-based testing so it is asserting there's a link on the page or asserting I can fill in a form and and it's all just built on the assertions that PHP unit already provides so it's pretty minimal code if you know Mink you'll be able to easily follow browser test base because it's just Mink code wrapped up sitting on top of PHP unit so that's what we're here today that's just recommending we replace the runout and framework of simple tests with PHP unit that we replace the browser code with Mink and agree on a way to do JavaScript testing with our functional testing and we're going to have a bof after this as well to continue talking about that and as Nick was saying before we're going to commit browser test base hopefully into core and people can start using it to write their functional tests instead of using simple tests and then we've got a point forward way from simple test and if we can get people to help port some of the existing tests to hopefully who knows even by 8.1 we can get rid of simple test so how we can get this in core that's our repo where we've been working on this and that's the issue where we've been posting patches and we'll post a patch later today that I've got working on the beta so there's some current blockers with getting this into core but we're talking this morning at modernizing test bot talk about how we're going to improve the testing infrastructure so some of those improvements and then they help with that they're sprinting on that this week so we want to be able to have it so we can run PHP unit instead of having to go through run test script we also want to be able to introduce some browser back ends so we can do the JavaScript testing so maybe bring phantom.js into the test bot so we can then have the test bot run these JavaScript based tests and you can help these guys out on the Drupal infrastructure ISE channel or myself and Ricardo are a part of the Drupal CI initiative sprinting on that so if you want to contribute one of this up after this so that's for watching the video these are the other talks that are on testing there's one from this morning about modernizing the testing infrastructure there's one on later today on automated front end testing and another one on BDD testing with Big Hat and then it's both what time is it sorry Rika 4.30 cool and we're also going to have a bof continue on this in G001 at 2.15 I'm not leaving that room so now we'd like to open the floor to questions and discussion around this topic if anyone wants to come up to the microphone here so it's on the recording so just wanting for you the browser test is there any sort of built in support for internationalization or language or anything about that translation strings yeah well because with MENC you can pass in labels fields you can just use the translation on your labels to find that field so you could go I want to fill in the username field and use the translate functions of Drupal so that's available I've got it to do that with the login form I fill it in based on what you see so when you read the tests they kind of read like what you'd see on the screen and then you just use the translation functions to give you the correct label for that language so if you wanted to test it in another language as long as you're writing your function tests to use the translate functions on your labels when you're finding elements you'll find them and fill them in so when can we remove simple tests from core yeah good question he works for us by the way well yeah so this as I was saying before going on backwards compatibility route so we could like remove it we're trying to do that for aid but it wasn't going to work out that way so in order to get rid of it now we have to start writing tests on browser test base and there's also suns brought in kernel test base so more and more tests are going to be written on kernel test base and then when we need functional testing we'd move them to this and then so hopefully over time there'll be less needing tests running on simple tests and then at that point we can go we can safely remove it so basically we'll set up a meta and then sprint on it just like we would have with the the page controllers and such like just everything in core that we've sort of converted over same process a meta that sort of lists all the modules and then we could either ask the maintainers to review patches and make sure that they're happy with the code coverage one of the big problems with simple tests aside from everything else is the way it tears down and creates a new site for every single test and the switching to mink wouldn't really resolve that which means the test suite would take essentially just as long to run that's true having you thought it all about how you can consolidate similar tests because these aren't unit tests so a lot of them could probably be consolidated into a single build that's what I'm going to plan on working on next speeding this all up because at the moment it actually goes through a full Drupal install and most of the time it's just a testing profile and I've seen people play around with this before so I want to just try and clone it just clone the testing profile and that should significantly speed it up because at the moment just for a simple test that one before it takes about 16 seconds to install a site that's just on my laptop though and on the testing infrastructure they use tempFS speed it up a bit more but it still takes a few seconds to set up and tear down the other thing is it's also how we test so changing from this what we've got right now where it's either a unit test or a web test as a part of the discussion with Sun the maintainers a simple test there was also this concept of sort of layers so first you would unit test do your really quick tests secondly you would test your database drivers so you would check does Postgres work, does MySQL work all good so now we'll move on to sort of functional tests so fast functional tests and I'll sort of compare and contrast in a sec but so yeah you go to your functional tests and then from there you do like javascript tests and like those really heavy browser tests and what that unlocks for you is the ability to nominate the fastest database driver for web tests for functional tests it also enables you to fail fast on a unit test or a DB test and just not even do functional tests at all there's no point running a half an hour test sweet if you know that your database layer is broken you'd be better off getting a feedback loop back quickly and fixing that and then seeing it go through so yeah that's the point that Nick was saying as we move on in the Drupal life cycle we're going to have more code coverage with unit tests and hopefully we can pick it up at that level before we even have to even worry about doing a functional test and it's also a good spot to start to have a conversation on how we test so definitely like bundling more things together in that one test getting a clear idea of what should be a functional test I think will minimise how many we have Lee we also have a colleague of ours Lee on the hangout so he may ask questions or heckle another thing that I don't know if you can hear me but another thing that we're getting closer and closer to is once we remove all the exit calls from core we should be able to do controller output testing via the kernel in the same way that symphony does so Cam was talking before about the mink drivers, you know there's a driver for browser kit and anything you want to do with browser kit is handle the request and so symphony use it for their testing their kernel just actually routes the incoming request through that and does all their asserts against that there's no HTTP in it at all so when we get there that's another way we can speed things up so your tests basically become web server agnostic you're not testing Apache or engine X or anything in that case but yeah that's sort of like splitting it up into those two groups so which ones do we really want a full stack test on this functional test and which ones do we want just to go through the kernel the other thing with the fail fast that you're talking about before Nick is when triple core controls the test runner via a YAML file or whatever then you could do that on a patch per patch basis if you're working on a particular patch that you do want to fail fast you could edit your YAML file as part of the patch because it's part of core and get that feedback loop and then when you get to the point where you want to start running all the tests take that out of your patch more of a mink question so we come from a long way with simple tests but what kind of restriction or difficulties we might run into by using mink because I don't know like which are the current users of mink and if there's something as big as Drupal using it with some weird case right I think mink is mink was on the market for quite some time we're using it extensively in B2D community in Bhat especially the only problem that you can run into is if you go rampant trying to write too many functional tests then you will have really really slow tests with it it's not an inherent attribute of mink it's an inherent attribute of functional tests other than that I mean it's it's averagely fragile again because it's an inherent attribute of the functional tests because you're making so many interactions there are so many layers in between your application and the actual user you have UI, you have HTTP, you have browser it's like they will be inherently fragile so as long as you keep the amount of functional tests to the bare minimum just to make sure that your application works like smoke tests for example you should be just fine I guess so I don't think there will be a big difference in regards to mink is limited in something in comparison with simple tests I think it's kind of on the same level I think it will also allow you to have more options in order to do things differently so this whole idea that sometimes you can use JavaScript, sometimes you can use good or like headless browser and maybe later on you can switch to the browser kit the whole idea of mink is that it provides single API for all of those drivers which means you can for example start from the tests which go through and make real HTTP requests like go to driver and then later on as soon as you kind of stabilize your interactions with kernel you can switch to browser kit and it will be completely transparent to the test switch so your test will just start running faster because you're not making real HTTP requests so benefits like this is just flexibility and more options I guess that's what you will get otherwise you still get pretty fragile tests and you still get pretty slow tests and that's what functional tests are in a nutshell thanks just a question we're talking about kernel tests as well so we could write a driver forming to do the kernel tests for us as well I think you already incorporated as far as I know the kernel into Drupal so I think browser kit driver which mink already has it shouldn't be that hard to adopt it to what you already have we've looked at that browser kit driver we've just got to override the do request just like the Google client we've just got to remove the issue that Lee said before about the exit calls so that's the blocker to that otherwise we could have that today someone remove the exit calls and making I actually have a question you guys said that you kind of came up with your own assertion library built on top of PHP unit or using PHP unit assertion mink comes out with its own like web assertion library which kind of is fine tuned to page interactions and stuff and it also throws exceptions which have more contextual information towards the page for example if it says element not found it can give you the context of where it tries to find the element this is a text around why aren't you using web assert part of the mink I actually looked at that and essentially copied and pasted them but instead of throwing exceptions I use assertions PHP unit assertions we need to find a way to do actually it's like the whole purpose of this will make sense if we'll start copy pasting absolutely I agree I agree on trying to remove that pledge even so how would you do something like that with PHP unit and I guess PHP unit would just pick up the exceptions all of those exceptions that mink produces you can easily identify them because they all extend from or implement one single exception class or interface you probably just need to extend PHP unit to know about those exceptions and what to do with them that's correct I was looking at that but I've been busy so far at this Drupalcon adding remote code coverage now that we're using PHP unit with our functional tests we can actually do code coverage and I was running it before just hitting a simple controller and it comes up that that controller's got 100% code coverage so it's awfully slow though when you do code coverage with functional tests XD bug code coverage is slow but yeah we will have to we'll have to put that as an item before getting serious reviews to switch to using exceptions to using those exceptions definitely sounds like that's the way to go on the general note actually I really like initiative I really like the fact that you're going more towards reusing the libraries that other frameworks are using and it generally will improve the quality of mink a lot and will hopefully help to improve quality of Drupal but at the same time it is important for you to be aware that this is not a replacement for unit tests or integration tests you need to push harder for having more unit tests because that's essentially where the beef of your quality assurance should be so functional tests because of those like inherited attributes of it being slow and it being fragile you want to keep those to the bare minimum so like I mean in trying to make it very approachable don't overdo it I guess people kind of need to start from unit tests I think actually having a meta and having some analysis on the tests at the time of conversion is a good reflection point to say is this the best way to be doing this test and there is as Nick was saying 75,000 functional asserts in core at the moment and actually having a broom and hold the hand of those and move them over to the new system is a great opportunity to scrutinize them a bit more yeah we're tackling the testing issue on multiple fronts one is we're going to have more unit tests and then Sun's been working on kernel test base which is doesn't set up a site, it just bootstraps Drupal so you can do test Drupal APIs and just install the module you just need to for your test so that's the second layer and then at the top you have functional tests when you want to test something that works from the end user's point of view for web stack with rich web view eyes and that's what we're gaining by bringing in Mink here that we don't have now with simple tests is we can test our website with its rich web view eyes and I think it's going to help both sides of the coin as well so with the Drupal CI side of things I think that's going to enable us to do this change a lot easier but then from our side if we're refactoring our tests it's also going to help the test runs because like Ricardo mentioned in the last talk for the modernizing test spot CPU is a big deal on our test spots so if we can definitely look at how we test the module and try and get way more unit tests in there then we're not spinning up like eight concurrent sites and clicking around and smashing our CPUs basically hello right, talking about functional testing and stuff like that, you know there's the B-hat extension and because again if you're going to be writing a lot of stuff in Mink since Mink is kind of underlying a B-hat it would probably be useful if the B-hat extension could reuse some of it just because again if you write it all in the test it's going to get buried in the test and then you're going to end up with people and contrary modules probably rewriting because they don't know it's there so would it be nice to sort of push it up a layer or at least expose it as a list of reusable components so allow us to extend say core to do certain things like our own modules and stuff that we actually have this set instead of it just getting buried which I think I can take this one I didn't look deeply into B-hat extension but I looked into it in different lifetimes and we talked with Jonathan this morning if somebody doesn't know so I think there's a lot of really smart and really cool things in it like the wall obstruction over Drupal the way to control or talk with Drupal through different drivers it's kind of along the lines of what Mink does to the browsers and I think this is something that you definitely can reuse and should reuse in your functional testing tools and I know Jonathan is thinking about structuring it into separate library outside of B-hat extension so it will be reusable I think he would like to get any help from you guys you can deliver to him I think it's a good initiative the last thing you want to have is to have segregated, isolated things that work in complete isolation from each other you kind of want to reuse amazing things that you create together I'm sorry I should get talking you definitely should any other questions oh cool since B-hat did come up I'd like to mention an idea I had which is I think it's great to do all of the functional testing we currently have using Mink and not convalume B-hat with trying to fill a gap but I think at the same time it would be a very worthwhile and valuable exercise to write B-hat tests, BDD tests for the standard installation profile as an exercise to see if as a community we can figure out is this thing behavior-driven does it do something and if not look really hard at why do we have a standard installation profile and maybe make it something that is understandable through B-hat tests okay so sort of test well I think I need to be really clear there I think using B-hat without doing BDD is a very inefficient way of doing anything so it doesn't force to even try to bring B-hat in if you're not planning to do behavior-driven development I think there is a benefit in doing behavior-driven development for a small fraction of Drupal for the core but I think evaluation should go around the methodology itself behavior-driven BDD not the B-hat as a tool because if you decide to go with BDD definitely B-hat will help there but if you just decide to go with B-hat just for sake of B-hat it will kind of damage you more than it will help so yeah because in my mind B-hat gives you you can create your own domain language about your site so if you're doing like a shopping site you might come up with terminology for that and develop a domain language around that and specify your tests in that domain language on how you're going to test the site and we're talking more just at a functional level and you're saying with the standard profile you might be able to come up with some terminology around that yeah and because I think there will be a huge I think there will be a huge benefit and it could streamline the way you communicate or the way you document the core behavior of Drupal but again that's a different conversation to what we're having right now I definitely well just to quickly extend I definitely think the standard install profile could be debated that it is kind of a product in some ways it's kind of like a distribution because it's like this is Drupal you install it this is Drupal and get started and go it's kind of selling it like a product or a distribution so yeah I'd like to be a part of that just on a anyone else brave enough to ask a question on the mic what are we doing for time still got 15 minutes anything else no, jibran so we are doing all of this because we want to test JavaScript and double clicking because right now in core there is no way to do that so after what stage we can do that after what stage can we do that so I'd say it's a bit tricky because like I mentioned there's two sides to the coin I'd definitely say once this browser test base gets committed everyone can do it they could most certainly use it for their projects as well it's a little bit trickier to get it going on our current infrastructure on our testing infrastructure to spin up something like phantom.js that's a bit interesting to make that happen and would take a little bit more work so that's kind of like the follow up after we commit this so and we would also need to have a discussion around drivers so how would you declare this is a phantom.js and this is a and I want to use this for this test like this other driver for this test so that's kind of a bit of a gap right now I would say so this is kind of just filling the hole of swapping out components right now so then we can start moving on to these kind of really interesting issues demo demo and just before about JavaScript testing I don't think we have a lot of JavaScript's bug on functionality it's always about some little thing that breaks the JavaScript and then it doesn't work so I don't think there's going to be a huge value well currently for JavaScript testing I mean there is when we refactor but now I mean we got through the worst of it for Drupal 8 so it's going to be better for the future but it's not urgent anymore it was I think in regards to smoke tests or general like in browser tests the job just running like something with requirement to have JavaScript enabled is just one side of it another side is when you're running headless tests for your application it's like it's essentially very far from the real usage that people will oversee with application whereas running through the browser it's getting as close as humanly possible with automation to run it exactly like people will use this application so it's not only about having limitations with JavaScript and headless browsers it's also about understanding that headless browsers or headless simulation of the website browsing is really really far from how people will use your website so if you have minimal amount of smoke tests running them through the browser can have some benefits even if you're not heavy on JavaScript and if you have unit tests on JavaScript so you said that you were going to do this more incrementally does that mean that it maybe it can be put in Drupal 8 now that the release cycle is different or is this still far in the future and we'll go into Drupal 9 yeah I think there's no reason it couldn't be in 8 like we're going to create a meta on converting the web test space that we have now and if people are keen and start porting them over there's no reason we couldn't get rid of simple tests before 9 it's just a matter of porting them over I also believe that the the testing suite changing that isn't classed as an API breaking change either and given it gets committed alongside it's not breaking anything getting committed like that either so it's definitely 8 worthy suite along those lines do you actually need to convert existing tests well no I mean can you just use this new setup that you came up with for something new or for the breakages that you find so this is more kind of natural way to upgrade to something else just like you still have your old test you still run it but for everything new or every time you break something old you migrate only this part to the new ones so this kind of will eliminate this crazy requirement for having incredible amount of hours spent or poured into this exercise which maybe never will have a value like you don't know if those things after transfer they will uncover some bugs right you don't know how many of those 700 or how many like 7 75,000 assertions actually save you from bugs like is it 90% is it 50% so just like transferring them and spending time into upgrading them or migrating them might never return an investment of this thing that's true yeah that's a good point yeah I want to try and measure how much our current simple test actually covering in terms of code coverage in that regards, I want to see if I can get that working it's all recorded buddy so you're contractually obligated to it now so Gibran asked how much time have we spent on it and it's a lot hundreds and hundreds and hundreds of hours because I've been going through and debugging what simple test does now and when we're trying to initially make it so it was compatible with that it's definitely 100 plus hours like 150 hours and it's what we're saying before how Dries was talking about reducing the costs and from speaking to son he's put so much time into simple test and it's something that's not even core to what we do with Drupal and so if we can get rid of that and use something that everyone's using in PHP and if we want to improve it we improve Mink for everyone not just people using Drupal so that's one of the motivators to why we started this because it's like we're doing it with the rest of Drupal we've moved to using external third-party libraries and trying not to reinvent the wheel all the time and that's what we decided we went this testing framework it's from 2008 it started and it's just one band-aid on top of another with that so 2007 so yeah so and assuming we don't put the old test to Mink like how fast would we get Mink you know available in core to write new test oh we're hoping to get it to core ISAP time frame because there's a sprint on Friday yeah yeah I still think it's kind of pretty heavily coupled to to like the PHP unit component and being able to run that on the test bots so yeah I think that's but we're making quite a lot of headway there as well so that's why I really care about that side of things as well so but we could look at some we could look at other options maybe maybe we can maybe we can even locally yeah yeah absolutely so we should definitely have a chat in the boff about that maybe from both sides I see Jeremy's in the back he's got room coming over chat oh well there you go have you just been hiding the whole time? sorry buddy sorry dude I yeah you're not really you're listed so here you go simple test just lost last maintainer in terms of getting rid of simple test I guess if we don't rewrite the test which is a totally valid way of doing it as we move forward and get more unit test and more kernel test base and then we go okay this is where we want do smoke testing and just write them in on top of browser test base and then at that point there's no simple test anymore I completely agree with the method doing it as things arise that's by far the simplest way of doing it as we move forward and do more unit tests and kernel lower level testing just port them as we need them that's how you would generally approach commercial project with legacy testing platform nobody would ever spend this amount of crazy money on just migrating the test with the only reason why your guys are considering it is because of your time is kind of free which it is not it's not a very efficient way of doing things like you don't need to do that yeah I've got to agree there oh you know it's easy when there's a couple of drinks and sitting down and hacking out some code 100 hours of drinks and code though I was thinking of automatically porting a few so if anyone's where Drupal module upgraded it can automatically convert code and that library I was mentioning that I'm the author of so you can get existing code and manipulate it and transform the code could even port some automatically so I haven't actually seen what I've got to do there yet you've got to put your mad scientist lab code on are you okay cool I think that's pretty much it if no one has any other questions I think that's smack on time now yeah okay thank you for coming there he goes I think it's like 11 o'clock at night or something like that yeah that worked out really well thanks for that