 I'm Andreas, one of the core committers, so hello, hello. I'm here to talk to you about what possibly is the greatest innovation in our lifetime, which is namely the web. And I think that everyone in this room cares quite deeply for the web. In fact, I think that the people in this room cares more deeply for the web than many other groups. So I want to start off by giving a little insight into the life which you have now introduced yourself, which is the life of the web developer. So of course, what web developers do is that they invent and they create the web, right? This guy is a web developer, apparently, really a modest bunch. We don't care much for titles, apparently. But unlike many other inventions, the web is not a static thing. The web is almost 20 years old, a little over 20 years, actually. The modern web is about 16, and we're not using the web as it was invented. The web as it was invented was a fairly basic thing. Now, the web is a living thing. It's something that evolves every single day, and it evolves in many different ways. It evolves because there's no single inventor, for example. Tim, the director, he could have said, he could have kept the web for himself, but he decided not to. He decided to open it up. He happened to get the idea of this electronic system of interlinked hypertext documents, but he chose not to restrict his use in any way. He said that it was not his to safeguard. In fact, he did the exact opposite thing. He decided to open the web up for everyone, and the web, consequently, evolves through individuals like you and me. So we have services like RSS, syndication, we have hosted web apps, all new innovations, big websites which we go to every day, and there are some things which we invent which are so important to the structure and the architecture of the web that it eventually gets put into a web browser. And you guys all know this because you're all web developers, but precisely because of this fantastic plethora of innovations and ideas, it's sometimes a little bit difficult to agree on exactly what is a good idea for our community at large and which ideas are destructive. Now as a web developer, we have a responsibility to test, a responsibility to improve the web. One way to improve the web is, amongst other things, through standards. Now my argument is that testing is quite a fundamental concept to software engineering in general. If you write a software program today, it's inconceivable not to test it. And the ironic thing is that that's not the case, or at least not until recently, has not been the case with web standards, with specifications. Specifications are sufficiently complex that you can't just define them and rely on the browser vendors to implement them. They are subject to a much, they need a much more thorough testing effort to actually make sure that they are as good as they can get. Now browser vendors, they do test their software, they do test their browsers, but they don't run the same test cases, for example, and they don't test in the same way. And everyone who's worked a little bit with CI environments, we know that the environment in which you test has a tremendous effect on the outcome of the tests. Now we have to ask ourselves, is this behavior from the browser vendors good for the web? Well, I mean, no, it's obviously not good. It means probably less work for the browser vendors. It means that they don't have to test as much, they don't have to write the tests. But for us as users, we suffer at the lack of interoperability. I'm quite sure that most people in this room have discovered some feature in some browser, which doesn't work in another browser. And that's one example of lacking interoperability. Now I'm here to talk about the web platform and how we test it. But the web platform, what exactly is the web platform? It's a fairly browser-specific implementation detail. So in short, I'm going to go through it. In short, all cross-parasite technologies that browsers like Firefox, IE, and Chrome build and ship, it's a collection of web technology standards developed by the 4G, W3C, the IETF, ECMA International, and Unicode Consortium. It is basically a big collection of technologies. And to illustrate that, I'm going to go through just some of them. And I don't think most people who develop for the web realize that this is actually the world's most advanced application platform. And it works, I mean, even if you give it invalid code or if the browsers aren't fully compatible or if something doesn't exactly work, it all works itself out. And this is because of the way that the evolution of the web, I mean, it didn't have to be like that. It could be that we would have more strict standards which would limit which websites would work in different browsers. So I'm going to talk a little bit more about that later. There are not many technologies that can boast the same level of backwards compatibility, for example. The website which was made in 1995 still works in the browser today, and that's pretty fantastic. But it's important for us as web developers to make sure that we don't regress the web, that we don't deprecate how the web works, and that we don't make any content on the web inaccessible. And that's one reason why we have these standards. Now, there are some core technologies, HTML, DOM, ECMA script, ECMA script, which is actually JavaScript. URL, XHR, course, it says encoding. But the list goes on. There are so many things to test. And each one of these words or each one of these has its own specification, and these are all technologies implemented in browsers. So you have animations, font positioning, WebGL, all of these things are foundational web platform technologies. Media, events and messaging, cross-document messaging, very exciting new technology. There are APIs for if you're using a device like this one for getting the battery level, the device orientation, whether it's upright or sideways, vibration, et cetera. And the list just goes on. And many of these things have been actually invented and specced in the more recent years. And interestingly, in the category other, it has categorized by the W3C, there's also a new specification which I've highlighted there called WebDriver. So exciting times. So documentation on this is found on these various URLs. If you just search for a specification, be a little bit cautious of what you find, because most of it is just outdated, and you need to go to the real sources. And developer.mzilla.org is an excellent reference. Now the web used to be a very simple place. We used to have useless standards that no one followed. We used to have browsers who did pretty much what they wanted. I'm not sure if anyone remembers the blink tag and the markweed tag, but that's like one example of the browser wars which happened about 10 years ago between Netscape and Microsoft. Around the same time, the W3C, the worldwide web consortium, realized that this was getting out of hand and they needed to do something. So they started the process of speccing XHTML, which was going to save the web from this mess that it was in. And that's true. The web was in a big mess, but at this point, it was also in a position where the browser vendors were saying, well, we have all of these users and we have all of this web content. We can't just deprecate it and don't allow people access to it. Pages must work. That's like a secret mantra of the browser developer. Now eventually, someone, the browser vendors in this case, realized that a spec describing reality was a much better approach. And so HTML5 was born. So the web used to be simple, but worldwide web has morphed from a system of simple interlinked documents into something which is so advanced that new approaches to testing is necessary. We used to test for regressions, but now we also have specs which comes with tests. So we need to find a way to test those. So the question is if we can do better than we did a year ago, two years ago, five years ago. And the answer is, of course, that we can do better. We can do better with automated testing. And I think Anand Bhagbab made some very eloquent, had a very eloquent talk yesterday about the need for automated testing and what it means. So I urge you to look up the video from that if you didn't attend. We need tests which are vendor-neutral so that we can share the tests between different browsers. We need them to be run in continuous integration environments so that we can get immediate feedback. Otherwise, it's really hard to find proper regressions. And it would be really nice if the tests were accessible to everyone, right? And a way, I mean, having access to the tests which the browser vendors use to make their browsers with means that we have a way of keeping tabs on our browsers to make sure that they are actually doing what's in the best interest for the web. And consequently, for the browser vendors, it also means that they get better interoperability. Now there is a project for this, which is called Test the Web Forward. And it's an open source project, and you'll find the link here. I really urge you to go and check out this project. It's a project by individuals who submit tests back to a repo called the Web Platform Test Repo. And it's situated on GitHub, so it's really easy to write tests. So the Web Platform Test Repo is, well, basically tests for the open web platform. They're collated by the W3C, but the W3C doesn't write any tests themselves because they rely on contributions from individuals and from browser vendors. They are continually updated, which means that, which just underpins my point about the web being a living thing. It's not a static thing which you spec and then implement. Specs evolve at the same time as you implement them. I'll upload the slides so you can get the links. Now exactly what do we test in a specification? Well we look for conformance requirements is one thing. And they're based on an RFC, which is 2.1.1.9, which is known to everyone who works in the browser business, which basically is like you have statements going must, must not, should, should not, may, et cetera. And these tell the browser vendors what they must implement, what they can choose not to implement, and what they can optionally implement. There are also some statements which are unambiguously requirements. There are some statements which are candidate requirements. Which basically says that, which is basically statements which go, you can choose not to test this. For number four, we have requirements which often, which often have stated, which are often stated with algorithms. The web drive respect has some awful algorithms, for example. They're really, really complex, but it, but it gives the testers a really good idea of what you need to, what you need to test. As a single requirement, a single requirement usually requires multiple tests. A single test case may be a test, may require a combination of multiple tests. A single test case may test a combination of requirements. But in short, the whole thing is very complicated. But more importantly, specs lie. Specs have bows, just like software. Specs change constantly, sometimes the specs themselves don't match reality, and this is precisely why the web is so bonkers. Three HTML5 specs were written first and then attempted, implemented. The reality was that no one implemented these specs. For example, HTML4 was not implemented because it was unimplementable. And it is also wrong, but that's another discussion. HTML5 isn't making the same mistake because it specifies what browsers are actually doing. It started out by just specifying what they were doing, and now it's like continuing to add new things. But I mentioned that the web is a massive place, which basically means that deprecation is very hard. So the first time, well, when you introduce a new feature, you can't just remove it. If a feature is used by only 1% of users, that's several tens of millions of users. So it's really hard to be speccing something like the web. In the web platforms test repo, there are several different test types. There's a script test type, which is driven by a framework called testharness.js, which is written by a colleague of mine, James Graham at Mozilla. There are web IDL tests. The web IDL is basically, I'll return to that later, and I can show you an example. Ref tests, we have ref tests, there are manual tests, and finally, a recent addition, a new test type to the W3C are the web drive implementation tests, which I just landed patched for a couple of weeks ago. You can start with the script tests. They are basically JavaScript. They run in context, in context of the browser window in the top-level browsing context, and this is an example of what the test looks like. So you have a basic HTML page, you load in the testharness, you have a test function, it takes the document title, which is meant to be document title, and it asserts that those things match. It's an example of a very, very simple test, but this is an actual test, which is located in the WPT repo. And the reason we have these tests is, of course, that if someone happens to break the title tag that we catch the regression before it ships. This is an example of this test running in the harness. You just spin up a web server to serve you the testharness.js script, and you go to the URL. Simple as that. There are a number of different testharness.js assertions. We have all of the usual suspects here. There are a few special ones, which are, like, assert-owned property, because objects in JavaScript can have properties. There's prototypal checks and a few other things. Because JavaScript is inherently asynchronous, there's also a need to test callbacks and event handlers. And I'll show you an example of an async test. So basically, you have to register a handler, and for each step, you can register an assertion. And then you tell the harness that the test is done so it can continue on blocking. This is an example of this test. So you register the load event on load, and that happens to assert that the event type is load. So that was simple. Web IDL test is basically a format for defining interfaces. They're typically JavaScript APIs, which are meant to be implemented by browser vendors. And the Web IDL tests are an extension, which you can put on top of testharness.js. So you load in this second JavaScript, and you can do tests like these ones. You have ref tests for when your test result is visual. So here we have an example of a biodirectional override. So yeah, you have the test on the left, on the left there, and the reference on the right using simple HTML. So this tests if the BDO thing works. There are also a number of things which isn't possible currently to automate. And these are simply manual tests, which you use when nothing else would work. A couple of examples of things that are untestable at the moment are on visibility change. So if your browser goes into the background, you have Chrome events, like if you resize the window, blur on focus and on full screen change. These are things that are hard to do with JavaScript, because the JavaScript doesn't control the Chrome of the browser. There are also a number of new web APIs, such as those for the Firefox OS device, which Mozilla is developing. There are telephony APIs, vibration, SMS, IDL, screen orientation, power management, Bluetooth, battery status, NFC, spell check, lots of different things. So all of these are things which you, on devices which support them, you can implement, you can use from JavaScript. There are also visual CSS tests, which, for example, this one. If the square is still red, please indicate that the test has failed. It used to be the case that you had a test genre, which you actually manually go in and click. Is the square red? No, it isn't. It's a pass. It's a very tedious process, right, if you have tens and thousands of these tests. When you have some helpers for tests, there's a requirement that you are able to speak to server sometimes to test some specs, like you have to delay the connection, for example. When you have to test XHR web sockets, perhaps you need redirect status codes. Perhaps you need to manipulate the connection data. There's a WPT serve library, which is in WPT repo, which allows you to do this. This is an example how it works. It writes a status code, and it actually sleeps for a second before actually sending the response. There are some implementation requirements for this. Tests must be pulled from upstream. They need to store the expected result of each test. They need to be super robust against misbehaving tests. Because tests are continuously upgraded, so the test needs to be really, really, really robust because they're updated very frequently. It can't require things that only make sense in one vendor context, like something which is Chrome-specific or Firefox-specific. Automation needs to be easy for the browser vendors to use. We need to have cross browser and cross platform support. It needs to be possible to run the WPT test, of course. They need to run fast, parallel with the ideal. They need to run in a normal browser, which means that they need to run in the top-level browsing context because some tests assume that, meaning you can't really run them in an iFrame container. We need to guard ourselves against misbehavior if the browser hangs. We need to kill it, for example, and we need some machine-readable test output so that we can interface this with third party systems which the browser vendors already have implemented, like their own testing systems. We're solving these problems in the WPT runner by pushing test scheduling into the harness. That basically means that we have control over the browser process. We have machine-readable output solved through structured logging, which is like a JSON-ish schema for storing outcomes, expected outcomes, debug output, anything. Then we need a way to communicate with the browser via some protocol, and that happened to be WebDriver. WebDriver happened to be at a very convenient point in time for solving these problems. The general design of this is that we have a Python test drama, which schedules tests. It typically uses WebDriver to drive the browser. For JavaScript tests, basically what we do is that we open a window at the start of the test run, then we call executeAsync script from WebDriver to start a test that we inject some JavaScript. That opens the test page in a new window, and then we have a callback for when test results are posted back via a channel from the open window. Ref tests are actually even simpler. They just use WebDriver's normal APIs. The WPT runner architecture is basically a test manager instance which owns a browser and a test runner. These things are fairly obvious, so I won't go into any detail. The browser is an abstraction over different products. We have one subclass for each browser. The test runner owns a test executor that actually knows how to run the test. The test executor has a subclass per test type and a method of executing the test. This basically provides us with the two axes of extensibility that we need. I'll just put this in here if you're interested, so you can look at this later. I don't expect you to see it. The current status of this is that it's working on Firefox and B2G, Firefox OS. It's even working on server, which is Mozilla's new attempt at re-implementing a browser, a parallel browser. It's working on Chrome via WebDriver and Microsoft are actually adding support for IE. Things are looking very interesting. An interesting thing I should mention is, of course, that the WPT tests have this concept of a manifest files, which lists all of your tests. It also has a concept of expected test results. This is because the browser vendors, different CI systems, they work slightly differently. For example, Mozilla's and Google's work in a way that they expect all the tests to pass the whole time, whereas operas, for example, will not expect tests to pass the entire time. So we need to cater for those different use cases. But essentially, that sort of defining whether a test is expected or not to pass is actually extremely useful in the context of testing the web platform, because it allows you to have things, to have tests for things which are not already implemented, and that way you can actually track your progress of something you implement. Is this test-driven development? Not sure. The reality is that the test suites are actually never complete. They're never feature-complete. They're almost always, they almost always have bugs, and they most certainly don't match reality. Yes. So do we have interoperability yet? The answer is sort of. Things are pretty interoperable. But the honest answer is that we really don't know until we start collecting test results. But the exciting thing is that WebDriver is being used in a very untraditional testing context. And it's filling an important role. It's really helping us to test the web forward in the truest sense in an automated way. So we've solved a simple problem with finding a way to automate all our existing sets of tests. How do we actually increase our test coverage? What about the tests, for example, that can't be automated? And to answer this question, why can't tests be automated, we need to ask, well, why can't they be? We need, I mean, one reason is that to test some of these things, we need to convince the browser vendors that we should take control over their computers, that we should disregard all security measures in the browser, and that we should have access to all hidden things inside of the browser. And that's quite controversial. What we really need is something which can emulate a user. We need something which is capable of controlling the browser instance from an out of process program. We need something which would let us instrument the browser. And we really do have that, don't we? We have WebDriver. We're already using it in WPT runner. We already have a WebDriver instance controlling the browser. So let's have a look at one of the un-automatable test cases. For example, if you have on key down, on key press, on key up events attached to your document, tests will, if those are examples of things which can't be tested currently, because, I mean, you can generate a fake synthetic JavaScript event, but that doesn't really mean, that doesn't really mean the same thing as injecting something into the event queue of the browser or into the operating system. We have tests, yes, so let's have actually a look at this test. Let me see if I can find it. There we go. So this is the test. Actually, I have the source code in my slide. There we go. So it's an async test. You load the browser, but there's no way of simulating the key event. So the test doesn't really pass until you press a button on your keyboard. And the interesting thing, I found this is an actual test from the WPT repo, which I found, and you'll note the first comment up there, that someone is thinking ahead. I haven't seen this before. So if we try to run this test, for example, it's now running. And I need to press a key, and there it completes all of the test files. If I just leave it running, it's going to eventually time out. But this is a great example of a test which currently isn't automatable. So let's have a stab at implementing this test. This is fairly standard Python. We load in a bunch of, for those who don't know Python, I'm going to just walk you through it. We load in a bunch of requirements, a bunch of dependencies. We need a way to define which, like this is how I'd implement it. We need to define which browser to test with. We need to get the capabilities of that browser. Then we connect to our remote web driver server, and we request the browser with those capabilities. We load in our test case, and we send a key. So far, so good. Then, of course, because JavaScript is asynchronous, we need to wait for the test results to come back. So we bring in a web driver wait, and we assert or we wait for the presence of the element by ID summary, and then we get the test result from that element. And then we print that to standard out, and we exit. But this thing is very verbose, isn't it? I mean, if a tester had to go through this sort of approach every time he wrote a web driver test, he would become very sad, because there's lots of boilerplate here. Granted some of the things we can remove, we can put into a library, which we load in. But in the end, you're going to end up with something roughly like this. And this is, by the way, a simplified test, because you also need to register the test events with a logger, which reports it back in some machine-readable format. So it's not easy. But the problem with people is that people don't always want to learn new tools. They don't want to learn a new framework, and especially they don't want to learn a new language. This is all fine, I think. Because when we had this discussion with the WattWG, this previous example is something similar to what the web driver working group ended up with proposing. And we went to the WattWG, and they said, I don't really want to use this. I don't want to learn a new language. They were very skeptical. And I can understand them, because they already have perfect tools for what their purpose was. They wanted something which was much simpler. So now we're, let's see. So if we take a step back, I thought long and hard about this for a few weeks, and I eventually came up with a solution. I should highlight that this is not the web driver working group official position. It's a proposal of my own. I haven't submitted it yet, but this is a taste or a preview of what, automating a web platform's test with web driver might look like in the future. So we have a remote web driver. And the remote web driver is, in all its simplicity, an HTTP server, right? It accepts some HTTP calls. You can send it some JSON as the argument. Use HTTP, because that's the simplest protocol which you can implement in literally any language. And we already have a web driver session. I mentioned this earlier. So the WPT runner already has a web driver session for maintaining control over the browser process. So the use case for web drivers often in combination with assertions that can be made using test.js. Like they want to do this key event thing. They have all other tests, but what they really need in addition is something which goes press a button. What if, conceptually, we had an interface from the JS test to web driver controlling the browser session? So what I'm going to show you is an in-document JavaScript, an in-document JavaScript script controlling its own browser session from an out-of-process driver through the web driver remote server. And I've included a couple of examples of what it might look like. I can actually show you how it works, but I mean, running tests in an automated is usually very boring because they just disappear right away. So I've included some screenshots of how it looks. So this is a JavaScript running in Chrome, controlling Chrome itself. This is kind of like inception, right? So in this case, the third test, the two first ones, which were just to make sure the thing actually worked. But the third test here, it says get title matches document title. And I've included the JavaScript console at the bottom here. And if you have really good site, you can see that it records the JSON payload which is sent to and from the web driver server. This is pretty amazing, right? I have another example in Firefox, which is a little bit more elaborate. So this isn't like an official test or anything. I've just thrown together some functions just to make sure that it works. And there's one failing test, but that's because of a bug that I know about. So you can basically in this way control the browser from JavaScript inside the browser it's controlling. It's amazing, I think. This is an example just to show you I'm not cheating. This is the inspector in Firefox. You can see all of the payloads being sent backwards and forwards between the Selenium server in this case. You can drive it directly against the Chrome driver if you want to. You should be able to drive it directly against the new Firefox driver, which is based on Myonet, once we have an HTTP server in place. And you can drive it against Selenium. So that opens up a lot of opportunities. So to wrap up, it is a brave new world out there. And the web driver specification is opening up completely new areas which have previously been untestable for web standards. I encourage you all to take a look at the Test the Web Forward project. It's a really interesting project. It requires help, the coverage in the web platform test is not great. And so you can all help. So thank you. I'd be happy to take questions, otherwise you can just find me afterwards if you have comments. So the question is, the question was whether how much money the big companies contribute to the W3C, if I understand? Oh, of the test themselves. Yeah, so they currently don't really contribute that much. There are individuals working at the browser vendors who do this partly in their own time, but the focus amongst the biggest browser vendors isn't great on the platform. So you could say like there are lots of companies contributing, but the focus isn't great. This is why we need individuals and individual contributors. That's great, thanks. All right, thank you so much for coming.