 We're going to talk today about writing some tests in plain English for the search feature of Wikipedia. We're going to be doing this all week, and you can join us on IRC, on Pawn Media, Wikimedia Dev at any time. So we're about to start making some changes to search, and we're going to need some browser tests for them, some automated browser tests. One of the ways we can do automated browser tests is we can find existing bugs that are likely to be fixed. We can make a test from the bug. And we're looking for bugs that really affect browsers. So I found this one, and I liked it very much. This actually seems to show a problem with search in different skins. There are the two most popular skins on Wikipedia are Vector and MonoBook. And so this bug, 24414, shows that the search behavior is different for these two different skins. So anytime we have a skins issue, we want to get a browser test on that. So what we want to do is we want to make a test that will define the behavior of search for these two skins. So once we fix this bug, we can make sure that it never comes back to virus. We're going to build ourselves a regression test. So what we really need more than anything else, we need a description in plain English language that talks about what the feature should do. And so I read that bug, and I culled the sentence out, and you know, I'll read it for us. In the Swedish Wikipedia, when I type A circle into the search box, then my results should start with A circle and should not start with A. This should happen in both the Vector skin and the MonoBook skin, and an Oumlat character should do the same thing. If you look at our Wiki page on mediawiki.org for the test backlog, you'll find this sentence there already. I added it, but anyone can add to the backlog, and the more potential automated tests we have in the backlog, the better off we are. So we do these tests in plain English using a tool called Cucumber. In Cucumber, we can define a feature, which is our overarching test. Inside that feature, we can have any number of scenarios, and we're going to name that with some aspect. And then inside each scenario, we have a very popular format of given, when, then. Given is like our set-up step. That tells us where to start the test. And when is always some verb, it's always some action. And then we should see some sort of observable result in the browser once we perform our action. I'd like to point out two things about how we're doing given, when, then, here. And the first one is that we like to use the word I. I take some action, and I should observe, because our tests, we think of our tests as users. Our tests are working the browsers exactly the way a user would. And so we want to use the word I, because that makes it really personal. The other thing is we always use the word should and should not, because those words point to some concrete result that we really want to check in some way. So we'll make a Cucumber test for our search issue with the two different skins. And we're going to call the overall feature localized test results. And our first scenario is going to be called search results match input and vector. And so we set up the test. We have to be on the Swedish Wikipedia, and our skin has to be vector. We can put an and statement after any of the given and when events. And Cucumber is great with that. So when I take an action, when I type this character, A circle, into the search box, then I should see a list of results that start with a circle. So this is what we call a happy path test. This is what should happen. But we're software testers. We're conscientious about our tests. So we also want to have a negative test. We want to go in and we want to make sure that our results don't contain incorrect characters. So once again, we'll set up, we'll be on the Swedish Wikipedia in the vector skin. And when we type A into the search box, we want to check. We want to make sure that we do not see results that start with the capital letter A. But there's still lots of things untested. We have two scenarios here. But if you recall, our original statement needed some other skins and they needed some other inputs. And we need to make it so that these tests are maintainable. Because the behavior of the software could change in the future. And we want to make sure that we can always keep up with any potential changes into the future. So once again, we're going to review our plain English test just to make sure we know what we're doing. We have to check the vector skin and the monobook skin. And we have to check that Oum Loud input as well. Again, anybody can add a plain English set of sentences as an automatable test to our backlog. Cucumber lets us do a really interesting thing. Many other similar tools do as well. But in Cucumber, we can define a scenario outline. And a scenario outline lets us call this set of steps, an arbitrary number of times with different parameters each time. So we're going to turn the name of our skin into a variable. And we're going to turn our input character into a variable. And then we're going to turn our expected results character into a third variable. And then we're going to call this set of steps as often as we care to using a table like this. So you can see we are checking that this is our happy path test. Our A circle should yield A circle results. Our capital A should yield capital A. Our Oum Loud should yield Oum Loud results. We'll check this for vector, and we'll check this for monobook. And of course this is great because someday in the future we may need to check another skin. Sometime we might want to check some other inputs other than just these two. Because there's a number of rules about which characters are considered equivalent and which are not. And we do the same with our negative test. We can see that our search results should not start with the wrong character. And we want to just flip those characters around. Our A circle search should not yield results that start with the capital A. It's a very fast overview. But in summary, what we need for our test backlog, more than anything else right now, are some really specific, concrete descriptions of how particular features in Wikipedia should function. We just need a plain English descriptions of how things ought to work. Once we have this, once we understand what it is that we need to test, we use Cucumber and we can make these given when then statements executable tests. And we take our given when statements and we make them as precise as we possibly can. Because when the tests fail, those steps given when then are reported in the output of the test failure. So even at a first glance, we know exactly where our test went wrong when it fails. And finally, we really need to feed the backlog. If you look at that page on mediawiki.org, we have a number of projects going on right now. Everything from search to the visual editor to the upload wizard for comments, image files to article feedback. We have all of these projects ongoing and we would love to expand the test coverage for all of them. So if you know something about features on Wikipedia or you want to learn, we strongly encourage you to help us feed the backlog and we will automate those tests in the priority order as soon as we possibly can. For more information, of course, our test backlog and contribute at will. You can look around our code repo in GitHub to see examples. The Cucumber tests we have right now are in features. They're called feature files and they're quite easy to read. You can see what we've done so far and you can watch what we'll do in the future there as well. We are always on pound wiki media dev channel in FreeNode IRC. Shalco Philippin, myself and Kim Jill are always, always happy to answer your questions and we'd love to collaborate with you to feed the backlog. Thank you very much.