 Hello. Hi there. So I guess everything kind of works, even I'm seeing here in stereo. That's a bit disturbing. Anyway, thank you very much for coming here today. This is our first QA workshop, and we are really excited. We haven't tried this before. So it's going to be two hours, more or less. And what we are going to do is go through the very basic steps of setting up test case scenarios for one feature, a specific feature that is running on. Hello. Hi there. So I guess everything kind of works. So I'm seeing here in stereo. That's a bit disturbing. Anyway, thank you very much for coming here today. This is our first QA. I thought it was a familiar voice, but I couldn't kind of, I didn't link that. Yeah, who's on IRC? OK, well, very good. But anyway, so there's many new people here actually. So we were thinking that it would be good to give you a bit of background, more in general. So how many of you are using Wikipedia? Can you please raise your hands? OK, that's not surprising. But how many of you actually have been editing? Wikipedia articles? OK. And how many of you have a technical profile? Do you work on technical works? And how many of you have done a technical contribution to Wikipedia? Right, so you have seen more or less the differences in hands here. And the context of this workshop is precisely all these efforts we are doing to raise the awareness that you can contribute content to Wikipedia. You can donate to Wikipedia, but also you can do technical contributions to Wikipedia. It is not just a website. It is actually a very complex project that involves, well, a website, which is the fifth most visited internet property, I don't know. So it's not just English Wikipedia, it's just a collection of all the Wikimedia projects. And this is pretty big. If you look at the other organizations in that ranking, they are basically huge corporations with very big teams and very big budgets. And we have none of this. But still, I think we managed to organize many projects and maintain a very good infrastructure quite well. But all this, of course, is done thanks to thousands and thousands of volunteers contributing content, donating. And we are aiming here to increase also the amount of people that contribute their technical skills with development, testing, documentation, and all the other tasks that are usually in software development. So in this context, testing is a very interesting area because you can be the super expert and contribute. And you can be someone without any clue about testing and actually do a very good contribution because that is precisely the users that we need. And then having users and testing specialists working together or under developers, of course, we aim then to increase the quality of our releases and being able to release faster, better quality software. So this is why we are focusing on testing. And within the different areas in testing, we also want to ramp up. It's more than ramp up because we have already started and we have something there. But we really want to make it more popular, the fact of contributing to automated QA browser testing. This is also a task that by the name would just make many people step back, thinking, oh no, this is not for me. But actually, we want to explain to users, editors, people interested in Wikipedia in general, that yes, you can get involved on this precisely because you know the software you're using, you know what to expect. And therefore, you can help us just telling the programs, the computers, to just do all these automated testing and saving us a lot of work. All these, Chris, will explain it a lot better. Is there any question? Do you have any questions like general? What's the Wikimedia Foundation doing? I don't know. Any questions you might have before we get into the specific topic of this session? Would you mind introducing yourself real quick? Yes, that's a good question. Sorry, but I was doing many things. So my name is Kim Jill. I work at the Wikimedia Foundation. I work in the engineering team. But my work is, well, it's a mixture of developer relations and community management. My work specifically is bringing more technical contributors to the project volunteers. It could be individuals, it could be companies, anybody willing to help all these engineering projects that we are developing. Any more questions? I'll give you a bit of information on logistics. So first of all, there's more people. So this is being recorded, or that's the intent, at least, is being streamed live or with a bit of delay as you could hear before. And there's also an IRC channel, Wikimedia Dev. And there's some people out there that are trying to follow this workshop. We haven't tried this before. We don't know how it will work. But in any case, here we are going to work together. But also when it comes to questions or speaking or saying anything, please have a microphone because this will help the people somewhere else today and somewhere else any day because this will be recorded and available. Second thing is, OK, these are instructions for the people here, those that are following online, I guess they know. But there's drinks and food down there. If you're at home, I guess you also know where that is. There's restrooms in this direction. So just follow the corridor, you'll find them. And now we're just in this setup, me talking, and then Chris will talk a bit. But actually, here we are going to be working. So feel comfortable. Just don't steal anything. But apart from that, you can sit in the desks that are more or less available here, here in the coach, whatever you want, OK? I think that's all from my side. I have just one question. Do you know what's wiki love? It's not what you think, or maybe it is, I don't know. Well, it's just a feature that I wouldn't even explain because Chris will get into that, but I will. But it's just a feature that editors or registered editors can just send wiki love to each other. So you're working on something. And then all of a sudden the next morning, you find out that paragraph that you were fighting with, you wanted to reference, do whatever. Someone came and did a great job. So you can just have your breakfast and do something else. So you go to that user page of the person that helped you, and you click on a little heart, and you send a bit of wiki love that you'll see how. So that's the future we're going to work on. I've made a lot of contributions to Wikipedia to the content, and I created a couple of technical articles as well. But I can't even figure out how to crack the door open as far as programmatic access to Wikipedia or contributing any kind of programs to the software. Yes. Well, there's many ways. And if you don't know, and you've been contributing a lot, it's awful, because you should know by now. And we should have explained it properly. There's a page at media wiki.org called How to Contribute. And that's a start. From there, you can see how to contribute from templates, bots, extensions, many different areas. But it's a white topic, and I think it's better that we can talk about this later. And perhaps we can open a door for you here tonight. Yes, I'm just here. But I really think we should get into the topic. And that's a white one. But for everybody here, media wiki.org, How to Contribute, that's a good start for all the other things that we can do apart from automated browser testing. OK, so Chris, all yours. I will be in charge, trying to be in charge of the audio visuals, and also will be paying attention to IRC just in case there's any questions, comments, anything, OK? Thank you, Kim. My name's Chris McMahon. I'm the QA lead here at the Wikimedia Foundation. I telecommute. I used to live in Colorado. Now I live in Southern California. And I'm not sure where I'm going to live next. But it's probably won't be San Francisco. This is a fairly new project. We started a proof of concept about a little over a year ago. It really got into gear back in the fall. It's had some ups and downs. And we're doing pretty well. I'm pretty pleased with it. And so if you know the topic, we're going to talk about cucumber. And we're going to talk about page objects. I see a lot of people here have laptops. And is there anybody in the audience that wants to walk out of here tonight with running executable browser tests? OK, I see programmers. OK, good. This is excellent. So I left some instructions on our meeting page about installing Ruby and downloading the software from GitHub and that sort of thing. OK, awesome. Everything work for you guys all right so far? Cool. So for those of you who aren't doing that, I'm going to suggest a couple of things. One is that I want to get these people set up real quick. I want to look over their shoulders and see what I can see. And if you want to like Mozi offer and look over their shoulders too, that would be pretty cool too. So let me just see what you have real quick just so I can understand. Oh, OK. Well, when you do, there are instructions. And what we've tried to do is make this pretty easy to install and to get up running on your local machine. We have a shared test environment, which is a default target for these tests. And I also left instructions. Let me go ahead and bring up a wiki page here for us. And another new project for us, not entirely new. But before I joined the foundation a little over a year ago, we didn't have a shared test environment. But it was really viable. We have a shared test environment right now. We can't really see the URL here. But it is called beta labs. And perhaps we could put it in the ether pad. One moment. And even if you're not programming, I would love it if you would just log in to beta labs. And let me make this larger if I can. Log in to, you can see the URL here, I hope, en.wikipedia.beta.wmflabs.org. Sorry about the long URL there. And the reason I'm going to ask you to log in, and if you don't have a user ID to create a user ID, is because I'm going to need your help. Because I know it's really risky. And I know all the pros say don't do this, but we're going to actually write code right here on the screen. And we're going to run it. And it's all going to be executable by the end. And the reason that I want you to log in to beta labs is that this only works if you're a user. You have to be a user to play with a lot of features on Wikipedia's. And in this case, we are in our beta labs environment. And you can't really go too wrong. You can't mess anything up. You can't really think. There's really, this is a sandbox for everybody. But it's a really robust sandbox. It's got a lot of extensions. It's got a lot of interactions. And the reason that I ask you to do this is because what I'd like you to do is when you log in, I'd like you to visit the Selenium user here if you can. If you'd like, you can visit me. Or you can visit some other user. Or you can visit your neighbor's user. Or you can visit my user page on beta labs. I'm using CMic Man here. I can't show you this because I can't do this on my own paper. But what I'm going to automate tonight is some tests for wiki love. And this is wiki love. And so if you log into beta labs and you visit somebody else's user page like mine, you'll have a little heart button up here in your tabs on the upper right. If you click the little heart button, you have wiki love. And wiki love is a really remarkable feature in a lot of ways. For one thing, it is a runoff project by one of our developers, Ryan Calderi, as I understand it. It is one of the most popular features we've ever released as well. Every feature we release has a certain amount of controversy. And this one is almost all controversy free. It's very nice. It's a good feature to use for browser testers. I have a list of qualities that make a feature good for browser testing. One of them is that it uses a fair amount of JavaScript. So there's a risk. It's going to look different in an explorer, in a Safari, and in Firefox and Chrome. Some of this JavaScript is not interpreted all the way. The other thing that makes a feature good for user interface testing, for browser testing, is that it requires you to navigate through the application to exercise different states in the application that depend on previous states. Because these are things that are very difficult to tell with a unit test, or an integration test, or an API test. So in this case, we actually have to click buying stars. We actually have to click food and drink. We actually have to click kittens. And it's kittens, you know? Yes. It's on the etherpad, and let me bring that up. You can see log into. Is that readable? I can make the signature. So I have been doing browser test automation for a very, very, very long time. It's something I specialized in, was the very first open-source browser test automation tool. It's called Watter. I was using it in the land, from Watter. And I was around when Selenium was released as well. So I've been doing this for a long time. And a lot of times, people come to me. And this happens to Adam, too. By the way, I introduced Adam Gaccio. He's visiting from Toronto. He's a frequent Selenium contributor, and one man about 10. Anyway, nice to see Adam. So he's a far more expert than me, this kind of thing. But anyway, people come to me, and they come to Adam. And they say, Chris, man, we need to do some test automation. We need to have some browser tests. And my first question is, I was, well, where do you want to test? I've had you want to test it. The answer is, I think we haven't thought very much about that. So that's what we need to do. So the reason that I ask you to log into Wikipedia, or log into the labs, is because I really sort of like to spend a few minutes sort of understanding, what does it mean to use working love? What does it mean to navigate your way through this application? How should this application behave? How do you think this application should behave? Hi, G. Nice to see you. Glad to meet you back. This is often a really difficult thing to say. What should the application do? How should this application behave? When I take action X, what is what happens after that? What should happen after that? This turns out to be really the hardest problem in acceptance test automation and browser test automation and user interface automation is specifying at a very close level the proper behavior. People are nodding, people look like they're open some eyes. Because really, truly, deeply understanding a feature and how it affects a user is the most important thing that you need before you can even start automating a test for a thing. You have to sort of understand what it does. Everybody sort of got an eye for Wikilev now, and you can send people some Wikilev. If you visit someone's user page, let me get rid of this so you can see the page around. On the user selenium user here, it's this little red heart up on the right. If you click the little red heart, it will have Wikilev. OK. Yes. This is Richmond. So for testing, I'm on the page on newwikilev.org. So yeah, so if you visit a user page and you click a little heart, you will have Wikilev. And so this actually brings me to my very first technical chunk of the presentation, which is that somebody's got to know what the software is supposed to do. And this is what Cucumber allows us to specify. In the nutshell, this is Cucumber. These are the existing tests that I wrote for beta labs, yes. Yeah. Yeah, we might. So if you click that, is there any? Yeah, you pretty much have to know who you're going to say nice things about. You can check with recent changes or whatever new action you might have with various users is why. People do nice things on Wikilev time. People improve your articles, people send you a message. People do all sorts of things. So as I said, the hardest part is actually, and particularly, specifying in a very close way what the software is supposed to do. And this is what Cucumber gives us. So I've written a very simple test for Wikilev. And it really was going to move. I'm going to ask you guys to help me write the next step. And we're going to walk through what that means to do that. So if you're looking at the feature itself, Cucumber gives us an ability to specify a background. This is the things that happened before each and every test that you had. So in this case, I'm going to, boy, I did. I'm going to visit the user pages so any user to click Wikilev. Every single test that I'm going to do is going to do these steps because they'll all show. Why write them more than once when you don't need to. So I click Born Stores. I'm going to say Born Stores Select Box. You should see a message text field. Born Stores is the simplest option. When I click Food and Drink, this is Food and Drink Select Box. The field's got to contain some baklava for you. If you go through those Store Pathels, there's an old joke around here about Store Pathels. We have a lot of Dutch, and we love their Store Pathels. And in this case, the Food and Drink option should have a message text field we're able to. And of course, when there's kittens, because kittens, you click kittens, you should be able to select an age. So you have a text field containing this text about a kitten, and you should see a message text field. So it's a very little test. It's a very simple test. And you may notice I have not yet actually saved a wiki of gift on the page. I simply decided that for this test, all I want to do is I want to check that the aspects of this feature are in place, and I can manipulate them, and I can address them properly. I haven't saved. I haven't decided if I want to do it. So let's run the test. For those of you who might have this up and ready to go, we are using a feature of Wiki. This is the Wiki one, as I was telling Adam. We're operating it such. I had some to tell you, oh, I don't know Wiki. I can't wait to run it, and I'm not going to run it. I'd like to sort of emphasize that Wiki brings itself to a very high level of abstraction. And what we're operating here is at such a high level of abstraction that the aspects of Wiki, the program, and the language itself are almost invisible. As you saw, that planing is preferred. So we use a bundle, which just simply makes sure that our environment is correct, without us having to even think about it too hard. Cucumber is a framework that allows us to tie all of these bits of abstraction together into a test. And I'm actually about to run the wrong test because I was doing some work on this one earlier. So I'm doing the proper test. I was actually doing something else in this environment. So we're going to use a pay-for-for Firefox. And hopefully the dinner grids are going to smile at me. And we'll see Firefox come up. And as I said in the background, we're going to log in, visit the user page, doing a cookie live. This is what my own start takes. I'm going to just check that this exists. I don't try to tap into anything. Not yet. I'll tell you more later. I just wanted to have just a really simple framework that's very understandable, so that I could show you and you can help out. Any questions while we are fascinated by watching the test run so far? Yes, sir. The instance variable log in? That is actually a marker. That's a little out of scope from my talk this evening. We mark tests that require logging with that. You can see the instance variable also have various environment names, like data obviously, if we are here, test to a production, we do some productionality with this. So as you can see, it's a little out of scope, but you can talk to me afterwards. I can tell you more about that. I have three scenarios, three tests, 20 steps, and three tests. So I'd like to go back to my cucumber file. I'm going to ask you, what do you think the next test should be? And you guys are experts at Wickey Lab now, right? You know everything there is to know about Wickey Lab. What do you think we should do next? We could do what? I was thinking actually, we might try, if you look at the options on the Wickey Lab page, I'm going to say, we might do that at the end. I'm just a little trepidatious about actually doing. One thing about shared environments is that things tend to be permanent there. Now, how I'm doing that, we have some very large number of very silly images, because we up there a lot of images for testing. And I'm not sure I'm going to up there a whole ton of barn storage right now. What I was thinking is maybe if you look at the Wickey Lab feature itself, there's four choices you can make. And I don't really have tests for three of them. That's a different question from IRC. It's possible to run the tests slower to be able to follow what's happening? I know you would. Not. We sort of want to run this test as quickly as possible. In the back of my mind, as I recall, there is an option to run tests slowly. I do not know what it is. I'm sorry, if I run it again, though, for you. But anyway, if you're looking at the cucumber file, what I think that we should do, so I think we should get some basic coverage for a fourth option. Because it looks a little suspicious, right? Because those first two options, they all have you just clicking and making a select one. The fourth option is really different than that. The fourth option has a whole different way of working. You actually have to fill things in and be creative and inventive and this sort of thing, right? You see what I mean? I was first two options, so I laid you through the garden path, though, but you didn't get the option here to make your own. Which, since we have our first three tests, we're actually doing a little thing now. We're doing a cucumber. But I got the fourth option kind of late as me. So I can stop this test. It's going to pass again, right? So what I think we should do is a new scenario. OK, sure. More. More separate? Yes, thank you. OK, I used to be a musician. I'm just used to eating microphones. What I like to point out, we're talking about cucumber. We're talking about page objects, too. And one of the things that cucumber gives us and one of the reasons that we choose cucumber in the first place is that, did I lose my screen cam? That's too bad. One more. One of the reasons that we chose cucumber, as I was saying, does that stop? Well, you are seeing cucumber. Can I'm going to reload this so I can come in back in? One of the reasons that we chose cucumber is that it affords us a very low barrier to entry. So what you so afraid I think it in is, yes? Frank, excuse me, I didn't read you what you're doing. You would say that you know quite a lot about Wikipedia, that you've been working with Wikipedia for a long time, and that you're exactly the kind of person we're looking for. Somebody who knows how the software works, who knows how the software's supposed to work, who can actually give us a file of cucumber scenarios and say, given I do this, when I take some action, then something should occur. This is actually incredibly useful a thing for us. So can I see your screen? But no one doesn't seem to be sure. Selenium, that's a really good question, actually. Selenium is a tool that drives a browser. Cucumber is a framework. I'm sorry. There you go. Now we're back. Selenium, OK, so I'm going to take a technical digression here. There are two versions of Selenium. There's a Selenium version 1.0, which is older. And Selenium 1.0 worked by opening a browser inside of a frame and injecting JavaScript into that frame to manipulate the DOM itself. And browsers would then jump around thanks to JavaScript being injected in a frame. Today, everyone, there is Selenium 2.0. Selenium 2.0 has an engine inside of a web driver. Web driver is like this. It's not approved yet, right? It's like web driver is this course to be approved as a web consortium standard, or W3C standard, as the standard interface for automating browser behavior. And this, of course, requires the browser vendors to agree to the standard. And to this state, the web driver protocol is fully supported in Firefox, Chrome, in the Explorer, Opera. I think that's all. Safari, thank you. The browser API is called WebDriver. The outward-facing implementation of that API is called Selenium. Selenium, the API, again, to my knowledge, is fully supported today in Ruby, Python, C-sharp, and Java. And Facebook only just recently, like a week ago, right? That's SeleniumConf released a PHP interface for it. Period. So not to hijack entirely, hijack somewhat. The face, there are about seven or eight different full PHP implementations of WebDriver. Facebook released one about two years ago and then ignored it for two years and released a new version that's completely backwards incompatible with the previous one two weeks ago. But the new one looks pretty nice, but I haven't played with it yet. So, yeah, WebDriver is a standard. OK, one more technical detail, and then I'll move back to Cucumber. Ruby WebDriver is really very nice for a number of reasons. Is that Yari Bakun, who maintains it, builds the Selenium Ruby interface automatically, constantly, and he does it off the standards. The WebDriver is built automatically off of the W3C standard, and the Outward Facing API is built automatically off of the HTML5 standard. So as these standards mature and evolve, the Ruby client interface matches them automatically as they progress. So this is a very nice feature that Ruby gives us. Cucumber is a way, all Cucumber does is it gives you this. Well, it gives you some other things. Cucumber gives you the ability, say, to specify in a form of given when and then what should happen in the software. Behind these given when and statements, you could have objects. You could have API calls. You could have meeting minutes. It doesn't really matter. Given I turn the handle on a door and I push, when I stick my head in, everyone in the room says it should sell out. I mean, you can use a Cucumber standard to do to implement anything you want. We happen to be putting Selenium Browser testing on the back end of this. So let me show you a thing. I have a new test step here, right? We might click Make Your Own. So I'm going to do a new thing in Cucumber. We don't need to have a test run, right? So I'm not going to run the whole thing. I'm just going to run the very last thing. Cucumber gives you the ability to specify a line. Remember, this is really kind of convenient. So I'm going to run my test from 9.20. So I'm going to run the third feature, and then I'm going to run my new feature. And notice I've had no code run so far, right? No code whatsoever. And let's see. Did I not share my screen property? Come now, people. There we go. This is better. OK, so what did I do incorrectly? What did I want to do? I hope the don't have had a smile on me. Like, I think they might not. There's a little slimmer silver here than I'm used to. But one of the lovely things about cucumbers and one of the lovely things about Selenium and one of the reasons that it's so popular is because you can watch the browser. I mean, I find myself just hypnotized by watching the browser just move all by itself. So I'm going to click kittens. And if I save the correct file, I should have not. One moment. Sorry about that. Here we go. This is what I was after. So you notice without actually writing any code at all, Cucumber has actually given me my next line of code for me to write, for me to use. And as I said, we do this by simply writing a line, given winner then, that is going to be our next test step. And when we run it, we will then get a line of Cucumber executable code. It's pending now, so it doesn't actually do anything. But what we can do with this is that we can add it to our steps file. And we can add it anywhere we like. And we can, as you can see, we express the regular expression above with the code that you wish you had. So let's do that. And so from the examples, you can probably see what we should do next. We should say on. And we're going to want to tell the page that we want to be on, which is the Wiki Loaf page. We're going to say. And here's, at this point, we start moving into the page object part of our world. Page objects are an abstraction of a page. Page objects are what we want to do. We identify an element of a page with some aspect of that element, like an ID value, or a name value, or a class value, or the text that's contained within that element. We have a number of ways to identify the page element that we want. And then once we have those identifiers in place, we can name that element anything that we want. I'm going to refer you, actually, to the documentation for the page object. Ruby Gem, it's on GitHub. It's maintained by a guy named Jeff Morgan, who is known to everyone as cheesy. And the documentation here is really absolutely top notch. I recommend, if you're going to dive into doing the browser test automation in our framework that we have here, I have this page open constantly. This is my reference for everything. And one of the beautiful things that it does is it talks about how you can identify elements. Like you can identify a button, for instance, with ALT, with a class value, with ID, index, name, source, text, value, x path. And then while that's not true in the raw Selenium IDE, using the water API, which is even a higher level API, we can specify multiple identifiers. We can specify the third instance of class x on a page. We can specify very, very closely exactly what element that we want to do. But in the meantime, all we got to do is we got to decide what we're going to call that thing, where you click Make Your Own. Because we got to click Make Your Own, right? So what are we going to call it? We'll just call it Make Your Own. That's what we call it. And you can see from some of the other examples that we have. And if you were to read the documentation for page object, you see that what we're really going to need is to make your own element. And then if you were really excited and you really were hip to reading this documentation, you would know that we would have to tell the page to click our thing. It's getting a little more complicated here, right? It's not playing English anymore. We have to actually think about what we're going to name our thing. We're going to have to think about what page it's on. And we're going to have to think about what it is we're going to do to this thing. And in this case, we're going to click it. And again, I'd like to sort of emphasize that one of the reasons that we chose the framework that we did and the implementation that we did is that the barrier to entry is quite low. If you are not a programmer and you were to submit a cucumber file that simply statements are given when, then, for your favorite feature, that's awesome. If you want to give us a step file that's full of things that don't run, that's great, too. Because look, I can run this. Cucumber has told me my next step that I need. So I don't have to think too hard. I just have to put something in place and then I have to run my step. And as usual, since we have a background step, we're going to bring Firefox up that's going to log in. And once it finishes logging in, we're going to navigate to the user page and click Wiki Love. We're going to click Make Your Own, right? Oh, no, we have a terrible, terrible, terrible red error. But that's OK because we just made up that name. We just made up the name, Make Your Own, right? So we get an error message. It's a very important error message that says, undefined method, make your own element for an object called Wiki Love Page. And remember, in our steps file, we said that we were going to be on the Wiki Love Page. And we're going to have a thing. It's called Make Your Own Element. And we get an error message saying, right up front, what we have to do next. Anybody, what we have to do next? We have to put Make Your Own onto the Wiki Love Page. So in order to do that, I'm just about to get there. The question was how to get the dump. Yes. I'm getting complaints from IRC that they don't hear the questions. I will repeat the question. The question was, how do you get the DOM for this? How do you identify this element? And as I said, we're not there yet. This is our next step. Because right now, as I said, we have told those developers, we've told me, you've told Jelco exactly what needs to be done. We need to click, Make Your Own. And we've given you a step. We've even called, we've given a name to that element. We've told you guys to click it. So now, we need to actually put the element in place. So in order to do this, we can do this several ways. Probably the nicest way is, I take it back. This is the nicest way. If you right click in almost any modern browser, you'll have a thing called Inspect Element. There are a couple of different DOM inspectors. There's Firebug. I don't know, anybody know what else there is? There are Chrome Developer Tools. It's F12, an Internet Explorer, right? OK, yeah. Anyway, there's tons of ways to do this. There's a DOM inspector. You can right click on an element. And what we can see is that this particular element, in this particular case, it's down here. It's a div, has a certain class, has a certain text. And I'm actually, I'm really pretty fun. In fact, if you guys remind me, I should really buy Ryan Caldaria beer. Because as I was automating this application, I found it's really very cleanly written. It's really very well designed. This element doesn't have an ID, which you might want. But it's quite nice. It's quite a straightforward way to identify it. And we'll identify this one the way that we identified all the other ones, which are very similar. We'll say, and this is, and right now we are talking about pure page objects. This is page object syntax. We want to make a div. We're going to say that what we want to do is we have a div. And now, our next argument is our totally made up name for this element. That every time we use this name, we're going to use the same name for it, which we call make your own. That is what we call it, right? Make your own, yes. Make your own. And we know that this div has text. And that text is make your own. So we save the file, and we run our brand new scenario, except I have to edit properly, right? It doesn't work if you put the colon in the wrong place. That live coding thing, right? We were just talking, Adam and I, the guy that maintains the Selenium WebDriver bindings in Ruby does this all the time. He is scary typist. He is the most accurate typist I've ever seen in my entire life. He can code for an hour in front of 500 people, and he makes no mistakes. It's eerie. And there you saw, it turned orange. We clicked, make your own. Our step worked. Our step actually worked. How about that? Any questions at this point? Is it fairly clear what we've done so far? We have plain English. We run plain English. We get the outline for our next step. We make up some stuff for our next step. We run that. We get our error. We put our element in our page. Bingo, we have a working test. Is there any part of that that's not straightforward, or any thing I can clear up? Any question, really? Now it's been one hour, so it's a good test also to see if you are following. So if you have any questions, now it's a good moment. Anybody actually writing tests along the way? Thank you. I was really hoping people would. What's that? Oh yeah, we haven't checked any of this in yet. We can do that. Oh, go ahead. Tell them what. Actually, can I take the mic? I haven't been writing new tests. I've just been following along with the ones that you're doing. That's fine, but you've been typing them out, right? You've been typing them out. You've been running them as they go? Yeah. Awesome. Thank you. This is, you are like, you are prepared to be part of my team right now. You are totally prepared to be part of my team. We have a mail list. Kim can set you up. Just let us know. Actually, now that I'm standing from this position, I can tell that I don't see many tweeters, et cetera. So people are really working here. So we can do something a little fancier. If we go back to our, I managed to close it. Let's go back to our, we call it a feature file. These steps in our, in our cucumber things. We can take a look and see in the barn star, right? I should, I have a message text field, you know, and the food and drink and kittens, you know, they have some default values, right? And this test is going to check the default value is some baklava for you. The default value is kitten for you. I'd like to point out one sort of nice aspect of cucumber as well because everything inside your step is a regular expression. We can check for header text field containing anything we want to. What this line does is we can call this step. I check for the header text field with any strings that we want. We, every time there's a similar element that has a string, we can just grab that string from the cucumber file. So if you decide tomorrow that, excuse me, that this should say some strobe awful for you. I mean, we can just do this right now. We can just, you know, it doesn't, but we can change this to strobe awful if we can change it to anything else we want. So, and our step files are going to be fine. We don't have to reuse those. There was a question. Hello. Both Maggie and I are getting way more deprecation warnings than you. And I'm on Ruby two and Maggie's on 193. There is a line, are you using RVM? Yes. Okay, there is a line on the meetings page on mediawiki.org where I point out at the bottom. Yeah, try, are you aware of the meetings page there? It says, slash meetings slash 2013, 627. They're at the bottom of the page under run and test. Yeah, that's what I'm testing. Yeah, there's a, you see the, you might have to do the source. You might have to source in your dart.rvm file. Try that and see if that gets rid of your deprecordification warnings. There's another question. Hi, yes. Have a seat in a while. Oh, wait. It looks like you might be seeing the wait until present depreciation warning, which I think has to do with some of the log-in steps. Oh, yeah, it could be. I don't think it's something to worry about. It seems to be working okay. We do have a few deprecation warnings. That could be one of them depending on version of the jam you happy running or the version of Ruby that you happen to be running. We try it, we try to keep things pretty up to date. So, actually, we keep things really up to date. I had a question. I was wondering about, you said about matching messages. When, do you have any support for internationalized that internationalized messages? So, presumably some of these things are actually coming from message files and depend on what the current language is. Yes, actually, we have, Jelka and I complain about this to each other mostly, that in what we would really like to see is a lot of ID values, a lot more ID values for page elements than we do see. We end up having to, like for instance, on the make your own, we have a class that is not unique on that page and we have the text and that is our only identifier for that element. So, we have no ID that we can work with. ID values, in case you didn't know, we're a W3C standard, is there's only one unique ID value per page. Name is not constrained that way, but name is also almost always one unique name per ID per page or per element per page. It is a concern, actually, it's a great concern because as we do, as you may know, we do internationalize our Wikipedia. It's in how many languages now? 100 and 300 and there's hundreds of languages. Our language team is incredibly devoted and incredibly competent and we do internationalize these messages and it is a concern, let me put it like that. And we would like to see more robust identification of elements in quite a few of the Wikipedia web pages that we see. We do have some tricks though. I mean, there's always another way to skin a cat. This is just a super simple demo. But yeah, very good question and it's a really big concern. Any other questions? Okay, well, I think we might try to run another, I think we might try to make another step. We'll be bold, as they say. So I'm thinking that if we click make your own, what do we do next? Given I click Wickey Love, when I click make your own and what should I do next? I'm thinking I need to type, right? Yeah, I don't have anything I can select, right? I just need to type some because that's what this aspect of our feature is all about. And we have to remember a good browser test is something where you have to navigate a path. So in this case, we have to click a thing, then we have to enter some text, then we have to enter some more text. So I think we need a feature that when I click make your own, then I should be able to type into, to type into, what was this called? Header, your own header, what do we wanna call it? Anybody? How about your own header? Is that all right? So one of the reasons we wanna be, we wanna take these lines, these lines in Cucumber, we want to make them as descriptive as they can possibly be. Because when this thing fails and some manager is reading this test failure, this manager does not have time to dig into the steps files, to dig into the page files. This manager wants to see what step failed. So we need to really be, we need to take some care. We need to take some real care with how we describe what it is we do here. So then I should be able to type into your own header. I think that's pretty descriptive. If this goes down, we're gonna know why. And as before, I'm gonna run, I can run out of line number, I'm gonna run my very last scenario. What happens next, anybody remember? When I run a feature file that has a line that's not been implemented? Remember what we're gonna see? We click it. And then sure enough, I get some beautiful yellow text in my particular shell that gives me my very next step. It tells me just exactly what I need to do next. So I need to write myself a step. And we can put these steps anywhere. We do this a lot. We have a convention. We keep our given when thens in alphabetical order. We keep the elements in our pages in alphabetical order because as these get larger, it makes less sense to have them in the order that they are executed. Cause you just wanna find them again. So we keep them in alphabetical order. It does not matter the order that these happen to be in. So if you remember, we're just gonna follow the conventions on the existing test. So we're gonna say on wiki love page. And what do we wanna call this? You know, we're gonna call it a thing. Gonna call it your own header. And just to be really safe, let's go check the documentation. We're gonna look up what old cheesy says about text fields and how we manipulate them. So text field. So Jeff tells us that if we type just a name of the element, we're gonna get what's in it. And we don't care about that cause it's blank, right? And but if we type the element equals, we can set that value. That's what we wanna do, right? We wanna be able to set the value because it's empty now. It doesn't do us any good unless we type in that. So we're gonna type in there. We're gonna use this right here. Cause Jeff tells us that just a good guy because it's lots of methods for our objects in our pages. So we're gonna say your own header equals, I think it's a string. I should use single quotes, not double quotes. And so remember, as always, if you get really, we like to do this. We like to, if you ever pair a program with Yelko, my colleague in the QA department who is the author of much of this architecture, this framework is a really expert guy. These particular set of tools. And if you ever pair a program with it. Let me just, I know also who is in Croatia now on IRC. So it must be, I don't know, three, four a.m. Yeah, it's about five a.m. in Croatia now, right? Five, 30, something like that. So he's following and active answering questions in IRC. Anyway, as we know, oh, quick make your own. So I made a mistake. I think I have two of these. Yeah, this is wrong. That's not what I should do. I did the wrong thing. Let's just get rid of all this. Actually, let me just do this, because I've lost my mind. You have steps, and what I called this was, I should be able to type into your own header. Sorry, I had to type in. The demo gods have been pretty good to me so far. And the error messages are really nice. The error messages always tell you exactly what went wrong. But as I was saying, if you ever prepare a program with Jelko, he's gonna force you to make your test fail first before you make it pass. It's an excellent practice. And I am often too lazy to do it. Jelko is quite a code reviewer. He's quite a task master. He doesn't let very much get by him. This is awesome. This is exactly the error that I wanted to see. As always, it's telling me that my wiki-love-page object does not have a method for your own header. So let's make that happen. In our page file, if you recall, we will say, and we have to find out what this is because it's a text field, right? I'm pretty sure it is a text field. Oh, and it has ID. This is lovely. See, we always love to see an ID value because we always know we're gonna hit the right element when that happens. So it's a text field and our own special name for it is my own header, as I recall. Is my own header? Oh, your own header. Excuse me, that wouldn't have worked. Your own header. And in this case, we have an ID and the value for that ID is MW wiki-love header. Which is a very lovely thing. As I said, IDs are awesome. If you're a Wikipedia developer, give us IDs. And if I have not made any outrageous typos, you should see my string actually populate. Make sure this is correct here. Lovely, automated test type that right in there. Everything's passed. This is a beautiful thing. I'm gonna stop here for any questions. Real fast. Anybody have a question real quick before I go? I'm getting a capture image when I run my test. I didn't get that earlier when I ran it from outside this location. It's a timing thing. Oh, that's why. That's why. I just told them this was a capture. Yeah, and this is a really great example, too. I'd like to point out browser tests are inherently fragile. You are inherently going to get what they call flaky tests. Any tests, and actually I'll tell you, having to log in at all is kind of a smell. We don't really like to log in with a test because it's kind of a waste of a lot of time and steps. But since we're running these in our shared test environments that are publicly available on the internet, we have to have a login step. And this is absolutely something that happens to me. When I run my suite and my tests do some failed logins and it doesn't work, I have a lot of a bunch of bogus failures because it's craft and cunning mostly. Good design actually eliminates a lot of these sorts of bogus flaky failures. What's that? Therein lies an entirely separate presentation, I'm afraid, my friend. I run the test from a coffee shop three blocks away, but I can't run them in here. That's pretty interesting. I don't really have an explanation off the top of my head except that it looks like whoever you're logging on has failed too many logins. That's my suggestion off the top of my head. I'm gonna do, if there are no immediate questions, I'm gonna do one last special thing. And it's kind of, I'd like to, actually I'd like to thank you all very, very sincerely for helping me write some real tests right here. And I'm gonna check this in right now if you'll be a bear with me. So we use get and make sure my files are good. Yeah, my files are all look really good. So I'm gonna get, just to prove that this is real. We're doing real work here and you guys are helping. Yes, ma'am, do you have a mic? Would you mind just opening each of the files in your text editor to make sure they're out? Yeah, one moment, I'll do this. Because we're a little behind. Let's do this. So I think you have this, yeah well let's do that. I'll do that later. This is more important. So opening these files. We have our feature file. We call this a feature file. The one, the important things about a feature file is that it has at least one string called feature and a name and at least one string called scenario and a name. This is pure cucumber right here. Okay, any questions about a feature file? Okay, steps file. Remember when we run our cucumber feature file with code we haven't implemented, we get that outline, that pending step. This goes into our steps file. On our steps file, this is where the page object starts coming in. We're gonna, that wiki love page are the pages that we're using to test. Our objects in Ruby, they have a name. They include a page object that follows by our own special name for the page element we wanna manipulate, plus whatever we wanna do to that element. That's our step. That's the action that we're gonna take here in our steps file. And then finally on our page file, this is where we specify what the page is. You can see it is the wiki love page. It includes a page object. It has a particular URL. We can use this, reuse this one URL in multiple test environments, including a local environment, including anywhere else. And then each element is specified according to the documentation that Jeff wrote for us here in the page object thing. So as I was saying, we have a very special page, my favorite page in all of Wikipedia. It's a TLDR page. So, okay, excellent. So get... That's all right. Excellent. Test steps. Written writ at WF with volunteers. Get review. So I am going, we have a code review tool here at the foundation. It's called Garrett. Code review is a really, really important aspect of quality at the foundation. All code undergoes code review. It is considered extremely bad form to check in your own code without review. Let me just go somewhere where I'm logged in so I don't have to go with this. As you can see, our tests are here. These are the changes that you and I made. I have a couple of... Jelco is going to have minus one me because I have some blank white space in here that he doesn't like, but that's okay. And I'm going to let Jelco review this. And then I'm going to leave a comment that says, these tests are pretty good. I'll clean up the extra white space later. Soon as this gets merged, our tests are going to be running at least twice a day in the test two, or in the beta labs test environment. That's all I have for you. I want to thank you very much for your help and for your attention. And it's been lovely working with you. So now it's logging me in okay, which is bizarre. Now I suspect that might have been what happened like on Facebook, if I log in in Berkeley and I come to San Francisco downtown, it says that I'm in a strange location and it won't let me log in. So I suspect that maybe that's what your site does if you're at a different IP address. Could be, could be a done out. It's weird, but it's logging in now. Awesome. Any other questions? Non-deterministic. Any other questions, demos? Wait, wait. First, I would like to say thank you Chris for this tour from zero to 100. So please. My pleasure. Very much my pleasure. And I think, I think there's some people here that ask questions. So I think we can, we still have like 20 minutes, a bit more. So I think it would be very good if, yes, we have questions, but maybe we can break a bit this setup and just go a bit more casual. There's also many of the people here that can ask questions, answer questions. So what do you think? Yeah, let's, you know, by all means. Just sounds like we've covered it. Sounds like people are pretty straight and find me or ask Yelko on IRC or talk to Adam or your name, man, in the yellow sweater. Yeah. Maggie knows what she's doing. Thank you all very much. I appreciate it. So now we will do this. Someone on IRC online, we will cut the connection and then we'll still stay here. There's some drinks, oh, there's a question. What can we do to get involved, contribute? Well, yeah, that's a question from, so. Please see mediawiki.org. Take a look at mediawiki.org. Do a search for QA. Yeah. Also the QA mailing list, there should be, is the mailing list on the meeting page yet? I will, well, you all go into questions, answer. I'll just update the wiki page. I'll put a link to the QA mailing list, which is exactly discussing topics like the one today. You can subscribe there, and this is the best gateway to get involved in anything QA, Wikimedia. List.wikimedia.org. You can sign up for the QA mailing list. If you'd like. All right. So, that's that. I will still follow up with the IRC. I will update pages, and here now, for instance, you have questions, so you can ingest. That's a really good question, actually. Can you repeat the question, Tim? The gentleman asked, are we planning to do test-driven development? And the answer is, I certainly hope so. We have not done that yet. And one, there are really two ways that we can go about using tests like this, using Cucumber in particular, and tests like them. It's called acceptance test-driven development, ATDD. And I keep coming back to this. The most important contribution we can have are people that truly deeply understand what the software is supposed to do, how the software is supposed to behave, and how to describe that behavior. So, in the scenario that you're talking about, we write the Cucumber test before we ever write a feature. We say that before we even code a line, here are the conditions that our feature must satisfy. In my experience and anecdotal evidence from my entire career in the last decade, plus a tiny, tiny, tiny fraction of people who do browser tests actually work in this way. Almost everyone uses browser tests as regression tests. And they're incredibly effective at regression tests. So, this is what we do at the foundation. Almost, well, so far 100% of the time is we are taking existing features and we are building coverage for existing features. We are moving hand-in-hand step-by-step with some of the features under development. We have constantly morphing browser tests for our visual editor, for example. We have an intern from the GNOME Foundation outreach program for women who is learning just as you are learning. And her target is the visual editor, a much, much more complex and tricky application than WikiLove. I cringed to try to show you how to automate visual editors. It's a very complex product. And Rachel is spending a significant amount of her time learning how to do this. I would love to do acceptance test development instead of only regression testing. It's not been a super high priority for us as yet, but I would love to see more of it in the future. Great question. Yes, sir. How come so many of the Wikipedia pages don't pass W3C validation? I have not been here long enough to give an experienced answer to that. My theory is that it's a historical accident. Wikipedia itself is only a decade old. The Wikimedia Foundation is, when did the Wikimedia Foundation start, Rob? 2002? Well, 10 years ago, yeah. Yeah, okay. We just had our 10th anniversary. But the foundation at first was like three people, and it was five people. Yeah, yeah. And it's like, it has not been a high, well, go ahead. Yeah, so on the W3C standards front, so what, like we do, I think have reasonably good HTML5 support. Like, so if you're looking for compliance with HTML5 spec, it's like, there are probably our bugs in there, but are there particular validation errors that you're particularly concerned about that many pages have? Yeah, practically every page I try, especially if I'm editing a page and I stick that into the W3C, I get validation errors. In fact, there's a Wikipedia page listing the things that are, that always occur. Yeah. This topic has come up on the Wikitech L list a number of times. I'm not sure if anyone specifically here can answer this question very well, but if you search the Wikitech L archives, you can probably, ah, we have someone here that can't answer this well. Hi, I'm partly responsible for that, I guess. I did not introduce it, but I did vouch to keep it. And that sounds a little weird, like why would you keep an error? So, this is so much controversial, but the way, I'm partially summarizing other people's point, so this is not entirely me, but basically the W3 validator, as you're assuming, you're referring to the validator specifically, is a useful tool to catch issues. It is, however, not a lint tool in the traditional sense for programming because it's not actually what browsers use. It is what browsers should be, ideally, but they aren't. And as such is, it has a lot of things that might be different, but browsers don't actually do. And as such, sometimes the most viable solution to fix a bug is to do something that browsers support in some cases, but the validator might not really validate. That is true, and that's actually one of the reasons that we run our tests in a variety of browsers. As of today, we run these in Firefox, Chrome, IE 10, IE 9, IE 8, IE 7, IE 6. And yes, we do actually identify significant behavior differences in different browsers with exactly the same college page elements. I think the most significant example is, in our lists, we have a UL, and some of the menus, we have an empty list, and the W3 says we shouldn't have an empty list. And this was actually fixed by switching to HTML5 because browsers have always allowed it, and they implemented it in spec. More often than not, it's the spec being outdated and behind what browsers have already done than the browser being behind and spec being forward. It's usually the browser being forward. And if you would check the value of the same page again, and a year from now, it might validate exactly in the same state. Any other questions? A little far field. Kim? I already closed one time the session. I think really, I think we can just leave the mics apart, stop the streaming, and just stay around having more conversations and just questions based on I have this in my laptop, et cetera. Is that the thing? Yes? So I assume yes. Thank you.