 Great, so hopefully I was in the right room. This is, how do I put Ores on my wiki? Yeah, so just a quick introduction in case anyone's wondering for people in the video who might not have the same background as us. Ores is a machine learning platform that is integrated with media wiki. And there are two main types of analysis that we're doing, one is of revisions. So that would be along the lines of, is this edit manualism? And the other type is of article quality, which is, it's, I can explain why that's useful, I guess, that previously to assess article quality has to be done manually. And it's a pretty labor-intensive process. It takes kind of, it also requires someone who's sort of an expert in grading articles to do it consistently as well. So then they take a rubric and they look at an article and see if it satisfies this rubric and then they give the article quality grade. And so to do this automatically allows us to do longitudinal studies and look at how article quality changes over time. That's cool. We have a robot running this for you. And then revisions scoring is, this is an idea that's been done for a while now. There are tools like Clubop and Hubbell, which they look at changes and they do pretty basic machine learning algorithms on them to decide if an edit should be allowed, it's been syncopated or not, and then they either lag it or revert it itself. So, course is an attempt to kind of bring all that under one umbrella and give it a consistent interface and make it possible to maintain the machine learning model across multiple wikis because I think that's a really big burden to ask some single-bot volunteer to do. So, this just got started two years ago, so yeah. And there was four 50 different language wikis already. And hopefully you're here because you're interested in extending the support to your wiki. It doesn't have to be a new language either. It could be an existing language wiki, a different project, so you could be extending this to the source, wiki voyage. And it's, sorry, I got lost for a little bit. Yeah, oh, I should move it to my side. So that you could see me. Okay, yes, so what is worse? You know, one thing that I, so, and I went through this kind of quick, but I was gonna help out with the, like, what do we do with part of the quality? And, oh, God, this has a, oh, it doesn't have a quality, so we correct. While you're typing, Erin recently wrote a paper that studied the quality of women's scientist articles on Wikipedia over time and was able to, maybe not prove the causality, but to correlate a very big change in article quality with the efforts of this amazing volunteer, and the Ludicula, and that's what it looks like. And so this is what's possible now with the article quality models. So previously we had, you know, one or two of these eight points. Literally, like, a few articles were scored a few times throughout time, and there was absolutely no way to see any trends in that data. And now you can just take a whole set of articles you can run through this and then grab them on time. I guess it's worthwhile explaining the difference between the revision scoring models also. That's probably gonna be confusing when we get started. The first type of model that we can put on your wiki pretty easily, and by pretty easily, I mean, there's a lot to do, but you don't have to run a large handling campaign. The simplest type of revision scoring model is the reverted model. The reverted model takes the examples of articles that have been converted and it does, I guess it's called supervised learning way, but it finds patterns in those articles that are recognizable enough that it can be extrapolated to other articles and other changes. And so this basically says, was the article reverted if it was, then articles like it, or changes like it should be reverted in the future. What you're looking at here is the actual JSON output of the API. And in here, yes, there's a top level prediction which is the easiest thing to use if you're ready to go to you to take advantage of Ores. Ores? There's some confusion with the pronunciation type. We think either one is okay, but initially it's Ores. Yeah, so this says that the change should probably won't be reverted and this is the probability that it falls into each of these classes. This is the won't be reverted class and this is the will be reverted class. Those are pretty close. It's probably an interesting headed. And there's no easy way to go from the original ID to the Ores type of singular question mark if equals. Oops. Yes, so the reverted model is problematic, however. It turns out there are a lot of reasons that articles are reverted. And so the analogy that I used to understand this myself is, imagine that you're doing an OCR about the local character recognition bot and you're teaching it like, is there income this page? Is this what you're doing rather than which letter is this? Is this the character A? Is this another script character? Because it's the crew's possible analysis of revision. Will someone stand for reverted? And because they're, yeah. So then a second problem with predicting whether something is going to be reverted is that you're actually perpetuating editor biases. So if an editor thinks that this particular type of editor like newcomers shouldn't have their work represented in public yet and they should have to have some sort of reverse first, then the machine learning algorithm will actually learn that prejudice and it will perpetuate its predictions. And we're concerned that that would create a vision cycle. Not something to have a lot in real quick so that we can show you where these kind of biases can make a difference. We're actually gonna show you orders working in practice so that you can actually use it to look for vandalism and recent changes less. Oh, let's do that. I thought I had the new, wait, so this is new recent changes filter interface which is fantastic if you ever tried it yet. I'll only look at it and it's that orders things that are very likely to have problems. So yeah, these are ones with the very... Oh, this one's gonna be great. Oh, this is actually a great example. Oh yeah, so this is interesting because we found the words but and ass and the machine learning algorithm determined that anyone who says that is an ass bot and shouldn't have to edit it in here. However, this article is about a cleavage because you can erase women's scientist articles but not about a cleavage article is great. Maybe we can find that here a little bit. Yeah, so we don't think you're probably another example. Oh, wow. Yeah, there we go. Probably, probably gonna do that one. No, no, no, if you're not good at it, it's good. She ate it, it says. Oh yeah, yeah, that's funny. It's not she. So now that we can't use this video for anything, this is good for anything. Okay, sorry. Yeah, so reverted is the most, it's the prettiest camera we have and but to even get a reverted model on your wiki, there are a few things you don't have to do but I said a few in mind. Yeah, so in order to get even that basic model, you'll have to at least identify bad words in your wiki. We use the English list because that seems to be really popular with families but we still need bad words less for your wiki. And if there is one compile order already, then you have to run a tool called the bad words detection system. Which is, I don't know if it's a bad word, it's really bad. You'll also have to, do you need a period of stop words there as well, so? So the VWES system will find words that are added in revisions that tend to be reverted but not added in revisions that tend to not be reverted. And so what it usually picks up are curse words, racial slurs and informal language that doesn't belong in articles but it's really just a big messy list. And so this instructions page is all about having a human go through this list and say, these words are bad ones, these are the informal ones and these don't belong in either list, they got picked up by mistake. And so that's essentially what these instructions are going through, the stop words. That's sort of like a weird information journalistic thing but it comes out of the same script. It's super, the most common word, it's in the entire wiki art, the list of stop words. And they contain information that turns out so it's a little bit tricky. But yeah, so you need to create a batteries list. What's the other list that you're taking? Informals. Informals, okay, so thank you. That's also done by the same system. Okay, so those are just things like you and things that don't belong in articles. No, sorry, room. So just, just tell me now, what else do you need to do to get your language supported? That's not distracting at all, okay, sorry. I mean, I'm bad too. Oh yeah, we're okay with that. Or did I just move the fish out of the room? Oh, I'm okay with that first. So what else do you have to do better? Oh, they give us, they give us a, if you wanna work with us, you give us language support and so you help us by finding out which words are bad, which ones are informal. And from there, we can take it away and build this reverted model by analyzing the history of your wiki and building the prediction model. So like Adam said, this is the course hammer. That's all you have to do though, I thought, let's see. Oh, sorry, that's article quality. Okay, great. So yeah, when you have that done, then you can train a reverted model and you can use that new wiki. There are two more nuanced models and we prefer these over the reverted model. They're called the damaging and good faith models and it's pretty much just like it sounds. The damaging model is supposed to predict whether something is actually damaging to the psychopenia which is certainly different than reverted. Things like your reverted for multiple reasons. Like it's, you know, it's, actually, I don't know if I can give you an example of this. Yeah, damaging is more specific and it's not because you think they're a newcomer and so on. It's not because the article is protected or it shouldn't be protected. It's still small. Fine, it's fine, it's fine. Okay, this says the author was trying to contribute productively, thanks. This is the editor, so what you're looking at is the big picture of the lection tool. This is a tool called wiki rainbows and if you want one of these more advanced damaging edit detection things, you need to actually hand code larger edits. I think normally maybe five, 10, 20,000 edits. And because of so many edits, we probably have the volunteer power to categorize each edit once, unfortunately. So this is like, even once you've done wiki labels it's sort of an exact answer to the question but it gives the machine a pretty good chance of figuring out what's good faith and what's damaging. Good faith is interesting because if you look at the intersection of damaging and good faith, the good faith edits which we're damaging are a good candidate for where to give someone a wiki ambassador. This is like someone who might need help to edit more productively. Nice. Yeah, so the long story short, we help you create a random sample of articles of revisions in your wiki. That gets fed to the wiki labels campaign and then you need to find for any people to hand code all those revisions. Once that's done then you can have the advanced revision detection or the advanced revision quality detection models and we'll probably turn off the reverting model because it's better to use these ones. And just a quick note before we move on from here, there are a lot of wikis that are currently training these sort of models now and if you don't read language codes, we have Arabic, Azerbaijani, Catalan, German, English is going through its second round, English, Wiktionary, Spanish Wikipedia, Spanish wiki books, Persian, Danish, French, actually I thought French was done. No, I guess we're still doing something in French. Hebrew, Hungarian, Indonesian, Italian, Japanese, Korean, Latvian, Norwegian, Romanian, Albanian? Yes. Yeah, that one's always a trick. That's the English name, but anyway, so the point is that we support a lot of these wikis and so I just wanted to give you this overview to know that and it's not just Wikipedia's, we also support wiki data, we support wiki books and Wiktionary, that sort of stuff, so don't limit it by thinking like, oh, all my wiki is a Wikipedia. Can you put the support table also? So something that we should do for you, I don't think we're doing right now, is that when there's a wiki labels campaign, we should probably let you know of yours that you don't get started on the second one. I'm just gonna save some time. Yeah, so that you have other people to find to coordinate that work with. Okay, I'm missing notices here and here, but otherwise it's mostly fault. And there are some of these that have no support yet, but we're already running campaigns, we just still don't have the link jacksets, like Catalan, anybody speak Catalan? So then article quality is similar, but there are a few different hurdles. For article quality, you have to explain what the different article quality levels are for your wiki so just in the scale you have to work on inventing one. And then you need to, is there a wiki label for that too? Actually, yeah, we can look at the one for wiki data. And that's what I think is still live, I'm not finished. And so it's the same as if you already have article quality, if you have examples of article quality that's been coded by real editors on your wiki, you can use those, if you don't have them, then you'll have to do some kind of campaign where you get those, well regrettably I can't even request the work set because we finished it and so it's like, no actually there's nothing left to do, we'll check this data, so we can't quite show this one off. That's just wiki data. Yeah, but if you don't like it's a good example of what we can easily support. Who here has heard of the wiki project assessment or wikipedia 1.0 assessment scale? So this is an assessment scale that English wikipedia's use to rate articles by their quality level between stub quality and featured article quality. So it actually goes stub, start, C, B, good article, featured article, and there are a few classes that I didn't mention that are kind of defunct now. And so some wikis have this relatively well developed and we can just train the models on the history and the history of these assessments. Other wikis don't have them well developed and so we actually have to do a specific labeling campaign and we like to aim for 5,000 observations. So you have to look at 5,000 articles and assess their quality level. Really fast for stub class, you can just glance at a stub and know that it's a stub. Much harder for differentiating between like good article and featured article, top two quality classes. But 5,000 is not that much if you have a group of 10 people who are working with you. It doesn't take that long. What do you say I was lucky about sort of being an expert at this level of decision to figure out what class in our post or is that pretty easy to figure out? So I think that at the very least for the upper levels, actually why don't I pull up the, so we had to develop a scale like when we first came to wiki data. They didn't even have a good description of what quality was or what quality is for wiki data. So we had to sit down with them and go through a consensus building process for deciding like what item quality would be defined as. Correctly, this is nothing to write in place. And so if you didn't look at the English wiki data quality scale before, this sort of structure would look familiar. But you can see if I expand, there's some very well-defined characteristics of what a certain quality level is. So if I click on one of these, it will scroll down to the bottom of the page where we get into details of what do you need applicable and what set of translations are important. In fact, if you really like to look at consensus discussions, I'm pretty proud of this one. There was a very long slot, but we made it through. For most Wikipedia's, you can pretty much just adopt the English Wikipedia one without too much trouble. But when you get out of Wikipedia and you're in Wiktionary or wiki data, then we should probably figure it out specifically for that wiki. And can you put the wish list? Yeah, I wanted to mention also, we're open to not only expanding to other wikis with these four models, but we're interested in people who have more ideas for what to actually analyze using machine learning. There's the edit types project that's happening. Is that something worth this, right? Yeah, I can't, can you? Sure. So, just change the URL report and we'll find it. Well, that's what we're gonna do. What is it doing? I can't just leave it, I can't stop it. Necklace and editor, match made in time. This closed source software, I don't know when I click on things with it. That's my closed source, it's just Facebook. So this is the main wish list. Like any time somebody comes to us and they have an idea for something that we might predict or something that we might have an advance over them do, we throw it on this artificial intelligence backlog because like we can't do everything, but we can certainly get it in front of other people. There are a lot of researchers who I work with who like to pick up these kind of projects that they know somebody will actually take advantage of. So coming back to the edit types model that Adam was referring to. So we have a model that works okay. We're not quite ready to deploy it yet, but it works all right. It predicts the difference between an edit that adds information to an article, removes information, makes a clarification, refactors the content in an article, and essentially what we're hoping to be able to do eventually is to put this on a history page so that you can look through the history of an article or maybe even look through the history of a contributor and see what type of work do they actually generally do in an article. This has some really fun implications too for what we can do with analytics. We can find out like, is there a specific type of edit that people do well with visual editor, but struggle with? Is there a particular type of change that people find much easier to do now that there is new functionality inside that new fan-coupled wiki text editor? Until we have this model, we couldn't even start answering those questions. Actually, let me see here. I can show you our taxonomy. It's because of the space. There we go. So this is the taxonomy that we had a big consensus discussion to work out. So copy editing, clarification, simplification, point of view. And clarification and simplification, really confused about that. DGG, help me out with that. I still don't understand the difference. Changing point of view, refactoring, fact updates, which are, and then elaboration is adding new content, but fact updates are sort of a special case of elaboration. Verifiability, dealing with citations, disambiguation, which is mostly fixing links so that they don't link to disambiguation pages. Wivification, which I guess is kind of obvious, using the actual markup. Vandalism, which we're all familiar with, common vandalism, which you know what that is, and process, like adding cleanup templates and that sort of stuff. And so we have a model that can do pretty decent predictions for most of these classes. It also struggles with the difference between clarification and simplification. But yeah, if somebody had a pressing need for that model, generally we just prioritize what people come to us with and say, hey, I would use this if you had it right now. Then we're gonna try and make sure that we have that deployed as soon as to right now as we possibly can. Yeah, so to nutshell, if you are one of the existing models and we have a workflow for you, we can just get started on that if you would like. And then you'll have your language supported pretty soon. If you have a brand new idea, we would love to hear that too. And we can work with you to figure out what that will look like. This should probably be a normal discussion and then we can start working on machine learning that too. So maybe we should break into groups now. And I guess one of us can work with each group. And yeah, if we can just try to help you get started on what are we here to do. Let me do one more thing before we do that. I will show just the list of wikis that we have any support at all for. So you may or may not see your wiki in this list. Even if you do see wiki in your wiki in this list, that doesn't mean that we have full support for your wiki. We still probably wanna talk to you. But if you see your wiki in this list, then we'd love to talk to you about the support we already have. Because maybe you wanna work on a tool. Maybe there's actually a tool that you already used that would really benefit from some of these scores shown down there. Would you mind, can we go around and everyone just give a couple of sentences explaining what they wanna do here? Because yeah, we can also work on tools if there's a group that wants to do that. You wanna say? Oh, I'm just, I'm just here to learn more about words. What's your name? Justin. What's your wiki? Uh, I don't know about wiki. Cool, cool. Be yourself. Yeah. I'm, I'm Justin from the English Wikipedia. I'm just thinking about what all these words are. Sorry, I can go back to that in a few seconds. Mic down. Sorry. My wiki is Korean Wikipedia. They're KWiki, so yeah. I was working, I've been working with Aaron with about Korean Wikipedia. More than working, but people are lazy and I am more late than them. So, like things are slowly, very slowly happening. That's all. And so just to borrow, this is an example. Korean Wikipedia is a good example of one where we only have basic support. We only have that reverted model available. So I'm gonna try and grab a review later and make sure that we talk about the progress towards getting that advanced support ready. I'm Stephen, I'm from the German branch. It's a quite nice tool, but I suppose you don't have enough people who can help our community as I think it's too small to maintain it and to check it. You could get the reverted model, for sure. And then, how and what labor do you say goes into 5,000 a day? So it's probably about on the scale of 100 hours. So if you have 10 people, then it's a day's work for 10 people to go through that, which is nothing to sneeze at, especially because people have other work to do and no one's paying them to do that. But yeah, even the basic support can do quite a lot. And for that, it would just be take an hour and review some of the bad work lists that we generate. He said it was a German Wiki Voyage. And so we already have language assets for German and so we might actually even be able to just stand you up a reverted model at this hackathon. If you want to create a fabricator ticket, there's I think a link here that will fill out some of the deal for you. Mm-hmm, okay. Get that get support page on the iPad. Okay. On Buns. So many more on Buns. Very few more on Buns. I'm actually, I just partnered with Media Foundation. And I was just here to learn more. Just get to see, oh, I like when three of us work in fundraising technology. So we just try to see, what was your little end of the way for? I could never get those sub-penis into my bank account. I'm not yet. I'm still a little curious that if getting support for the projects will help out new editors that are writing the wrong kind of stuff that we will be able to get, but I think we could be somewhere else. So I was thinking that we'd be waiting as one of the places that sort of more so definitely agree. Well, that's cool. So this is like to direct newcomers to the appropriate project? Well, I have things. Yeah. I'm looking forward to having some brainstorming sessions over at, I think it was like all fans or something. So that was one of the things that we were able to do is like score a revision in other models and see if there's some place where that change would actually be welcome. I mean, there's one, it's not quite on those lines, but there's another thing that you might wanna look at. We're hopefully going to a project which will categorize revisions or categorize articles which we keep project we belong to. So it's basically a way of steering people towards the community. Right here. Okay. The domestic community. Yeah, the domestic community. My name is Gildan. I work in Albanian Wikipedia. I've worked with him. I've worked with you guys in Albanian or in Albanian Wikipedia. I want to know how is the process to add new batch words or other labels? Oh, like to add new language. So we have a list of languages now, but after some time there are more words, or bad words, we want to fill it up more. To be able to understand. Morning, you will see regular expressions with bad words in that one. Which you might not be able to pronounce. So this is generally, like we track bad words by having regular expressions to match them. And in this case, we just took a line word list and threw it in as the list, but we can also do things like, you know, interpret all eyes as the number one and people tend to personally speak to that sort of thing. And so this list can always be added to it. If you get a food, that's great. And if you don't get a food, emails and wiki posts and that sort of thing will make the updates ourselves. So it doesn't sound like as we were going around the room that we had any brand new languages, but we didn't have one new wiki and another wiki that we wanted to get more support for. Did I miss any other, and maybe a wiki that we want to extend the bad words list for. So for I think it's for finding these new. Yeah, yeah, but the two good ones. Do you want to do the overview and I'll go start talking about a new wiki? Sure. Okay, and that is the end of our coordinated discussion. The rest of this is workshopping.