 I am going to share with you some of the latest results out of the why we read Wikipedia research, and I know I have five minutes, so I try to keep it down to three minutes, and then any questions you may have, I can answer in the two minutes left. So let me just share. Toby, do I have more than five minutes? It actually turns out that there have been some cancellations, and you do. Okay. So I shoot for 10 minutes then. That will take time. Yeah, so I think we're just going to, yeah, so why don't you talk to a quarter after? Sounds good. Yeah. Let me just share my screen, and I won't see, I think the chat anymore, but I just want to hear from you. Do you see my screen, like, the presentation? Yes. Okay, great. So why we read Wikipedia? Most of you know about this research. It's a research that we started in 2016, and basically the goal of the research was to learn about Wikipedia use cases and read their behavior across languages. The way we approached this problem was that we have the web request logs that capture basically the reader behavior in a very atomic level in our servers. And basically, you notice that at an eye, the user searches something on, let's say, a website like Google, they end up coming to Wikipedia, read an article, they traverse another article, they go back potentially outside of the Wikipedia world to a search engine, and they come back to Wikipedia. So what we did is that we used basically this window of opportunity that the user is on Wikipedia, and we surfaced a survey to the user, a series of surveys over time back in 2016, and I think the more novel part of this approach was mixing the survey results with web request logs to make sense of the use cases of Wikipedia for the readers, but also understanding a little bit more about the behavior of readers when they're on Wikipedia. A lot of the focus on 2016 and early 2017 was to build the survey, what type of questions we wanted to ask people, and many of you have seen it, we came up with three sets of questions. One of them would ask the user about their information need, basically the user would be questioned. So the user is on an article, we pop up a quick survey widget and we asked him to help help us improve Wikipedia by answering three questions, and these are the three questions the user sees. I am reading this article to do a quick fact lookup, basically, reading for an overview, or I want to basically go into an in-depth reading, and this kind of information gives us insight into how deep the user is planning to engage with the article or the series of articles they are reading. We asked the user whether they had prior knowledge of the topic related to the article before reading the article, and here the user could say whether they are familiar with the topic or not familiar. And we also asked the user about their motivation. What was the trigger that brought them to this article, to Wikipedia and this specific article? And the user had some choices. They could say basically a piece of media was the motivation, meaning that they watched a show, listened to a radio program, read a book. They need to make some personal decisions. They would need to buy a book, buy a laptop, choose a travel destination. They are bored and randomly exploring Wikipedia for fun. The topic has come up in a conversation with a friend, with a colleague. They have a work or a school-related project due, basically. They want to know more about the current event. Current event can range from, you know, terrorist attacks to soccer games or earthquakes, someone's death. Anything that creates basically a sudden spike of user attention. And the last thing is basically intrinsic learning. The topic is important to me and I just want to learn more about it. And we left another field in case the user, in case we're not asking the user for a kind of motivation that the user wants to communicate with us. So we ran the survey in 2016 and we had some results. And many of you have already seen them. I'm not going to go over those. Those were heavily focused on English Wikipedia. In 2017, we ran what we call a great grand survey. This survey was ran during a one-week period in June 2017. I think it was June 21 to 28. 14 languages were involved. These were Arabic, Bengali, Chinese, Dutch, English, German, Hebrew, Hindi, Hungarian, Japanese, Romanian, Russian, Spanish and Ukrainian. The survey was run through a quick survey in both mobile and desktop platform. Sampling rates varied for each of these languages depending on how much we expected to receive both views or impressions of quick survey, but also engagement with it. The survey was shown on article pages and to those we do not track off. And we collected 215,000, 16,000 responses in total across these 14 languages. So our job in the past months has been to basically dig through these responses and try to regenerate what we did for English Wikipedia last year for these now 13 additional languages. The first question is, is there anything burning questions that you want to ask right now or shall I move on? I want to see the results. So the results that you see are going to be the results from the first part of the analysis which focuses on the survey responses, the bias, the part about the behavioural component. We are still working on that, but this is as of this morning. I don't show you all the languages, but I do have for all the languages. So if you have specific questions, just ask it. First one is Arabic. Our sampling rate was one out of 10 and we collected close to 2,200 responses. The user that helped us with the translation and community engagement is about. And here are the responses. So starting from left, what you see is that intrinsic learning is basically the highest reported motivation for coming to Arabic Wikipedia. And then with a drop, you see media and being border random. All the other reasons seem to be kind of, you have a pretty gradual decline in slope here. Intrinsic learning seems to be a strong factor for Arabic Wikipedia. By the way, if you're looking at these plots for the first time, the blue or what you see on the left hand side is left hand side of each of these bars is this actual survey responses. And the green is the device survey response. You see that in most of the cases, the biasing didn't have a huge effect. You see a little bit here in person decision, for example, for all intent and purposes, you can focus on the green bars here. The information need is pretty much spread evenly. So in depth reading, overview and fact checking is used pretty evenly in this language. And the split very similar to what we saw in English last year in 2016 is almost 50-50 for Arabic speakers. Next. Hey, Leila. I don't characterize the differences or like in general the comparison between English and Arabic. I think a lot of folks have kind of already kind of taken in the English results. I'm mostly curious kind of how Arabic is different. Right. So in English, I think what we saw was basically media was the highest. And then it was, sorry, I hear a little bit echo. I'm not sure if somebody is trying to say something. No. So for English, what we saw mostly was media, intrinsic learning. And I believe it was conversation. I need to refresh my mind from that here. So for Arabic, I think what is interesting is that intrinsic learning comes higher than everything else. For English, another difference was that the slope was pretty constant. You didn't have this sharp drop that you have between media and intrinsic learning. What this really means, we need to look at the Holocaust logs a little bit deeper to understand what this drop can actually mean for Arabic versus English. For now, based on these responses, there is a drop. And it seems intrinsic learning is more important for Arabic Wikipedia users when compared with English. Okay, let's now look at German. We had 28,000 responses for German Wikipedia. And here is what you see intrinsic learning again, reported as the top reason for coming to German Wikipedia country, followed by media. This slide followed by conversation. The drop, the slope is much smoother here. You don't have a significant drop. One thing you see in German is that fact checking is higher. So basically, the distribution is not as spread as let's say Arabic or even English here for German. You have a lot of fact check errors and in-depth reading is as much as we saw for English Wikipedia in 2016. Prior knowledge seems to be pretty split also for German 50-50. Moving on. For Spanish, we collected around 40,000 responses, one out of five page views. I think one thing which is interesting about Spanish is that work or school comes as a motivation, which is in top three motivations, which is rare across our languages. There is something going on here that we need to dig in deeper. There are either education-related programs that are being more active or for whatever reason it can be that media is not, for example, as active of the source. And there can be multiple reasons why this is happening, but it's definitely different than other languages in this sense. Fact checking over you and in-depth are very similar to Arabic. They're basically spread, the distribution is pretty spread. I'm not going to go over all 14, worry not. I'll show you a couple of more for Hindi, 3,000 responses. What we see here, again, intrinsic learning is reported as a significant factor for motivation for coming to Wikipedia, followed by conversation. These are kind of both interesting. One is that learning seemed to be at least in terms of self-report is a strong motivator. We actually don't see such a sharp decline from one motivation to the next, I think, in any of the other languages. And the other thing that is interesting is that conversation seems to be a big factor for initiating people to come to Wikipedia, which is pretty nice. Users report a lot of in-depth reading, and we need to dig into a web request box to see if this indeed is the case. So we can come back to this, but this is the highest in-depth reading reported across languages. Leo, this is insanely different from everything else, which is fascinating insight. I think that Hindi editors, I'm sure, would love to see this. Their readers are actually behaving the way that all the editors believe. It's interesting, again, we need to check in web request logs and the length of articles and all that to be able to see if we can explain why this is happening. It can be that, for example, for fact-checking, most of the information is available in English, but not in their language. So people are switching to English for that kind of content. Yeah, but more on this, there's definitely something here. Sorry, go ahead. I'm actually going to... This is... Have you seen, this is fascinist. That's... I don't know. I think I know that you should... I think it's fascinating that that split is so... So different from the other way. Right, and I think what would be really interesting is also, for example, to see if intrinsic learning behavior is actually similar to English or not, right? Because we can characterize what it means for a user to be an intrinsic learner. It would be interesting to see if that's different between English and Hindi or, let's say, Japanese and Hindi. Something is going on here. It can be that either people feel like the self-report is the layer that causes this gap or it can be that indeed that's a very strong factor for coming to Wikipedia, which would be really good to know because that means a lot of emphasis... That can mean that a lot of emphasis should be put on the learning component of Wikipedia as opposed to other factors in Hindi. Or we need to advertise that we're a fact-checking source better. Right. Depending on how you want to manipulate that. Yes. Okay, last but not least, I'm going over time, is Japanese. For Japanese, I think one thing which is interesting is that before then random cups as top three, I think for two languages this happens, one of them is Japanese. And actually, the report is pretty high. It's 25, well, let's see, 22, 23% come for this reason to Japanese Wikipedia. The spread across overview in depth and fact-checking is kind of, I would say it's the same. These differences are not really huge. Yeah. And I have some right up here just to say that media conversations and intrinsic learning seem to be kind of the main motivations, the top three motivations for people coming to Wikipedia across the languages that we looked at. There are some differences where Spanish and Romanian, it seems, work and school-related projects can substitute one of the three top motivations in Japanese word and random exploration does that. And in Ukrainian, the events basically replace one of the top three motivations. But overall, it is interesting to see that these 14 languages are not so different from each other in terms of the motivations, except for the few that we need to understand better. And I think that's all on our end. Yeah, I'll stop here. Yeah. All right, so can you just like fill us in a little bit on the next steps in terms of. Because with the English, I know we dug deeper on a few dimensions and curious. Yeah, so at least in the six or seven languages where we have enough data, we want to dig deeper and basically connect the responses to the web request logs for the survey participants and be able to characterize each of these groups in these languages and then compare them with each other to see if there are differences in terms of how we characterize users for these specific pockets in different languages. If they're not, then that's a fantastic news because it really simplifies the way we think about readers across languages if we can say that these users are actually the same across these languages. If it's not, then obviously it's more complicated and we need to look at nuances as we develop or think about developing certain products for different users in different languages. And the next big thing I think for this research in general is to add demographics component to this research so we are interested to understand if this kind of line up the research that we do with the results with different if you have information about the gender of the user, the ethnicity of the user, the socioeconomic status, the education of the user, like how these results change. That can have implications on how the surface content and the type of content that we create across projects and languages. Like on our end it's like, how do we use this information to make our product that, like the fact that, you know, bias or whatever side that you would be so different from people coming to the, you know, so different from people coming to the you say they are like, what do we, what do we do with this? Do we get this information to the editors and be like, Hey, this is what your viewers want, like, I don't know, but I think this is something that, you know, we have to start, we have to start thinking about. Like, I'm sort of thinking of all of our projects as being sort of like this sort of one giant plane of, you know, knowledge or whatever is certainly not accurate. We start figuring out what we do with information about various segments. Yeah, and whenever you want to brainstorm about those and you find our input kind of more directly useful, please pull us in because we can't think that far without your help so.