 I thank you over one second. Good morning everybody. I'm Dario, I'm the head of research, and I'm going to be your host today. Apologies for the delays. We had some technical difficulties with Hangout on Air, so hopefully you're all on the right Hangout. I want to say a few words before I introduce Leila, who's going to present today. This is going to be a special edition of the showcase. We're going to have a full hour for presenting these results, and the reason why we decided to allocate all this time to this project is twofold. First off, this is the first time that in years, at least since I've been around, we're going to present results on a large-scale reader segmentation. The reason why there hasn't been any such research in the past is twofold. We didn't have four years like the technical capacity to collect and analyze this data, and also, I guess, from a strategic perspective, a focus for the past couple of years has been on editors primarily, and there's been a renewed interest in understanding readers and our reach most recently. I want to say the past year or two years have been particularly focused on this aspect of our audiences, and you probably attended the new readers' presentation that looked at new audiences, especially in the global south, that we're trying to better understand, and today we're going to focus on a study using a combination of mixed methods, both quantitative and qualitative, on Wikipedia. The second reason why I'm very excited about today's presentation is that this is a multi-center presentation, so it's actually a collaboration between Wikimedia Research, Stanford University, GASES, and EPFL, and it's been a project in the working for about a year now, also in collaboration with the reading team in the foundation, so I want to call out the fact that many people work on this and are very excited about the results we're going to hear from Leila. So with that, this is yours. Thank you. Thanks, Dario. Yes, my name is Leila Zia. I'm a Senior Research Scientist at Wikimedia Foundation's research team, and as Dario said, this is what I'm going to be talking about today is the study that we did over the past year to understand Wikipedia readers, formal collaboration with folks at Stanford University, GASES, and EPFL, as well as the reading team at the Wikimedia Foundation, with huge support from both security teams and legal. Okay, let's get started. So why readers? So Dario touched on this a little bit. I'm just going to show you a few numbers, and we do an exercise together, and then we move on. 610 million people, page views came to Wikipedia on October 30th, 2016. 85% of these page views were from users, meaning humans, versus bots, and spiders, and we here are pretty much used to these numbers. So we look at these numbers, 610 million, and we see a lot of it, and we're almost, I would say numb about it, we're used to it. So I would like to walk us through one exercise. So blink your eyes once. And now during this time, 2000 Wikipedia pages were requested by humans, and we don't know why. So this will have impact on the way we provide educational content for the people who come to Wikipedia to access the content that we are providing to them. And it will also have impact on the way we disseminate this content globally, both for the Foundation and also for the communities who are working on creating this content and working on disseminating it. I won't spend much more time unconvincing why it's important to focus on readers, at least for a bit. So let's understand Wikipedia readers. We're not operating in a vacuum. So there is a literature in research, and more recently in the Foundation to try to understand the intentions, motivations, and behaviors of users in general. So on the web, motivations and user behavior has been studied extensively. You see a lot of studies around the web itself, the search engines, social media, such as Twitter and Facebook. And Wikipedia, we mostly have focused traditionally on editors for a variety of reasons that Dario mentioned, a few of them. We have studied editor motivations. I have, we have also studied their editing behavior. We know much less about Wikipedia readers. We know there's a little bit about their content preferences. We know the search queries that bring them to Wikipedia. We also know their navigation patterns through a series of studies. And then more recently, the Foundation has started the project on new readers, trying to understand who are the readers that we haven't reached out to. So basically these are people who don't know that Wikipedia exists, or if, and we want to bring them to Wikipedia, or they're using Wikipedia, but they don't know that they're using Wikipedia. So that's the focus of the last research. So our contributions to this literature are going to be the three following. We're building a robust taxonomy of, for characterizing the use cases that people come to Wikipedia for. We use this taxonomy and then through a large series of large scale surveys, we quantify the prevalence and interactions between these use cases. And then we go one level deeper. We enhance our understanding of Wikipedia readers by looking at web request logs on top of the survey responses to try to understand more who are these Wikipedia readers who come to Wikipedia. So I'll talk about the first part, building a robust taxonomy. And by the way, this talk is naturally divided to three parts because of the three contributions, also at the end of each section. And you can ask questions so that we don't collect everything for all the way to the end. Also, if you have questions that stop you from understanding what I'm saying, just stop me at the point that I'm saying it. Otherwise, keep it for the break at the end of that part. Okay. So we want to build a robust taxonomy of Wikipedia readers use cases, but we don't know where to start. We have a few places we can start from. One of them is web request logs. So each time that you request a page on Wikipedia, the foundation servers receive a hit. And in that hit, there's information about the IP address of that request at the page, the user agent, and a variety of other kinds of information where the request has come from, what time of the day the request has been made, and so on. So we collect these logs. And the problem that we have is that these logs don't necessarily tell us why people come to Wikipedia. We're receiving around 100,000, more than 100,000 requests per second to our servers and looking at this body of data and trying to make sense of why people come is not very easy. It's impossible unless you know how you have some hints about why people are coming. So the other way we can get the information that we are looking for is that we can ask people at least maybe if you ask people, they will provide some information. And with that information, you can go back to the logs and make sense of the immense amount of data that we are receiving. So we're going to be using surveys for the first part of this talk. You want to build the initial taxonomy using a survey. We run a survey on English Wikipedia for four days at the rate of one out of 200. So for every 200 requests, we take one and this request will be eligible to participate in the survey. The survey will run on English mobile and desktop and only on article pages. So we're not going to be showing the survey on talk pages, search pages, main page, anything like that. We run the survey and we collect 5,000 responses on desktop and then 1,655 on mobile. So let me just show you how the survey looked like. So you're on Leonard Cohen's page on English desktop or you're on his page on mobile. If you have been selected to participate as part of the survey, what you will see is at some point there's a widget will pop up. If you're on desktop, the widget will pop up on top of the info box on the right hand side on English Wikipedia. If you are on mobile, it will show up in the bottom basically after the info box. So you have to scroll down before seeing the survey. The survey will tell you one thing, answer one question and help us improve Wikipedia. You have an option to say visit survey or say no thanks. You will see in the bottom that says survey handled by a third party because the question will be handled on Google site. And there's a link to the privacy statement for the survey. If you say that you want to visit the survey, this is what you see. See why are you reading this article today? So we're asking you to tell us why you are on Leonard Cohen's page today. And you have 100 character limit to explain to us in words why you are here. There's a reason that we asked the question this way as opposed to showing options to the user that they can choose from, which would be much easier for the user. The reason is that we don't want to basically imply our own biases on how people should be using Wikipedia at large to users. So we want to ask them why they are here. So just to give you a sense of the kind of responses we received, here's how it looks like. So these are actual user responses. Personal interest about conflicts in Middle East. So I can see the country's population. New York Times today mentioned Operation Wetback alluded by Trump in a debate and wanted to learn more. Confirming address for shipment going to this town. Studying for my med school test. Interest and curiosity. Because I'm in a very boring art lesson. To find out more information about this aircraft, someone came by my desk talking about the last man on earth movie. So I looked it up. I had previously edited it to see a movie summary. So I didn't show you a lot of the funny ones that we received. So we received around between like three to six percent also funny responses. But I'm not showing you those. The numbers were pretty low but they're also fascinating on their own. But this is to give you a sense of what we saw. So what can we do with this? We received a lot of text from people in English and they're very different. So usually when you have a lot of text available to you, you do hand coding. So what we did is that five researchers paired up and we did one round of hand coding with the following approach. We chose 20 responses, 23 form text responses at random from the 7,000 almost 7,000 that we had collected. And then each of us went over them individually and then we talked to each other about these 20 responses to see to basically create an initial alignment among ourselves. Then what we did is that we decided that each of us are going to randomly choose 100 of the responses. So this would be 500 responses and then we're going to generously assign tags to these responses. We did that exercise and then we reviewed the tags and what we learned is that there are four main trends that people try to communicate with us when we ask them about why they're on Wikipedia. And then we took these tags and we basically went to a third stage. We did another 500 of hand coding and then we checked whether any new tags should be added. What we learned was the answer was no. So basically and also we could reduce the four tags to three tags. So let me show you one example. So this person had said to evaluate technical description of Bosch fuel injection system installed on a car I'm interested in. So one of the researchers who was assigned to tag this had said deep dive shopping technical. And then in the last round we decided that this is about decision making and in-depth reading. So we did this for 1020 responses and the output is here is what we learned. What we learned is that when you ask people why they're on Wikipedia they communicate with you about three dimensions. One of them is information need. They will tell you how much information they need. They will tell you whether they're here for a quick fact lookup, overview or in-depth reading. They also tell you about their prior knowledge, some of them. They will tell you whether they're familiar with the content that they're reading or not familiar with it. And the last thing they talk to you about is their motivation. They talk about whether they have a work or school project coming up. A personal decision whether a current event has triggered their visit to Wikipedia, media trigger, a conversation trigger. Somebody walked up to my desk and said this and now I'm looking it up. Whether they're bored or randomly exploring Wikipedia for fun or whether they're here just because they like learning and they're doing intrinsic learning. So these are the three main what we call dimensions and each of them what is inside a dimension from now on I will call it category. Correct. So the question is that how did we infer motivation? Because some of the responses that we showed you could not tell anything about what the motivation of the person is and we did not assign motivation to those. So that would be not available. If that information is not available for that we would just say it's not available. Okay. So now we have these three dimensions plus their categories but we don't know if these are robust or not. So we ran the survey on English Wikipedia. So what we did is that we went to two other languages Persian and Spanish Wikipedia and we ran the same survey on these two languages. We asked people in their own language why are you reading this article today? And what we learned after hand coding of the responses from those languages is that at least in these two languages the tags the categories and dimensions that we observe from English also apply to these languages. It can be that the distribution changes when you go from one language to the other but the same tags apply. The other thing that we wanted to test is whether the same the same categories and dimensions identified would stay robust in English Wikipedia. So we ran a three question survey on English Wikipedia. We asked people about their the depth of information they're looking for their prior knowledge and their motivations. And as options we gave them the categories that you saw in the previous slide as options. So now they had to deal with a multiple choice question. What we did here is that for each of the questions we add an other tag. So this would allow the user to say none of these motivations that you're showing me is the one that I'm looking for. I'm going to choose other and I'm going to tell you what that other is. So there was also a form text associated with this other that they could tell us. What we learned is that only 2.3% of the responses chose other which meant that for us that the categories and options that we were making available to the users were robust and we also studied what is in the other when people choose other what other responses they give and and we realized that all of those responses could be captured by the categories that were being shown to the user. It can be that the user didn't really have enough time to read it or for whatever reason they didn't choose the option that was available to them and they went with other. Okay so this is the end of the first part. To conclude we built a robust taxonomy of Wikipedia readers through a series of large scale surveys. We showed that there are three dimensions that are important when we think about readers in for their information need their prior knowledge and their motivation. Now I'm going to move to the second part right now but before going to there I would like to pause here for a few minutes and let you ask any questions also for people on IRC. If you have any questions Ellery can take questions. Any questions for the room? I have the mic here for IRC. So I guess we'll continue. Okay all here. So we built a taxonomy. We have these three dimensions and the categories in them and now we want to understand what percentage of the reader population falls in each of these categories. So we want to quantify the prevalence and interactions between these use cases that we identify. So what we did is that we ran another survey. This was the duration of the survey was one week. The sampling rate was one out of 50 requests on English Wikipedia, mobile and desktop. On article pages and this time if you had a do not track option on your browser we would not show the survey to you and we would not basically study your data. We collected 29,372 responses from users and here are the questions that the users saw. So basically after they would approve that yes visit the survey they would see three questions. The top is basically the description of the survey. Why are you reading this article today? The first question is I'm reading this article to look up a specific fact or get a quick answer. Get an overview of the topic. Get an in-depth understanding of the topic. So this is kind of the depth of information. Prior to visiting this article I was not familiar with the topic and I'm learning about it for the first time or I was already familiar with the topic and user could also not choose anything here. And the last one is about motivation. I'm reading this article because you could choose. The topic was referenced in a piece of media. I need to make a personal decision based on this topic. I'm bored or randomly exploring Wikipedia for fun. The topic came up in a conversation. I have a worker-school-related assignment. I want to know more about the current event. This topic is important to me and I want to learn more about it and then other. And just for your information we randomly shuffled the question sequence and also the responses just to make sure that you know if people are randomly responding we can capture that either when we do the analysis. I'm going to show you the results of the responses to these questions plus some of the correlations but before going to there I want to have one slide talking about bias. So as any other empirical analysis we will suffer from bias in responses. The bias can show itself in many different forms. For example those who had longer sessions had a higher chance of seeing the survey widget. So this means that for example if you are in Wikipedia and your motivation is a conversation you had with your friend and you quickly want to look up a fact we will have a lower chance of showing you the survey because you were there only for one or two pages. There's also the fact that other kinds of bias. If you have a deadline tomorrow and we ask you to participate in a survey you may be more likely or less likely to participate depending on whether you want to procrastinate or not and there are other kinds of biases that will show themselves in this data. Some of them we will be able to capture an address and some of them we won't be able to. So just keep that always in the back of your mind bias is there. What we did for the part of the bias that we could control for was using inverse propensity score weighting. I'm not going to go through the details of the method but just think about it this way. Suppose a user saw population for example students were twice sampled in the survey pool then they truly exist in the reader population. So now we have 50% more than we usually have students in the sample that we created for the survey. What this approach does is that it takes the responses from the students and it will basically multiply them by a factor of half. So it will fix for the fact that these people were overrepresented in the sample that we took and then you can see the result of this fixing in the following slides. Okay so the first question was about the information need. On the x-axis you see the three options that were available to the user overview fact and in depth and no responses. On the y-axis you see what percentage of responses had come from which category for which category. So for example you see that let's say 21% of people had said that they do in depth reading. The two different colors basically blue which is the left bar is the actual survey response and the green one is the survey response after we fixed our bias. So you can see that in this plot there's not much difference between the bias correction responses and the true responses that we got from the survey. So for the rest of the slides let's just focus on the corrected responses. Okay so there's one interesting thing that we see here and that is only 21% are coming for in-depth reading. 77% in total are coming for overview and quick facts and also keep that in the back of your mind this is based on English Wikipedia. Prior knowledge the population is pretty split 50% are familiar with a topic that they're reading about and 47% are not and now motivation. So in motivation what you see basically on the x-axis you have media intrinsic learning conversation all the seven motivations that we saw plus no response and what you see here is that let's say 32% of the responses in the survey were motivated basically visits to Wikipedia for that article were motivated by media. So the person has read a book watched a movie and using newspaper or whatever other source of media and that's the reason that they're coming to Wikipedia. And you can see one thing that is interesting to see here is that Wikipedia is being used for a variety of use cases and no use case is really dominant. You don't see a sudden drop anywhere we have a gradual drop and it seems that the website is being used for all these different reasons. It seems media is pushing for a lot of it but the drop is pretty gradual. The other thing that you will observe here is the effect of correcting for bias. So you see that people who are coming for conversation if we would not fix for the bias issue you would see a lower representation of them in the survey data but after the fixing we see more of it and that's our hypothesis is what I said earlier. So basically those who are in conversations with other people they want to do a quick lookup and they want to go. Now if you show them a survey they won't participate or they will have a higher lower chance of even participating seeing the survey in the first place. Okay so how do motivations change by day and time? And by the way we looked at the prior knowledge changes by day and time and information need changes by day and time those don't change so much so I'll be just talking about motivation. So generally let me actually first walk you through what this plot is. So on the x-axis you see the days of the week Monday through Sunday on the y-axis you see the share of motivation for that day. So on Monday if you look at this blue bar on top, what you see is that 30% of the people who come on Wikipedia on Mondays are motivated by media and then you can see for example 10% of them are motivated by personal decisions. What you see generally is that these figures don't go that much up and down so the motivation is pretty much stable with a few exceptions. One is worker school related and that's the drop that you see here. So basically during the week people come for worker school and then as you hit Saturday it goes down Sunday it goes up a little bit and then it goes back up on Monday. You also see that conversations start picking up on Friday so that also goes with what we expect to see. People go Friday nights, start socializing more, seeing more friends and all that and then on Saturday and Sunday you see also a conversation being higher chance of being a trigger for people coming to Wikipedia. And then the last one which has some difference is media itself. So on Saturdays and Sundays it seems people have more access to media and that seems to be a stronger trigger than during the weekdays. We did a similar analysis for hours of the day so this is basically midnight all the way back to midnight. Again the motivations are pretty stable except for work school. I mean you see that pretty much starting 6am things start picking up and it stays pretty stable until 6pm and then it starts dropping. You also see media basically starting to go down as you go towards the early hours of the day and then after basically starting 6pm it starts going up again. Okay so now let me show you a few correlations here that can be interesting. What you're seeing here is motivation versus information need. So here you have the seven motivations that we have and here are the three information needs that people could choose from in the survey. How you should read this is as follows. So let's look at the first line together. This is telling you that 38% of the people who chose media as their reason for coming to that article were there to do a quick look up or a fact check. 19% were there to do in-depth reading and 43% were there to do overview. The number in parenthesis will show you the lift. So this says that people who are doing overview you're 12% more likely to be coming by a motivation triggered from media than everybody else. So there are a few interesting things to observe here. Those who come with conversations as their trigger they're mostly looking for facts, quick lookups. Those who come for work in school they're also looking for quick lookups. For in-depth reading basically people who are doing intrinsic learning are mostly focusing on that and overview is mostly represented in people coming from media and those who are bored or randomly exploring Wikipedia. And by the way the last column shows whether the result is significant or not and the degree of significance. So the next one is motivation versus prior knowledge. So basically did the same thing we just changed the column to prior knowledge familiar unfamiliar. As you can see the lift is not that different in this case. There are two exceptions. One is that if you're doing intrinsic learning you're more likely to be familiar with the content that you're reading and if you're coming because of media as your trigger you're more likely to be unfamiliar with the topic that you're reading about. Okay I'm going to finish the second part here. So what we did here was that we ran a survey on the taxonomy that we built in part one and what we observed was that English Wikipedia is consulted for a variety of use cases. There's no single dominant use case that people come here to Wikipedia for. We observed also the shallow information needs meaning overview and fact lookups are 77% of our requests and are part of our responses versus 21% which is in-depth reading. We observed that readers have nearly identical shares in being familiar with the topic that they're reading about versus versus being unfamiliar. And we also observed that extremes there are differences between extrinsic and versus intrinsic motivations. So extrinsic triggers are media conversations work in school and current event and the number in parentheses shows you what percentage of the responses came from there. While intrinsic triggers were intrinsic learning or then randomly exploring Wikipedia and personal decision with the associated numbers. I am going to stop here and give you a few minutes to ask questions if you want about the second part which was running the survey and understanding these numbers. Yes, so we have several questions on IRC so I'm going to ask Ellery to relate them and then we can take questions from the room. Sure, so the first question was why we chose Persian and Spanish Wikipedia to sort of verify the taxonomy. Right, so we wanted to do this quick and we wanted to do this in languages that we can quickly hand code. That's one reason and Persian is a language I speak and Spanish was a language that we could easily use Google translate and also we had resources internally to ask for translation. There was another reason for the choice of these languages. We wanted to have at least one language which is pretty much contained in one country. So that would be Persian and like Iran the associated country a country which is potentially culturally different from let's say for example US and Spanish would be another language which is similar to English. So many people from many parts of the world speak Spanish but this population has probably a very different culture than the general representation that we have in English Wikipedia. So these were the two reasons. And then Giovanni was asking about a bit more detail on the University of French City score waiting. So one of his questions was how did we choose the weights in the yeah. Okay, yeah yeah I can expand on that. So basically what we did is that we built a prediction model. So we divided so we have two populations people who participated in the survey and everybody else the general Wikipedia reader population. So what we did is that we built a prediction model based on the set of features that you will see actually in the next section that I will talk about. And using those features we predicted whether the user belongs to one group versus the other. Basically whether the user is a survey participant or not. And then the weights basically the probability that they show up in the survey come from that prediction model. Good question. We tried to keep that under direct for this part. Okay let's see from IRC. We have one question from the room. So in the second part in the survey you had an other box. I was just curious about what other things came in to that other box. Were there any patterns there? Yeah good question. So we actually check those again to see if we can identify anything around the robustness that would question what we did in the first part and we couldn't identify anything. So it seems that people were still using it around I would say three percent four percent and what was in the box for other was still something that they could choose from the options that are available to them. That's another topic that there may be sometimes language barriers. We say something we say for example intrinsic learning but what does that really mean. We tried to make it as simple as possible for the audience but especially for English Wikipedia because the audience is so broad people interpret different questions differently and that can be part of the reason that they choose other. Another question from the room. I have a question about the second result about the shallow information needs are more frequent than in-depth right. So this is about per participant right per participant and I'm just kind of curious if any insights or maybe this is for the next section when you discuss the traces but how that would look like a way by overall amount of usage time spent or pages viewed because I understand you can only I can only participate in a survey with one page view but if I come back if I'm a power user with 50 page views I want to get asked once and if I'm a user with one page order get asked once. Any insights on this. Yeah I wish I could tell you more and I'll talk about it in the next section but basically for information need and familiarity with the topic as you will I will talk about in the subgroup analysis in the next section but there were no strong statistically significant differences between the people who were doing in-depth reading and other types of knowledge seeking reading that we could identify from the data but yeah I will mention this in the in the next section. More question from RSC. Namely what was the response rate for the survey? That's I think Ellery can correct me if I'm wrong but we don't know this right because we did not collect who did not respond to the survey. Well we could look into the number of survey impressions there were and compare that to a number of responses. I don't know. Yeah so the yeah so our answer is we don't know it. Okay good let's go to the last part. So now we did the survey on English Wikipedia and then what I showed you in the previous section was basically the response distribution the correlation some descriptive statistics about what's happening what was happening with the survey responses. Now we have the luxury of going deeper we can look at the web request logs of the people who participated in the survey and try to understand more about these people because now they have provided a valuable set of information to us about how much information they're looking for and also why they're here so maybe if you go back to the web request logs we can do something more. So we used four categories of features to make sense of web request logs and the survey responses that we were seeing so basically from the survey itself we have three features motivation which tell us why the user is here information need what depth of information the user is looking for and prior knowledge. Because the user has requested the page on Wikimedia servers we also know the country the request was associated with the continent the local time the user was in in the week the local time in hour the host that the user has requested has sent the request to and the refer class. We also know on which article we have shown the survey to the user we also know what other articles the user has visited in the session the survey was shown to the user. So we have a variety of information about the kind of articles that the user has read more specifically we know the in-degree out-degree page rank text length page views topics and topic entropy. I won't be talking about these last two in this talk but basically we had to build topical models for the articles and measure the entropy of the articles for saying some of the things that you will see later. And the last feature that was available to us is the session or activity of the user right so we know the session length these these are all related to the session in which the widget was shown to the user. We know the session length we know the session duration average dwell time average page rank difference average topic distance refer class frequency session position number of sessions and the number of requests. So we have all these features available to us and what we want to do is what is called subgroup discovery. So here's how it works for each survey question survey question response deform a target. A target is basically let's say as an example people who have chosen motivation equals to work and school. This means that we're going to focus on this population right now and we're trying to see what kind of behavioral patterns we see in this specific population which is different than the rest of the population. So we focus on work and school and then we look at everything that's happening in work and school and we compare this with everybody else and see if work and school has some characteristics which are different than the rest of the people. So for example for work and school it can be that you observe that they have larger share of long sessions when you compare them with the rest of the population. So subgroup analysis and this goes I think partly to what Tillman was asking when you do subgroup analysis and information need basically you have three targets one of them are people who have chosen the information to be in-depth reading one of them who have chosen information need to be summary or an overview and the last one the set of population that has chosen quick look up right. So for each of these we did subgroup analysis and what we learned is that the subgroups are pretty homogeneous across these three different groups. One thing we observe is that users from Asia describe their information needs significantly more often as in-depth. Now I don't know why this is happening we know that a lot of requests from Asia come from India so this may be something that we want to keep an eye on but because we have a lot of representation from India it can be that for some reason this result we cannot say that this is true for Asia we need to look at the data more we may need to do further analysis and this is something you should keep an eye on and bring it up if you're more interested in but this is one thing we observe and there is this kind of anecdotal not evidence almost like the stereotype that we think about right we think that in the US people do a lot of quick look-ups and fact checks right and we hypothesize that maybe in other countries people do more in-depth reading right and this seems to be signaling in that direction but we don't know it we need to look into it more. We also know that obtaining overview is more common among desktop users this is again something which is not very intuitive if you would ask me before running the survey for example I would say people who are on mobile should do more of overview than people who are on desktop this can be partly because of the features that we offer on mobile devices but there may be also other reasons there and we also see that fact checking is more often observed in topics that are related to sports so these are the kind of the significantly different results that we could see in the information need category then we did subgroup analysis on prior knowledge so we have two targets here people who are familiar with the topic and people who are not familiar with the topic for users that are familiar with the topic we notice that topics that are read by these users are more spare time oriented so there are sports 21st century tv movie and novels we also see that topics that are read by these users are mostly popular topics these are topics that have a lot of page views on wikipedia they tend to read articles that are longer and are more central in the link network of wikipedia now this is pretty much what we can gain about understand about prior knowledge when we do subgroup analysis about prior knowledge and information need when you go to motivation more interesting things happen now i'm going to show you on this slide two examples of what you can learn one when you look at motivation so let me first walk you through how to read this table so this basically t is the target it says that the target is all the people who have chosen motivation equal to work school and it says that the probability basically the share of these people in the survey was 19.5 percent 19.5 percent of the people had chosen this motivation as their reason for coming to that article when we took the survey now let's focus on a subgroup one of the subgroups that we have are articles that are around the topics which are focused on mathematics what this is telling you here 7.9 percent is telling you that 7.9 percent of the articles that the survey widget was shown on was on the topic of mathematics and then what you have as p s given t or t given s these are basically conditional probabilities of the subgroup given that you are in that motivation target or vice versa and you see the lift so basically what you see here we are only showing you the significant results we tested basically we looked at say hundreds of subgroups and this is what comes out people who have chosen motivation their motivation to be work or school tend to be mostly focusing on four categories of articles or topics these are articles which are focused focused on mathematics war and history technology biology and chemistry literature and art what you see also here for the two top categories is that the lift is more than two which means that you're twice more likely to be reading a mathematics related article or war history related article when you are motivated by a work or school trigger we also see that you're coming mostly from desktop so that's another thing we observe we observe few other things you do your session duration is more than six minutes these numbers are in minutes and average time difference is telling you how much on average the user spends time on an article which is between three minutes and nine minutes basically nine up which is telling you that these users are spending quite substantial amount of time on articles they the number of referrer being searched is telling you that and being greater than two is telling you that these are the users who go in and out of Wikipedia often so these people don't stay on the site they go probably to google and come back to us with another query now let's look at the other extreme these are this subgroup analysis is for the users who have been motivated by board and random as their motivation for coming to Wikipedia and what you see here interestingly is that these are the users who stay on Wikipedia they don't tend to go in and out so much they come to Wikipedia they stay here and they read they use the internal ways for navigating Wikipedia to learn more or to read they also tend to be fast readers they have the number of requests that we receive from these people is more than eight meaning that they go from page to page to page they focus on three main topics and these are sports 21st century tv movie and novels and the average topic distance for these people is more than one which means these people actually change topics also very frequently they don't necessarily stay on one topic they don't start with sport and start stay in sport they start from sport and they end up somewhere else okay so I throw at you a lot of data and knowledge let's step back and summarize what we built is that we built a taxonomy of Wikipedia readers we know that we need to focus on the information need prior knowledge and the motivations of the users we quantified the prevalence of each of these dimensions and the categories in these dimensions and the interactions between the use cases and then we went a little bit deeper by looking at the web request logs to try to understand these people in the context of different sessions articles or requests that they have sent to us and now what are the implications of this kind of research well and our future direction one thing I want to say loud and clear on behalf of all the people who have worked on this project we want to hear what we should do more it is not actually very clear right now so one thing we know we should do is that we should repeat the English Wikipedia survey the large-scale English Wikipedia survey in at least two other languages so what we will doing in the next couple of weeks is preparing another survey you want to go and see if the if any of the results that I showed you will change for example if you run the survey on Korean versus English we tried during the span of this research to build prediction models that can predict the motivation of the reader and this was unsuccessful we can do a little bit better than random guess and one of the hypothesis we have here is that basically the issue is that the features we have do not capture the motivations of the user's problem and one thing we observed which is probably true is that we are asking people about their motivation on an article and people can change motivations when they start the session you you will start as a work or school project but you may end up reading about the movie which it doesn't have anything to do with the project that you have to deliver right one thing that we are interested to do to focus on next is to predict motivation at the article level and that's something I would like to hear more from you all about so suppose for every article I could show you the distribution of motivations from people who are coming to that article what would you do with that information would it be useful for you to know that only 5% of the articles sorry people who are coming to read your article are reading it in depth would you change anything or would it be useful for you that if you know more students read your article than people who are motivated by media I don't know if it should be useful to you but like I like to know it and this is a little bit on the philosophical but also pragmatic and what is the movement's role so now we know that 20% of 21% of people who come on English Wikipedia do in-depth reading what is our role as people who are working on a mission with a project that has immense implications on what happens on the internet you can have a role in defining whether this number should be more or less and you should not overlook that right and then when I say you I mean the people who work in the foundation but I also mainly I'm talking to editors the content that you provide the kind of context that that you provide can have impact on this number do you want people to read more then how should we do it should be able to measure this should you be able to change content and we measure and tell you how that's changing these are the things that you should be thinking about and there is another thing you're probably writing Wikipedia because you want to share the free knowledge and you want people to learn probably please for some of you now if you know for example that people do a lot of context switching if you know that people have come here to learn something and now they're learning something else do you want to help them to focus or no what is our educational role here that's another thing that we should think about the screen size and content that's another important thing we we know that many of our users are only on mobile we know that many of our users will move to mobile more and more we know that the screen size is small and the question is what kind of content do we want to show to the user so now one thing would be that we say okay we know only 21% of people are doing depth reading so let's just cut the article and just show them in snippets of information but by doing that we will have impact on how people learn and this needs to be intentional we need to understand if you want to encourage people to read more or we want to encourage people to read less and this is especially critical on smaller screens and there are many more things that we talk about now on a daily basis when it comes to this project so my question to you is what else could we do and I'll stop here with this and we can take probably around 10 minutes of questions if you have questions thank you okay any questions in the room yes I hope I didn't miss this as you were talking Laila but did you guys walk through any use cases like I know Abby's group has done sometimes for instance this person was bored they were on mobile they went to something about a movie and and try to look at that empathy-based part as part of the motivations did you sort of work that all the way through with any of these so we didn't look at we didn't go through any individual responses right so pretty much everything that we did was kind of at the algorithmic level doing at like classes of users and letting the algorithm decide what are the subclasses that are interesting to look at now in the context of the new reader's work I think it would be very interesting to actually go a little bit deeper and just read like basically print sessions of people let's say from Mexico Nigeria and India and then let's look at those and then let's see if we can make any other insights out of the information that is available to us specifically for Nigeria and Mexico we don't have that much that many people from these two countries that have participated in the survey right so we may need to design follow-up surveys to target these two populations further to get more data out of it because basically with 30 000 data points once you go at the you know you start you start slicing and dicing not much will be left in each category to be able to say something of significance but I think that's something we should be doing I have questions from the audience before I ask my question okay go ahead it's actually more of a comment but so you asked what the foundation can do versus what the movement and volunteer contributors can do and I found specifically the main takeaway on different types of motivation or information needs is a function of the topic to be extremely relevant for the way in which we structure the content of an article again content is not something we give the foundation as any control of this is entirely community driven there are some aspects related to reading recommendations or the interface that we're in best position probably to address but it strikes me that when as a volunteer I write on a topic I often don't have any guidance about who am I writing for what are the needs of my of my audience and there are many cases in which decisions about structure information what to put at the top what to put the bottom are really uninformed by by data so actually I was talking yesterday to James about one case where the decision is to whether or not to put some expert related information towards the top or the bottom of the medical article can actually have some pretty big implications for audiences right so just a comment that I think to me that is one of the most promising areas in terms of actionable actionable input for it so I want to comment on this comment it's it's a very important point and it's not an easy point it's it may be easy to say what are the motivations of the people who are coming at the article level so we are going to try that next to see if we can predict but the tricky part is that suppose I tell you that 80 percent of the people who are coming to this article are coming for a quick lookup is now the conclusion that we need to show quick facts on top of the page or the conclusion is that no these people should actually be reading more of the content and going deeper and that's the part that is hard and I'm not saying it's not solvable but I'm just saying we need to be very mindful about every action that we take based on this knowledge because this will have implications on how people learn on the internet I have a couple questions from IRC first question is would it be feasible to repeat the study but focus on a few articles and track basically the answer distributions over time yeah so I saw that that was from Lodovike yeah so this will be feasible you tell me which articles so I'd be happy to talk to you Lodovike after the talk to see which articles you are interested in I can imagine that even in Doc James case probably there are medical articles that we can focus on so I have some ideas in in that area but if there's any specific area that you want to focus on please bring them up to myself Ellery or anybody else who has worked on this project and we would be happy to think about it at least brainstorm with you to see where to look for and I do believe that there are articles for which this kind of analysis and information can be useful for example if you're providing medical content in certain parts of the world like you know people won't be able to parse the whole article what are you going to show on top of the page right and we may be able to experiment with interesting things there and learn then we have a sort of suggestion for further work by Ziko so in this work sort of we've measured you know people's the depth of information that people need when they enter the article and so an open question would be how well does the article satisfy those various needs and how does the quality class of the article sort of you know impact how well it satisfies those different needs yeah very good comments Ziko and I'm happy you're here you made it so about satisfaction so we started with the notion of satisfaction what we gave up and then we gave up a year ago but that doesn't mean we're not going to go back to it the reason for that is it's a little bit hard to ask for people's satisfaction without interfering and making them aware of what they're doing and then the responses will have that kind of awareness will have impact on their response so a few ideas that we had talked about was basically one thing websites do when you land in there they ask you at the end we will we may ask you a few questions would you be willing to participate and that's not very good in our case because that makes the users conscious the other thing we can do is something like mouse tracking for example so you can track the mouse and as the user wants to close the page completely you will know that the session is ending basically and from those users you can basically pop something up and ask them did you get the response for something that you are looking for now mouse tracking is is a technology that I mean we don't use at the moment but it's something that I think we should be open to if you have these kind of questions and we want to look at the satisfaction of the users and then the looking at the quality and that was also a very good comment I think this is something we did not look at right now partially because the quality levels and differences across languages are different but that's something we'll make a note of to look into further another question from Giovanni on IRC so you mentioned that based on just looking at the log data and the features we extracted from that it was quite difficult to infer people's response eventual responses to the survey so Giovanni is asking what type of other feature engineering you think would be promising in that direction is it possible at all from to predict people's survey responses given what's in the logs that we currently collect yeah so again Ellery at least who is on the call can correct me if I'm wrong but my understanding at the moment is that we don't have very high hopes for being able to predict motivations for the readers and again like part of the reason is that people are switching motivations as they are in a session so we need to continuously collect information there may be creative ways for doing this I can't think of it at the moment but definitely something that we will continue thinking about and Giovanni if you have ideas about how we can potentially do this please let us know yeah so one idea that we're thinking about exploring in the future that they'll actually mentioned is it seems quite plausible that the article itself holds a lot of information about readers intentions for visiting so if we could get sort of enough survey responses for a large sample of articles maybe we could basically infer what the response distributions would be for articles that we don't have data for it just seems likely that you know an article about you know television actress would have the motivation distribution would be like pretty skewed and not as diverse as it is averaging over all articles yeah Daria here I just want to make a follow-up comment on this there are obviously reasons related to research to try and understand if there are features that are highly predictive when it comes to motivation but they're also actionable aspects of doing this kind of like pretty predictive modeling of readers and I just want to say that the vast majority of features that were used from a behavioral standpoint from this any motivation would not be features that can be treated or extracted in real time correct me guys if I'm wrong but the amount of like a processing of data will be needed to make a real time classification for example of readers as a function of motivation compared to what you just said about the articles which are fairly static and can be preprocessed is not something to be used in an actual way for building better recommendations on the fly based on the on the little data that we collect and retain can you guys comment on that yeah I can comment on this so I think it's fair to say that we're not going to be doing I'm not sure about Abel or not it sounds if I have to bet I would say what Dario said it's correct we won't be able to add an online fashion predict motivations of readers as they're in the website what we can however do is that we can let the reader tell us why they're here for so suppose you're a reader you know here you're here for a work or school project tell us that you're here for work or school and then we can provide services to you that are associated with that especially again on the mobile screen we we talk about like preview of the next article what we show as a preview of the next article if you tell us why you're here for that information maybe changed based on your motivation so I don't expect us predicting online the motivations of users but if there may be things to do if users tell us why they're here for what they're here for Casey did you have a question my question was have we considered or would it be useful I mean I think it might be to include questions about to the users how concerned they are with data cost or download time and if that affects their behavior between maybe how long they stay what they search for so then the new readers project affordability is something that we're interested in yeah definitely that's something very interesting that in this area definitely we didn't touch because the focus was something else but yes if you want to decide what kind of content for example we show it what part of the page we need to understand the affordability issue especially as we go to other countries and that can have impact on our decision about how to slice and dice content one more question from the room is it on okay um as I recall daily you you talked about how the the quick checks were desktop and typically were google to wikipedia article and then all the way back out was that right so I think these were for people whose motivation were work in school they were mostly on desktop and they were going in and out a lot and and when they came back to wikipedia it was also through google so um I said google as an example but for majority of cases is google um yes they will come back from that refer which is google so in many cases they're basically just going out doing a search and coming back as simple as that yep and I guess what I wondered was the I guess they would be the left brain users the folks who are browsing and reading and stuff were they staying on wikipedia and were they doing our hyperlinks and our search bar were they not going all the way back out so I don't know about we don't know anything about they're the type of user beyond the three levels that we have right so the only thing I can tell you is that for the for the people who were here motivated by boredom and randomly exploring wikipedia these people tend to stay on site ah okay so the quick check they're going all the way in all the way back out or many are and for the browsers they're staying and reading us like a magazine there's they're staying on there right any other questions okay thanks so much everyone for staying around also started later and you'll stick around thank you so much see you next time