 All right. Good morning. Good afternoon everyone. This is Dario from Wikipedia Foundation and I'm delighted to have you here for our April edition of the research showcase. Today we have two external speakers who will be presenting on their research, actually very recent research that it's about to be presented in today's conferences. So we have Nick Vincent from Outwardson who's presenting a study looking at relationships between Wikipedia and other online communities. And we also have a native land charge from University of Washington presenting results from a replication study looking at the impact of norms and quality control systems on open communities, replicating results from Wikipedia in other Wikipedia's communities. Very excited to have you here both today. As usual, this is going to be two presentations about 25 minutes each followed by Q&A. And we have a discussion IRC and Jonathan is going to be our host. Thank you for any questions from us. So with that, you're welcome to start the talk. All right. Here we go. All right. Hi everyone. I'm Nick Vincent. I'm a first year PhD student at Northwestern University in the People, Space and Algorithms Research Group. And today I'll be talking about some work that I did with my co-authors, Isaac Johnson and Ben Hect, looking at how Wikipedia matters outside of its immediate contents. So Wikipedia has been immensely important to computing research. A huge body of literature has used Wikipedia to answer questions about social computing, about peer production, and even to train state-of-the-art intelligent technologies. But Wikipedia doesn't live in isolation as we've seen recently in the news in a surprise move aimed at concerns about misinformation. What YouTube started linking videos about conspiracy theories to Wikipedia articles. And YouTube is not the only internet giant with Wikipedia dependencies. Facebook similarly started adding Wikipedia links to news articles that are shared by users. And this is really recent. The date on there is April 4th, in case you can't see. And last year at ICWSM, the McMain had all demonstrated the immense value that Wikipedia adds to Google. In their lab and field studies, the inclusion of Wikipedia massively increased click-through rate. But I really want to emphasize that Google-Wikipedia relationship. In the study, the drop in click-through rate was incredibly large. It went from 26 to 14 percent, which means that Wikipedia basically increases click-throughs for Google by 80 percent, which is just massive. Together, news and research suggests that Wikipedia is important outside Wikipedia. And that means that the volunteer work of Wikipedia contributors has value and impact outside Wikipedia. And it also means that the research on Wikipedia, such as the positive effects and also negative things like bias, these may have effects outside of Wikipedia as well. And this is all there's a big advice versa here. Things that go on outside of Wikipedia matter to Wikipedia. For example, if Google were to change its search algorithm, that could be a really big deal. But we're still left with a lingering question. What about other entities in the internet ecosystem? And that's what we looked at in our study. So I'm guessing that some people in this hangout might be familiar with Reddit and Stack Overflow, either because you researched them, your users of them, or both. In fact, if you're like me, Stack Overflow may have helped you finish your recent work, such as this paper. And Reddit may have helped you unwind afterwards with some top pictures. Although they're quite a bit different in purpose and scope, both Reddit and Stack Overflow are large online communities and both of these sites include links to Wikipedia. On Reddit, Wikipedia links are often used to share interesting facts, like this sentence about the ancient Egyptian calendar. And on Stack Overflow, Wikipedia links often act as references or further reading, like in this very popular branch prediction answer, which when I wrote the paper, this was the most popular answer on all of Stack Overflow. So in this paper, we specifically looked at a sample of submissions on Reddit and submissions are like the first post in a thread in a traditional form, if you're not familiar with Reddit, and answers to questions on Stack Overflow. But throughout this presentation, I'll just refer to both as posts. And to show you what they look like in context, here's a quick screenshot of just a page of Reddit posts. And here's a screenshot of a question on the top and an answer on the bottom, the big arrow points to the answer. So to frame our investigation of the relationship between Wikipedia and these online communities, we identified two research questions. First, we asked, what effect does Wikipedia have on Reddit and Stack Overflow? So how does Wikipedia add value to these other online communities? We operationalize this relationship with four engagement metrics. So that's user-voted scores, number of comments, views, and estimated revenue. In other words, to posts like this affect user voting, commenting, viewing, and money in the end. And then secondly, we asked, what effects do Reddit and Stack Overflow have on Wikipedia? We operationalized this metric, this relationship with metrics that capture value to Wikipedia, so article page views, edits, new editors, and editor retention. We especially focus on edit behavior because a major concern for Wikipedia is declining editor base. So now that we framed our research questions, let's dive into how we answered RQ1 and what we found. To start with, we first consider how many links to Wikipedia actually exist. And it turns out that Wikipedia links appear in a small fraction of posts, but Wikipedia is one of the top external sites that get linked to on both communities. So Wikipedia is the fourth most linked to site on Stack Overflow, but the top three are all just sites that were code and documentation lists. So specifically, it's GitHub, Microsoft documentation, JS Fiddle, which means Wikipedia is actually the number one conceptual resource on all of Stack Overflow. And on Reddit, Wikipedia is especially prevalent on the widely subscribed to subreddits, mainly doing to the Today I Learn community, which has a lot of Wikipedia links that have really high scores. And in addition to that, there is some past work suggesting a causal relationship between Wikipedia and Reddit specifically. So in 2015 at ICWSM, Moira all had a study showing that Reddit posts in the Today I Learn subreddit increased Wikipedia page views after they're posted, and that was a causal effect. So with this context in mind, we can take a look at the average metrics of posts with Wikipedia links. So here on this slide, we're just looking at two plots. On both plots, the y-axis shows the mean score and the x-axis differentiates between Wikipedia linking posts and posts without Wikipedia links. And there's an icon to reinforce that. On the left, we have Reddit posts. And on the right, we have Stack Overflow posts. And right off the bat, we can see Wikipedia links are well above other posts in terms of score. On Reddit, posts with Wikipedia links have about five kinds of score of other posts, which is quite large. And on Stack Overflow, it's about two and a half times the score. So these are pretty big effect sizes in the associative statistics. These stats certainly suggest that Wikipedia adds value, but we want to take this analysis a step further to really understand how these Reddit and Stack Overflow posts might look in a world without Wikipedia. Because just from associative stats, we don't know if there's other factors at play. Like maybe people who write long posts also happen to post Wikipedia articles a lot. And it's actually the long posts that are responsible for increased metrics. In that case, in a world without Wikipedia, nothing would change. So to answer this question, to dive into this world without Wikipedia, we identified three counterfactual scenarios with increasingly conservative assumptions. And when I say conservative here, I mean, we're airing on the side of assuming that Wikipedia is more easily replaceable. So I'm going to walk through those now. And to walk through them, I'm going to use this example post that I brought up at the beginning and talk through these posts and how we might estimate the treatment effects in each scenario. So in our first scenario, we assumed that Wikipedia is so critical to each post that in this world without Wikipedia, every Wikipedia linking post would just be gone. It never could have been authored in the first place. So that symbol to imagine this post would just be essentially deleted from the database. It never existed at all. And that means that anything that came out of it, such as score, discussion, ad revenue, would be entirely attributed to Wikipedia. We could also imagine that perhaps roughly the same post exists, but now they became a post without any external link because Wikipedia is gone. So in this scenario, this post would be the same, but instead of linking to Wikipedia, now it's actually just a Reddit self post. And a reasonable way to estimate the value of Wikipedia in this scenario would be to compare Wikipedia linking posts to similar posts without Wikipedia links. Finally, for an even more conservative scenario, which we can call a lower bound, we can imagine a world where every Wikipedia linking post exists with an entirely different external link. For this example, the post might instead link to the Google Books page about the topic. And as a side note, this particular book is actually the citation for the fact in the article. So this isn't so impossible. To estimate the value in this scenario, one way we could do that is to use the difference between that middle ground estimate, the value added by Wikipedia compared to similar no link posts, and the value added by non Wikipedia linking posts to similar no link posts. And I'll expand how we did this in the upcoming slides. So just to emphasize again why we're going through these counterfactual scenarios in this methodology, it's because we want to understand the value that Wikipedia brings to Reddit and Stack Overflow beyond just association and correlation. And one way to estimate these treatment effects corresponding to counterfactual scenarios is causal analysis. So specifically we use propensity score stratified regression, which is a method that's very popular in economics and medicine, but somewhat new to human-computer interaction. And the general idea of this method is that we want to make an estimate by making comparisons between data points with reduced covariate bias. So the covariates in this case would be things like textual features such as text, user characteristics such as reputation, or contextual features such as when the post was made. And to put it more formally, we want to minimize the covariate bias in our comparisons in our estimation of the treatment effects. So I'm going to walk through exactly how this works. To set the stage, let's say we have some data in our case posts represented here by red and blue circles. So in this case red indicates a control post which has no Wikipedia link and blue indicates a treatment post which is a post with a Wikipedia link. For each post, we're going to calculate a propensity score, which is now shown on the slide. And the propensity score captures how much a post looks like a treatment post. In practice, this is actually a logistic regression where the input is all the visible covariates such as post length and the output is a probability of whether the post is a treatment post or not. So any post with a high propensity score such as 0.9 is a post that has covariates like posts that actually have Wikipedia links in real life. So based on the propensity scores which say whether the covariates look like a Wikipedia linking post, we can separate our data into strata based on whether propensity score falls. So the post with low propensity scores will go in the low propensity stratum and likewise the post with high propensity scores will go in the high propensity stratum. Then we perform a regression within each stratum where we'd expect the covariate bias to be smaller and then take a weighted average of the regression results with the regression outputs weighted by how many treatment posts were actually in each stratum. And this produces an estimate called the average treatment effect on the treated or the ATT as you might see it in the economics literature. And just to get a quick sense of what all this actually means in terms of Reddit posts here's two examples from a high propensity stratum. So both of these posts were actually in our high propensity stratum from our experiment and you can see they have long capitalized punctuated titles and also the posters had a very high reputation in terms of karma. So that's why these posts made their way into the high propensity stratum. To go back a little bit to actually estimate the effect of Wikipedia in our middle ground scenario we use this propensity score stratification method to compare Wikipedia posts to posts with no links at the middle ground estimate. Then to estimate the effect of Wikipedia in our lower bound scenario we find the difference between our middle ground estimate and the effect of non-wikipedia external links compared to posts with no links. And that's going to output a lower bound the value added by Wikipedia in the scenario where every single link got replaced with an alternate external link. And after this procedure we find that in our lower bound scenario the value added of Wikipedia is smaller but it's still quite high. So this new bar on this plot the same plot we had before but now there's this lower bound estimate in the middle this shows the value added by Wikipedia. So entirely attributable not the value of the post but the value added and it's still quite high after adding this value Wikipedia linking posts are still over four times higher scoring on Reddit and two times higher scoring on Stack Overflow would suggest that Wikipedia really is bringing value to these online communities. We also did a similar analysis to estimate the effect of Wikipedia article quality on our value metrics and we found mixed results so quality mattered a little on Reddit and not so much on Stack Overflow. And finally to contextualize all this analysis we performed a revenue estimate. So corporate revenue is a bit of a black box but even black boxes are vulnerable to the mighty back of the napkin calculation so the core fact that enables this estimation is that both sites Reddit and Stack Overflow mainly make money by selling ads with a fixed cost per ad view. So actually went in and looked up looked at these costs per ad view and the revenue is basically proportional to the views. We have views from Stack Overflow in our data dump and on Reddit score correlates with you. So roughly back at the napkin estimation revenue from Wikipedia is proportional to the views added by Wikipedia which we just calculated and although this estimate required making some assumptions we tried to err on the side of being as conservative as possible. And the key conclusion here is that under our conservative estimates Wikipedia contributes $100,000 per site per year just from article links that increased ad views that were posted in Stack Overflow answers and Reddit posts the Reddit submissions. And it's worth re-emphasizing that these figures are likely only a tiny fraction of the value Wikipedia adds all over the internet because this is just two communities which are themselves free to use. These are just yearly figures and these are conservative estimates. So now we can dive back into RQ2 where we asked what effects do Reddit and Stack Overflow have on Wikipedia. And to look at this effect it's actually the methods are much simpler. We just compare the Wikipedia metrics for each article from the week before post was made with that article to the week after a post was made with that article. So how many views before how many views after etc. And we do see an increase in page views as in previous work from where at all. And we do see some edits from Reddit only but an aggregate the amount is very small. So for a quick comparison it's 0.4 edits per post and that amounts to about 0.002 of Wikipedia's 10 edits per second that are going on right now as I'm speaking. And we saw almost no edits from Stack Overflow aside from a tiny but albeit statistically significant increase in edits to low quality articles. So a small amount of Stack Overflow users did edit low quality articles after they got posted on Stack Overflow. And finally we didn't find any evidence that Reddit or Stack Overflow contribute to the editor base of Wikipedia either through edits from new users in the week following a post or through an increase in retention of these new users. And these results were somewhat surprising. The large score in comments we saw suggested the users were really engaging with Wikipedia linking posts. So inspired by a previous talk from Dario at the Wikimedia Foundation we suspected that reuse of content might be at play. For example in that today today I learned posts I've been using as an example the text on Reddit already contains information from the article. Here's the actual article. So if I'm a Reddit user and I'm just interested in obtaining a fun fact and moving on I don't really have any incentive to click on this link and visit Wikipedia. And Dario Torrebreli identified this as the paradox of reuse. Perhaps Wikipedia's permissive content licensing is mitigating the benefits of Wikipedia. We saw a really strong version of that in the Google Wikipedia relationship study and it seems like it might be going on here. So to investigate this we did a small qualitative coding exercise and we found that on Reddit 79% of the posts had content quoted or summarized from Wikipedia and on Stack Overflow only 33% did. So this suggests that on Reddit in particular reuse may be limiting the extent to which users are engaging with Wikipedia from other communities. And overall this does provide more evidence that addressing the paradox of reuse is very important for the sustainability of peer production. So now to sum it all up let's return to that diagram from the opening slide but now we can replace those question marks with some findings. So we saw that Wikipedia adds substantial value to Reddit and to Stack Overflow and they're both on the order of $100,000 per year or more. And we saw that the return effects were much smaller just a small amount of edits from Reddit and likewise from Stack Overflow. And it's a bit tricky to directly compare value the value to Reddit and Stack Overflow which are companies and the value to Wikipedia. But one way we could do it is we just compare the total fraction of score that Wikipedia provides to the total of fraction of edits provided in return. We can see so 0.7 to 0.02 there's over an order of magnitude of difference. And so this relationship can be characterized as quite asymmetrical. There's a lot of there's definitely a lot of factors at play here but for now I'll focus discussion on the important implications of these findings and some possible opportunities. So first off these results provide more evidence that Wikipedia consisting entirely of volunteer created content creates revenue for corporate entities. And this adds to a growing body of research looking at data as work which is a new perspective that's going to have long-term implications for relationships between online entities and even our ability to estimate economic measures like a gross domestic product GDP. What does Wikipedia mean for that? And there's some asterisks at the bottom here. I have to add a disclaimer looking at hyperlinks is just one way that Wikipedia interacts with other entities in the internet because so much computing literature relies on Wikipedia as a corpus there are likely even deeper algorithmic dependencies between Wikipedia and any platform using natural language processing of any kind. So that definitely warrants a lot of future work. And additionally we see some potential opportunities in our findings. What we see is the improvements to Wikipedia stand to benefit Reddit and Stack Overflow and vice versa. So collaboration between communities could really be great for everyone. The platform owners can make more money, users can get better content and Wikipedia can advance its mission. So this might take the form of campaigns started by community members which could build on existing ideas such as this Reddit post calling for fans of the video game Stellaris to update the Wikipedia article about it or it could come through resources like this existing Wikipedia project page which provides a very detailed list of possible cross-community partnerships among other resources. And together these collaborations could help make those arrows that we were looking at much bigger and make everyone happier. Another option is design interventions and I'm very excited about the new Wikipedia page previews I just read about yesterday because this is a perfect example of a great opportunity. In the actual Medium blog post by Nerzar Pankerkar, he says Wikipedia page views previews, Wikipedia previews can be adopted by other publishers as a means to provide quick context around topics. And this might be a great way for Reddit and Stack Overflow designers to basically implement a quick design intervention to help users actually engage directly with Wikipedia for the mutual benefit of everyone. And finally, these collaborations may be especially helpful to improve Wikipedia's coverage issues where a small number of edits relative to all of Wikipedia could actually be a really large number of edits relative to an undercover topic. For example, articles about rural areas which are often undercovered. So to conclude, there's a lot to be done to further this line of research. Future work could look at more nuance to these relationships using additional metrics or more close looking analysis. It will be important to move beyond back of the napkin estimations and provide more formal financial estimates. And another really valuable line of future research will be looking at other communities. And in light of recent news, two especially come to mind as being very important to study, and that's Facebook and YouTube, which I'm sure will be the subject of some scrutiny in the upcoming days, weeks, and months. So with that being said, that's actually the end of my presentation that looks like I'm about five minutes early here. Thank you so much for listening. I'd be happy to hear any feedback. One note I'll just add is that I'm actually prototyping this talk right now, and I'll be presenting it again at CHI factor on the conference on HCI research next week in Montreal. So any feedback regarding the presentation or the research would be much, much appreciated. Thank you so much. Thank you so much. That was fantastic. And yeah, thanks also for giving us the exclusive of your talk before going to the conference. I'm very excited about the design of research. I have a few questions. One of you asked first, the room or IRC, if there's any discussion or questions for Nick, they're about actually a bit more than five minutes right now. Awesome. Yes, we have one question from IRC from Tillman. Tillman asks, how was the impact of Reddit or Stack Overflow on new account creations calculated considering that these are not tied to a particular article linked from those sites? That's a great question. So the way that we did it is we actually basically used the individual revisions as the main source of all this data. So we counted up all the revisions from the week before, counted up all the revisions from the week after, and then looked at how many of those revisions actually came from a user who registered a new account during that exact week of time. And then we took the difference between the number of revisions that came from new users who made an edit to that article. So there's definitely some limitations to this methodology, but the general idea is to look at the difference between the week before and the week after to see if we can detect an effect in terms of people navigating from Reddit to Wikipedia, creating an account and then editing that Wikipedia article. Awesome. So it's actually new accounts who edited it. Sorry, could you repeat that? So it's actually about new accounts who edited that we're counting. Yeah, exactly. So it's specifically it's edits that came from newly registered users. Cool. Thoman also wanted to point out that this paper was recently reviewed in our research newsletter, and I put the link in there. It's also an IRC. I can send it to you if you'd like. Awesome. Yeah, I have, I think I saw that, but I would love to take a deeper look. Cool. And that's the only question so far that we have from IRC. Okay, we have a I think two questions here from the room. Yeah. Hi. I was wondering if I'm not sure if Reddit and Stack Overflow are publicly traded companies, but I'm wondering what percentage that $100,000 is of their total revenue? Great question. So first of all, as far as I know, they're both private companies. And those basically the way that we did the calculation, the percentage of revenue is actually just the percentage of score in the case of Reddit and the percent percentage of page views in the case of Stack Overflow. So for Reddit, it's 0.5 to 0.7 percent. And for Stack Overflow, it's 0.4 to 0.9 percent. Okay, so is there I'm not sure if I'm following, but is there any way to tie those scores actual revenue for the company? I mean, is there any way to know what that value is objectively? I guess. Yeah, no, that's a great question. So the logic that we followed in the paper is to basically, so we actually do have Reddit has released information about their revenue in the past. And for Stack Overflow, we basically did our own estimation based on looking at the actual price per ad impression and then calculating how many ad impressions appear on a page and how many total page views occurred in the year. And we added a coefficient there to factor in for the percentage of desktop users who use ad block and a couple of other things, if I recall. So we produced the Stack Overflow number. And basically, we came to the conclusion that the annual revenue for Stack Overflow in 2016 was about $21 million. And for Reddit, it's about $19 million based on their own admission in a blog post from the CFO, I believe. Is that answering the question? Yes, totally. Thank you. I have a question again, jump on the queue. So, yeah, first off, thanks for laying out and not just the results, but also the consequences that I bet this presentation is going to get some attention from from our executives because of the of the of the current debate around the reuse of the comments and the symmetry you pointed out very eloquently. I was I was not surprised to see that the the returning it's like the side of this of this loop is is so small. We, I mean, I think a number of internal and external lines of research have shown that this kind of like drive by engagement from external communities doesn't doesn't result in a in high retention of contributors doesn't result in a significant amounts of edits. Well, significant depending on the scale, of course. But the one question that I had for you or something we looked at also in the past is the question of the quality of edits. And I was curious if you've done any analysis on these edits that come in from red and the second floor. Are they high quality edits? Do they get reverted? Do you have any sense of what they contribute to Wikipedia? Yeah, so we actually did this is informal and I I could pull up these numbers from a CSV file somewhere hidden in a directory. So we did do a brief analysis because one of the one finding that I didn't discuss here and it's not in the paper is one of the communities that did provide a lot of edits other than just such that I learned our slash they learned is one of the big communities are slash Wikipedia is one of them. But also the Donald, which is the Donald Trump subreddit had a lot of edits and I was we're kind of surprised by that. So we did actually run an analysis to see if edits from certain communities were being reverted at a greater rate than other edits. And I'm fairly certain that the the result was that overall the reverse the rate of how often edits were reverted was actually quite low. It wasn't something staggeringly high, although maybe my my frame of reference is off there. So my informal answer is that it seemed like the edits were not being reverted in mass or anything like that. We also did one other analysis that's not reported, which was to see if there was any phenomenon where people were editing Wikipedia articles and then posting them on other sites like to basically create their own, you know, short term credibility before someone could revert the vandalism and we didn't find any evidence about either. Very interesting. Thank you. One other comment from IRC. This is from Leila. Leila says I very much appreciate the effort to estimate the financial value of Wikipedia. This can help copyright discussions and negotiations with community foundations or chapters take part in. I also want to highlight that this kind of citable information is key for some of our communities. I can speak to the Persian Wikipedia community where I'm from. We do get the kind of open free license. So what question in that community? Awesome. That's great to hear it. I totally agree. All right. I think we're at time for the first first slot. So thanks again Nick. Hopefully you can stick around. There's additional discussion at the end of the of the hour. And I want to pass it on to Nate who's going to give a second presentation on the replication of the rise in the Kai study. The stage is yours. All right. Okay. Thanks a lot, Nick. It's really good. I'm looking forward to seeing your talk again in the Kai when we're in the same session. Yeah. So I'm going to get my presentation up. Oh, wrong version of the slides. Sorry. Yeah. So I made a mistake when I made my slide, but we'll make them again. But I'll just get started anyway. I'm a PhD student at the University of Washington. And I work with Benjamin Hill who some of you may know. And we're part of the Community Data Science Collective, which is like an interdisciplinary research group between the University of Washington and the Northwestern University, the Department of Communication. And so I collaborate with Aaron Shaw and with Mako on this paper. And I'm really excited to be here today and talk to you about it. It's about, as Dario said, it's about revisiting the rise and decline of Wikipedia. And I'm going to abbreviate the rise and decline as RAD because it's a RAD paper and the others are RAD as well. Some of them are listening. That was Aaron Helfacker, Stuart Geiger, Jonathan Morgan, and John Radle. And they were interested in the abrupt transition from growth to decline in the history of Wikipedia. So everyone may be aware that there was this amazing period of growth that was so exciting. But then it ended in 2007 and Wikipedia has been sort of on the slow decline trajectory ever since. And Helfacker and his colleagues really wanted to understand why this was happening. And they developed a theory and a model that was centered around the survival of newcomers to the community and that's a sort of necessary part of sustaining a population of active editors to Wikipedia. And the idea was that in order to sustain a population, some people are going to leave the community. There's always going to be some attrition. But as long as you can bring in new people to replace them, you can sustain growth. But the population of the amount of newcomers that were surviving was falling and they attributed this to the quality control systems that Wikipedia introduced in order to prevent vandalism in large part, prevent vandalism. And one of the main things they found was that the newcomers who got rejected whose reverts were edited were much less likely to survive. And then making matters worse and more complicated, the quality control systems that were increasing newcomer rejection became more difficult to change. They calcified in their term. And they were especially calcified for newer editors. So the editors with more tenure were more likely to succeed when they attempted to change the norms. The norms meaning the policy pages as well as essays and guidelines that sort of do the work of structuring how Wikipedia is supposed to operate as an organization. They also saw that communication with newcomers increased their chances of survival. So if a newcomer got a message or if they got a message if they were reverted, that would make them more likely to stick around. But the rise of what I call algorithmic governance tools, which include both fully automated bots, Clubot, as well as tool assisted editors, they reverted newcomers. And the newcomers who were reverted by these tools were less likely to survive. And they also decreased the amount of communication that the newcomers had. And then tying it all together is this sort of the temporal process of the wiki. And so that as the Wikipedia matured, this transition took place that increased newcomer rejection, increased the algorithmic governance tools, and made it more difficult to change the norms. And I conceptualized this as wiki age because I'm going to apply this theory to an analysis of a large number of wikis that were created at different points in time. And so it's that makes it possible to sort of think about the life cycle of a wiki, not just the sort of history of Wikipedia as happening like in a course through time. Okay. And so I'm going to talk a little bit about like what's the rationale for like, doing this replication study. And one of the rationales is that the like this paper is a lot of influences influence conversations across such computing and even beyond. And these are some quotes from a variety of people who also sort of say that like the quality control mechanisms of Wikipedia can be restrictive and biased against changes introduced by new editors. The rise in algorithmic monitoring and increasing rigidity of policies have been shown to reduce number of new contributors to continue editing. And the bottom up process of revisions by the community is far from perfect. And so people are taking this story and applying it to and also these these people are not thinking about wikis here. They're thinking about open street map. They're thinking about how how should we like do quality control on social media like Twitter or Facebook? And how should we think about the the like ways that we're also thinking about Kickstarter, right? And like crowdsourced fundraising. The third quote is not from Havakardal. That's just I left the citation to Havakardal in there just all of these people are citing this paper. Okay. And then and another rationale for replication, not just because the study has been influential, but because there it was not like a like no study is perfect. There are always like alternative explanations that you can't rule out. And one of these is that like maybe there was something that happened during Wikipedia's trajectory or during Wikipedia's history that you know caused the decline, right? So maybe the rise of social media like Facebook actually you know created a competition for the attention of the editors. And so people maybe just you know at one point in time Wikipedia was the really exciting thing and everyone wanted to do that but then Facebook was the really exciting thing. People wanted to do that instead. Another important part is that like the sort of original rad story is like the importance of vandalism, right? That Wikipedia became popular but that attracted lots of vandals and that required you know quality control systems that would that contributed to the decline, right? But what if you know your project is smaller and there's like lots of important projects that are smaller that don't attract the same levels of vandalism but maybe and maybe those kinds of projects will be resilient to the rise and decline. And then the broader rationale for this for me is I want to like contribute to the science of peer production in the sense that we can you know that we can like Wikipedia doesn't just like provide a free source of knowledge, it also inspires us to think about how you know we can collaborate together and produce important valuable as like Nick just showed like very valuable and sources of knowledge and not but not just and public goods sorry like very valuable public goods and we can and so by generalizing the rad study we can learn more about that. We can learn if the rad process generalizes to other peer production communities or not and we can learn if the sorry and from that we can learn like are these sort of quality control problems related to newcomer retention across different projects. Can we expect open collaboration to encounter these types of difficulties in general? Okay so I break down my analysis into sort of three parts or three mini studies. The first one focuses on the sort of aging process of the life cycle wikis through the and this I just replicated like three of the plots from the original rad paper right so this is a study of wiki age and the relationship of age to newcomer rejection newcomer survival and active editors. The second study focuses on newcomer survival as an outcome and through this I reproduce a logistic regression model from the original rad paper to predict newcomer survival and then study three focuses on the attempts to influence the norms on wiki right so this for this I use like wiki age and tenure and another logistic regression okay so my data comes from wikia and I'm using data from 2002 to 2010 this is and then I take all the wikis in over that period and then I just select the top 1% by the number of registered editors so this gives a set of relatively large wikis though some of them are pretty small with as few as 98 editors and then there are also some that are fairly large with as many as 300 000 editors and they're also very diverse across topics right so they include all like pop culture video games fandom and so on they also have a lot of variation in their quality control systems so the number of reverts ranges some of these actually some of these wikis were surprised that no reverts some of them use bots but most of them do not the edit the project name space you know they most of them like most of them use the product name space most of them don't use it very much but some of them use it a lot okay so and then I'm going to talk more about like how we measure each of these variables right so and doing this I'm trying to follow the original rad study as close as I can there are limitations I'm not able to do it perfectly and we'll talk about that later but to measure but active editors are just editors who made five edits or more a month a newcomer survived if they edit after 60 days a newcomer is rejected if they're reverted and attempts to influence norms I use this measure of calcification and for this I just use edits the project name space and this is a sort of proxy for attempts to edit policy and then editor tenure is just the time since an editor is first edit a communication with newcomers is just whether the newcomer has been messaged algorithm and governance tools are whether a newcomer has been reverted by a bot and wiki age is just the time since the first edit to that wiki and now I'm going to log through my results so this plot shows the proportion of reverted newcomers on the y-axis and the wiki age on the x-axis and the original rad studies predicted that wiki age would be possibly associated with the comer rejection so as time passes more newcomers will be reverted and you know this is this turns out to be like a pretty noisy data and a lot of that is due to the rarity of reversion in these wikis right so if you see the thin red line at the bottom that's of the median so actually the median wiki were to known newcomers in the given uh six month period but um but you know you there is like a pretty dramatic increase in the proportion of reverted newcomers early in the wiki's life and then again uh near the end of their life and uh spearman rank correlation test uh shows this change is actually significant even though from these like uh box plots looks really noisy okay um and here's the original plot from rads we can do a little bit of a qualitative comparison between the two uh and it is it actually is a quite different story right there is a bit there is like a pretty dramatic transition between um you know 2006 and 2008 on wikipedia uh and you know there's there's sort of this noisy overall trend on wikia but um so this one's sort of uh you know there's some nuance here to think about um you know wikipedia maybe there's a bit more of a discrete transition between phases that doesn't always happen okay um so for newcomer survival uh i plotted the proportion of surviving newcomers over time again uh and here the trend is very uh it stands up very um clearly um and that newcomers tend to survive less over time as wiki's age yeah so uh and here's the plot from the original rad paper we see the again we see that sort of abrupt transition um but one thing that really stands out is like how high the proportion of surviving newcomers was during the earlier period of wikipedia compared to wikia right so here the the median um is always like less than zero point two uh and wikipedia uh it doesn't fall below zero point two until 2007 okay um so here's a sort of famous like rise and decline plot uh the uh the so in order to make these comparable um i standardized each um month uh with the or sorry standardized within each wiki so like the the units on the y-axis are in standard deviation units um and this is so that you can like sort of take that the aggregation of all the different wikis and do a single data point um but that means that this uh y-axis is it's sort of on a on a like a exponential or sorry like a logarithmic scale so this period of rise is uh actually like translates to like a pretty like uh substantial amount of growth during the early period uh and then in year three we see the transition from rise to decline um and here's again that plot from the original rad paper and again we see a sort of this sort of like very different qualitative um like abrupt transition that gets smoothed out because we're averaging across a lot of wikis but the the like sort of overall story um of like the typical experience of wiki is uh is like pretty similar on average okay so here's results on the the logistic regression that relates wiki age to newcomer survival um and so we found just like the original rad paper we found the negative effect of wiki age on newcomer survival and one thing that's pretty astonishing is like how similar these retro coefficients are um you know like negative 0.37 uh in our analysis to negative 0.4 in the rad analysis um like that I think that um that's pretty surprising anyway um yeah and then uh if you think about like odds ratios this corresponds to like basically newcomers who get um sorry this is reverted so the reverted we have again very similar coefficients negative 0.72 negative 0.68 and this in odds in an odds ratio sense this is means that like in both cases newcomers to get reverted about half as likely to survive um as newcomers who are not new the effective message is still is positive again um so newcomers who receive communications are more likely to survive and um the one thing that we weren't able to replicate was the effect of algorithmic governance tools and survival um and this is purely a function of the rarity of bot reversion on wiki age um it's just there's just not enough uh bot reverts to for us to be able to observe a relationship here um which also tells us something about like the difference between wiki and wikipedia right that like that these that these other dynamics of reversion and uh communication and the life cycle don't seem to depend on the development of the algorithmic governance tools okay um now the question is like do the do the norms calcify uh and they do uh the editor the effect of editor tenure is negative um in our model as in theirs um and uh they and the norms also become more difficult to change over time okay now i'm going to talk about the limitations uh briefly um so we go back here they have this uh all right we have reverted they have deleted and reverted uh and that's because we just can't observe uh deleted pages on wikia we just don't have that that data um we weren't able to observe like tool assisted editors um they're either we think they're either very rare or uh they don't exist really on wikia and as we as i said before like bot reversion is really rare and so these two factors contributed to like our failure to observe that the relationship between bot bot reversion and newcomer survival and then we use the project namespace as a proxy for the policy pages so the original rat authors were able to um you know tease apart different types of policy pages so they looked at the policy pages looked like essays and they think guidelines on wikipedia but the the norms around policy page use or like name project namespace use on wikia are just not as consistent they vary a lot between different wikis um and we just um and so we just use the project namespace as a proxy for that for all the different norms um this is like a limitation but i mean like other people who've studied wikipedia um and govern or sorry wikia and um i've done the same thing so and then the other thing this one's pretty important is that we don't try to identify the good faith newcomers from the vandals uh this was an important part of the original rat paper um and because i mean and they like had to manually code a sample of newcomers as being good faith or vandal or or like ideal newcomers and um that was to show that like you know that this was really affecting like this these quality control mechanisms really affecting like the the desirable newcomers um and uh this was this wasn't something that we really tried to do uh the you know 740 different wikis and different languages about different things uh we didn't you know the um and then also the rate of reversion is so low that um we it's maybe not as big of a concern um yeah all right so in conclusion we set out to ask if um the rat narrative generalized similar projects uh and the answer is yes the overall story um about the mechanism the quality control mechanisms affecting newcomers seems true you know uh then and then also wikis tend to rise in decline right the these overall trajectories um like the sort of like idea of a life cycle seems pretty consistent um rejected newcomers are less likely to survive newcomers become more likely to be rejected as the wiki's age they become less likely to survive as wiki's age and then norms tend to classify uh and and that means that they become more difficult to change as wiki's age and become more difficult for or and are more difficult for less experienced editors to change uh and in general I think that the the main takeaway of this study is that the rat narrative isn't just a story about wikipedia um it's a it's a story about the about like these other wikis that they're in my sample um and I think that it's this also points to like a broader idea that you know even when groups try to be open and try to have um you know collaboration without hierarchy or without a lot of um like official rules or power structures that you know culture develops norms develop and they and they tend to become um a barrier to the participation of people who are outsiders right the big the maybe they create like a barrier that people have to overcome before they can come active participants um and I think that's something that we you know still need to think about and have to wrestle with as we like you know um think about like how wikipedia inspires us to think about you know the possibilities of collaboration producing public goods but it also you know reminds us of the the challenges that arise and that there's this tension between quality control and um sustaining an active body of contributors and that's it awesome thank you Nate um so there's been a lively discussion of this on IRC um only a couple of the things from that discussion have have yet been framed as official questions or comments for you um so I'm going to start with start with those and I'm going to uh insert one of my own as well um because I'm curious what you think about it so first of all um uh Aaron Haffaker notes that um when we're looking at survival rates over time um we were actually looking just at golden and good faith editors um so we were actually excluding vandals and non-serious editors and we used a content analysis approach for that yeah um did you in your sort of survival uh analysis did you also um exclude vandals or was this overall survival this is overall survival we weren't we like I I brought this up in the limitations but we didn't try to exclude vandals um that was I mean I think when I tried to like identify is this a vandal or not on some of the wikis that I looked at I actually had a hard time uh but even the ones that were in English because it would have they would have like you know like there's there's wikis in here that are about like role play and about um and about like collaborative fiction writing and I when I'm not I didn't feel like I was like a qualified member of those communities I would have had to be a qualified member of those communities in order to be able to make those judgments effectively and I just it wasn't really tractable to to do that for 740 wikis right right um that transitions a little bit into my question so I'm going to ask it uh so one of the original uh there were a lot of different hypotheses um uh kind of arm chair hypotheses in a lot of cases proposed for the editor decline back when we were putting together this paper and some of them were exogenous and some of them were endogenous and one of the endogenous ones that I I thought actually had some resonance to it was that the low hanging fruit hypothesis basically wikipedia filled up to the extent that all the major topics that most people um wanted to you know would want to edit as newcomers uh had already been written about and so it was harder for people to find stuff to work on um and and I think that that's that raises kind of an interesting point about the difference between some of these wikia wikis in wikipedia because in in wikipedia that hypothesis is basically in my opinion kind of well it's it there actually wikipedia wasn't full up it was just full up of the things that a lot of the people who tended to join maybe wanted to write about yeah but I think it may have more of an absolute effect in some of these wikia wikis where if you take like a fandom wiki it it may really be true that everything in that particular fandom universe that needs to be covered is kind of covered at a certain point on the wiki so it's harder to say you know write about your favorite character and I wondered do you would it be interesting to look at the uh you know the the completeness of these wikis at particular times in their trajectories um and how that might correlate with new editor survival um yeah that would be uh that would be an interesting thing to look at um I think that there's a there's a few other of these like sort of well okay so um yeah that would be interesting to look at I think that you know of the wikis in this sample um a lot of them are fandom but a lot of them aren't right things that are like you know there's a lot of fan fiction and there's a lot of um like you know collaborative writing and there's humor and I think for those there's not a sense that I in my mind that they would be full right um but I think in the in the case of like wow wiki which is like one of the biggest wikis in the sample uh or like you know some of the some of the there's certainly gonna be um you know I think that the the idea of absolute completeness I think maybe does hold up a little bit better here but the there's always this sort of idea of like things can always be better as well right and so it may be more difficult for my idea is actually that the substance that is complete but that like newcomers will find it more difficult or harder to change content that's already there and so even if the content on even if like the page already exists and has something written on it then that they may not improve it even if it's not like complete yeah that makes sense um so a question from Tillman uh you mentioned wikipedia has been in decline ever since 2007 but the rats the rat rad graph ended in 2012 uh more than half a decade ago now has the trend continued since then yeah so you can go to um I think I should fairly you're there's like a stat there's like a stats wikipedia you guys know what you know I'm talking about uh yes that has that same plot basically like adds adds a point to that plot every month um and to go there you'll see that it does continue though it slows um it's like it actually is kind of like uh pretty classic like logarithmic curve um seems to be seems to be like so yeah basically it keeps declining uh the second derivative is positive cool so we have a couple more questions I think that are going to be asked via the room so I'll pass to Dario actually I want to check I'm not sure there's anyone here who has a question I do have one I want to check first uh I don't go forward with mine um yeah so um I I I do want to emphasize that like this question of a replication across languages and um um so corroborating the results with um addiction analysis outside of English Wikipedia uh there's still a lot of work that actually could be done to better understand uh you know whether well uh the rise of the client paper identified was a primarily uh the permanent result of like a of uh maturing communities with um quality control systems or if it applied also to other language communities and if it did um because of what because of uh attention spillover because of the application of these quality control tools across the board there are many many questions that I think uh could be addressed uh from a more multilingual perspective um in a in a future extension of this the one question that I wanted to ask you um is similar to uh the the issue John was bringing up around uh it's like the um uh the the gold rush hypothesis and an idea that there might be uh a few of your topics to work on I was actually curious if you guys have any evidence uh of uh notability policies on Wikipedia basically preventing the creation of uh fandom you know like uh articles related to more popular culture topics that resulted uh in a shift of attention and contributions from Wikipedia which for a while actually was a place where people uh at least like globally I don't have any specific um evidence to point at but I I think Wikipedia used to be a place where many um you know people excited about uh comic series tv series were uh were contributing I'm wondering if at some point as a result of uh policies or general like attitude towards uh contributors in these topics um uh there was a shift in attention and contributions from Wikipedia to some of these communities on Wikipedia so you're asking like if the if I understand the question so you're you're you're asking like if the um there's no room to the key uh like sort of grow because Wikipedia pushed them out and then we keep uh still basically like compete with Wikipedia over editors correct yeah so it's because there's a lot of changes in in policies around uh you know uh topics in popular culture tv series comics and whatnot mm-hmm now I don't really know the answer to that question uh I think it's an interesting question I mean I guess um yeah I didn't really look at changes on I didn't look at anything about Wikipedia for this project right so it could be it could be something to look at yeah and makers are saying if he has some programming analysis on this so that is a follow-up with him um yeah cool um thank you I also wanted to open up uh the discussion to any question about both presentations since we're uh technically the end of the hour so there are other questions also about um Nick's talk uh it's a good time to ask him and you'll see Nate if you get a chance to see the IRC log there's a huge discussion going on yeah I see um Nate I have a question about if you observed anything qualitatively to accompany the quantitative findings in the study about whether because I remember in the rise and decline paper I think there was a qualitative component kind of maybe I saw somewhere else verifying that that editors did in fact react negatively to these things that quantitatively appeared to be pushing them away did you detect any of that in Wikipedia whether there were alienated new editors um so I I mean I didn't do any like sort of systematic qualitative analysis for this project I you know like based on like spot checking and like seeing what's going on um I mean I can't say that I saw any of anything like that that was very clearly like that to me um but that doesn't mean it's not there I didn't look for it very hard um yeah no no questions are on IRC just to establish they're gonna check around here in the room because I know like I sure so yeah for Nate um I I know you said you didn't do a qualitative analysis but did you find that there were any um any wikis on wikia that didn't follow that trend that didn't have the drop off the oh yeah there are definitely wikis that don't follow the trend I mean I think they're yeah um I mean there's so much there's a huge amount of variation within wikis um or sorry between wikis uh there's just like I think that's another thing I barely could have emphasized that more um but that like the these are like this is sort of the average story about what happens but um I think that there are there's so much like variation between the different wikis that you can find like you can find a wiki that doesn't that doesn't decline during the period that I observed it for sure do you have any sense about like what kinds of qualities those communities might share like is it is it just sort of random or are there certain kinds of topics that maybe lend themselves to more sustained editing um I don't think I have I don't have any information about that um I think the I would say that I think that there I think one thing is that there will be like different communities while they're on time scales right so some will be like sort of flash communities that um you know form build the wiki really fast and then sort of dissolve right and then um there are other ones that you know I think they're the wikis that I would I my guess is the wikis that I have um not seen decline just haven't declined yet that they're like I think given a long enough period that I think I don't think that you'll find a lot of wikis that will never decline that's my own little like a little pessimistic story like belief but I don't really know and I'm looking around and there's no other question I want to thank again uh Nate and Nate for joining us today uh thanks for the fantastic discussion and presentations good luck with the official presentation next week at the conference and I guess I'll see you all um next month um on the 8th of May for the next showcase thanks everyone thanks everyone yeah I'm here for to see you all again soon someday yes