 Okay, we are live. Hello everybody. This is Jonathan Morgan for the Wikimedia research team and welcome to the July 2019 research showcase. We have two exciting presentations today. First up we will have a presentation on characterizing civility on Wikipedia by Elizabeth Whitaker. Elizabeth is a PhD candidate in the School of Information at the University of Michigan. And then following that up we will have Lou Show presenting on hidden gems in Wikipedia discussions, the Wikipedia's rationales. And Lou is an associate professor of information studies at Syracuse University. Handling IRC today will be research scientist Isaac Johnson. If you are participating in the IRC channel or are watching live on YouTube and you have questions, you can just post them to the YouTube comments section or to IRC and ping Isaac. And Isaac will put your question in a queue and then we can ask that question for you to the presenter after the presentation is done. I think that covers all of the logistics. So without further ado, I'm going to turn things over to Elizabeth Whitaker. Elizabeth, it's your show. Hi. Thank you all for having me. Again, I am Elizabeth Whitaker. I'm a PhD candidate at the University of Michigan. Most of my work is around online in civility, harassment, hate speech, just all very fun things. So I'm going to go ahead and get the presentation going. So before we get into the nature of in civility on Wikipedia, let's talk a little bit about why in civility might matter and let's talk about deliberation. So deliberation can be thought of as a problem-solving technique in a democratic or collaborative society. However, what exactly deliberation looks like is a little bit fuzzier. There are a number of different conceptualizations of deliberation and perhaps one of the most influential is Juergen Habermas. Habermas states that deliberation should involve the exchange of information and reasons, should be inclusive of all of those who might be affected by the decisions being discussed, and it should be free of external and internal coercion. So thinking about this definition, it's relatively clear that deliberation as a concept is intended to promote equal discussions and just governance. Sorry to interrupt. If you press F11 on your function keys, I think then we will see just the slides. Right now we see the slides and also the URL bar and bookmarks bar of your browser. Okay. F11. There we go. Perfect. Go ahead. So deliberation is intended to promote just governance. However, we live in a practical world and that may not always be the case and deliberation may sometimes be constrained, which is, of course, where civility comes in. Civility can be thought of as something that could facilitate deliberation, something that makes it possible for people to discuss issues and come to solutions or conclusions. If civility is able to facilitate deliberation, then incivility should be able to prevent deliberation. However, it's not just the presence of incivility, but also how we define and understand civil and uncivil speech that can either facilitate or hinder deliberation. Depending on how we define civil and uncivil speech, we might impact who is able to speak and what they are able to say. One of the clearest examples, not necessarily my favorite example of this is Pew Research Center put out a report last year stating that 80% of black Americans believe that social media helps highlight important issues that might otherwise get attention. Well, 80% of white Americans believe that social media discussions distract from the important issues. So this is one group of people saying that they are able to participate in the public sphere and get attention while another group of people are saying, well, this isn't necessarily what we want to be talking about right now. Maybe the speech should go elsewhere or be elsewhere or does not be important. So our definitions of civility and incivility not only impact the nature of the speech, but who is able to speak and what people are able to say. Now, much like deliberation, defining and operationalizing civility is difficult and incivility. We've seen a lot of work in computer science and machine learning around detecting incivility automatically. And of course, when you were doing this sort of automatic detection, you must have a very clear operationalized definition of the thing you are trying to detect. And because civility can be so wide in scope or incivility can be so wide in scope, people tend to narrow the scope of the problem in order to be able to just detect something consistently. We've seen some people narrowing the scope of the problem to hate speech. Some people will focus on personal insults. We see a lot of focus on harassment. One particular study used users being banned from a community is sort of a proxy for bad behavior. However, when we narrow the scope of what we're looking at, we run the risk of accidentally missing some kinds of interactions that are really important to users and that might prevent them from speaking or just make things very uncomfortable for them. So, the definition of incivility that I tend to go with is ZZ-Papacherese's 2004 definition, which states that incivility can be defined as behaviors that threaten democracy, deny people their personal freedoms, and stereotype social groups. So, you will notice that profanity, hate speech, insults, those aren't listed here, but it is things that might prevent people from participating in democracy. So, it's not necessarily unpleasant speech. It is speech that would prevent democracy or collaboration in the case of Wikipedia. And one thing about online incivility that is particularly important to understand is that it's not just language happening in a space, but the space is going to influence that as well. Online incivility is shaped by its online environment. When we're talking about online environments, we are, of course, talking about affordances. And one thing I would really like to highlight about affordances is an affordance is not only what you can do with the technology physically, but it is the routines that develop around it. So, if technical constraints initially constrain the technology and drive development so that X piece of technology is created and then certain social practices develop around that piece of technology, those social practices will shape the piece of technology that is developed next, which will then shape social routines and so on, sort of gradually building layers of technology and social processes into technology we use every day. One really lovely thing about Wikipedia is that it does require so much collaboration. When I've been talking about deliberation, that is very much what happens on Wikipedia. Because Wikipedia requires or has the Wikipedia page as the ultimate goal, the article is the goal of editing, you're not just in a space talking to people about things. You are working towards something and you have to make decisions and come to consensus and do these deliberative processes, which means that civility is very important for allowing that. And in civility potentially might disrupt that and what is considered civil and uncivil by editors and experienced in those ways may influence who edits what. But Wikipedia is absolutely collaborative. And when I talk about the sort of social norms and social routines being important to how technology is developed, I think it's relevant to point out that Wikipedia is sort of built around the norms that guide encyclopedia creation as well as the norms around the scientific community and free and open source software, which may be relevant towards the end of this talk or in Q&A, but not necessarily right now. So in doing this work, my research question was what interactions do Wikipedia editors perceive as disruptive of editing? In order to answer this question, I conducted semi-structured interviews which were consisted of cognitive walkthroughs. So I had participants bring me a page, a talk page for an article that they said contained civility and then one that contained in civility. And that was really about as much guidance as we gave them. So whatever they brought was what they brought. And then I had them walk me through what made the civil page civil and the uncivil page uncivil and what exactly that looked like. I recruited from a project dedicated to closing Wikipedia's gender gap and analyzed the data through a grounded theory method. We had 15 participants. Okay. So the results, what did we find? Well, we found roughly three categories of in civility on Wikipedia. These categories were policy weaponization, technology weaponization, and content based in civility. A brief reminder, although I'm sure none of you need it, Wikipedia's policies are the neutral point of view policy or NPOV policy which dictates that information included in the Wikipedia articles must be neutral, not biased in some way towards some particular point of view which shouldn't be putting forward any specific viewpoint. The verifiability policy just means that anything put into an article should come from a reliable source and the no original research policy prevents me from publishing my work directly to Wikipedia which is probably a good thing. So policy weaponization was by far the most frequently mentioned form of in civility. Most of our participants experienced this and it was also one of the more distressing forms of in civility they reported. What policy weaponization looks like is taking a policy and then sort of manipulating it in order to get the other person or in order to prevent the other person from editing. Participant 13 explained it relatively well. So if someone accuses you of violating a neutral point of view that immediately is a red flag for Wikipedia. And of course it's very difficult probably close to impossible in some ways to prove that you haven't done that. So you have to have diffs. You have to have examples of things that do represent what happened. Participant 13 felt that you just couldn't argue against an accusation of violating neutral point of view policy and that was sort of seen by most participants as a way of just immediately shutting down that discussion. Now interestingly one component of the policy weaponization that came up very, very frequently is that this is a gendered phenomenon. Participant 5 explains it really well and she said I'm applying the same guidelines that anybody else would to any article but because it's a woman wanting to retain women's biographies there becomes this extra layer of is she just doing it because she's a woman and she likes hearing women's stories out there? Is she this feminist who wants to have women's stories everywhere? So one thing I heard again and again from participants is that just being a woman then people will assume that you are a feminist and somehow feminism is a form of bias and so feminist points of view can't be included or shouldn't be included or suspect in some way because they are biased. We also saw the weaponization of Wikipedia's technology against editors. What happened here most frequently was Wikipedia is a fairly technical platform when you go to edit there's HTML markup if you want to insert links or do something with pictures, citations it takes a little bit of learning to figure out how to do that it's not necessarily intuitive so if one more experienced editor encounters another less experienced editor and more experienced one is able to sort of run rings around the other technically speaking. I'm just going to sort of read the end of this participant for was just saying that it's difficult and people who are more experienced are not so competent with language and technical language they struggle they may have a good point but they have difficulty expressing and the other people tend to run rings around them because of their lack of technical competence. So we not only saw that on the pages like the article pages with the markup when you go to edit but we also saw with the edit summaries people would hide nasty little comments in the edit summaries and it was then much harder for the admins to go in and figure out what was going on and arbitrate disagreements so evading detection and evading censure was a large part of what was happening with the technical weaponization. Finally we have content-based instability this is sort of the more traditional form of instability we can think of this as profanity, insults, hate speech, name calling that sort of thing. There wasn't too much new with the content-based instability but it's still there. So in conclusion what we found were that content-based instability was not the only form of instability present on Wikipedia, policy weaponization and technology weaponization were also present and policy weaponization was by far the more commonly reported form of instability and so if we think back to how we are defining or conceptualizing instability if we were looking at only content-based instability we would miss a lot of what's going on and a lot of what our editors reported was driving them away from editing if we were only to look at name calling, hate speech that sort of thing. It's important to consider the affordances of the platform and affordances do mean technical features and the social routines developed around them in order to understand instability on the platform. This is not simply a language problem there are social and technical things happening that need to be understood as well. This is the end of my presentation. Thank you. I believe questions now. All right. Thank you, Elizabeth. So first, I'll throw it to Isaac. Isaac, do we have any questions from IRC? Yep, we've got one from IRC from HACC asking if you're going to try to detect policy weaponization computationally what kind of markers would you suggest looking for and thank you for the presentation. So if you were going to try to do this computationally I would suggest either links to the policies or the names of the policies in parentheses themselves what you typically see is somebody will cite the particular policy and sometimes they will link to it. That's still going to get you some things that's not entirely perfect because a lot of the time there is policy linking that is really productive and somebody is getting off topic and you need to rein them in it's going to take a little bit of finagling I think. Thanks. Any other questions from the room? None from IRC and none from YouTube right now. Thanks a lot. So following up on Erin Haffaker's question I wanted to know are there particular policies that you noted were used kind of seem to be used disproportionately weaponized what are those weaponizable policies in the context of the kind of incivility and harassment that you were looking at? By far the neutral point of view policy verifiability was sort of the second runner up in that people would start questioning sources if they were to social justice but it was by far neutral point of view. Interesting. That actually so Isaac jump in if there are other questions that are coming in Eric go for it. Sorry about that. So yeah I'm curious you know thinking about past work in trying to model harassment I jumped on that question about trying to model policy you know weaponization because I thought that that was that was something that stuck out in your talk as something that would be hard to model using like techniques that have been used in the past that are looking at hate speech and that sort of stuff I wonder if there's anything else that you found in your work that you think might be a productive direction to either explore you know through modeling or maybe through interface changes or something like that that might make it easier to detect that kind of in civil behavior. Hmm interface changes so in terms of detection there's nothing else that's particularly jumping out it's a lot of I think what I'm focused on right now is more how do we deal with it how do you I think the neutral point of view policy it makes sense from an encyclopedia perspective but at the same time how the community understands neutral is sort of a normative thing I also know in the past there have been pushes to make Wikipedia's interface less technically challenging and those have been pushed back against by more technically adept users so you can do a lot with modeling and automatic detection I don't think it's going to get us all the way there. Patrick go for it. Just want to thank you again Elizabeth it's really nice to start breaking civility into pieces I think that like harassment it's one of those problems that you know folks when you look at the broad label folks say well it's just too complicated to figure out or it's just too complex and so I really appreciate you taking the folks you talk to in terms of remedies or outcomes you know like how they dealt with the situation when they were in a situation that they found someone was being civil did they have any feedback on that you know what they did after they experienced it so the most common response to experiencing in civility was taking an editing break stepping away from the page moving away some people would just abandon the page entirely I think one person took like a two month break from Wikipedia right and often that's the advice given by you know people in online communities is oh you're having a bad experience maybe you should just take a break right that's not terribly helpful okay thank you we've got one more from IRC if I can get that in Jonathan go for it Isaac great so the question is how would you distinguish between weaponization of policies by marginalized groups in order to try to reclaim space for them to function in this case to edit participate for always members in the community versus weaponization by the dominant groups that's a very interesting question and thank you to whoever brought that up unfortunately I don't have a good answer for you we didn't see a lot of the policy weaponization by marginalized groups typically the responsive marginalized groups was to just not be on Wikipedia or if they were on Wikipedia it was centralized organizing on like wiki project pages yeah they tend to use different strategies I didn't see so much of the policy weaponization there it was more I'm having a problem with this person can somebody get an admin can I maybe get some help of course that does border on the dog piling and calling and friends and that's never good and if we are going to do this computationally those things are going to look very similar and that's part of the problem I think with some of the computational approaches is there are power structures and power differences going on that a computer is just not going to get necessarily I don't know what the answer for that is sorry no thank you I have a question I know this wasn't the purpose of your research but I wanted if you if you learn anything about the process did you especially in the context of policy weaponization that form of incivility or harassment did you get a sense of what the goals or the motivations or the intense were of the people who were who were performing these behaviors what were they trying to accomplish it was very much yeah this wasn't something we explored so I can only sort of comment on it from what I saw during the cognitive walkthroughs looking at it and my impressions gathered there it looked mostly like they were protecting turf usually that page was maybe important to them they had done editing on it or they just didn't want this particular phrasing perspective something in there it seemed very possessive and territorial to me just this is mine go elsewhere I have claimed this and I know there's a little bit of a problem with people like squatting on pages and doing that sort of thing so I think that fits in relatively well with what might be going on there that makes sense any other questions go for it yeah thank you for that that's really interesting I have a different question but I just wanted to say in my work looking at what predicted somebody getting reverted removing some content that somebody who's around just added regardless of your experience level and your past quality level and all that sort of stuff is one of the most strong predictors of getting reverted so yeah this ownership stuff definitely shows up in the data too so my actual question though is about future work you know I was struck by how excited Patrick was about like splitting up like the components of civility and in civility and it sounds like like a lot of what you've done with your work is splitting out some specific types that are they're hard to understand without talking to people and exploring their experiences and looking at examples with like real eyeballs and real empathy you know I wonder what do you see the next step you know it's the next step more along the lines of turning this sort of qualitative understanding to quantitative understanding or is there more to explore and more to break out or something else entirely so I think right now this is really early work and break components of civility on one platform and I don't think it's going to look the same across every platform I think especially if we're talking about the technical features of a platform like Facebook it does not make me deal with markup language to post a picture of my dog it's a very different space so I think as researchers we need to have a much more fine grained approach to platforms and understanding not just like the words but also the norms I think we need to go to sort of an approach maybe a little on the sociology side with looking at norms and governance and sort of I know there's been a turn towards communities and Facebook groups and sort of smaller closed environments and sort of ground up moderation which is actually what Wikipedia has so like clearly there is no one size fits all solution I think it's going to be a lot of experimenting with policy changes technical changes hopefully getting something with norms although that's much harder to change on these platforms and I think it's going to be difficult and I think we're going to see resistance to it and I don't know what the next concrete steps look like necessarily because I think they are going to be very deeply informed by whatever work you do at this stage if you don't know exactly what the problem is you can't really do anything about it sorry that's not the most optimistic answer well sometimes isn't the most optimistic well thank you very much I'm assuming you don't have any other questions if we do we can I'm sure we can take it we're good on both channels excellent thank you once again we look forward to seeing future work in this area so it's wonderful to see this work presented here I think now I will turn it over to Professor Liu Shou from Syracuse University Liu take it away tell us about article for deletion debates you've done a great deal of work in this area if I'm not mistaken okay thank you Jonathan so let me first share my screen perfect okay okay thank you first of all I want to thank the organizers for giving me this opportunity to share with you some of my research projects related to the Wikipedia context and specifically today I would like to talk about several projects that we have done looking at the Wikipedia's rationales in Wikipedia discussions and we define rationales or the Wikipedia's rationales as Wikipedia's justifications or their own justifications of their views votes their suggestions in the discussions why am I interested in doing this actually I've been working on projects related to rationales and the effects of rationales in other context as well such as in virtual group work in crowd sourcing online crowd sourcing collective activities from those work we have found that rationales provided by others they can make an impact on the individual and on the collective activity for example being aware of other people's rationales can contribute to one's awareness of their intellectual contribution their domain expertise can actually influence this individual's reflective thinking process can also help monitor and control the group work's quality so we've observed advantages and certain also some downside which I will talk about later in the Wikipedia context in terms of providing the rationales and sharing the rationales in group activities in online collective activities so then I'm very interested in studying this rationales and specifically the role of rationales in this particular Wikipedia discussions the article for deletion discussions so as we know that if a Wikipedia article is proposed to be deleted there are several deletion mechanisms one of them is through a discussion so in this discussion there is a period of time when any Wikipedia can participate offer their opinions their views on what to do with this article to delete it or to keep it or merge with existing articles or other suggestions and at the end of the discussion the decision regarding this article needs to be made based on the rationales that people shared in the discussion according to the policy as opposed to just a simple count of the majority vote and so this is the context that we mainly focused on in studying the rationales in Wikipedia discussions also because Wikipedia requires that the participant in the after discussions they need to provide not only their view and their rationales so to understand how participants' rationales play a role in those discussions we look at several aspects so we conducted a questionnaire study to probe those perceived effects by the participants and then we analyzed the rationales we wanted to see what type of information is included in the content of the rationale we also examined the sentiment of the rationale and aggregated to the sentiment of the discussion I wanted to see how that correlates with the discussion outcome then we also explored the language used in the rationales to identify some potential indicators that might indicate some specific aspects that can be useful in the discussion so in this talk I will go over these projects very briefly given the time and for those of you who will will be at Wikipedia in August I hope to see you there and I will give a more extended version of the talk at the research track and so the questionnaire study we wanted to see what participants think about those rationales shared in the discussion we disseminated our questionnaire through four mailing lists, Wiki Research L Wiki Media L, Wiki NL and Wikipedia L in about a month we collected 41 valid responses as you can see here that we also observed the gender gap and the analysis of the questionnaire data shows that most respondents they have been a Wikipedia editor for over five years and only one person had less than a year of experience and talking about the benefits they considered in terms of seeing other people's rationales they acknowledged that this helps people better understand other people's votes help people control the quality of the discussion outcome and also help educate participants about Wikipedia's policies some respondents also made a note of some of the drawbacks of sharing the rationales in AFD discussions for example you can make it more challenging to propose different opinions and sometimes there is a misuse of the policy or processes and then this type of debate can be quite lengthy and wasting time in some cases the respondents will also ask to provide three types of most inferential rationales in their judgment and these three types of rationales are the rationales that are about the article's notability the rationales that cite Wikipedia's policies and those rationales that are from established Wikipedia editors so you will see that in our later project some of the results we observe here are consistent with our later studies so then we the participants view on the effects of rationales we decided to also analyze the content of the rationales to do that we first conducted the traditional content analysis study so we collected the three days of AFD discussions and we manually annotated those rationales through an open and iterative coding process which I won't be able to go details on our data analysis processes but these are detailed in our paper and this is a table that is obtained from the paper that shows our coding schema through the open coding process so as we can see here that we have we have several coding families some of them also have subcodes so for example for a rationale that we coded as it talks something about notability we further coded that there might be different types of situations that related to how people talk about notability for example some people mentioned about Google hits mentioned about coverage in media as a way to explain whether this article is notable or not notable and certainly the content of the article and some other aspects so through this human annotation through this coding analysis we found that most participants indeed offer their rationales even when they just want to express their agreement with other people in some cases and they followed the Wikipedia policy on voting and also many votes in our analysis they were coded with multiple rationales which means that when a participant offers their view to delete this article there might be several aspects they have considered different multiple rationales included in that argument and there is a good portion of rationales that refer to Wikipedia policies or refer to the topics and notability is the most frequent rationale mentioned in the data and it far outweighs the second most frequent rationale which is about credibility this is also consistent with the literature something we find very interesting is the participants not only provide their views and their arguments the rationales but they also take actions to help improve the articles and in some cases they really want to help to make the articles acceptable for the community which we find interesting and also through this content analysis study we realize that it is impractical to use this human annotation to analyze the content of the rationales because there are just so many AFD discussions out there so we started to explore the possibility of using computational and visualization approaches to help analyze the content of the rationales to analyze at a much larger scale and we developed open dataset for this purpose this dataset contains two years of the discussion data and with the purpose of making it ready for later computational content analysis we removed about 5% of dataset that caused the difficulties for automatic processing especially that those difficulties happened when we were trying to parse the HTML content to extract the parts so we have about 39,000 discussions and we created SQL database to organize the discussion content to provide the structure to provide additional information in the database such as the total number of comments in the discussion and we manually looked at the category the category of the article based on the hierarchical structure about the categories in the Wikipedia and we did the same for the policies that are mentioned in those rationales and provided those hierarchical structure of the policy in the database then based on this dataset we developed several interactive visualizations so for example on the left side we have this some burst visualization that shows the policies mentioned in the participants rationales and this is interactive visualizations if you go to this URL you can actually interact with it it shows through the color coding it shows different policies that are mentioned in the rationales also shows for those rationales that mention a specific policy what's the percentage of votes for deleting the article and the percentage of the votes for keeping the article more for other suggestions on the right hand side we have this visualization that try to map the articles categories and the policies that mentioned in the rationales of those articles so here you can see that we have music highlighted here and it shows the percentage of different policies that are mentioned in those discussions that are about articles in the category of music so our purpose of offering those visualizations is because from previous studies it has been shown that new editors, newcomers they may have difficulties of being getting familiar with Wikipedia's policies so when they added the new articles or when they participate in the discussions it can be quite challenging for them so we hope by providing but we hope that by providing those visualizations it may help the newcomers to get some understanding of what are the kind of policies that tend to be mentioned in certain types of articles and so next we look at the sentiment of the discussion and see whether that sentiment of the discussion correlates with the outcome of the discussion we use the same data set and we use Vader a sentiment analysis tool that gives one of the three labels positive sentiment, negative sentiment, neutral sentiment to a given short input text so we we obtained the sentiment for each rationale and then we aggregated the sentiment chose the majority sentiment as the sentiment for that discussion and yes you can see from this table that how those discussions are distributed over the different labeled sentiments and also it shows the different discussion outcomes so number of delete discussions refer to the total number of discussions that have the delete outcome in the data set we conducted a high school test to explore whether the sentiment of the discussion has some correlation with the outcome of the discussion and we found that statistically there is some significant observation that as expected discussions that had a delete outcome or that had a keep outcome they tend to have more negative or more positive sentiment but what's very interesting to us is the neutral sentiment so for the for the discussions that had a delete outcome they have less than they have more than expected a neutral sentiment and it was the opposite case for the discussions that had keep outcome so what this indicates to us is we consider that when participants they prefer to delete the article and the article ended up being deleted even though there is some even though there is a kind of negative aspect to it like in terms of deleting the article but their rationale is tends to be more neutral stays more neutral and when the discussion is more about keeping the article really hope to keep it in the community they actually are more intended to use the positive sentiment there is less neutral sentiment and then we look at the language use of the rationale first we look at the imperative use imperative statements are the statements that basically the speaker or the writer makes a request will ask the listener or the reader to do something this is like a command and we in our preliminary analysis we notice that participants tend to use imperatives in this discussion context so we were curious what type of requests people made in the discussion and whether this might be useful for newcomers and so we analyzed about one week of Wikipedia FD discussions for each month of 2013 in total we have about 4,600 discussions and that is not practical for us to manually analyze them so we used our imperative extraction tool to first extract those imperative statements then we further analyze those imperative statements and kept about 1200 statements for further analysis and we applied open and iterative coding processes and we identified 5 types of requests of those requests we find that actually close to one third of the requests or the commands or the suggestions are about telling people how to evaluate the articles so we consider this might be useful for newcomers who really need this kind of advice they also imperatives also talk about discussion for norms, practices like what should be done or shouldn't be done in the AFD discussion context and also there's a good percentage of imperatives that made references and made reference to Wikipedia policies like suggesting other people to refer to this policy or to read this policy etc so we found this findings to be positive in the sense that it can be useful for newcomers if we can extract those information and somehow present it in a way to help newcomers and the last aspect in the language use of the rationale was we were curious about what type of linguistic features are there that make some rationales seems to be more powerful than the others and specifically we were interested in those rationales that actually justify the view that ended up being the same as the discussion outcome versus those rationales that has that advocates the view that's the opposite of the discussion outcome so with the same data set from our open data we identified those persuasive comments and non-persuasive comments we then used a language analysis tool called linguistic inquiry and word count analysis tool which calculates the percentage of different language categories in the input text and we found that the persuasive rationales we wanted to use more punctuation marks they use negative emotion words they may have those words that represent higher power and risk but there is no difference in terms of the length of the comment between the two groups and so this is a very quick overview of the projects we have done in the AFD discussions and one of our future work includes understanding the participants aspect in this process for example whether gender makes a difference in terms of types of rationales people provide what about the status of the participants and also the temporal aspect in the discussion how the rationales develops over time so if the whole time period whether the rationales shift across different discussions and how the topics of the articles make a difference in those rationales or the strategies of providing rationales and that's the end of my presentation thank you great thank you very much Lu okay do I can keep your slides up or you can switch back to your face whichever you prefer sure I guess I can stop it excellent so first Isaac do we have any questions on IRC? none for Lu from IRC or you too I have a question if there is anyone else in the room with one though well let's you go first I don't know if you have any early insights on the relationship between topic and the discussions that you mentioned at the end I'd be curious to hear them so the topics and the discussion outcome I guess yes so actually in our in our early study looking at the content of the three days of AFD discussions in that study we also analyzed the types of articles in terms of the categories and we did analysis to see whether there is like certain types of articles may tend to be deleted or tend to be kept we had some finding there so like for like for people's biographies they are more likely to be deleted and for events or locations they are more likely to be kept like when they are proposed to be deleted and in our later data set the bigger data set that we developed we have visualization we have a visualization that also shows the categories of the articles in our data set and also the percentage of those deletion delete votes or keep votes and we in our paper we also suggest that for certain categories that seems to be more delete than other like I think now I need to go back to my slide because I think I actually had a slide about it the three categories that have the highest percentage for deleting are martial arts football and sports people and the three categories that tend to have the higher percentage of let's keeping them the keep comments they are history crime and architectures great thanks okay I've gotten the word that I can jump in next so thank you for this presentation I think that you know what you've discussed is one of the most thorough analysis of AFD discussions that I've seen I'm curious though if you can reflect on what you've learned about policy usage and what we heard from Elizabeth specifically I'm wondering if you've seen any clear sort of power dynamics at play with policy citation in these deletion discussions that's actually what I was thinking when I was listening to Elizabeth's presentation and at this point we have not really gone that far to tie this two aspects together so what we have done is to look at the like the percentage of those policies like we can say which policies seem to be mentioned more than the other policies and which policies seem to be more relevant to a particular category so I'm sure that in the wikipedia structure that is presented there now through our visualization we hope to reflect whether this is the case in the AFD discussion I think the reason I mentioned that I was thinking about that too I think the next one next step could actually be further analyzing the content in the way that identify those incivil rational or those negative or the one center that Elizabeth mentioned and then see what type of policies tend to be mentioned more and make a comparison that task I think will be somewhat limited by I think Elizabeth also mentioned limited by the performance of automatic detection because given the amount of data we have it will be less likely that we can just process manually yeah thank you so any other questions from the room or IFC if not I'll jump in with mine so at the beginning of your presentation you mentioned that one of the factors that when you talk to wikipedia editors one of the factors they highlighted as being most persuasive about a particular rationale was actually a factor that didn't have anything to do with the rationale itself that had to do with the authority of the person giving the rationale so that reminded me of some I think related work from last year study led by Jane and Amy Zhang looking at requests for comments where and then they actually built a it was a mixed method study but they culminated in building a model that would try to predict whether an RFC would be closed keep or delete and Elizabeth is excitedly nodding her head when I say this you might actually move the details of this study better than I do despite the fact that I was essentially involved but I think that one of the things they found is that one of the strongest predictors for which direction an RFC debate would be closed or even whether it would be closed at all were predictors related to the level of experience of the people involved on the different sides of the debate basically if you had more experience for expedience on the support side then support had a much higher probability I wonder if you what you think about that and how and whether you intend to look at features like that in the context of these AFD debates or I might just open the question up and say if you were to look at those kind of features in the context of your data set what would you expect to find? Right that's a very good point and that is also somewhat relevant to our later analysis of the language use of the persuasive versus non persuasive one so in the persuasive ones they tend to use more knowledge related to the power and one next step, one next thing which I put in the future work was looking at the participants that I had the gender and status I think it would be very interesting to examine whether the higher status or the established status the way they provide the rationales is somewhat different from the new editors which I think well from what we see from the language use that perhaps with more established status they will use more powerful language because they feel more comfortable at the list in the environment and more there might be there and also first to see the types of information or the type or the perspective that they offer in the rationale I think would be interesting to see that and hopefully that help us to better understand that going back to your original question that is it really just you know I see that I know this person is higher status so I'm going to say this is influential or is it also because that this person is higher status and this person is better at elaborating and knowing better about the norms and other things better at framing which I hope is the second case rather than just checking the status right awesome Elizabeth did you want to jump in with a question or comment at all I wanted to give you that opportunity I was just nodding very vigorously Jane in as a friend of mine she's here at the University of Michigan I'm just happy to see her get a shout out so I I guess I had one general question this is kind of oddball and this could be this could be for kind of both of you so I've been listening a lot to very wonky political podcasts recently and one of the there's a lot of discussion about whether whether rhetoric how why rhetoric is persuasive right is it persuasive because of the content because the reasons the rationale the logic or the evidence provided are themselves persuasive or is it persuasive because because of something else in the way it's being presented and one of the things that people in the wonky podcast for a listen to talk about is social signaling so particular you know making arguments that that signal that you are a part of a particular identity group and so people pick up on the fact that you are maybe in Wikipedia contest context this could be you know picking policies or using words that are associated with like deletionism which is kind of one of the one of the overarching philosophies of Wikipedia editing versus arguments that are more associated with inclusionism and those being kind of too salient identity groups on Wikipedia how are those did you see since you're both looking at people making arguments about what should stay what should go did you see these kinds of appeals to identity groups as being a strategy that people sometimes used kind of maybe alongside their rational arguments but maybe even maybe even ending up being a more important part of the argument than the reasons behind it itself it's a very long and leading question but I'll just throw it at you and see if you have anything to say I guess I can say first then so I think it's very interesting you mentioned about this strategy as you're talking about the strategies I was thinking of Chowdini's six influential principles and to me he was talking about the six basic aspects or basic principles that people can use as a way to influence other people contextually not necessary content wise and so I think the social signaling in the specific AFD discussions were if I mention policies then I'm kind of signal that I'm credible in a way I think this can be a well this certainly can be a strategy that people use to persuade other people and I think that's also one explanation that in our questionnaire study that some respondents were complaining that sometime there is a misuse of policies and processes and so it's a waste of time and so forth so I guess in those misuse of policies some of it might be because the editors are merely using the policies not necessarily knowing that this is the right one it's marking it and so in the Chowdini's principle I was thinking which one might be a good one to explain this phenomenon I was thinking maybe liking because I think to use this type of strategy it's almost like you need to have an environment that this type of behavior is widely accepted, is highly respected or is highly regarded and so I want to be like them so then I will make myself more persuasive or more influential, more powerful but there could be other related it could be other related reasoning to use this strategy to me this is like I want to be like be one of those that's exactly the kind of lines I was thinking Elizabeth do you have any final thoughts on this I do actually I'm listening to you talk about the identity sort of grouping and talking about that I am immediately thinking about affect and not necessarily like rational thought and not even necessarily like breaking down into groups and then using that as signals but there's some sort of affective thing happening here I'm not entirely sure I understand it but I think it's there I also one thing that is occurring to me now is my participants when they were talking about experiencing like the policy related I think they are the one who is linking to the policy if an administrator comes along and looks at that and tries to like moderate that the fact that you have posted the policy that's a very strong signal of like belonging to Wikipedia culture and I think that in some ways can like play a part as a cue of identity to the admins on behalf of whoever is posting the policy links so I think that's an interesting way that can work too yeah on the rare occasion I try to get involved in any kind of talk page debate on Wikipedia I've noticed myself using a lot of jargon maybe and maybe that's a not particularly sophisticated but an attempt to kind of signal that yeah I know this stuff I don't have much of an edit count but I still know my way around policy if I can talk about COI and NPOV and things like that so at least one person probably does this on Wikipedia and if one person probably does it then it's probably happening elsewhere too any final questions or comments from anybody there was a question from YouTube for Elizabeth that I can get in let's do it yeah there was a request if they had any insight into responses other than walking away perhaps by the experienced editors that can be used as a tool for learning to deal with disruption so what my participants reported if they didn't walk away they just sort of kept going and ignoring the thing if you can push through whatever unpleasant thing then that's fine I'm just going to caution you about burnout I've also seen people calling it administrators calling in friends or not necessarily walking away from Wikipedia as an edit space but going to somewhere a little bit more welcoming sort of learning your way around Wikipedia a little bit not ideal but it's something okay alright well I think that does it for us this month this has been the July Wikimedia Research Showcase quickly I want to first let everyone know who's still watching that there will not be an August Research Showcase we're just kind of going to kind of let Wikimedia take on that burden for a month so we will next see you in September and then finally I wanted to thank Jana Layton, Cameron Ross, Isaac Johnson and our speakers today Lou Scho and Elizabeth Whitaker and that's it have a great afternoon evening or night everybody thanks everyone thank you everyone bye thank you very much both of you this was wonderful thank you for having us