 Mania, it's exciting to see all of you join. We'll give a few moments for everyone to teleport into this room before we begin our session on theory fact. If you are joining us right now, tell us something about what you most enjoyed from the last session you were in. We're seeing a number of people hop in and join the session. Some we've seen in other sessions, some we've seen at other Wikimanias, and it's great to see you all, whether it is morning or evening, or whether you're joining us from some other planet and we don't even know how to think about the time zone. Welcome. Okay, well, let's get started on this session. This is theory fact or fake, a social for global Wikipedia research. Our goal today is to learn the things we don't know, see if the things we think we know are supported by the evidence, and most of all have a good time and meet others connected to the movement. Today's trivia is hosted by the citizens and technology lab at Cornell, who work alongside communities to create evidence for a world where digital power is guided by evidence and accountable to the public. We have myself. I'm Nathan Beteas, a professor of communication at Cornell University, who's been editing and participating in various aspects of the movement for quite a few years now. And also, Sarah Gilbert, if you could wave at us and say hello. Hi, yes, I am Sarah Gilbert. I am cat labs research manager. This is my first for communion. I also mostly just a reader of Wikipedia. So I'm going to pass it off to a parajita. Hello everyone, I'm a parajita and a PhD candidate at Cornell University and this is also my first with me and you. So at cat lab, we work together with communities to create knowledge that's practically useful to communities, contributes to wider debates and tech policy, and also advanced both computer science and the social sciences. And we do that by producing research that involves community organizing at events like the research summit. We held at Wikipedia and Stockholm in 2019 and by actually creating software that makes community research possible. We've seen some of our work over the last few years along many different language Wikipedia's to test effectiveness of things like thanks, evaluating outreach campaigns and also testing ideas for the retention of newcomers across multiple Wikipedia's. We're also getting started on a new round of research with a continued focus on non US non English context. So look out for announcements in the next few months, but you didn't come here to learn about a research lab. You came here to learn about yourselves, each other, and about what is known and importantly what is not known about Wikipedia movement and how to understand and improve it. And on that line, I'm going to hand it over to Sarah. Thanks Nathan. So thank you all for joining us for our wiki or our wiki trivia game this morning. So we are playing in a fashion that is similar to the sort of trivia style called two truths in a lie, except we're sort of substituting that and really reframing it as two knowns and an unknown. So what we're going to do is we are going to send everybody off and to break out realms, ideally around five people or so per room. And we will send you a form which we have like sites here we will also like it in the chat so that you don't have to type out the URL there. But that will have some introduction questions for you to ask each other so you can get to each other when you'll tell each other your name where you're located and we've got a sort of an icebreaker question. If someone visited your region, what food would you serve them. And so you spent about five minutes doing that. And then you can spend the rest of the time or however you'd like in the in the room playing the game. So we'll have about 25 minutes for that. So we will send you that Google form or show you a series of three statements so 10, 10 series of three statements. And so two statements about Wikipedia will be established through research. And then one statement about Wikipedia that research has not yet uncovered. And so your job will be to identify that which of those three statements is that one about Wikipedia that research is not yet uncovered. So just to note that if somebody is in your group is vision impaired. It might be more fun rather than having them rely on conversations and also a screen reader for somebody to read it out out loud. So that everybody can can sort of play together without having to navigate a couple of different couple of different attentions. So we have about 25 minutes to guess the unknowns and then we will come back together to review the answer. There is no, there is no winner competition, but we are at the end, planning on writing a blog post. So we'll share some of the analytics or the analysis to see where the funds are like what everybody knows or what everybody doesn't know. And then we'll come back with a community afterwards. So there's also a link or a question or space in the form so that we can share the blog post with you. Also, if you notice that we've made any mistakes, you can feel free to correct us also in the form as well. So first we've got an example just so you can see what we what we mean. So there are three statements here. The first receiving thanks from editors on multiple language Wikipedia's leads volunteers to express more thanks to other Wikipedia's as well. So Wikipedia contributors, more time, more time mentoring newcomers across multiple language Wikipedia's and the third adding an image to a Wikipedia biography will increase the views that the article receives. Does anybody if you want to know which one is the unknown you can drop that in the chat we won't spend too too much time because I know we want to get to the game. But Nathan, do you want to advance the side to the bottom one is the unknown. Well, we do know that there are efforts to increase visibility of biographies particularly biographies of women by by adding photos there's actually no evidence like solid research scientific evidence yet that demonstrates whether or not that that that relationship between a photo like a photo and a biography actually results in more views even though it seems like something that makes sense. Nathan do you want to advance the slide. I like I like the the the conversations in the chat yeah hopefully it'll be really challenging and you'll learn a lot about some of the research that's that that's out there and that's not will also be providing citations. I'm not sure how, but hopefully we can make the slides available to you after we'll put them online so that you can consult all of the references that we have at the end as well. So I am going to just drop the link in the chat if you do not have access to Google forms. We also have a PDF. Oh I see Nathan has there's that's a that's a very awkward link there. And then, and then Nathan thank you Nathan has dropped both in there as well. And so, she is going to put everybody into breakout rooms now. And if anybody has any questions, we're going to stay out here but if anyone has any questions, there should be a, there should be a way to call us in for help and then we can kind of pop in and and clarify anything if needed. You can also feel free to pop into sort of the main room and ask us questions and then rejoin your room as well. I was just wondering, Sarah, or even the Cal. So there's a lot of people who are not assigned to a room right now that are joining but I'm having trouble. I'm having trouble creating a new room without closing the ones that already exist. I hear you. Just one second. Let's figure out what to do. There's 17 unassigned participants. I understand that you're assigning them manually. Yes, but I was thinking it might be easier to just create a new room. Yes, you can also do that. You are assigning each participant to a breakout room, so they cannot join by themselves, the breakout room, so you need to assign them. Everyone who's currently unassigned, yes. Correct. For those of us who are just joining us now, we are organizing a trivia and you will be assigned to a break room where you will have the opportunity to look at a list of knowns and unknowns and debate and discuss with your fellow room members what questions, what statements you believe are known on the basis of research and what statements are unknown. So we are assigning people to rooms. Thank you for your patience and hopefully you will have the opportunity to get to know your fellow team members and come back and see what the answers are. Welcome. Sarah, I was delighted to see that some people were able to guess the two or at least research supported statements and we actually surprised people with which ones were known on the basis of the evidence. So I think it's going to be a fun challenge for everyone. And also some people from the breakout room are asking for the questions. Yeah, we can also thinking include them in the main chat. I will share them and now. Also read up tip reduction tip. There are some breakout rooms that change the amount of participants that they're having so if there's one person alone in one breakout room, it would be good to reassign them. Okay, sounds good. Also monitor. I should broadcast that message to all. Exactly. Yes, if not, I can also do it. So Sarah, there are going to be people in this room on their way to sessions and on their way into rooms and so forth. So I wonder, Sarah, if you can. Perhaps we can talk about we could either talk about the challenge of creating this quiz, or we could try to go through some parts of the quiz in the room itself while we are here. Maybe we could go through some of the quiz questions while we're here. I can, I can open up. I don't want to necessarily change what I'm sharing on my screen. But what I'll do is I will just move my tab down to the PDF of the form and we can talk about some of the questions and be behind everyone so I don't think there will be any surprises for people. Okay, Sarah, we have this question. Which of the following statements about Wikipedia is unknown? We can reliably identify the gender of registered wiki users based on profile information. Contributions from tour users that slip through detection. So tour is a encryption software are similar to contributions from unregistered new editors. The reduction of mobility during COVID increased volume of people seeking information on Wikipedia. However, once mobility returned to normal, the volume also returned to normal. But the kind of information people looked for did not. Which one of these was most surprising to you, Sarah? I think like the two knowns were most surprising to me. That is that the contributions from tour users are similar in contributions to unregistered users. I don't have, because I'm not a wiki editor, I don't have it too good of a sort of a sense of like what contributions are like from different kinds of groups, but I have read that unregistered users tend to have lower quality and so I assumed that perhaps tour users were doing so in terms of tour perhaps. Seriously, which is I think why also they have the policy in that it does so it was I was actually pleasantly surprised that somebody out there had done this research and had looked at the the contributions from tour users and found that the ones that tend to slip through the cracks are similar to those from unregistered users. Of course, this was a this there are some challenges with a study like this. They are the ones that just slipped through the cracks. There's probably lots that the bot is actually getting who, you know, and for good reason, and for good purpose that they are using vandalizing Wikipedia. Thank you, Sarah. I'd like to interrupt briefly. I'm noticing that there is, there are two rooms, room number one, and remember six, but only have one active person in them. I am wondering if it would be a good thing to move people move people between rooms that are good. Thank you. Thank you. Sorry to interrupt you, Sarah. No, that's fine. The other one I thought was very interesting, the other known, which is the third option there, that the reduction of mobility during COVID increased the volume of people seeking information. But then once mobility returned to normal, the volume also returned to normal, but people's information seeking did not. I am a moderator of a subreddit on Reddit that is also very information seeking heavy. Lots of and we noticed that there was a change in volume on COVID as well. More people going online and asking questions about history. And so it was very interesting to read about how COVID changed people's information seeking on Wikipedia as well and how interesting it would be to do similar studies in different contexts. The unknown is the top one. And that was not so surprising to me because I know how difficult it is to identify gender information based on what people say about themselves or what people don't say about themselves. And that gender is often not just binary, that there are all kinds of folks who identify as transgender non binary or gender fluid, and that, that can be difficult for those to identify. Nathan, do you want to talk about the second one since I've been talking a lot. Sure. I had another quick question. Are, are you, I'm keeping track also, but are you keeping track of when we need to bring people back. When did we start with the session. When do we have until we have another 25 minutes. Okay, so we should bring people back. In about 10 minutes. Great. You have indeed 25 more minutes. We should, we should leave 15 minutes at the end. So let's bring people back at 615. Our time. Yes, our time. Great. So yeah, this next set of unknowns and notes. I was really fascinated to read the first study. So in all of these, we should have said all of these have a citation. So there's none of these are unsighted claims. I was fascinated by the first one here when I first read the paper about this question of whether requiring account registration helped improve the quality of the wiki or the quality of the wiki. It's a really interesting study because it looked at not just individual accounts, doesn't it, you know, are you able to get individuals who make high quality quality edits by requiring account registration. It looked at the overall quality of the wiki where you have fewer people if you require account registration. And that also has large scale implications and it was a very cool study because it looked at 136 wikis, not part of the wikipedia movement, but part of this site called wikia. So that was a really, really interesting study that I first read two years ago. I really enjoyed that one. Oh, I think we actually brought forward a question from our own research here as well. So I think it's a great inspiration with French, excuse me with German, Polish, Persian language, and also Arabic wikipedia, this one about wikipedia editors who report spending or who spend more time monitoring wikipedia for damaging content, feeling more emotionally and also more positive about their contributions. It was a really fascinating study and I hope that it inspires more research about the feelings of burnout that contributors sometimes have, but also how the things that maybe make editors feel most burned out are the most important things that they care most about doing. So I really hope that that inspires further research about supporting editors, especially people overwhelmed by all of the things that are involved in studying, excuse me, in actually editing wikipedia. I just moved to the next one. Sarah, what was most surprising to you about this set of questions? I was very happy to learn about the last one, which isn't one of the knowns. We did a project on the representation of music artists and how they are represented on wikipedia. And what they did is they looked at the most popular artists on Spotify, and then went to see if those popular artists were then represented on wikipedia. They looked at thousands and thousands of music artists. And even though female music artists tend to be underrepresented in the music industry, they actually found that they were overrepresented on wikipedia, which shows that, you know, some of the initiatives that groups have been taking or being taking to increase the representation of women and women's biographies on wikipedia seem to have been working. And to the point that now actually like male music artists are comparatively underrepresented. There are still more profiles of male artists, male music artists on wikipedia just because there are so many in the industry itself. But it does show that when you look at the representation in terms of like how many popular female music artists there are out there versus on wikipedia, that representation is actually really, really good. And so that was really exciting to see that some of the efforts that people are putting into into these projects and reducing gender gaps is working. So those are great, great study by the team at Spotify Research who study these things, not just on wikipedia, but also on their own platform. For those of you who are just joining, you are experiencing hopefully the most boring room in this activity. We have breakout groups all across this zoomscape who are discussing and debating the questions that are the statements that we have chosen between knowns and unknowns. If you're just joining us and there's still five more minutes to join a group, you have the opportunity to jump in, meet some other people at wikipedia and discuss and debate these questions. So welcome, and we look forward to coming back in just under five minutes to talk about what everyone guessed. Let's move to question four. I will admit that I was really encouraged and surprised to read the finding from the very first item here that over the last decade the coverage gap on wikipedia between Europe and Africa has reduced by roughly five times. This is a new result from the folks at Whose Knowledge and the Oxford Internet Institute that showed that efforts to broaden wikipedia's information about African knowledge and places have actually been fairly successful. And actually it's a helpful reminder that sometimes research becomes out of date because previous studies on this, of course, have found very different coverage gaps. And so it was very inspiring to read about that. There's so much more to be done to improve knowledge equity on wikipedia. That was the thing that I found most surprising and a real testament to the great work of so many wikipedia contributors all across the across Africa and diaspora communities who are contributing from around the world. I see a comment in the chat from Goodness, who has said that they are in a breakout room but that they are lost. Can one of us join the breakout room to help out? I can join the breakout room. It's possible that no one in the breakout room has received the briefing at the beginning. And because I was, we were talking through questions. Okay, I will join and I'll be back in two minutes. And then had control over the slides and I just realized that we can now not see some of the questions to go through that I can share my screen. Sarah, let me know when you want me to close the rooms as well. We have one more minute left. And then we will close the breakout rooms. Thank you for managing those share my screen there. So we can keep talking about some of the responses and some of the responses. So, that's except you can't see the whole thing. So for this one, we still can't see the whole thing. Yeah, I will stop sharing my screen then. So it's cutting off the questions for some reason. Okay, so I'm going to go ahead and close all the rooms. And so within a minute, everyone should be back. Okay. Because it's the way that I am viewing the questions, but I just can't read the whole whole statement for some reason. If you are returning from your breakout room. Welcome, welcome, welcome. You had a puzzling and interesting time. And I will continue to vamp until more people are here and everyone is back from the breakout rooms. Parajita, can you let us know when you see that everyone is back from the breakout rooms. And as we, we will read and go through the answers shortly. And as a reminder to our co organizers, and not everyone here is English first language speaker. So if we speak clearly and slowly, that will be most accessible for everyone. Welcome back. Oh, all the breakout rooms should be closed by now. So we should all be back here. Okay, Sarah, I believe you are setting us up for what is next. So the next part of the trivia session is that we are going to go through all of the questions that you just saw, and let you know which ones are the knowns, and which are the unknowns. So Nathan, do you want to start us off? Absolutely. So as we talk about each group, I ask you to post in the chat what you thought the unknown was. So is the unknown about gender, about information seeking during COVID or contributions from the encrypted internet service tour. The first one, while researchers have found that some better than random success in identifying gender editor from editor user pages, to do an analysis of Spanish Wikipedia, automated tools can't make judgments about the full spectrum of gender identities, making this an area of continual unknown. Next up, Parajita. You are muted. So is the unknown about editor burnout, registration requirements or information inequality. So if you want to put in the chat, and then the unknown is actually about the third one, so information inequality. While many data scientists have made it attempts to measure information inequality and Wikipedia, the 2030 strategic plan for both media research includes trying to create something that works across the movement. And this currently is still an unknown. So what is the unknown about diverse geographies, edited fun motivations for female music artists. You can quickly put in the chat, your guests now. And the answer is the first one about diverse geographies. And this has been successful attracting newcomers who live in Africa, Asia and Latin America as the 2021 community insights report or as of that insight report. There has been no overall improvement in the geographic diversity of those tenured editors since 2019. This may have changed in 2022. Most recent research shows this case for now. Many researchers, including the growth team at the Wikimedia Foundation have been testing ideas that can help this out. Okay. Is the question about volunteer contributions during COVID, financial support for editors, or the coverage gap between Europe and Africa, an unknown. The answer is add your idea, which one is unknown. It is the second qualitative research conducted with Indian language Wikipedia has found that financial support did not lead to active participation, but it's not clear whether a different approach to financial support would have worked or whether it would be effective elsewhere. Still a big open question. Perjita. So is the unknown about women in India recommender algorithms or biographies. And so the answer is the second one recommender algorithms. The taxonomy of knowledge gaps so Wikimedia Foundation suggested that recommender algorithms might helpful knowledge gaps an area that's still kind of developing an exciting area of experimentation by the movement and researchers. Sarah. So is the unknown about reader motivations, transgender and non binary articles or the motivations of editathon organizers. And the answer is the last one. The researchers have worked alongside organizers of editathons. By March and descriptive based on interviews with many editathon organizers strongly encouraged future researchers to work with non English organizers to build a clearer picture about motivations in the majority world. We are finding that we were also stumped by many of our own questions. So it's fun to see your responses. So next is the unknown about Wikipedia reader gender reader patterns or indicators of conflict type in the chat. The unknown here is the third one, while some researchers have created prototypes of fully automated metrics for conflict on Wikipedia. Social scientists who looked closer found that what sometimes looks like conflict can actually be collaboration. A parajita. You're muted a parajita. So the first one is the unknown about search engines, notability or feelings of empowerment. And the unknown is the first one. So research has found that while interventions can increase content about women, they don't necessarily decrease gaps in info box content. While early research has investigated how Wikipedia influences search results, the best way to actually do that is not yet clear. Sarah. So the unknown. A question about harassment. Urban and rural users or the reach of wiki data. And the answer is research among urban and rural users. Survey research suggests that urban research readers tend to be overrepresented among readers and contributors to Wikipedia. Therefore, less is known about the experiences of rural readers or what might prevent people in rural areas from reading and accessing Wikipedia. And so for the very last question. Okay, so is the unknown in these these three things, a question about how people use Wikipedia, showing newcomers view counts, or a trust meter. The answer is to can we increase participation by showing newcomers how many people have viewed their articles and helping them see the value of their contributions. That's an unknown. It's something the growth team is testing right now. So contact them if you want to test it with your language Wikipedia. I want to thank you so much for joining what is probably quite a challenging but hopefully an interesting quiz about Wikipedia. Someone has commented if this were my final exam, I would not have graduated one of the things that is especially fun about being a researcher who works with the community is that there are many important questions. There are many hard questions. And as scholars, what we don't know, and the search for what we don't know is one of the most important parts of being a researcher. And that's something that often we as scientists learn from talking with all of you. So we're grateful to everyone who has come to Wikipedia, who has talked about your experiences and your challenges and your work, because that is how scientists learn what questions to ask. So we encourage you in the last few minutes to name someone from your group who you especially enjoyed meeting, or who you think was especially good at coming up with answers. Other things to know about the session is that we hope to publish a blog post with these questions soon. And please, we may have made errors. We might have thought something was known or unknown and got it wrong. So we had an item in the form for you to let us know about our errors. And then if you are interested in connecting with researchers to do your own research about your community, please check in with the Wikimedia researchers list. Look at the Wikimedia research showcase, which is a regular gathering hosted by the foundation of research. And obviously, we're always interested to talk to Wikipedia communities about collaborations with you. So many thanks to everyone who submitted ideas for this. We will post the blog to our website, citizensandtech.org, and if you add your email address to the form, we will send you the blog post, the answers, and all of the other relevant material. And I believe, Sarah, we have it set so people can update their answers or submit another form entry if you didn't add your email address the first time. Thank you everyone. This has been fantastic. And in a true Wikipedia fashion. Here are the citations for our questions. Have a great Wikimedia.