 Okay, we're live. All right, thank you. My name is Emily Lesak, and I'm the Senior Research Community Officer on the Wikimedia Foundation research team. I'd like to welcome you to this month's research showcase. Our showcases are monthly convenings organized by our team to recognize and share recent research on or relevant to the Wikimedia projects. For those of you who are joining live, we welcome you to ask questions in the YouTube chat or on IRC. My colleague, Pablo, will monitor these channels and pass questions to the speakers at the ends of their presentations. And we kindly ask that attendees follow the friendly space policy and the universal code of conduct. Before we begin, we do have a community announcement to share. That registration is now open for the 10th edition of Wiki Workshop, which will be held as a standalone virtual event on May 11th, and I'll drop the link to register in the YouTube chat. And with that, I'll pass it over to Miriam, who will introduce this month's theme and speakers. Thank you, Amelie. I have a very quick introduction for you. So I'm gonna share the screen and I'm assuming unless anyone tells me anything else that you can see this. All right. Hi, everyone. Welcome to the April 2023 Wikimedia Research Showcase. My name is Miriam. I'm part of the Wikimedia Foundation Research Team. And I'm honored to be the host today. Today is going to be a very special and very visual showcase. And we are, because we're going to talk about the role of images on Wikipedia. So let me just give you a little bit of context of what we're going to see today. When we talk about images and Wikipedia, we talk about a beautiful place on the web, called Wikimedia Commons. Wikimedia Commons is Wikipedia sister project operated by the Foundation. And it's one of the largest repositories of free visual knowledge on the web. Just to give you a few numbers. The community of Wikimedia Commons is quite big today. There are more than 15,000 active editors, some of which are very amazing. And they're starting creating artwork with the Wikimedia Commons logo using the latest AI image generation technologies, as you can see in this slide. The result of the work of this community is quite astonishing. Wikimedia Commons today hosts more than 90 million multimedia files, audio, video, but mostly images. And it has been in steady growth in the past few years with the volume of content on Commons growing by 400% since 2014. And Wikimedia Commons is really the essential infrastructure of free visual knowledge because the large majority of images on Wikipedia are actually hosted on Commons with 17 million file used to illustrate articles in many, many languages. And so Wikimedia Commons and Wikipedia, they really work together to create the free knowledge ecosystem that we see every day on our sites. And Wikimedia Commons is important for Wikipedia because it actually enriches the encyclopedia. For example, just to give you an example, articles on Wikipedia that are illustrated, they're generally more popular. Illustrated Wikipedia pages get on average four times more views than articles without images. And know that this is not a cause of relationship. It's not because they have an image that they are more popular, but this gives an idea as the exposure that images get once they enter the Wikipedia ecosystem. And Commons is important for Wikipedia also because readers like images a lot. And this is a topic of the first presentation, so I'm not gonna spoil any numbers. I'm just gonna tell you that readers engage with images very much and much more than other parts of the page. And then Daniela is going to tell us more about his study on how readers interact with images on Wikipedia. And while images are important for Wikipedia, we're still missing many, many images on the encyclopedia because there exists this large visual knowledge gap. On average, on a Wikipedia project, there are 40% of articles that do not have an image. And if we break down this number by actual Wikipedia project, we see that the largest encyclopedias such as English or German, for example, they have more than half of the articles that do not have an image. And while this visual gap is a problem in itself in terms of accessibility and the type of consumption of knowledge in different parts of the Wikis, it can actually have repercussion on existing inequalities in the projects, such as, for example, the gender gap. We know from research that it is important to have images of underrepresented genders in knowledge ecosystem in order to bridge the gender gap in society. And so it is important for us to understand the state of the visual gender gap in Wikimedia projects. And Pablo, in the second talk, is going to give us an overview of what is the visual gender gap in the 10 most spoken languages on Wikipedia. And with that, I am super excited to welcome today's speakers. We have Daniela from University of Turin, who is going to give us the first talk about how readers interact with images on Wikipedia. And we have Pablo from Catholic University of Chile that is going to tell us about the visual gender biases on Wikipedia across different languages. I am also very excited that the room on Zoom is very crowded today with many people who are interested in this topic. We have almost everyone from the Wikimedia Foundation research team. We have friends from and colleagues from Wikimedia staff. We have Fiona and Giovanna from the Grammar and Culture team. And Jasmine, who is the product manager for a product that have been working on bridging the visual gap. We also have some collaborators of our speakers. We have Titzena from Stanford University who's been collaborating with our first speaker, Daniela, and Pushka from Cognizant who has been working with Pablo. And with that, I hope I can go to the next slide. It doesn't work. Okay, the next slide was just about enjoy the showcase. I'm sorry, this is not working. Yes, here it is. So I hope you enjoyed the showcase today. And with that, I can call Daniela on stage for the first talk. Thank you. Okay. I hope you see my slides. Please just stop me. Hi, everyone. Thank you, Miriam, for the very quick... I mean, not yet. Yes, now, yes. Okay. So, hi, everyone. My name is Daniela Rama. I'm a postdoc at the University of Turi in Italy. Thank you very much, Miriam, for the kind introduction. And thank you for inviting me here today to present our project, which is about a large scale study of reader interactions with images on Wikipedia. This work has been done in collaboration on the Wikimedia Foundation and Rossano Schifanella, who is actually my PhD supervisor, professor at the University of Turi. So today, we are going to talk about the visual side of Wikipedia. In the last 20 years, the volume of images and, in general, multimedia content has exploded online, most of all thanks to social media and image sharing platforms. And the reason is mainly because images facilitates communication among people from different cultures and speaking different languages. So images are really changing the way people communicate and share and use information online. Also, in the scientific literature, there is evidence from psychological research that images and visual content play a key role in text comprehension, reading, and learning. But so far, little is known about how images support knowledge sharing and learning and information seeking in online context. In online context, Wikipedia is actually the largest online resource for knowledge sharing. Wikipedia is not only of textual content but also on visual content. Wikipedia has more than 50 million articles and 14 images spread across more than 3,000 languages. However, so far only a small fraction of the articles as Miriam was outlining before are illustrated. For example, in the English Wikipedia, only around 50% of the articles have at least one image and this percentage decreases a lot in the other language editions. And also, little research has been done trying to understand how images impact the readers on Wikipedia. And that's why we need to investigate the visual knowledge gaps in the context of Wikipedia. So our main research question is what we want to do in this project is trying to quantify and characterize a reader's interaction with images on Wikipedia. In practice, we would like to answer to these three research questions. First of all, we would like to measure interactions and measure to what extent readers interact with images. Then we would like to understand what are the factors associated with the engagement, with interest, with respect to images when reading articles and in conclusion, we would like to understand if images satisfy somehow readers, the reader's information needs when navigating Wikipedia. The data we use in this project is a large-scale collection of the data coming from articles and images of the English Wikipedia. We also collected all the image-related readers' interactions that happened during the month of the image collection, which is March 2021. We collected page views, clicks on images, and image per use, which we will see later, both from desktop and mobile. And we only use anonymized version of this data for privacy reasons. Now that we see what data we're going to use, the first research question is to what extent are readers interacting with images on Wikipedia? First of all, we need to quantify interactions and find a way to measure engagement with images. To do so, we define the click-through rate. The click-through rate is usually defined as the ratio between clicks and impressions. In this case, we measure the global click-through rate with respect to images, which in our case is defined as the ratio between page views that at least one click on any image over the total number of page views that happened in the period of our data collection. And from a certain point of view, this is kind of the probability of observing an interaction with at least one image, even a random page view. It can be interpreted like so. Let me give you an example. This is, for example, a reader reading a random page. There are images spread all over the text. And let's suppose the reader wants to interact with the image, the reader can click on that image. And then the image shows up in a new, in a different window called the media viewer. We collected all these kind of interactions and measured that the click-through rate is 2.7%. Or in other words, one in every 30 page views results in a click of an image, which may seem a very small fraction of page views, but it's actually, for example, 10% higher than engagement with the other kind of interactive content of the pages, for example, with the citations. So images elicit a good amount of interaction, actually, on Wikipedia. Now that we have seen that the images are actually engaging, we move on and ask ourselves, what are the characteristics of the images that makes them interesting for readers when navigating articles? To do so, we extracted a set of features coming from the visual context and the visual content of the images. The visual context of the images are basically the pages where the images are. And those features are the topics of the page, the page length, the popularity of the page, the readability of the page. Also, we extracted the length of the caption if present for any image and the placement of the image, which can be the offset, meaning how far from the top of the page the image is and if the image is placed within the info box in line in the text or in a gallery. The visual content, the features coming from the visual content of the page are extracted taking inspiration from scientific literature and are the quality of the image, the number of faces and the presence of outdoor settings, which are three features that are usually associated with high level of engagement. Using these features, we wanted to train our gistigic collection using the image-specific click-through rate as the target variable. So in this case, we computed the click-through rate for each image defined as the number of times an image was clicked divided by the number of times an image was displayed. So appeal to the reader. We binarize the image click-through rate splitting into each value into high and low, according to the median of the complete distribution. And also before running the logistic regression, we extracted a matched dataset in which we balance the page length and popularity of the between the two classes of high and low image click-through rate because from previous studies, we know that page length and popularity are two features that have a high impact on the in general, on the click-through rate. After extracting these two balanced datasets, we train the logistic regression using the finalized image click-through rate as the target and the features and the image characteristics as features in the model. We actually train two logistic regressions. The first one was a train on the topics of the page and what we see is that the topics that are associated with a positive impact on the click-through rate, meaning that in those topics, images are more likely to be clicked are transportation, visual arts and geography. On the other end, the topics where images are less likely to be clicked are entertainment, sports and education. On the logistic regression, we use all the remaining features and we saw that the outdoor feature was the feature most positively associated with the click on images, which somehow is in accordance with the fact that also on the other side, there are geography and transportation as main positively associated features. And on the other end, the offset is the most negatively associated feature with the click-through rate, meaning that the lower the image is on the page, the less likely it is to be clicked. We also note in this case that the presence of faces is negatively associated with the click-through rate, even though we know from previous literature that usually the presence of faces instead positively associated, I mean, the presence of faces show high engagement with images. And this is something that we will dig deeper later. We also perform a cluster analysis trying to identify groups of images with homogeneous characteristics using the features that I described before. We identified 23 clusters. These are only some examples. Each face displays the mean value of each feature for each group of images. And for example, the groups that are identified with high values of click-through rate are groups of images coming from visual arts, geography, biological topics, transportation, and also unpopular biographies. On the other side, images associated with low click-through rate are biographies of sports people and also popular biographies. We here starts to see that the presence of faces has been disentangled by this analysis, showing that faces are sometimes associated with high click-through rate when they are showing unpopular biographies, but on the other side, are associated with low level of click-through rate in the case of popular biographies. We want to investigate more on this aspect and perform a matched analysis, dividing images into two groups, images with faces and images without faces. This plot shows the click-through rate as a function of the popularity of the page. And we see that up to a certain point, images are more likely to be clicked when showing faces. So when pages are not so popular, images are more likely to be clicked when they show faces. And it appears to be the opposite in the case of highly popular pages. Our hypothesis is that probably people tend to click more on faces, on images with faces, only when they don't know the person of the people they are looking to. Finally, our last question is, do images satisfy readers' information need when navigating Wikipedia? Here, we explore data about a different kind of interaction, which are page previews. Page previews are little pop-ups that shows up when the mouse goes over a link within a page. And in this case, little box opens, which shows a short preview, a short textual summary of the destination page. And the preview could have or could not have an image complementing the text. For each preview, we completed the conversion rate, which in this case is defined as the ratio between the previews followed by page view. So, the previews were used, the readers actually clicked on the destination page over the total number of previews displayed. We compare the conversion rate for the two groups of previews, the one without images and the one with images. And we see that across all the level of popularities, the conversion rate is held for previews without images. So, it is more likely to visualize the next page if it has no image in the preview. Again, here our interpretation is that probably images tend to have a, tend to play information tend to be at an information support or support for textual information. So, when readers see also the image, they are less likely to click and go on and explore the next topic. Let me summarize the main takeaways of our research. So, first of all, we have seen that images tend to have quite a high level of engagement, at least 10% with respect to other type of interactive content on pages, such as references. We also see that clicks on images occur less often in longer and popular articles and when images are placed at the bottom of the page. With respect to topics, images are more likely to be clicked in pages of visual arts, transportation, and geography and also faces are engaging, but on Wikipedia only when showing less popular people. Also, finally, images have to satisfy the reader's information needs when navigating Wikipedia. Our results can help editors to include visual content in areas of articles where that we think are more engaging for readers. Also, from our results, other researchers can start building models to find the right images to be recommended to Wikipedia articles. And also, our results somehow provide a justification for investments and initiatives designed to improve the visual side of Wikipedia, such as Wikipedia Loves Monument and Wikipedia Loves Science. Thank you all for listening and thank you again to my collaborators, Tiziano, Miriam and Lozano. Okay, thank you very much, Daniela, for your presentation today. We have some comments on the YouTube channel like people are saying, like the cluster you have presented was very interesting. I don't know if we have questions here in the room to Daniela, I have several. Let's place two others to start the discussion. Okay, if not, I can initiate the discussion with one question. So in what is like classic in research cases, like we typically ask speakers, like this is our work that focus on English Wikipedia. So we typically ask the speakers how much their results are expected to differ if they extend their work. Ah, thank you. And so that Leila has a hand up. Leila, go first, please. No, I think you're asking part of my question. Go ahead. Ah, okay. Yeah, for some reason I cannot see reaction. I think I have a problem with the Zoom version. So please, if you have questions, please tell me in the chat. Yeah, I was going to ask like, we typically ask what, how much the results will differ? We go for other languages, but in this particular occasion, I'm more curious by sending this work of analyzing the same image on different language versions of Wikipedia and maybe not in the same page or not in the same context on how the results are affected by the future from the image or future from the community, the specific community on that level. Thank you for the question Pablo. Actually, now I don't have many expectations on what the results could be extended these work on other language editions. Actually, also because I don't know how many images, I mean, it would be interesting to know first how many images are present in, I don't know, a good number of language editions. Like, I don't know the most, the 10 most popular, because I don't know which is the per, how they are spread across different editions. What we started to see sometimes ago was to try to understand how the results of the logistic regression generalize across countries in the world, which is a bit tricky to interpret because, you know, there is very, very different levels of, there are different levels of volumes of people coming to Wikipedia from one country to another. It's difficult to compare, I don't know, people coming to the English Wikipedia, which is billions of people and people coming to, I don't know, some version in some particular country in Africa, because there could be way less, the popularity could be way less and also results could generalize, could not generalize, and we can introduce biases. But yeah, that would be, for sure, something that we would like to investigate more. You have a question? Leila? Yeah, thank you so much, Daniela. Also, Rosano, Tiziano, and Miriam for this research. My question is to build up on what Pablo asked. I think, as you know, we have a concern about like the different languages of Wikipedia. And particularly in the case of English, we know from older data from 2017, 19 time period that around 75% of the page views are by men. Now, we also know that there are topical, we have some indications of topical preferences between what men or women read on Wikipedia. And when you shared the statistics about topics in which images were clicked more often, I naturally thought about this issue that there can be correlation between basically the number of page views that is received by one gender and then what happens with the topics and then the data that you see here. So one suggestion I had is that for future research, you all consider including a language in which, one or two languages in which we have a smaller gap in terms of separation or smaller gender gap, basically in readership. There are languages like Polish or Romanian where the gap is smaller. Of course, you have to answer the question of what are you have sufficient images and all that. So I appreciate that. But I think it's important to consider that because let's say if you go to the case of Romanian, you will end up with roughly 60% of the page views being by men, which is significantly lower than English. And you may be able to see results that can be different or not, right? We will learn something on that front. I don't know if you have any reactions to it, Daniela or others who are on the paper and research. Yeah, that's a very, very interesting comment. Thank you very much. Indeed, that may explain also, I mean, at least partially why we see military, for example, as the most positive coefficient for the logistic regression, assuming that military topics are mostly associated with the main readership. So, yeah, for sure the approach you suggested would be very, very useful in our case. Yeah, I mean, the military topic was something that appeared a lot in our analysis we did. Also, analysis that we are currently running. So, that may be one of the reasons for sure. Thank you, Daniela, for the question and for the answer. We have time for one more question. I think Jovana has a question too. Yes, thank you. So, I was wondering, because you said that the articles that you have more clicks in images are the ones about transportation, visual arts, and geography, just like the slide that you are showing. And the ones that have less clicks are the ones about entertainment and sports and education. And as far as I understand, entertainment and sports are articles that are very popular. So, it's like the opposite, right? And so, I was wondering, I know that you have the hypothesis that maybe people click less because especially entertainment, that popular articles have, they are about people who are more known, so their faces are more known as well. But I was wondering if it's also about the details that are depicted in the image. Because transportation, visual arts, geography, I tend to think when I think about images and just about those subjects. I think about images that have a lot of details and that I would like to see up close and to visualize it in a bigger way, especially geography with maps. So, I was wondering if this is something that you were thinking about as well, something that appeared somehow. Yes. Yeah, thank you for the question, Jovana. And, yeah, it will be for sure interesting to somehow define kind of image complexity somehow and see how people tend to engage with respect to this aspect. We didn't think about it when we performed this kind of analysis. We only computed a feature which is image quality, but it's kind of a different thing, which is this quality is related to the image quality category in common. Yeah, but apart from that, we didn't explore in details the details of the images. I mean, we didn't look further into the content of the images apart from the three features that we extracted, which are the quality faces and all those settings. But, yeah, that would be interesting to investigate. Jovana, thank you for this question. This is a topic that has been actually investigated both by the cognitive psychology researchers and experimental psychology researchers who wanted to understand the relation between visual complexity and the interestingness of the image. And then these kind of studies have been translated to a computational approach by computer vision researchers to measure complexity and its relation with visual interesting on a large scale. So what happens, so the results of this research show that there is an inverted shape relationship between visual interestingness and complexity. And so that means that up to some point there is an upward relation between interestingness and complexity and then it goes down when the image is too complex. And then there is another study to do to understand whether it's the complexity of the image that actually arise interestingness and what is the component of making the image bigger versus actually the image is interesting for the reader. So previous research from experimental psychology will tell us that there is actually an interestingness component, but it will be interesting to decouple those two. So thank you for raising this. This is very interesting, at least for people and some of these things for you. So thank you so much. Thank you. There is just one clarifying question on IRC. Andrew, you want to know if the geographical location of the user was considered as a variable in this research? Sorry, well considered as a variable in the research in the analysis. If you were considering the geographical location of the user? We didn't consider the geographical location in this paper. We started to investigate also the geographical location of the users. But again, it was a bit difficult to, as I was saying before, it was a bit different to interpret the results for people coming from very different areas because there could be several bias also connectivity issues, availability of information communication technologies to different kind of people around the world. So we, at the end, we didn't consider the geographical aspect of the leadership I mean. Please thank you Daniela. Okay, so I think it's time to thank you again Daniela for your presentation and to go with Paolo's talk. Thank you. Should I introduce Paolo? I'm going to introduce Paolo. Paolo, Daniela, if you can stop sharing your screen. So Paolo can share your screen. Alright, it's Paolo Vecchia from Catholic University of Chile. He's going to tell us about visual gender biases and Wikipedia. Paolo, please. Okay, thank you very much, Miriam, for this introduction. They really mean our research team for the kind of invitation. I'm very happy to be here. I'm presenting this study representing a fantastic team of Pushkal, Agarwal, Miriam Reddy and Videk Singh. Let me talk a little about the context of this investigation. For some time we have known that Wikipedia tends to register and communicate content disparities in multiple dimensions like gender, geography, culture, historical period and so on. Particularly well-known fact is the low percentage of particles of women. Usually less than 20% in different variations. Some studies found that the gender gap in content is more complex than that. We have multiple asymmetries in the text. For example, the lexicon topics, sources or network centrality. This is relevant to Wikipedia users and volunteers and several initiatives have attempted to balance the content between genders. However, there is little research. Sorry, a few interesting studies have addressed gender bias as a joint manifestation of multiple biases. However, there is little research on the visual aspect of the content gender gap. That is how images are used to depict genders in Wikipedia. The goal of this study was to evaluate the content gender gap with an integral approach. We tried to analyze visual and non-visual disparities in content from a comprehensive and systematic perspective. But what does it mean? First, we wanted to consider biographical diversity. We know from previous studies that the content biases in Wikipedia are different for each language version and also for biographical domains. For example, articles about people with the same occupation. So we assume that the bias evaluation should consider this content diversity. Our second intuition was that a proper bias evaluation needs to expand the types of content analysis. On the one hand, we need a multimodal approach considering written and visual content. On the other hand, it is relevant to observe the biases generated in multiple stages of content production. Here, I need to explain more about what involves this multi-stage perspective. Think of Wikipedia as a continuous cycle of knowledge organization. So in this cycle, we can recognize some big editing stages. We identify three. The selection of the topics for articles, the building of the content, and the positioning of the edited article within the information system. Each stage is linked to more specific editing processes and a way of framing communication. And we can connect most of the content asymmetries with these three stages of content production. The selection stage is the beginning of the editing cycle in Wikipedia. One editor proposed an article to edit, for example, a specific biography. And editors with experience can accept or reject the proposal. Regarding communication, the representation of social groups in Wikipedia is a state in this selection stage. The literature on gender gaps has studied in this dimension the article coverage of each gender and the deletion processes. The second stage of content production is the building of articles. This is the moment when editors create biographies adding writing and visual content. Also here, we have the discussion processes as a background of the generated articles. In terms of framing, in this stage, Wikipedia creates a collective characterization of each gender or social category. Some gender asymmetries related to this stage include writing length, lexical, topical, and source biases. The final stage in this cycle is the positioning of content related to processes like content justification, the connection between articles through hyperlinks and the multilingual dissemination of content in Wikipedia. These processes frame communication by giving a structural placement to social categories, that is, ordering them into positions with better or worse probabilities to get communicated. In the gender gaps literature, positioning has been studied by analyzing variables like classification structure, network position, and multilingual nodability. In this study, we understood the content-gender gap in repeating this complex way, considering the multiple stages of content production. In this model, the content-gender gap is a composite phenomenon, an integral result that comprises as cooperating content layers all these gaps and their mutual relations. Now we back to the question in what sense did we offer a systematic approach? We can clearly respond to this by comparing our perspective with previous studies. In most cases, research on visual gender biases in Wikipedia has focused on one language version, English or German, and they analyze it in a sample of biographies the building stage with one metric, the number of images. By contrast, we designed a multilingual research in the 10 most spoken languages, considering all biographies and exploring the three editing stages with a multimodal approach. How we did it? We started compiling the complete list of biographies from Wikipedia in any language with the gender, occupation, and birthplace. Then we classified the biographies into gender categories and 10 occupational domains. The quality of the article was calculated with an automatic classifier based on the structure of the article. We then collected all the images of the biographies in the 10 selected languages and estimated the visual quality by training an image quality classifier contrasting with media commons and higher quality images. The next step was to calculate eight metrics of content asymmetries, representing the three stages of content production in a multimodal way. Each metric is a content ratio between female and male biographies. We created similar indicators in textual and visual content to compare both communication modalities as much as possible. In the selection stage, we calculated the ratio of female versus male biographies. To have a similar visual indicator, we selected only the article with images and calculated the same ratio. In the building stage, we estimated the text quantity by comparing the average number of character in biographies. As a visual counterpart, we compared the average number of images in female and male biographies. Regarding building quality, we estimated ratios of article quality between female and male biographies. We also compared between both genders the average image quality. As an indicator of content positioning, we compared the average number of languages in which each gender has its biographies. And to set a comparable visual indicator, we selected only articles with images and calculated the same ratio between female and male biographies. So what were our main results? All our metrics are ratios contrasting female and male content. So if these ratios are close to one, there is no content bias. When the metrics are over one, there is a female bias and when they are under one, there is a male bias. We also extinguished with colors the metrics related to visual bias from the indicator related to the text or general content. Our main results are related to multilingual analysis. You can see that the most salient male biases for the creating and visual appear in the article selection stage. When editors decide which personalities should have a biographic. Russian, Arabic and French are the languages with more male bias in the selection of biographies. The situation is different in the building stage. Women tend to have more text in biographies, but there is a contrasting visual trend and male bias in the number of images. Regarding content quality, women have better articles and images with better quality. In the positioning stage, female biographies have better multilingual coverage in some languages like Russian and Arabic, but that is not a general rule. We also have languages like Indonesian and Chinese where female biographies average a lower multilingual coverage. So to sum up, the most salient male biases for written and visual appear when editors select which personalities should have a will be a page. The trends in written and visual content are dissimilar, so we cannot assume the same pattern in both modalities, especially in the building stage. Women tend to have more images across languages and female biographies have better visual quality on average. So how can we use this data to fight against gender bias or what are the practical implications for editors trying to balance the content? To use our data in the best possible way, you can see this challenge like a battleship port. In this old game, players attempt to sing rival chips by sending ballots to specific cells. So you can order our gender indicators in a similar matrix according to two variables, Wikipedia language version and occupational domains. This is the matrix for image quantity ratios. In our data, we have 10 languages and 10 occupational domains. There is a female bias where we see a green cell in this matrix. For instance, in India, notable women have more images than men when they have religious or army occupations. In the country where the cell is brown, there is a trend toward a male bias. This matrix summarized 100 content ratios in one specific metric, the image quantity. But we studied eight of them, so you can observe the big picture in a multimodal and multistage perspective that involves 800 content patterns. This is how you can see the content gender gap from a multimodal and multistage perspective. Each bore is related to one metric. Certainly, I cannot summarize now the main pattern, but I think Wikipedia editors can use these multiple bores to search gender biases in their own language community and locate them in specific occupational domains. Finally, from a cultural point of view, we can also use this data to reflect on what Wikipedia versions have similar composition in their gender biases. For each language, we have 80 gender metrics, ratios by 10 occupational domains. In future studies, we can estimate the proximity of languages, specifically regarding the gender bias production. This is an example with hierarchical classroom modeling. We can represent the language version's proximity and display the results in a tree diagram. You can see in this model that the Western Europe languages, English, French, Portuguese, and Spanish are too close regarding their gender biases structures. Russian appears like a bridge to a second cluster between Arabic and Chinese. And the widely spoken languages around India have also a relative closeness. That's all. Thank you very much for watching this presentation. It was a very interesting talk and an impressive work that you all have conducted. We have some comments on YouTube, so I will start with them. There is a participant who is curious about models to find out which images are needed, in particular, what kind of images. For example, in an article about airplanes, that person is interested in whether there would be images about the type of airplanes or more details about parts of the airplane, like cabin or more about the social content, like the airfield log. I think this goes to how to make those findings actionable in the space of recommending the system and how we can learn about those biases and what is the specific question, how we can learn about airplanes? This is an example. The main purpose is how to discover images that might be needed. This is how you were presented in the end. Yes. At the end of the presentation, I was showing some matrix related to our database. We have an open database with this information and you can create these matrix. We extract information, for example, on the languages, the biographies, of course, the gender, the occupation and also the birth place. You can select if you are interested, for example, in Spanish Wikipedia and you are editing in Spanish Wikipedia, you can create the matrix and see if in Spanish Wikipedia we have a gender bias related to people that were born in Latin America. That is an example. Or you can see if in Spanish Wikipedia there is a gender bias related to religious people. So we don't have all the occupations or topics that you can find in Wikipedia but we have some relevant information and you can combat the bias in that form. Also the participant on the YouTube chat has your answer. That participant is always bringing another point that I think it relates to both presentations. That participant mentioned maybe there are images that are ethically problematic that affect their personality. In the context of biases to what Pablo presented but also in the area of engagement with Daniela presented, I don't know if you have some thoughts about this visual content that might be problematic. We didn't do that kind of analysis. We did it with a few studies so we tried to offer a general picture of that. The most general picture is related to the quantity of images, the quality of the images and so on. But we didn't analyze the stereotypes that I think need the most specific approach and most educated. For example, in sociology you have the studies from Erwin Goffman analyzing these kind of stereotypes and he used six kinds of stereotypes but I think the computational approach that we use in this study cannot give that kind of quality details. So it's a challenge, it's a big challenge to have the methods to give that kind of information with stereotypes, for example. Pablo, I think Milan also has some ideas. I just wanted to give a perspective on the comment around having models that can suggest images to be added to articles. And actually Jasmine is here. We've been working with a few of the product teams to develop models that can suggest images that can be good matches for articles. These models are largely based on what is already existing on Wikipedia. So we're able to, for example, align articles across different languages or align sections across different languages and say if this image is good for this section it's good for language being. I think what is so we don't have a very good way to prioritize what are the most important areas in the article where we need images and I think this is where Daniela's study might be useful as well as Pablo's mattresses where we say there are missing images in these important aspects on knowledge equity. The other thing is from the audience I'm just giving a generic perspective is that because we use existing patterns on Wikipedia we use these existing images to avoid further potential biases but at the same time we are also replicating any bias that is already on Wikipedia. So there is a lot more work to do in these models and I don't know Jasmine if you want to add anything else from a product perspective of how these things have been working and etc. Yeah, you covered a lot of the Mary and the only thing I would add is that looking to the future marrying things that are putting together things like the topics API for example or allowing some sort of like filtering just a step in the right direction when it comes to these but I think there's still like some more work to do to be able to get to that place but it is a direction especially within the apps and perhaps on like the other platforms that we're interested in. Thank you. Jovan I think you have something to comment on then Leila. Yeah I have a question about something else actually about the part about the building part of the process because there you said that there are more images depicting men and less images that are higher quality that depicts women right and so that immediately made me think about the problem that not the problem but I mean this is the characteristic and what is good about our projects that the images they need to be open access or in public domain right so that creates the problem of maybe like the fact that those are old images you then have more men depicted on it like this is the case in museums and galleries as well like that bias exists especially as the order you get right in the collection for example. So I was wondering if you saw that represented somehow and like for me it would explain I think why higher quality images are for women because they are just more recent I would assume and if you saw that there is a way for us to work on that I know that there are several campaigns that I think kind of go into that they're actually visible with women and whose knowledge and all of that but I was wondering if you have some sort of perspective related to that. I have the same hypothesis so probably there is the residency the key to understand this yeah so if you see this kind of information in a historical point of view yeah the gaps are huge so the beginning of the history you have maybe and this is interesting you have more female biographies but then you have around 25 centuries with few biographies female biographies and maybe after 1950 around 1950 you have more famous women or notable women in the record so you have a huge historical ranch without female or with around 5% of notable women so that is part of the problem in that range historical range of course the image quality is not the best but now in the last 50 years you have better quality so yeah probably is a good explanation thank you Pablo we have time for our last question and Leila has raised her hand Leila we can not hear you okay there are some big issues okay so I will pose some I have this last question so I think the work that you all presented today is super relevant in the space of knowledge gaps in my case I'm focused on other areas that the team is working in knowledge integrity and how what can we learn from all this work on images that can affect the integrity of knowledge like for instance some members of the team that have presented today work on engagement they also work on engagement and citations that apparently the engagement is lower than the one on images so what can we learn from the images space to improve the area of citations in the aspect of biases like how can we learn from biases on images that we can improve existing biases that might exist in terms of source reliability this is a complex question I do expect you to have and if not we can continue on this conversation and keep these ideas so I think if there are no further questions let's go up to the session all right thanks Pablo that was a really interesting session thank you so much to Daniela and Pablo for sharing your work with us today we appreciate your accepting our invitation to be here the showcase today was made possible thanks to coordinated efforts between myself Pablo Aragon as well as Diego we'd also like to thank Miriam for introducing today's topic and speakers as well as the Q&A on YouTube and IRC big thanks to Emerald for providing the AV support as well as Janet Venteria for coordinating internally and thanks to all of you who tuned in live or are watching the recording there will be no showcase in May since we'll be hosting the 10th edition of with the workshop so this is another friendly reminder to please register in advance our next showcase will then be in June and it will focus on representation of non-binary and LGBT people on Wikipedia so I hope to see you there I'd also like to share that this unfortunately will be my last research showcase I am leaving the foundation at the end of this week to pursue another opportunity but I would like to take a moment to thank all of the speakers that we featured over the last couple of years for taking the time to share your work with a broader community and also the supportings and contributed to the discussions so each month I just want to share that I'm really inspired by the featured research and its potential to have a positive impact on the broader community and highlight that the success of this showcase as well as the other initiatives is really doing a large part to all of your thoughtful contributions so if you're a member of the Wikimedia research community and if you have any immediate questions or need support please reach out to Layla, the head of research or you can join us at one of the upcoming research team office hours