 Good morning Singapore, hello and good day wherever you are in the world joining us online. I'm really excited to be here with this amazing panel talking about one of the things that I've been occupying all of us recently, which is generative AI and specifically how it connects to what we all do on the wiki world. So I've gathered a few friends to kind of have a good chat and the idea for this panel is we've just heard in the morning panel from the wiki media foundation and what it's been developing and I think this panel is more about community focus but I still wanted to have in the room some representatives from other stakeholders. So we have Leila here from the wiki media foundation and Jimmy that no one needs to introduce kind of representing the board but the rest are community folks and this is what I'm hoping for this panel. I'm hoping that this sparks a discussion that is way way beyond this panel essentially because I believe that as a community that is global and dispersed we're all you know into doing our own things sometimes but what is happening now is massive, it's disruptive, it's changing the world, it's changing the way that we're consuming information and producing it and I really honestly think that we as a community need to have a global discussion about it and be more strategic. So what I have in plan for us today is to start a small collaborative effort to first of all map out what are the current uses that we're already being seen in the community happening naturally, organically, what are some of the challenges that we've seen but mostly what should be the next steps that we take as a global community together and hopefully this is the part where you can join us and share your thoughts and share your questions online and here in the room. So without further ado I want to begin with a really short introduction of our panelists and I'll start right from the right, at least my right, Andrew Lee. Please explain, like share who you are, although... Yes, hi, how are you doing? Hi, I'm Andrew Lee, I'm user Fos Edo on the projects in the community. Some of you may know I've written a book about Wikipedia and for the last few years I work as the Wikimedian in residence but also the Wikimedian at large at the Smithsonian Institution, which is the kind of our National Museum of the United States and with the Metropolitan Museum of Art. So I've been working a lot with those institutions on big data projects and most recently on AI style projects. So happy to talk about this topic with you all. Thanks so much Andrew, Leila. Hi everyone, my name is Leila Zia, I'm the head of research at the Wikimedia Foundation. If you're not familiar with the research team, it's the research is the team that develops models and insights using scientific methods and helps strengthen the Wikimedia research community. I'm here because as you can imagine the research team is very much involved in thinking about and also developing models using generative AI and large language models. And also we are part of the conversations and collaborations within the foundation with other teams. So I'm going to be sharing some of the perspectives of our team as well as other teams as we are going through this conversation together. Looking forward to it. Thanks Leila. Calder. Hello, I'm Calder González, I'm a member of the BAS Wikimedians user group and also a staff member there. I work mostly in education programs but also other things. And I think I'm here because I'm interested in this topic and I have been talking with this with Sani. So let's see if we can introduce any topic or something related to education. Thanks Calder. Jimbo. Hello, I'm Jimmy Wales. I have a very keen interest in all of this area. I'm not an expert in anything but I'm constantly thinking and testing and trying things with chat GBT and the other large language models. And I see a lot of opportunities for us in this and obviously some risks as well. Personally though, I'm really looking forward to the day when my role can be completely replaced by an AI that would be fantastic. Thanks Jimmy, Netha. My name is Netha Hussain. I'm a volunteer on English and Malayalam language Wikipedia since 2009. I also upload photographs to commons and I'm also active on Wikidata. I'm basically using chat GBT and mid journey, these two generative AI models to make my volunteering work efficient. And of course, I believe that you have, if you are using these models, you have to be quite, you have to use your discretion before you put your content up there online. So I will be speaking more about how to center yourself, how to have human centric AI to be able to use your discretion and put up and understand your content and then place it up on the internet on Wikimedia projects. And last but not least, Tillman. Oh yeah, my name is Tillman Bayer, username HAB on English, German, Wikipedia and some others. So I'm the editor of the Wikimedia research newsletter where we like summarize academic research about Wikipedia, a lot of which is about large language models or natural language processing and even before chat GBT came along. Also I'm involved in design posts with the English Wikipedia newsletter where we report a lot of going on, including on such concerns. Thanks so much for being here, all of you. You really do come from different portions or parts or corners of our movement. And I think it would be really interesting to begin with simply trying to map. What have you, what are some of the uses that you've already are either experimenting yourself with with Jenny, Jenny, I in our work in the wiki world or have seen witnessed with the caveats, right? There's one caveat. We're not going to map everything like I'm sure there are things that are going to be missing. And we're counting on you, the audience here in the room and also online to help us map out all the other uses that are currently used happening that we're not going to cover here in this panel. So let's begin with Netta. I'm actually very interested to know what your experience has been so far. Well, in my case, I experimented with mid journey. It's any generative AI technique. It's a generative AI software that will help you to create images. So I tried to create historical figures, animals, you know, I just played around with things and figured out that it would it produce, it produces pretty decent images, but not a photographic quality. So I would get portraits of, you know, Alan Turing or Adelaoules or, or any of these, you know, famous personalities. But they are not exactly as is in the photograph. So the small blemishes on their faces or, you know, the small features, they are not always adequately represented in the images that mid journey is creating. And I uploaded this content to Wikimedia Commons. And in the end, I found that they are widely reused externally, not it's reused within Commons to an extent, but externally, people are using it on postcards. They are using it on mugs, t-shirts, wherever. And I even found a website which is selling this content for money. So that's something I've been seeing with mid journey generated content. I also tried experimented with shaggy pity to try and see if it writes a decent Wikipedia article. And it really wrote me a convincing article about a medical condition, but it was completely wrong. And the references that I got, they were completely made up. So it, it, the articulation, the way the article was written was so convincing that it even lured me as a medical doctor to believing that it was true, but it was almost 90% false. So I find a use case where you can actually use shaggy pity to rephrase sentences to, you know, summarize some content, but not in the context of creating an entire Wikipedia article. I also tried shaggy pity with writing small pieces of code. So I wanted to find out I had a list of articles related to, you know, COVID-19 and how it affects different countries. So there are articles about COVID-19 in Argentina, COVID-19 in India, COVID-19 in USA and so forth. And I wanted to find how they compare with each other when it comes to bite count and article quality. So I wrote a simple code using Python. It was completely done by shaggy pity. Chaggy pity wrote a beautiful code for me to go through all these articles, find their bite counts. And I found that the article about COVID-19 in Yemen is 10 times less in terms of bite count compared to the article about COVID-19 in the USA. So these kinds of smaller research that you can undertake on your own, it could be offloaded to shaggy pity. But as I always say, it has all these, you know, generated via technology has to be used with discretion. And you have to put your discretion first before you, you know, use any of those contents. Thanks so much, Neta. These are quite different uses. Some of them with pictures, some of them with text. In a sense, demonstrating that this is a quite a vast world that we can be using in various different ways. And I think another maybe different aspect may come from some of the uses that Andrew has been experimenting with. Andrew, do you want to share a bit more? Yeah, hi, thanks. In the telegram group, I put in a link to some of my documentation of playing with chat GPT in the early days. So I'm not alone. As many of you probably tried, I think a lot of us are people who like direct experience with these types of tools. So I think my first experiments with it are similar to what you probably saw, which is write a Wikipedia article about X. And hopefully that X was something we don't have an article about. So you would actually be testing chat GPT's ability to scrape and collate rather than just regurgitate in existing Wikipedia article. I think most of you found out very quickly that no matter how much you try to tell chat GPT, please cite your sources. I absolutely positively do not want you to make up any sources. They must be true. Do you understand? It says yes. And it spits out an article with fake sources, right? What we call hallucinations. So yeah, in certain situations, if the content you're asking to write is very much based on press releases, very factual information, it doesn't okay job. But anytime it goes outside those boundaries, the hallucination problems and those types of things take root. And I was talking about this with some tech folks, it's like, it's almost useless at that point, right? Because to try to fix a problematic article by checking every source, you might as well just write from scratch yourself, right? So that's, that was one of the early experiments we saw using that. But that doesn't mean I'm down on the technology. It means you need to, you need to ask it the right things to get good results, right? So in terms of being creative and being advisory, I think chat GPT is huge potential. Then we went to chat GPT and said, here's an existing draft of Wikipedia article given our policies around PR speak about advertising copyright. And we said, like, I don't want to see these type of words, I don't want to see something sounds like a press release, do you understand? It actually did a pretty good job of saying, you know, is this draft good enough to be in Wikipedia? And we actually were playing around with folks in the community like saying, maybe we could use chat GPT for new editors to say, how adequate is my draft for Wikipedia? So I think that's a better use case than just saying mimic Wikipedia editor, which is not a very satisfactory thing. And the last thing I'll add, I'd love to hear other situations here about how these are used is when used in the area of wiki data, I was very impressed with what chat GPT could do out of the box because in the museum and library world, we often have what we call kind of fuzzy metadata, right? You have an art historian saying, this artwork was created in the first half of the 17th century. And you're like, how do I put that in wiki data? Because wiki data doesn't like first half of the 17th century. It wants to say start date, end date, and then what we call sourcing circumstances like circa, maybe probably those types of things. And if you feed chat GPT, like 50 examples, saying that when you see first half of 17th century, you should put this into wiki data, what I did using quick statements, if you know what that is. And you fed a lot of things like, here's the exact date, here's the probably date, here's the first half date, here's the latter half of the 20th century. And you gave it a lot of trained data. When we fed chat GPT, new dates had never seen before, it did an extremely good job at spitting out wiki data, quick statements that satisfied what we wanted to do. So that's my, I guess my bright spot story in that for transforming data, huge potential, especially for saving our folks in our community that don't really know how to program or don't want to learn how to write Python scripts to say, hey, chat GPT, here's 50 examples. If I give you a new thing, can you do the right thing? And it has done the right thing in many ways. And that's really exciting, at least for me, to see that this is a bridge for people who want to get the advantages of coding without needing to learn how to code. And that's exciting to me. Thank you. So we've heard so far from only two panelists, but you can already appreciate the array of different things that we've been kind of doing. And I'm wondering before we move on to the next topic, if any of the other panelists have an additional use that we haven't mentioned that you want to mention, Jimmy? Yeah, I just wanted to talk a little bit about a use case outside of the Wikipedia, Wikimedia contact. So at Fandom, which most of you know is a for profit wiki company with pop culture and so on. For a long time, the community has been very eager to have quizzes, the sort of thing we wouldn't really consider having a Wikipedia, but the quizzes are fun. The community enjoys taking quizzes. They enjoy competing with each other and so on. And so, but they're very, very, you know, even though the content at Fandom is more casual in a way than Wikipedia, they still like at Wikipedia, they don't want you to get anything wrong about Star Wars. So the community enjoys taking quizzes and they have said they would like to make quizzes, but actually not many quizzes get made. It's not that fun to make a quiz. It's a lot of work. And so we've been experimenting with using ChatGBT to generate quizzes where we feed it an entry on Darth Vader to be simple and say, give us a 10 question multiple choice quiz with answers in JSON format so that we can quickly import it into a quiz tool. And again, it hallucinates. It gets things wrong. In the earliest attempts, it would get sort of three out of 10 questions would be just completely crazy and wrong. But with some refinement with some yelling at it, as Andrew sort of described, the team is getting it up to like 98, 99% accuracy, which is actually good enough that then the community can go through the quiz and just delete the ones that don't make sense. And so suddenly there's this ability to generate huge amounts of quite interesting and entertaining content that's reasonably accurate and so forth. So it's a different kind of use case from what we're using or what we're thinking about. But I thought it was very interesting. And obviously in our context, maybe there are some uses in educational settings and things like that for that that kind of sort of repurposing Wikipedia content in a new way. Absolutely. And I think what I've been hearing from, you know, just the beginning of this panel of the current uses to me says, you know, we live in a really exciting times in that sense. It's the rise of the machine people. And we get to see it live and experiment with it. There are some benefits. There are some challenges. For sure, there is a learning curve, right? And I think some of what we are seeing here is that we as a community, first of all, need to really experiment with these tools and then try to understand how we can apply them to what it is that we are doing. And we see all sorts of innovation, including quizzes and just things we wouldn't have thought about originally. And I know Gaulder hasn't spoken, but your Wikipedia has been creating mythological pictures using me journey to illustrate articles. And you've been experimenting with summarizing some of the text, etc. But I don't want to ask you about that. I want to ask you about some of the challenges because I want to move us to some of the issues that we've been seeing using these different platforms. And certainly it's not just chat, GPT, right? There's such a huge array of tools out there from create. Yes, the chat, GPTs and Claude and Bard and all of these, but there are also ones that create PowerPoints and quizzes and pictures, etc. And it's really endless and voice. They're they're really automated animations and really things that we haven't really considered properly for for our world. So I want to ask you, as someone who has been experimenting quite early on with these types of tools and doing quite innovative work in your community, what have been some of the challenges that you've been seeing? Okay, so I had prepared my first experiences, but let's go to the challenge. And one of the challenge I wanted to mention here will be let me introduce a little story. My elder daughter, she's six years old, and she came home this year like they had four first time computers in the school because we six years old is the first time they had it. And she came very excited, saying, do you know that there is a page in internet where you can know everything? And say, oh, wow, which is the page? No, it's Google. So she said, oh, you can ask anything at Google and it will give you the answer. So I would. Okay, let's ask. So she asked it where this is her actual first question. Where do panteras sleep? That was her first question. And it was a really strange question. And the funny thing is that if you write it in English, you will get an answer. But if you write it in Basque, you won't get an answer. If you read the Wikipedia article about panteras, you will have a really huge taxonomy article. But not an answer. And if you go to the actual panteras he was thinking about, that is the Jaguar, you won't get an answer. There is no information about where panteras sleep. But chat GPD says it correctly that they sleep sometimes in dens and sometimes in trees. So you can't get the answer from Wikipedia and you can't get the answer from Wikidata and you can't get the answer from Google if you don't type in English. You actually can't get the answer from Commons because we have a lot of images of panteras sleeping. That will be one goal how to also find these kind of answers. The second one is related to this is languages. You can have really good answers working in English. You can have fairly good answers. I have tested working in Spanish, French, and I assume that there will be also good answers in German, Russian, Chinese, Italian, maybe. I mean, the most spoken 10 languages, but you won't have really good answers in all the other 7000 languages in the world. So whenever we are talking about let's do something we have to be aware that there is a highest speed road for English. There is like a middle road for other 10 languages and there are no roads for like most of the languages in the world. And the other problem that we are facing is that as we are using automatically created text everywhere like automatic translations and so on and so on. The next models will be trained in the things we have created and the things we have created has been created by a machine. So the next language model will think that the machine created test is the correct one because it's the one which you are training with. I don't know if you understand this, but every iteration will be the language poorer than before because it assumes that that is the correct way to say things. So if we only train, if we create a whole Wikipedia with chatGPD, the next iteration will be based in what chatGPD thinks is the correct way of writing. That will be very good in English, it's great, but it's very poor in other languages. So I think these are some of the things we have to be aware. Absolutely. And I think in that sense, you know, talking about some of the challenges that go really beyond just different languages, right? Maybe the most essential issue for all of us Wikimedians is provenance, the fact that there are no references, right? Like, to me, as an educator, as a researcher, and as a Wikimedian, the first thing that I look at in any text is show me the reference. Where does that come from? How can I verify that this is true? And in essence, what all these GNI tools are doing are creating another barrier between the consumers of knowledge and the information itself or rather the source of that information. And that has been a huge, huge deal for our community. We're now trying to tackle it, at least experimenting with it slightly. And maybe Leila, you can say something about the plug-in for Wikipedia and other efforts. Sure. So the Wikimedia Foundation, as some of you may know, is experimenting in the space of LLM, large language models and generative AI. The foundation recently launched the ChatGPT plug-in for Wikipedia. And if you now go to the ChatGPT tab corresponding to plug-ins, you can get the Wikimedia plug-in. All the Wikimedia plug-in does is basically it creates a prism that you can basically use to focus on Wikimedia content. So as a user, you can ask a question of ChatGPT and then you can also ask the question of ChatGPT using the Wikimedia plug-in and then get a sense of what would happen if ChatGPT would focus on the Wikimedia content. In other words, what is Wikimedia's perspective on the answer to the question that the user is asking? So if you have not experimented with it, I encourage you to experiment. This is definitely an experimental plug-in. The purpose is to try to understand how people engage with this kind of technology and what are some of the affordances that we can have. As Shani mentioned, the issue of provenance and verifiability is core, of course, to the work that we all do in the Wikimedia movement and particularly around Wikipedia. So on that note, I will say that you're looking at applications of large language models and generative AI that can help us innovate basically in a shorter-term, medium-term and also think about the longer-term problem of provenance as we move forward. In the short-term, we think that text summarization is an area that we should invest in and we actually are investing in right now. Text summarization can have many different applications. You can think about, I mentioned this in the previous session, the perennial sources discussion on English Wikipedia, for example, has been going on for more than two decades. When we were last in Stockholm for Wikimedia, one of the requests from the community, parts of the community on English Wikipedia, was can we receive like summaries of these discussions? Because if you're a new person going to that page, you have to read basically two decades of discussions to understand where we are, unless the editors have done the manual work of creating summaries in the perennial sources. So this is one of the places that large language models can be utilized using the existing content that is on Wikipedia, in this case in the discussion pages or in the policy pages, and then surface the summaries of what has been discussed and also say where the information is coming from, because in this case it is very clear. They are basically the technologies processing the pages that are on Wikipedia and it can actually point back to where the information is coming from. So we're looking at, you know, text summarization as one piece of technology. Of course it has many applications. It can be related to vandalism detection or other, you know, article description, summary of edit descriptions or other applications. Where are the places that we can use this technology to reduce some of the load that is on the existing editors on top of thinking about future audiences? Thanks, Leila. And I think, Tillman, I know you've been experimenting quite early, right? As early, if I remember correctly, as July 2022, even before Charge.GPT was deployed with all sorts of Gen.AI platforms. And I think you can tell us a bit more from your perspective about the different challenges that you've encountered. Yeah, I mean, I was referring to an experiment that the signpost did by myself, but which direct us to Leila's point about text summarization. So the signpost has these arbitration reports, which is about reporting what the arbitration committee decided. And it's always very formalistic, like they sanctioned this user because of this evidence, et cetera. And actually went with GPT to those before the release of ChatGPT and in work pretty well with corrections, of course, not unsubmarised. So I think that was an early example of the value of summarizing our huge amounts of discussions we have on Wikipedia, right? I think that's kind of what Leila is referring to. I do want to mention, I think that ChatGPT plugin, yes, you should try it all out. I think it's generally only available to paid subscribers still, right? We pay $200, but I think the foundation has this offer that if you email that can send you an email with a signup link for free, is that correct? So just keep that in mind if you try it out. So you don't necessarily have to pay the $20 to try out the Wikipedia plugin. And lastly, I want to mention a general remark about these limitations and challenges. And we really need to be aware that this is a field that's really moving very quickly. And for example, yes, it's right that ChatGPT doesn't cite sources and it's outdated. But we've had the Bing AI, which Bing is actually citing very current sources, right? So if you want to try out a writing Wikipedia article, you should actually not use ChatGPT, you should try Bing, which at least tries to give you current references, which may still be hallucinated, but you have a better chance of getting this to work. And also looking forward, it's again, things are moving very quickly and the people who are involved in this field, even the full-time AI researchers are kind of overwhelmed, right? There's several artists who are like, oh, God, I don't know what's going on. And every day, there's new research results and new products. And we all need to keep this in mind, what's not working today may work tomorrow or what's not a problem for us tomorrow. Today may become a problem tomorrow, so I just want to advise some general caution and open-mindedness. Thanks, Thielman. And I guess this is again, just the beginning, the tip of the icebergs in terms of the challenges that we're facing as a global community. There have been additional ones mentioned in today's session, including attribution, including the whole issue of copyrights. There's a whole issue of the general public simply not understanding that they need to critically look at the content. So there's a digital literacy or rather, generative AI literacy that we need to kind of attend on a global level to new audiences communicating with these tools and how it's impacting us because more and more they will go to different tools and less to our platform. So it raises all sorts of questions. And I guess that brings me to maybe the core reason why I wanted to actually facilitate this session and have this global discussion and for you to start helping us mapping more systematically both what is already happening but also the challenges is that we can also then take the discussion one step further and kind of talk about what are the needs. Leila kind of prompted us this morning to ask to request to and to request we need to kind of agree on what it is that we need. And I can start us off by saying, you know, Andrew mentioned wiki data and I see Danny here in the room. Right now, generative AI, large language models are not using wiki data at all. They're just using wikipedia, tiny portions of wiki source and tiny portions of one of our other projects. I'm not sure which comments maybe but they're not including a huge amount of what is it now over 105 million items on wiki data last time I checked and now wiki functions launching. So there's the whole discussion of how this whole world of linked open data is connected to generative AI that to me is an unsolved issue that we still need to kind of grapple with. So that's another issue down there down the road that we kind of need to explore. But I guess to me it has to begin with simply mapping out what's happening and maybe it's a good time to say we've been doing again stuff in different corners of the movement. There are some wiki pages. There's a telegram group. You're all welcome to join. There's a Facebook group. If you don't like telegram or vice versa, there are various corners but we don't have one space where we all come together to talk about these and I think we should. I really do. So this is about you, all of you including the people at home and we want to open the floor in a sense to hearing what do you think should be in the next steps. I'm going to start us off actually with the panelists. So I want to ask each of you to say one thing that you think is like tomorrow morning, what do we need to do as a movement to start strategically thinking about us? What do we need? What is missing right now for us to tackle it in a more strategic way? I'm just going to do it by order. So Andrew. Oh. Not to put you on the spot or anything. Yeah, I think I guess this is looking beyond the chat GPT horizon which I guess I don't know lately you can tell us more about it but when I was at the Hackathon in Athens earlier this year, I was very encouraged to hear that there's going to be a adoption of a framework to experiment more with machine learning, right? This is called lift wing. I believe it's the first time I ever heard it was in Athens earlier this year. How many people here have heard of lift wing? Okay, not that many folks, right? So I thought it was pretty cool. So you may know that we already have some AI being used in the movement at scale like ORS, the Objective Revision Scoring System we have is its own machine learning system and there's other machine learning systems. So I think lift wing is a way to have a common platform to plug in and have these places in one space and potentially that'll be a place where community members can write things that plug in as well. So I think that's what I'm most excited about is providing the same way that we have things like pause is like a Python platform for our community to easily write Python scripts. If we have a framework like lift wing that allow people to very easily not have to buy a subscription somewhere or define computing power on Google Cloud and we actually have a framework for people to start experimenting I'm excited about that potential. Thanks, Andrew. So I'm hearing a platform to experiment with as a first step, Layla. Yeah, and maybe just to follow on what you shared, Andrew. So if you haven't checked out lift wing, you can. It's online and there is, I think, I believe there's also a session on the topic. I'll come back to you with that. However, what I wanted to say about lift wing which is a technology developed by the machine learning team in the Wikimedia Foundation is that the team is really centering their thinking around how to enable the community to govern the machine learning models that the foundation or AI models that the foundation or the community develops. This really is anchoring right now around the one piece of way of thinking which is called model cards. These are some of the latest recommendations by experts in this space that it is important for every machine learning model or AI model that goes to production to have a model card where actually it is clear where the data for this model is coming from, what is the technology that is being used for developing, for training the data and developing the model. What are the potential implications of using this model? So these are all documented. So if you look at for model cards, Wikimedia page on meta, you can see which models are right now in production and what are the model cards corresponding for these models. And I really applaud the work of the team on this front because it's really opening this space for everyone to engage with this piece of technology in terms of governing it, by the community members, for the community members and the projects. Maybe if I can go back to your question, Shanee, about what is one thing that we should talk about? It's a hard question. It's in my heart to say it since the previous session so I say it here that I think we haven't really talked about Wiki Source and the importance of Wiki Source in this conversation. As we're talking about languages and the potential of leaving languages behind, we should remind ourselves that a lot of the content from a lot of the knowledge that we're talking about is not actually available anywhere on the web. And Wiki Source as our online digital library is going to play a key role for helping that kind of content to even come to the internet and get connected and be on the web so that we can talk about the kinds of technologies that we're talking about. So eventually we can build if we want to and if it's useful, large language models for other languages. I'm really hopeful that Wiki Functions at some point will help us address that. No pressure, Danny and James. But yeah, I think that in the further future we will have to address the language issue more systematically and Wiki Source is part of it, Wiki Data as well. And by the way, how cool is it that chat GPT can now give us sparkle and like, I love sparkle, but we don't need to know it anymore to actually write queries, just in parenthesis. Galdor, what is your wish? So in Buzzweek Media, we are now with a project of multimedia content like pedagogical videos, say, added to articles and which are especially expensive. So we have been testing an idea about what about automatic podcasts? I mean, you fit an article and you get a podcast with a song logo, we have a song logo. So you can hear it like every morning in Spotify where you are driving because you can't do that. I mean, it's possible to do that. Well, this is possible in English actually. We have this tip and it's great, but only in English. Well, it's actually possible with some other languages, but not for us. So, could we do that? Could we do something like that, like the featured article, for example, in any Wikipedia could be summarize, text-to-speech, add it with a music, you select free music from Commons and then you have it like in other media, like in Spotify or whatever podcast or whatever you need for that. And this is something we are researching, like not heavily researching, but we are trying to do that. But is there a specific need that you see right now? Yes, the problem is that we can't do that in other languages that are not English. And the problem is that summarizing in other languages is very difficult. The problem is that text-to-speech is available in Basque, but it's not available in most of the languages in the world. And you have like 40 different speakers in English, like with different accents, but this is reduced to one and very computer-like robot-like in most of the languages in the world. So, I think that if we want to do something like that, we need a lot of steps that are beyond what simple Wikipedia's, even if you have some money to invest in that can do, because we are talking about very large models, we are talking about technology that is out of the scope of any user group, and it's something that Wikimedia Foundation could invest, but not only the Wikimedia Foundation, it's something that goes beyond Wikimedia Foundation. And I think that, well, I don't know how, but let's dream on the kind of ideas. To me, by the way, that also implies having better tools to monitor the content that is, the flux of content that is being produced by these machines, like in a more efficient way. I mean, we've been doing some of it already, but in a sense, maybe working on that as well to make sure that our volunteers, I think it was mentioned also in the morning, making sure that volunteers do less mundane technical stuff and do more things that require actually a human brain. So, helping our community with tools would be some of the implications, I think, of what you're talking about, but I'm not sure. Partly. So, so. I don't actually know. But what do we need to do that? I mean, it's beyond my... We can imagine things. Actually, we can do it by hand, and we can fit an article, summarize, translate, fit two artists to speech, then add somewhere in commons, and then build a podcast in Spotify. But that's not the idea. The idea will be something like, for example, anyone in the world can open an article, just click at the button and have a good summary of what they are, I don't know, cooking or driving or whatever, or just working in the factory and they can hear knowledge. I mean, that idea will be different, and I don't know which infrastructure we do need for that. It's way beyond my knowledge of what the infrastructure we have, we can have, and they are actually possible for the Wikimedia Foundation. Thank you. And now that Mike is moving to Jimmy, I want to invite the audience to start writing questions if you do have them, so we can see it and address it in a few seconds when we finish the round. So if you do have questions, post them. Jimmy, what is your wish? Well, I mean, I think I want to leave enough time for the audience, so I don't have anything specific to add. Other than to say, I think we all need to be really actively, intellectually engaging with what's going on in this space. It clearly is going to have huge impact on our work, there's huge opportunities and huge risks. And so I'm happy that we're having this session, but I also think in two years, three years time, we're going to be having a load of sessions and a load of different things going on that in some way are impacted by this generative AI. Thanks, Jimmy. Netha. Yeah, I have something to add on to what Galdar and Shani said earlier. This is concerning myths and disinformation. So we kind of know that with generative AI, it's quite easy to turn out a lot of content which looks really convincing. And in future, or already in the present, there are probably people who are generating content from ChargeGPT or any other generative AI platforms and trying to put that content up on Wikipedia. And it's quite hard to detect if that is myths or disinformation because they sound very convincing and it's not exactly blatant misinformation. People are probably putting that there in good faith and they don't really know that this is misinformation. So my wish is to have more tools to be able to detect this kind of subtle misinformation and also to revise our policies and to establish new practices where you can easily detect, more easily detect misinformation and reduce the burden of volunteers to keep on the watch, be vigilant for this kind of vandalism. Thank you, Netha. Tillman, last but not least. Yeah, I want to encourage us all to think a bit more what it means from the reader perspective which we haven't talked a lot about. So we tend to have a good understanding of how people read Wikipedia articles. There's been a bit more progress in recent years. But I've seen several anecdotal reports of people saying, yeah, actually I find myself only going to chat with you instead of Wikipedia now. And yes, I know it's hallucinating, but you know, Wikipedia has vandalism too. And also, yeah, we're not the only website on the web where people go to for information. And if you look at, for example, Khan Academy, which is a bit, which is like our competitor, but if you want to learn about the topic right, you can go to Wikipedia or you can go to Khan Academy, watch their videos. And they jumped head into this, why they built an AI assistant based on GPT last year. So there's a big contrast to our movement, why we like kind of set back and let it happen. And we'll look at Stack Overflow where we have been very concerned about where their readers go and there's a discussion which is happening. So yeah, my wish is to get a bit of better understanding if the readers are leaving us for chat to what happened. I mean, maybe the foundation has looked at this. I mean, it wasn't mentioned as far as I'm in this morning session, but if there's data on this, what if you're losing readers traffic and then secondly, just experiment more with reader tools. And there's some things like I cash people to check out, for example, Santos question answering bot experiment, maybe I don't know if you'll mention in the talk, but there's some experimentation going on, how we could integrate generative AI with our articles for better reader experience. Thanks, Thielman. Thanks to the whole panel. I guess I wanna throw my own ask and mine is quite a humble one. I think we're at the very initial stage of like this global discussion and we should be much more proactive. I just want to have a space like one designated place where we can curate all the work and make sure that we engage everyone for a starter. So that's my ask and I wanna include a comment that we got from our online audience and this is someone saying, thank you first of all for engaging. Should the movement play an influencing role highlighting responsible tech or AI for tech organizations of all kinds? I think that's a great question. I think we've been already doing it and will continue to be doing it, but maybe Jimmy, you wanna say something about that then Leila? Yeah, I mean, I think that the question, I mean, to me that's an obvious yes. I think we have a strong voice about technology and ethics and being responsible and being thoughtful and I think we are already are. I mean, when I had a call with Sam Altman and my very first question to him was, how is the work going in terms of not hallucinating references because that's for me one of the most problematic aspects of the current thing. The answer is it's a really hard problem, but yeah, I think we definitely should play a positive role in talking about positive uses for technology and education and so forth and responsible uses. Yes, and in that sense, responsible to me expands to free also because I think one of the highlights of what we do is making sure that every single human being can access knowledge and I think what we've been seeing growingly because these tech companies, all the chat GPTs and et cetera are managed by for-profit companies. The reality is that some of it is free but many, many of the attributes, the possibilities are locked or require further payment. So to me, that creates an even wider divide in the world between those who can pay to get access to accurate knowledge and those who can't. So that's another thing for us to kind of consider as well as advocates for freedom of knowledge as a basic human right. Leila and then Galdur. I think my answer is also yes. I think the question is like how much more and in which ways some of the work that is already happening as some of you know, I mean, we as a movement and Wikimedia Foundation as a foundation, we take our values or guiding principles seriously and we put those into work in the context of AI and large language models. For example, the legal team in the foundation is working with the research team and machine learning teams on developing AI human rights checklist. This is a checklist that is going to help us understand what type of technologies are we going to basically productionize and put on lift link or other platforms that we offer in the future. And this is going to guide us and this is the kind of thing that at least by doing it we can help influence others and lead other companies or organizations. Communications plays a key role in this space. We can do so many amazing things but if we don't communicate and we don't communicate proactively and when I say we, I mean all of us with all the different affordances that we have where we are in different parts of the movement. So I think communication is also a key aspect for influencing but we should definitely do that, yes. Thanks, Kaldar. Okay, this leaves us roughly 10 minutes to hear from the audience. So any questions, any comments? Yes, you, can you bring a mic to him? There are mics, okay. Hello. Hi. Jonathan, Director of Engineering at Wikimedia Deutschland. I work in AI as well and with AI ethics sector. I was wondering about the future of not just Wikimedia but our effect on society in general. Two factors, positive and negative was curious to your input on this. One is not just all large language models ability to proliferate disinformation and when I started at Wikimedia I found it to be a balancing act for disinformation. I know that some can be passed through Wikimedia or Wikipedia but I was wondering if there's a way I think the AI checklist would go along with this as well but just the idea of how we could be in elements against disinformation now that the large language models are hugely like can emphasize it dramatically and in a slightly different direction and pick your choice but on what's like larger population Wikipedia articles if you want to retrain a large language model you only need a small amount of data and so you can take what the English large language model is doing and then convert it to many many other languages depending on that scale and I agree. I was wondering if we've looked into how much of our data can be used to do that. How many Wikipedia articles do we need to retrain an existing large language model on BASC since you're here. Whichever direction you wanna take. Who wants to take the first one? There is a small problem, not only about languages it's also about what each language or culture will highlight on a topic and I mean we have been doing that for example summarizing science articles for children in Chiquipedia and our children's encyclopedia where there is not a large bias like I mean the article about Jupiter will be slightly similar in every language in the world but not for other topics and maybe introducing a cultural bias will be also a problem there but that's something we have to figure also with Wikipedia and with many other things. Yeah I guess in a sense to your question to me it goes beyond misinformation, to me there's an unholy trinity of misinformation, disinformation and actually missing information and I think our community has been trying to do slowly, slowly, by and by in various languages consistent work to close different gaps. We have tons of work ahead of us but I think it only highlights what's happening with Gen AI is only highlighting the importance of what we do and why it's so crucial. Netta is sitting here, she's systematically self-handedly curating medical content related to women and consistently working to close gaps in that space right and that's just one tiny example. There's so many other gaps, so much missing information also and that gets perpetuated and internal bias that we all know from society, from our history as a human race gets perpetuated by these machines that don't know any better so in a sense our role has never been more important than it is now and to go back to the first comment of do we as a movement have a say? Absolutely, who else if not us really? Like who else is able to tackle it on a global scale than all of you here in the room? I think it's our job and I would argue actually that we are probably better equipped to do this than other folks just because we have the understanding of hey, what is references? How do we construct knowledge properly? So any other comments from the audience? Yeah, Doug. Thank you, I've got a comment and a question. The first comment is you mentioned sort of the list of platforms which the community has to discuss artificial intelligence. I just wanted to add one more. Since 2018 we've had a artificial intelligence group on Meadowiki and we've got a plushie, the dodo. The question I have is could anyone just summarize what is the current resembling consensus on how to use AI to edit Wikipedia at the moment? Is there even a consensus? I know there's a lot of discussion on this issue. Okay, I can try to tackle it because I've actually been beginning or attempting to research the topic in various studies that I'm conducting at the moment. To my knowledge, which is limited and here all of you are to help me if I'm incorrect but to my knowledge there is no policy. There are individuals working and experimenting but we don't have policies. We don't have case studies like bits and pieces of it, right? Which is why I wanted a panel. Like we need to be so much more coordinated in this work and so much more strategic. So it's in a sense just the beginning. We're in an experimental mode and different people have been experimenting and sharing their experiences in various corners but I haven't seen just yet like full policies around changes of policies or tools at the beginning. We see the beginning of experimenting with tools but nothing so massive coming from the community just yet that I'm really hoping will spark that, yeah. I mean, if I could just say a lot of our existing policies and practices already helped quite a lot. In other words, things like when you make an edit to a Wikipedia entry it should be well sourced. It should be accurate. It should be true and you're responsible for what you've done there. That remains true and that's actually very helpful. So it's a reason why we don't necessarily need a rule that says don't just ask chat to BT to write an entry and then just copy paste into Wikipedia with made up sources. That's already against the rules because it's gonna have made up sources. That doesn't mean that there won't be any sort of tweaks. I mean, I think certainly we have bot policies that vary across languages. Can you use a bot to do X? Well, can you use a bot to spell check? Well, yeah, but there's parameters around that because spell checkers are sometimes wrong and so forth. Can you use a bot to whatever? And I think that those will begin to emerge. The kinds of things I'm interested in is using chat GPT and AI to suggest things to editors. I think there's a real opportunity there and I can imagine someone making a bot that only writes on talk pages that says, hi, I'm the bias bot and I found these statements in this article that don't seem to be supported by the sources. Maybe a human might want to look at that. I think people might welcome a bot like that as long as it's generally not a waste of time and so forth and we don't yet have a policy or need a policy about things like that but it's the sort of thing that I think is going to emerge over time and that fortunately nobody thinks, great, we'll just use bots now because that would be a disaster. Thanks, Jimmy. Yeah, I would just add that. Just a second, we have two and a half more minutes and I promise, Thomas. If you go to the English reviewer, W, colon, LLM, that's a policy job which is pretty far along as further reading to cover some of these use cases. Thomas. So there's been some mention about how LLMs are not amazing for generating new content from scratch but they're all rights currently reformulating existing content into different formats. One of the things that Wikipedia in particular traditionally is terrible at is video content and there have been some experiments in the past but they've tended to be quite manually intensive. Have you seen any experimentation using generative AI in any sense to reformat Wikipedia articles specifically into video? We are doing it at Buzz Wikipedia. I can't talk to you later because we have one minute left. Talk to you later. Just a quick note, I admire Bass, Catalan and those Wikipedia's can move quickly and try to experiment with a lot of stuff. I don't think you have a hope in English Wikipedia that they have a very big bias against visual content. We need to fix that on another occasion not during this panel but I think Thomas to your question we're at the very beginning of experimenting with let's say generated avatars or generated videos even from text which now there are specific tools that can do it to illustrate certain articles. I'm sure the community will discuss and continue to do that but that's a good question, thank you. Last remarks, we are less than a minute from finishing the panel so I want to thank everyone who participated. I think we've just scratched the beginning of this and I want to first of all say a huge thank you to Andrew and Leila and Galdor and Jimmy and Netta and Tillman for joining today and sharing from their experiences. I'm hoping you're all going to help us curate some more information online and that we can continue as a movement to have this discussion of what it is that we actually need to have as next steps. Please stay in the room if you're interested in the topic because we're going to have another session connected to AI just now. Enjoy the rest of the day and thank you. Shani, one last thing. There was interest in people wanting to continue the conversation so if you want to do lunch back there in the roundtables you can grab a bento box and people want to talk about AI you can lunch back there, I'll be there and some other folks will be there too. Perfect, thanks.