 Hello everybody. Hello everybody, welcome to the Metros meeting. Thank you all for coming. Today we're going to start with our usual welcomes and anniversaries. And we're going to have a strategy update from Lila, community update from Lewis, a research presentation from John Morgan and product demo from who again? Mitri Brands. That's who. That's who. Thank you for reminding me. And then we're going to close with UNA. So first things first, welcome to our new hires, Liz Lard and Tyler Kulcher, Lindsay Ann Frankenfield over in editing, Aaron Palmer and Beagle and Joe Madazzoni. So in editing. And then we have new, and we are also welcoming Jessica Lagarde and Eibishan patients. And we have some anniversaries to celebrate as well. Anniston with us for five years. Anniversaries are next to UB for four years. That's flashing for three years. Boys, Nuriya, Shara and David Chan for three years. And T. Marco Mirzar in three years. And with that, I will hand it to Lila for a strategy update. Thank you. Good morning everybody. Thank you for being here. Very, very brief strategy update today. As you remember about a month ago, in the last metrics meeting, I put in front of you the three key pillars of the strategy. Since then, you're giving me a lot and a lot of feedback. So I've taken that feedback and here's what I've advocated in the two major areas. Most of the requests basically came in asking for one more transparency. The lack of visibility into the process and collaboration with the process created some of a lot of confusion. And so if we increase transparency and increase collaboration, we should reduce the confusion about the process, the process and ultimately the outcome of the strategy that we want. So that's one of the things that we're going to address and we have started on that path already. The second one has to do with more focus in the strategy. Because we're such a large movement and we have so many opportunities, we can really do so many different things. And that makes it really hard for us to make decisions. So what I've heard from you, what you said to me is the strategy when we put it together needs to enable us to make some decisions and make those decisions clearly and at every level within the organization. It also needs to be obviously aligned with our vision and with the current and urgent needs of our users. So what have we done so far? On the transparency front, we've actually made some movement already. We've set up and met with the collaborative Flatbus team. Those of you who have volunteered for it, thank you so much. You already have provided guidance and we're going to be coming back to you for more. Maggie, a special thank you to you for drafting a possible strategy process and Kevin for helping us move forward. And we took that process and we evaluated possible timelines that this process may take. Then at the group, you, the Flatbus team, those of you who have participated, decided to place that process on meta. Place is important. It needs to be public. It needs to be visible. It needs to be easy for everybody to participate from different projects. So that has been decided. I have spoken with a possible strategy consultant who can help us through the collaborative process in the next quarter if we decide to have help like that. If we decide to go that route, a few of you will speak with her and interview her and make sure this is the right course of action. Finally, determining which process to follow. Really important as well. And this is something that we need to make a decision on and move forward. There are two possibilities. Some of you already know. One is a long process. It will take 12 months and give us a lot of details and tactics. And another one is a shorter process that will give us strategic direction, including our top priority goals, which each one of your teams can then take and build your own priorities and your own project plans again. So that would be distributed to your teams and give you the ability to make the decisions. So on the guiding focus, what's next on that front? There is a, as you remember, the three pillars. And these are actually the same pillars that you see in the previous strategy for similar, very, very similar ones, is reach, community, and knowledge. And of course, all of those are fundamental to our movement overall, to what we do. We can't neglect any one of those. From perspective of the WMF as a part of the overall movement, as our organization, we need to decide where we can make the most impact and really create a guiding focus in that area. And that is something that's going to need to do in the next couple of weeks. So what are our next steps? We need to choose that top-low priority. We need to get it down by the end of the year. That will help us all make choices and set priorities. That is true to our vision, is where we as an organization, as a WMF, can make the most impact, as of course focused on our users' needs. Where is the most urgent user need today? And in parallel, we need to start planning the community consultation and that's the work that the Platypus team is doing, that the consultation will take place in Q3 to make sure that we have full participation around actually building collaborative and transparently strategic goals against the top priority. So that's where we're at. I wanted to give you this update. I wanted to let you know what changes we have put into place to help respond to some of the feedback. And here's the timeline for getting it completed. Thank you. Oh, but yes, this. So while we are continuing, I mean, is this my right mark? Awesome. The world outside these walls continues on without us. While we're very correctly focused on things like fundraising and strategy. So here's a quick glimpse into that. The WikiSource conference, some of you may have heard about this in the past few weeks. The WikiSource community together for the first time ever as an independent community, funded in part through our grant making process, to talk about the future of WikiSource, right? This is something, as many of you know, it's a project that has some very passionately involved people, has some great support, but hasn't always necessarily, you know, hasn't had the kind of growth you would like to see. So they wanted to talk, so they wanted to get together and talk, and they, by all accounts, it sounds like they had a really good conference. They had external partners helping to give them context about how those partners might use it. They talked specifically about identity and mission, our own, a soft help to drive that conversation. They talked about the technical needs of WikiSource. Some of you in product will be seeing some wishlisty kinds of stuff out of that, and I think we'll be having some interesting discussions about how we can help support that. You know, this is the kind of thing that we support because we feel that it helps that getting the right people in a room physically together, IRC and talk pages are awesome, but this kind of thing can really help set a community on the right path for the long run, right? By building those face-to-face bonds. I'm pretty excited to see what comes out of it in the coming weeks and months. Many of you here in the office saw this, but for those of you out on the stream, we had the stewards in a couple of weeks ago. This is a picture of, I think not all of them, I think several of them had left or didn't want their photos seen, but we discussed again, you know, I mean, folks are careful about their identities. A lot of talks about how we can support them technically. A lot of talks about how they can improve their own internal workflows, which I thought was really interesting, right? There's also, I have to admit, sort of interesting, simply watching some of them walk through the workflows, the complexity. I sort of knew abstractly how complex some of these workflows were, but it was really fun for me to see them in person. Again, the bonds that are formed with these kinds of things are really important for us, so I was really glad to see them here in the office, and we'll be thinking about how we can do more of these. We did, as some of you know, we do a feedback survey every year to see, you know, speaking of showing that these events matter. We'll, we do a feedback survey every year, and this year we did, we got some pretty cool numbers out of it, right? A huge number of people agreed that it was helping them gain knowledge from others. Critically, do I get the, oh yeah, I actually get a laser pointer here. A huge number of them made new connections that they feel are going to help them work in their projects over time. Things that were, unfortunately, I realize we have two spelling errors in the names here, but the favorite sessions to come out of it were bringing free education to the world. Many of you saw this. It was a keynote by Luis Van Aan, who's done a lot of really interesting, and to me, it was a very thought-provoking talk. My life is an autistic Wikimedean by Guillaume, which I didn't get to go to, but for those of you who have not read it, there's an essay version of it online. It's powerful and important, and you should read it. And the fun one, the coolest project is the Wikimedia chapters, right? Showing to each other what they're doing. You know, we were also, we did this survey in part to feed into an upcoming consultation on what Wikimedia is, why we're doing it, how we can improve it in the future. Keep an eye, as some of you will have seen on Wikimedia L just this morning, we're aiming to take that live in mid-December and run it through January. So keep an eye out for that. This is a fun one. So, John Cornyn, Senator, US Senator over there. I don't agree with all his politics, but, you know, there was, his article had some stuff he didn't like on it. He reached out to Wikimedia DC. You can imagine a whole lot of different ways that this could have gone. Wikimedia DC ran with it and turned it into something really awesome, right? They worked with the Senator's staff to simply educate about how Wikipedia works, how it can be improved, and they're working now on planning, hopefully, an education session for all of the Wikimedia staff, right? For all of the Senate staff, Wikimedia staff, hopefully, shouldn't need it. And that's the kind, you know, I think this is a really great example. I talked in the last slide about how one of the most popular talks at Wikimedia was people seeing what other chapters were doing. This is the kind of great thing that can scale across a lot of places and a lot of governments, right? You know, the local chapters like to talk to government, like to educate. We've seen similar things happen, for example, off the top of my head in Italy. I think it's a, you know, this is the kind of thing where boots on the ground do things that we as WMF can't necessarily do. Another example of the kinds of things we can't necessarily do, cross-chapter collaborations. You all have seen me talk a few times about Ibaro co-op and the Central and Eastern European group where folks are working with each other, helping each other out. We've got a couple new examples of that. I think it's a really interesting and growing area of community collaboration, right? The examples here are German speaking and Serbia and Bulgaria. You know, working together on projects of shared interest, which I think is a great way for them to multiply their impact, which is an important theme for all of us. Finally, a couple of fun milestones. Ukrainian Wikipedia. How many of you have edited in the middle of a civil war? Nope, nope, nope. For those of you who couldn't hear, Lisa said she feels like she's doing that every day. However, the Ukrainian Wikipedia has been doing amazing stuff of late, despite what's been going on, you know, in that country. There's 600,000 articles, which I think is amazing. So a big round of applause for them. You know, and that's on the big end of the scale. On the small, the still impactful end of the day scale. Some of you have heard some of your coworkers talk about 100 wiki days before an initiative where people just commit to write a new article every day for 100 days, right? On the one hand, it sounds really small, but on the other hand, it's 4,000 new articles that have been created, right? At least some of them with the help of the new content translation pool. So we have both these big picture numbers, you know, big active Ukrainian community, and also, you know, smaller groups of people taking their own initiative and turning out some pretty cool articles. So I admit I'm never going to write 100 articles in 100 days, but one of these days maybe I'll get to at least 100 edits in 100 days, and that, you know, hopefully will be a fun milestone and something that I think everybody in this room could easily do as well. So on that note, I will pass it off to research. I don't... Brennan, is that your switch? Yes. J-Mo. Hey, Lewis, how's it going? Can you hear me well? Yes. Perfect. So I think that I'm going to share my screen, and that way, let's see, that's not what I want to share. That way I can read my notes while I talk to y'all. All right, can everybody see the research slide? Yes. Perfect. All right, I'm Jonathan Morgan with the Design Research Team. Today I'm going to share a story with a happy ending with you. It's a true story about editor retention, collaboration, community health, and about what it feels like to be a Wikipedia newbie. It's also about a place called the Tea House. You're probably already familiar with the general background of this story. In order for Wikipedia to survive, a relatively small percentage of the people who read the encyclopedia need to try editing it, and a relatively small percentage of those people need to continue editing regularly for months or years. Unfortunately, while our readership has grown, the percentage of people who try to edit, and the percentage of those people who go on to become Wikipedians has been shrinking over the past eight years. In part, this is because there is a steep learning curve of becoming an editor. You need to learn how to use the technology and how to play by the rules, both of which are rather idiosyncratic. Another reason is that new editors are increasingly having negative experiences when they try to participate. Their articles get deleted, their edits reverted, and their user talk pages rapidly fill up with stern warnings from people they've never met. Even good faith newcomers who are trying their best are likely to be reverted, warned, and even blocked for mistakes they probably didn't know they were making. All right, so that's the background. The tea house, which I'm going to talk about today, is an attempt to address some of these issues, and thereby increase the number of newcomers who stick around and work on the encyclopedia. The tea house is a populated, user-friendly help space designed to provide peer support and positive experiences for new editors. When they visit the tea house, new editors can get answers to their questions, experience-friendly encouragement, and learn about the community behind the encyclopedia. The tea house was created in early 2012 as a fellowship project aimed at increasing new editor retention with a specific focus on narrowing the gender gap by recruiting and retaining more women editors. Although the tea house was launched with foundation support, it has been entirely a community run for the past three years during which it has served more than 8,000 new editors. The type of service it provides is reflected in a set of features that serve as the foundation for the tea house model. First and foremost, the tea house provides a question-and-answer forum. Here's an example of a question thread from last week. In this thread, a new editor, Kat Brewer, asks how to add sources in order to keep an article they've created from being deleted. Kat Brewer receives prompt and friendly responses from three experienced editors called tea house hosts who go out of their way to explain how to add sources as well as the importance of reliable sources. One of them even helps the newcomer by editing the article directly. The Q&A forum was created before we had flow, so it uses a simple gadget that allows new editors to participate in discussions without having to edit wiki text. In addition to a Q&A forum, the Wikipedia tea house provides a page where new editors can create public profiles sharing a little about who they are and why they're on Wikipedia. Guests can also view profiles created by their volunteer hosts. These are three of our longest standing hosts. When a Wikipedia decides to become a tea house host, they agree to live up to a few basic expectations. They agree to welcome everyone to be polite and patient with new editors and end with each other to keep their answers simple and easy to understand and to answer questions directly rather than simply linking to a bunch of policies. They also agree to notify new users that they've responded to their question since many new editors aren't familiar with concepts like watch lists and page histories yet. These expectations are not rules. Hosts aren't required to follow them, and there are no consequences if they don't. The final piece of the tea house model is direct outreach. Rather than sitting back and hoping new editors find their way to the tea house, we send a bot to place a friendly invite message on their talk page. So that's the basic tea house strategy for engaging and supporting new editors. In order to determine whether that strategy worked as advertised, in mid-2012, the original tea house team launched a set of surveys with tea house guests and hosts. Survey results like these here suggested to the team that the tea house was providing value to new editors. These two responses from our survey show that new editors appreciated the quality of the information they received, as well as the overall experience. Respondents remarked on the welcoming atmosphere, the ease of use, feeling encouraged, and feeling like part of a community. We also found that a lot of female newcomers were participating in the tea house. At about twice the rate we expected, suggesting that the tea house was especially valuable for underserved and underrepresented populations. But the pilot project was too short for us to definitively answer our main question. Does the tea house encourage more editors to stick around and become Wikipedia's? Fortunately, a few months ago, Aaron and I had the opportunity to start trying to answer this question. Even more fortunately, the tea house has prospered under community leadership over the intervening years, so the answer was of more than purely academic interest. We designed a controlled A-B test to evaluate the impact of being invited to the tea house on a new editor's likelihood of surviving as a Wikipedia. Our data set consisted of 14,000 new editors. 11,000 of these editors had been invited to the tea house. 3,000 of them could have been invited, but the invite was intentionally held back. That's our control group. For both groups, the criteria for receiving an invitation were the same. The editors had made at least five edits in the first couple days after registration. They hadn't received any red flag talk page warnings or been blocked from editing, so we could be reasonably sure that they weren't vandals. We examined how many edits each editor in our sample made during the three-week, three windows of time after the invite date. Three to four weeks later, one to two months later, or two to six months later. And when we counted how many people in each group had made at least one edit during each of these periods and how many people had made five or more edits, we found that a higher percentage of editors in the tea house invitee sample met our one and five edit criteria across all three time points. For two of the conditions, the difference between invitees and control editors was statistically significant. So in other words, we saw more invitees making edits across the board, but only in two of those cases could we be 95% sure that the increases we were seeing were more than random chance. In the other cases, we could only be 80% to 94% sure, which is pretty good, but not enough for science. So all focus on the findings were most certain about. Three to four weeks later, 10% more tea house invitees made at least one edit compared to those poor neglected editors in the control group. Two to six months out, the picture is even brighter. 16% more tea house invitees were making five or more edits. They crossed some invisible threshold and were much more likely now to continue editing for months and years to come. Now, a 16% increase may not sound like a huge deal immediately, but it's actually pretty exceptional. And here's why. New editor retention has been a recognized issue within our community and a high priority for a long time. There have been quite a few foundation and community-led initiatives focused on this problem, but we have traditionally struggled to show positive proof of impact. So it's especially exciting and inspiring to me anyway that a joint initiative of the Wikimedia Foundation and the English Wikipedia community has been successful at retaining new editors. Part of that impact is probably due to the design of the tea house. The tea house model I talked about earlier. But I credit the participatory design process by which the tea house was created for its ability to sustain its positive impact over time. So there are many ways to do participatory design. Here's how it took out in this case. The team proposed the project on the Wiki. We got approval to experiment with solutions to the problem we had framed. We involved editors early in the design and development, and we iterated on our design in public. We evaluated the success of the pilot project against our stated goals, and then we disseminated the results as far and wide as we could. At the end of our pilot, we quietly stepped back and let the community take over, offering support when needed. Over the past few years, the English Wikipedia community become more and more invested in the tea house. This chart shows growth in the number of links to the tea house from user talk pages over time, as tea house links have been included in welcome templates, and as more editors have gotten involved in reaching out to new editors in need. The tea house has become an important part of the overall Wiki ecology. Ultimately though, a 16% increase in retention isn't enough. Every year, fewer and fewer people become editors of English Wikipedia, and we can't maintain the quality of the 5 million articles we have, let alone continue to create the articles we need without retaining a larger, more diverse set of new editors. And we can't forget our other encyclopedias and sister projects, many of which also struggle with editor retention for similar reasons. With that in mind, here are a couple of examples of how we could scale this success to support more new editors better. Since the success of the tea house is at least as much about people as it is about technology, my first example focuses on programmatic work. The tea house was only able to get off the ground because of a dedicated team of staff and volunteers who shepherded it during its early days. Other successful, self-sustaining programs within the Wikimedia movement, like Glam, Wiki Loves, and the Education Program, have also followed this general pattern. A movement-wide tea house program helps support Wikimedia projects that want to build on Wiki new editor support tools of their own. It would help local partners organize their community around the initiative, deploy the necessary tools, and customize their tea house to suit their local wants and needs, and possibly even assist them in using the tea house as a launch pad for other editor engagement initiatives. We can also build on the success of the tea house as a product and incorporate aspects of the tea house model to our products and features. Here are two examples. Unfortunately, most new editors don't get to benefit from the tea house because they abandon Wikipedia before they get an invitation. The vast majority of people who make one or two edits never edit again. Currently, we don't invite people to the tea house until we can be reasonably sure that they're not going to be hostile or disruptive. Unfortunately, that has meant waiting until they'd survived in the icy waters of Wikipedia for eight hours before inviting them because we don't have a crystal ball that lets us tell a good editor from a bad one just by looking at a couple edits. Fortunately, we will soon have a way to do precisely that thanks to another foundation community collaboration. The Rev scoring system that was announced this week will make it much easier for us to distinguish between a good-faith newcomer who is struggling and a vandal who is intent on messing things up and reach many more valuable new editors sooner before they give up without attracting vandals to the tea house or wasting volunteers' time. However, even with better tools for identifying new good-faith newcomers early, we still won't be able to serve every new editor who needs help. The tea house is a finite resource. There are only so many hosts. We can't bring 5,000 people a day to a single Wiki project and we can't effectively support our increasingly mobile-first experience to every new editor. One way to do that could be to provide self-service Q&A. This is kind of like how new programmers use sites like Stack Overflow. You think of a question, you type it into a search box, find that 10 other new programmers have asked similar questions, and then you scan through the results to find the best answer. There are currently 12,000 questions in the tea house archive and more are added every day. We know that those questions have high quality and that they're phrased the way newcomers would phrase them. We also know that they're more likely to be up-to-date and easy to understand than help documentation. We could use Elastic Search to make these questions and answers easy to query and provide this functionality via a simple gadget that we activate by default for every new editor. Anyway, those are just a couple examples of how I think we might be able to build on this project. I'll wrap up this talk with a couple lessons that I've learned along the way in my fellow and in a volunteer capacity. The tea house reminds me every day that our projects are living systems. Each Wiki is a densely interconnected complex of software, content, and people. Building within such a system affords certain opportunities and it also presents particular challenges. At our best, we in the Wikimedia Foundation have been good faith collaborators with our volunteer communities and responsible stewards of software, content, and community health. There are many awesome successes, not just the tea house. So I want to close by inviting you to think of other successes, big and small, that you've experienced or witnessed during your time in the Foundation and in the Wikimedia movement and reflect on what we can learn from those successes and how we can use what we've learned to play our role even better. Thank you for your time. And yeah, if we have a couple minutes, my collaborator, Aaron Haffaker, and I are happy to answer questions. We're actually doing questions at the end. Oh, cool. In that case. Thank you. So next up, we have Demetri Brantz, who's going to do your products. Hi, everyone. Demetri here. I'm an engineer on the mobile apps team and I'm the product owner for the Android app. And we just wanted to highlight a few new things that we've been working on in our Android world. What I'll show is actually pretty similar to a lightning talk that I did a couple weeks ago. So sorry if I'm repeating myself to those of you who watched that talk. This looks like a slightly bigger audience than the lightning talk audience. So I think I'm good. Let me just shuffle some things around. There we are. So the Android team has been pretty busy this quarter. One very important thing to mention is the new content service that we've been working on. The content service that's built on top of REST Base and Parsoid and that's what the app is now using. Without getting too technical about this, the goal of the content service is to return the article content but also augment it with all kinds of useful metadata about the article and structure it in a way that makes it really easy to consume it in a single transaction. Not just our app but any other client that might want this kind of data. I'll just touch on the rationale for this. As we started adding more and more nice features to the app, what we found was that many of these features were requiring us to make additional network transactions to get all the information about the article that we couldn't otherwise get easily from just the article contents. Here's an example we show lead images at the top of articles in the app. In order to do that, we need the URL to get the proper resolution of this image for this device but then we also need the URL for a smaller version of that image to go to serve as a thumbnail for this history list that we have here. It used to be that we need to make separate requests to get those URLs whereas now with the service it can give us multiple resolutions of the image in one go which makes this a bit faster and more efficient. Another different example here if, for example an article has a recorded audio pronunciation of the title like this one does here the service can just provide the URL for that audio recording instead of us having to hunt it down in the HTML of the article. So all we need to do is drop that URL into a native component like we have there and just have it work seamlessly. You'll have to trust me that when I press that button I hear the voice in my head saying Titan. Another one of our newish features is link previews so when you tap an article you get a little preview of that link. So the idea is that you get the short excerpt of that article as well as a little gallery of images related to the article and more often than not this will provide enough context for you that you don't need to move on to the full article to understand what it's about. Of course you can continue the full article if you want to read it all the way through but this way what we're trying to do is encourage you to not be shy in clicking as many links as you want and just get the gist of them while being sure to not lose your place in the article that you were reading originally that's the key. And so once again thanks to the content service we're building link previews can be even faster and more efficient because it's combining all these bits of information into a single response. So we can see that these link previews are pretty much as performant as they can be on this android device. I know it probably looks a little choppy over the hangout but on my physical device here it's totally smooth. So as we speak we're working on developing additional features that expose and bring together more content like for example we're working on integrating with Wixionary you'll be able to highlight any word in the article and it'll automatically pop up a little short definition of the term from Wixionary we thought it would be a pretty natural combination of the two projects to be able to surface Wixionary content like that. Another thing if you go to an article that has a geolocation associated with it like a city or a place you'll have a button that will be able to pinpoint this article on a map or even give you directions to it. And all of this is once again done through the content service. So the android team has been sort of owning this. Got some echo there. And we're actually just in the process of rolling this out to a percentage of our beta audience to see how the service performs under increasing load but our hope is that a lot of other consumers can start benefiting from it like other native apps or any kind of future web applications and so on. And I was just mentioning maps and pinpoints wasn't I. So the one other thing I wanted to highlight is our new nearby screen. So previously the app had a nearby feature but all it was was a list of articles that happened to be nearby your location or like near your GPS position of your device. But now when you tap on nearby you get a full blown map with pinpoints on articles nearby you and in case you're wondering this is using our very own Wikimedia Tile Server running OpenStreetMaps you can pan and zoom this and you can rotate it however you like and when you move to a new location the collection of markers gets updated and when you tap on a marker you get the same kind of link preview that we saw when we tap on regular links in an article. Very simple. And really we should probably stop calling this nearby at this point because really you're able to zoom out as far as you like and go to a different part of the world and try to zoom in anywhere we want. Let's see what's going on here. There we go. And actually this can even work without the location permission that you usually have to grant to apps when you install them in the new version of Android you can selectively grant permissions to apps and even if you deny the permission location to our app this would still work and it would show your location in relation to these markers but that's all it would work. So we're actually deploying this to production or to the Play Store next week, end of next week at the latest. So yeah, this is the new nearby. So anyway I think that should do it for now. So if you have an Android device and you have the Wikipedia app installed you're missing out and I say that without any kind of bias or ulterior motive but seriously do install the app tell your friends and we welcome all your feedback. Thank you. I'd like to invite Lila and Louis back up front to answer questions and people can pose questions to the remote presenters as well. People locally here in the SF office please line up by their question mic over there and people attending Remote League can ask questions in IRC. Speaking of that, James, do we have questions from IRC already? Hello. Yes, excellent. I have several questions none of which this microphone, there we go. So the first question is for Lila it's from Erin. Why are we so concerned about focus? How do we balance focus and long-term planning with opportunities and adaptability? I think there are two questions in there. I think focus is working. I think there are a couple of reasons why we want focus. The range of possibilities of what we can do and the range of opportunity that we have is enormous and if we look back at our previous strategic plan one of the things that we see is that because it was hitting every single criteria kind of on the list it was really hard for us to deliver on all of them. So focus would provide us with ability to test and concentrate on impact in a particular area. And I think there was a second part to the question. How do we balance focus on long-term planning with agility and adaptability? That's a great question. Focus should provide us with think of it as a direction. But how you get to that direction is going to be up to every different team. When we speak about focus we are talking about the three pillars. We are saying should we focus more can we impact more our reach our ability to reach every human being can we impact more what we can do for our community to increase the number of editors should that be is that where we can have the most impact or can we have the most impact on actual knowledge that we've reduced? And when we think about that it's whether you pick one or the other the actual detail strategies and tactics and experimentation that needs to go in there that's going to be up to our teams to do in collaboration with the community. Okay. Last question. Let me elaborate on that one just a little bit. I was actually when we were interviewing a CFO candidate he asked about how you all have so many opportunities how do you decide, right? And I think for me as the leader of a department you know narrowing focus a little bit helps us choose I mean literally with all the different opportunities we have to do work with our communities I could have a staff of 500 people just in my department product could be a few thousand we need to have yeah don't anybody get any ideas and if I gave Lisa a stroke I apologize you know and so a good strategy tells you not just what your opportunities are but also tells you how to prioritize them right so that given the limited resources we do have and given the limited resources we will have you know that that helps us narrow down and do a few things really well rather than being scattered and doing a lot of things you know inconsistent right so that's really I'm looking forward to that certainly as a leader in the group so. I suppose the question I'm next going to pass by giving to you and I'll see already what does the procedure for communicating with German speaking people or whatever the chapter slide said so unfortunately the answer to that is actually I am not entirely sure that was actually added to the slides fairly last minute and I unfortunately don't have the detail there but I will find that out I know is a general matter especially the German chapter has been experimenting a lot with how they get information out of their communities so that they can turn it in actionable information right so for example they've been doing a lot in the tech work which our community tech team has also been working with them to learn from what they've done there so I'm sorry I don't have a more specific answer on that one usually pretty on top of the details in these slides but this week you caught me out Any questions from the room or I've got a hold on Oh boy okay I've got all questions for you and I will leave mine to last so the first one from Joe as isn't it unfair to not invite potentially great editors to the tea house just so they might troll group this study yeah no I I felt really conflicted about that that's why the control group was so much smaller than the experimental group it actually impacted our ability to make claims there are a lot of a lot of things that we failed to get significance on by just a hair that I believe we would be able to get significance on and learn more if we had a bigger control group so we're actually doing it again so to get to the original kind of the personal moral or broader ethical question of it this was done on a volunteer basis it was not part of my work for the foundation the data was that we're actually collected last year and not for this particular purpose and I think it's a total shame but at the same time I believe that if showing that the tea house has a positive impact on editor retention allows us to scale the tea house or make it get the word out there then it's worthwhile Hello I was just wondering if it's possible to after the experiment was done send them invites afterwards ship is generally sailed by that point yeah okay next you said the tea house is at least as much about people as it is about technology how do you know tea house treatment is many things invitation to participate in safe space chance to ask questions and get answers portal to Wikipedia adventure opportunity to attach to the biographically blurbed of editors and so on how do we scale it unless we know which properties the tea house are responsible for if they deserve so I'll make one answer because he has some other answers to this so first part is we have a lot we gained a lot of insight from our survey data into what aspects of the tea house we're engaging particular sets of participants that people found most helpful and then there's some other analysis that we can run and Aaron can give a quick answer to that yeah so I think that this point is really critical especially as we consider how we extend technologies we really need to know how to sorry I'm getting a ton of pings all of a sudden making sure it seems that somebody's adjusting my audio level anyway we really need to know what aspects of the the interventions that we're doing actually work so that we can have we can move in the direction that will have more significant impact this requires that we build theory about what's actually going on Jamo and I are invested very heavily in a research program to explore who the tea house benefits when receiving the invitation has the biggest impact and what other effects receiving different messages have and so we've already got some preliminary results there that we're going to present at a research showcase on the 16th of December and so if you want to see our updates in about this then please attend that research showcase I have a question for both of you this is I'm waiting to hear about the results of this it's really exciting and as you mentioned yes of course there's technology the most important part is that there are editors who are nurturing new incoming community members now I'm wondering if you guys have built a model in terms of some of the possibilities that you've identified so for example if we were to invite more people or even the same amount of people but invite them earlier how much how many more editors would need to actually participate in an age with those new editors in the tea house and if we actually have that many editors in in our movement to support that well so one so I see two kind of two areas where you know where we have kind of where we may get blocked one is just the number of people who are willing to engage as hosts and two is just the kind of the limits on the infrastructure of a single wiki page I don't think that we have I don't think we I'm not sure if we log edit conflicts but having a bunch of people asking and answering questions on one wiki page there's kind of diminishing returns beyond a certain point so if we wanted to Q&A we'd have to either take it off wiki or have multiple forums that said I think that it's certainly possible to kind of like it would be possible to put together some estimates for how many we could reasonably scale to at given the current infrastructure just before I ask the next question yes we log edit conflicts but we don't have enough with that except the next next question I'm wondering is the 16 growth enough to also test the other goals that were mentioned for the details like increasing diversity and if so did it no it didn't we didn't in this in this study we didn't learn anything about the demographics of the participants it's other than through surveys we don't have a good way of asking people personal characteristics I think that if we decided that that was a priority that it would be possible to combine a quantitative study of editing patterns with survey research and get a better sense of that but we haven't crossed those streams yet thanks now I'm going to abuse my authority I've got a question so my general question is whether retention is all that important compared to for example helping editors be more effective or efficient or guiding editors to make better edits in the 6 edits they do make yes well so so for the basic reason that every editor stops editing at some point and so there has to be a replacement rate we have to bring on new editors who continue editing at the rate of the people who are naturally deciding to stop their wiki careers the other issue is the issue around the diversity or general lack of diversity within our editing communities the systemic bias of our content is directly related to the interests and characteristics of the people who choose to edit and people with certain characteristics are more likely given the current technical and social climate wikipedia to stick around and become editors so we need to kind of remove the blockers that prevent otherwise valuable contributors who are underrepresented in our editing population from leaving or from editing so really new editor retention particularly retention of a more diverse set of new editors is more important than eliciting quality contribution from people who are not likely to apply to become wikipedia I would say one other note that I'd add to that is in our analysis of the results of this experiment we actually did see that not only did people survive but they also did substantially more work in the invite conditions and so I think that really when we're measuring this long-term survival and we're looking at productivity overall we're sort of measuring two sides of the same thing so one quick note about the relation between retention and diversity is true that we don't collect data from graphics registers but we do have some data for example about the time when they register and one in direct effect increasing retention of cubies as you guys remember from a news presentation last time is that I retained this user to actually change the underlying composition of the total active edit population so by adding four people who are retained after we registered it means that we're trying to answer the current underlying graphics which as of today still see post edits heavily dominated by the relation of mostly registered in 2006-07 Next up I have a couple quick questions for Dmitri I'll join you together the link previews have you done user testing of the link previews? it's a new kind of direction so you can't necessarily draw their perspective and the other part is on iOS sorry we've done user testing on link previews I think this was done at the beginning of the summer and we've actually been collecting data on link previews after having deployed them on Android we're still sort of looking at the impact of the data and what it really means so we're still working it out and on iOS I'm not sure when I think Josh Minor can speak better to that if he's in the room because he's the PO for iOS I don't see okay I've got one last question from Yuri what's the process of deciding what are these and how do we decide what that process is one thing that we've changed based on your feedback is we decided to create a collaborative process we have a proposal for that we're going to decide what the proposal is today and how to amend them together and no, we've not decided yet we're not going to have a final solution on how that process is going to come you have some ideas, if you're part of the process you'll probably have seen some stuff in the proposal then the next week or so you'll see some more presentation proposals so but if you want to comment on the iOS I saw you walking sorry I'm going to take the miniature for two before we roll out the feature to any of the other platforms except in the back I will do it if we are interested actually, that's us at time actually, it's 12 o'clock so thank you everybody for coming to this day presenting and see you all next month and next week's meeting