 Hello, and welcome to this session on inclusive product development. I'm Marguina Botany, Vice President of Product Design of the Foundation. Last year at Wikipedia, we presented a new product development framework designed to be more inclusive and to result in more equitable outcomes for our users. While our current product development process is already user-centered, there is more we can and must do to improve how we build products that can reach and serve more diverse audiences. As you may recall, the Inclusive Product Development Working Group who is developing this framework is composed of people from the product and technology departments at the Foundation, and they represent the full spectrum of functions that are required in software development. Over the past year, the working group has begun to put its new software development playbook into practice. Seven of our future teams volunteered to beta test this playbook and incorporate it into their typical processes with the goal of providing feedback on what works, what doesn't, and why. The working group has used this feedback to revise and improve the playbook. In our next step, we will be rolling it out to all product teams and to beta test it with technology teams. If you are interested in learning more about product development at the Foundation, you will be sharing a link to a more detailed overview in our session information on the Wikipedia site. So for now, I will turn it over to Jasmine Tanner, a leader of the working group and product manager for the Android team. Jasmine will be speaking to a panel of product managers, designers, product analysts, and engineers, all of whom participated in the beta test. They will be sharing with you some of their experiences in trying to use the playbook in their day-to-day practices. Take it away, Jasmine. Since we last presented at Wikipedia, we've taken the first version of the Inclusive Product Development Playbook ahead seven teams tested out for at least one phase in their product development lifecycle. Testing took the form of teams implementing each step or attempting to implement each step in the playbook where they could and sharing their feedback on if the step was helpful or not clear or challenging. Each month, teams testing the playbook met and shared at least one thing that was helpful and one thing that was challenging. The challenges that we, the working group, could help more immediately with, we did, and the challenges that were a longer-term fix were noted for version two of the playbook, which will be released next week. We have six members from seven teams here to give you a brief idea of what they experienced during the testing process. And the first person we'll hear from is Peter. Okay. Hey, everyone. My name is Peter Pelberg. I work as the product manager for the editing team. You must most likely would have encountered our work most recently through what we've been doing to evolve talk pages to make them easier for people across experiences, experience levels to communicate with one another on Wiki. And so, yeah, I'd say that one of the most impactful things that the playbook has had on the editing team is bringing an extra level of rigor and specificity about who we are specifically centering in the work that we're doing. So we have a practice in history of thinking about target audiences based on things like edit counts and projects they participate in. And I think the playbook has equipped us with both languages and tools to introduce other factors like geography, devices that people are using, the type of their connection, their race, their gender. It's not only equipped us with language to talk about and incorporate these different elements into our product development practices, but also giving us the tools to actually bring those people in. And so a very tangible way that this has showed up in our work is in developing a series of design concepts for the design changes that we're making to talk pages. Something that we did that was different in the past is translated all of the designs that we were making into seven different languages. Excuse me, running central notice campaigns across a variety of wikis to make sure that the people were showing up and reviewing the designs were from a widely representative, you know, widely representative group. So I think that was a really tremendous impact that the playbook has had. And then in terms of challenges, I think it's kind of the flip side of what I just shared is that that process required a lot of kind of like manual work to start. And that makes total sense considering it was really the first time we were doing this kind of work at that scale. And so I think in version two, we're going to see a lot of these kind of common things that different teams needed to do on their own. Probably like codified or centralized in a way that makes us like really gives us as teams a new capacity to involve a whole new range of people inside of our product development practices, not just folks speaking languages that we're most comfortable in. Thanks. You know, Peter, a lot of what you shared about editing as their product manager was helpful to hear about segmentation of different languages. So maybe shaking you drill down on that point a little bit more. Thanks, Jaz. I'm the data scientist for the Android and iOS mobile app teams and the Android team has been working using the playbook. And so I get direct insight all the time about working with the playbook and working without it. The Android team relies on data from users to determine success for new features and help decide where we will focus our design and engineering efforts in the future. Apps generate 2% of all page views on Wikipedia. So the data is perfectly sized for us, but tiny in comparison to the rest of Wikipedia. It's been a great test bed, I think for following the playbook. The biggest advantage of using the playbook has given us the impetus to look deeper. And it gives us many more dimensions of data from our users. We found that in comparison, analyzing all user data together for specific user activity was limiting our insight and leading us to miss important nuances that affected app usability. So we started segmenting users by region and language. And one of the challenges was determining how we were going to break out those segments. And Jasmine led a lot of that research based on user feedback, research from papers written about wiki communities, the potential size of our audience, and where our users are with criteria needs we may be overlooking. And from there we determined our first set of user segments. And I can share a slide from our presentation on building with not for case study. The primary challenge for working with this data has been maintaining foundation standards for privacy and data protection. And this data is anonymized but it means that we have to just be very careful in tracking users, especially from smaller regional groups. Additionally, we only store 90 days of data. So we have to be very thorough about collecting our baseline data while preserving foundation data privacy. And we plan our okay ours at the start of each project explicitly because we learned going along that if we didn't map that out exactly, we would end up at the end of the development cycle without all the data that we needed to compare from start to finish. But we've learned a lot and I think it's been really helpful in our planning and development of what we're going to analyze. This is just applies to any segmented data analysis, it's maintaining data cleanliness and clear delineation across multiple data sets. One of the other advantages as segmentation has revealed bugs and data anomalies that we would never have seen if we weren't breaking the data out into smaller groups. I included some examples of inclusive data utilization that we use we have been analyzing a bunch of different feature developments across six user groups. And one of the tests we conducted was an ABC test for a notification bell. And it was very edifying because we saw different results across these groups. When we first started we weren't sure if we were going to see any differences in these groups, but we definitely did and it helped drive our development further into what we were going to use that most users would adopt most readily and easily. And I am a quantitative analysis analyst but we've also been conducting qualitative data analysis using a quite thorough user survey. And that has been interesting as well we are currently collating all the results from that survey. But we are getting a lot of extra insight that we just never would have gotten if we didn't break out this data into these smaller user groups. Thanks Shay that's really helpful and I know that the community tech team which Natalia is the product manager for they have been doing the work without a data analyst so it would be really or data scientists rather and so it would be really interesting to hear their application of the playbook while not having the luxury that the Android team has of having someone great like Shay. Hi everyone, my name is Natalia Rodriguez and I'm a product manager with the community tech team. And indeed we did have challenges I'll start with those first and then I'll talk about the benefits that the playbook afforded our team. So the challenge we had, as Jasmine said is that we have no data analyst in our team. So those of you who eventually gets read the playbook and you go through the steps, you'll see that a lot of the steps that are part of the best practices for inclusive product development require you to use data to check your assumptions and check your biases against. So we were faced with this playbook that had a set of best practices but we didn't have an analyst, and I think a benefit that the playbook afforded us and all the efforts of having better inclusive practices afforded us was we were able to take our time. So it created space for us to move timelines around and to use our engineers and to use our product managers to step up to the role of analyzing the data. And I'm part of the community tech team and we're the wishlist team. So for those of you who may not be familiar, it's a wishes that we run every year where engage contributors to the platforms can come and propose ideas. And we saw through analyzing the data that for the past six years there's been an under representation of engage contributors that come from Spanish speaking communities and from Chinese speaking, Mandarin speaking communities. So the time that it afforded us gave us the space to then go through the data, come through all the surveys in the past, and then switch our strategy based on that so we can use those users. Just to connect and switch it over just a little bit. Let's go over next to Gergo who also works on a team that here's a lot of feedback from the community. Specifically thinking about newcomers but can give us the perspective from an engineer point of view. Hi, I'm Gergo. I'm a software engineer on the growth team. Our team works on helping the editor communities grow. We focus on the first few week of people's experience when they sign up to be a Wikipedia editor. And we try to give them a good user experience and try to help them find their footing and get familiar with the policies of the site. Which means that we need to, we need both a lot of feedback from the new users point of view because they aren't familiar at all with Wikipedia yet. And then we also need a lot of feedback from experienced users because we help the new users make edits and we need to make sure that these edits do not cause a burden on the established community. And the playbook has been very helpful for us as a generic guide. Currently it contains, I think, something like 50 items which is way more than we could keep in our head. So it helps to make sure that you do not forget anything and it helps to make sure that you can be disciplined. Like, for example, we did think about accessibility before we had the playbook, but we usually started thinking about it in the development phase, which is often too late. And now with the help of the playbook, we can make sure that it's integrated from the start and we consider accessibility during planning and during the design phase. And it also helps to focus efforts because everyone is doing the same thing. For example, the playbook is recommending a pre-mortem. A pre-mortem is a way of thinking in a structured way at the start of how things could go wrong. And there are a number of practical questions of how you structure such a discussion and because every team is doing a pre-mortem now, it's a lot easier for each separate team to figure this out. In terms of challenges, the playbook has a lot of different items, so they are necessarily very short in how they are defined. It wasn't always easy to figure out what, when we should consider something done, whether we have done enough to check it out. And a more specific challenge was around multilinguality. So the playbook recommends multilingual usability testing, which is something we found very helpful because sometimes we get different feedback from different language communities. But it also brought up a lot of practical challenges because that means you need to write mocks in multiple languages. You need to maintain those in parallel. You need to translate the usability testing documentation to other languages. You need to find speakers for every language who can conduct the test and the interviews. You need to translate everything back. So it's pretty involved. And in practice, what we did is we relied on multilingual speakers in the design team, which is not a really scalable approach. So that's something we still need to figure out. Thanks, Gergo. That's really helpful. And to dig even more into the technical point of view, we did have one team that was a back-end team that originally had a product manager and was able to connect the team's work to some front-end user thinking and then through some shifts that changed. And so Desiree has stepped in on that team and to help, you know, the platform engineering team know how to do the application of the playbook and we'll talk more about that experience. Great. Thank you so much, Jasmine. So my name is Desiree Bodd. I'm the director of platform product management here at the Wikimedia Foundation. And I work with a lot of different teams across technology and product in what I call our foundational technologies. Back-end is another way of putting it. But essentially these technologies are what power our sites, Wikipedia, keeping it up and running commons, et cetera. And so without a lot of the work that teams like the platform engineering team, the sites won't run. And so often historically, we never really thought of these technologies as products. And so that's been a real challenge, trying to then say, well, okay, let's plug in some product management to this, which we started for the first time. And now let's adopt this concept of this playbook about how to build these products in a diverse, equitable and inclusive way when we're just starting to think of them as products. And so we had a few key challenges as a team kind of rolled through this. The first was making sure we had the right roles and functions and building the connections into the community and thinking of our work as products. So many engineers, for example, struggled to think about how would something like an API, which we're building for many different internal teams and the movement and the community and tools be thought of as a product. And so we were trying to go through the playbook and we're not a feature team. So oftentimes the typical things that you would think of for features, usability testing, et cetera, we had to translate them a little bit in order to make them relevant and digestible to the team. And so some of that included shifting the way we were thinking about the products. And so this was a real benefit in the team where we started to say, what are the dominant languages that people are using? How do we make APIs language agnostic? How do we think about service and performance for different users across the world? And how do you guarantee that? How do you build so that everyone has equitable access? And so it was a really different way of thinking that we had to start looking through the work instead of just saying we built an API and it's done and it meets a specific use case. We actually had to balloon out our thinking quite a bit to think about how do we make this that we can use for many different use cases and that the movement consume more readily and more easily instead of just for a specific product or feature that we're trying to develop. And so it's a muscle that we've been building and we've had ups and downs through the process. Some of it, a lot of it has been this translation and I think we're going to keep going through this translation journey of what is equity? How do we think about applying the playbook and the scope of our technical and technology products? Thanks, Desiree. That's really helpful and definitely something we'll continue to work on and I'm excited for us to all continue to partner to move that forward. And so the last person that we'll hear from here is Alex, who will share the perspective from a design standpoint and who has been working on a really big project and also applying the playbook at the same time, which I think is a really big feat. So Alex, can you share what that has been like? Yes. Hello everyone and thank you Jasmine. I'm Alex, the designer on the web team and as Jasmine mentioned for the past two or so years, we've been working on Vector 2022, which some of you are probably familiar with. So yeah, I'm excited about the playbook. I'm proud to say that prior to the playbook, our team was already somewhat focused on testing in multiple languages, testing with people in multiple parts of the world, and also focused on accessibility to a certain extent. However, these focuses weren't particularly formalized in terms of process. They were sort of things that were important to individual members of the team, not necessarily something we would formally plan around, or yeah, have kind of a systematized approach to. So I think those are two clear areas that stand out to me that the playbook has helped us with. Prior to using the playbook, we were testing things in maybe like three languages on average. And our most recent feature we tested in six or seven different languages. And part of what has enabled us to do that again is the playbook and then planning ahead of time, which has helped us to build sort of tools specifically in terms of how we build prototypes and make those prototypes available in multiple languages to streamline some of that process. As Gargo was talking about, it can be overwhelming to prepare mock ups in so many different languages. So we've actually switched to using prototypes that use the Wikimedia APIs and allow us to do that a little bit more easily. And then on the accessibility front, Bernard, who's an engineer on our team, has kind of taken the cues from the playbook and started to set up some automated accessibility tests. And I know he's also working to get a sort of consultant or expert reviewer to come in and work with us on accessibility stuff. Again, these things have been prompted by the playbook. So, yeah, it's been exciting and interesting for sure in terms of open questions and potential improvements for the future. One of the things that came up in our team review recently is actually that there are a lot of different accessibility standards available online. And even within accessibility standards, there are various degrees like AA, AAA, etc. that you can adhere to. So I think we wanted to clarify what the sort of exact guidelines are that we want to be following. And then the second thing we talked about is potentially having better integration with the knowledge gaps, which is coming from the research team, specifically the technical fluency knowledge gap. We have found in our user testing that there's quite a range of technical fluency and that sort of variation leads to different needs that people have. And so we're wondering how to better build that as a dimension into the playbook and have it be something that we explicitly call out when we are developing and testing features. Thank you. Thanks, Alex. That's really helpful. Now that you have heard from us, we would like to hear from you. Please share with us in the etherpad, which has also some helpful links based on some of the things that the panelists have shared here. What barriers you have had to overcome when engaging with our projects and what tools would have been helpful to you to overcome those barriers. We're hoping to take those use cases and apply it to the playbook before sharing version two of the playbook next week for all of the teams in product and technology to test against. Thanks again for joining the panel and thank you to all of our panelists.