 Hello everyone. I'm Abby Ripstra and I'm the lead design researcher here at the Wikimedia Foundation as many of you know. I'm the first design researcher here and so today I'm going to take some time to talk with you about how I'm thinking of kind of fitting design research into our product development process. I'm going to talk a little bit about the high-level process of innovation and product development. So could you switch to the slides? Okay, so design researcher at Wikimedia Foundation. So I am here because to build products and to innovate good products, successful products, you need to have those products be desirable for the people who you're building them for. You also need to have viability. We need to know that it's in accordance with the business which is our mission and we need to make sure that we can actually build them. Is it feasible? So we need technology to make those things happen. So my role is about desirability and how to bring the users needs and feedback and contributions into the process. So, oh I want to hide this little. Okay, so understanding desirability isn't only my gig, it's a lot of people are understanding this. So community engagement which Rachel is here, the director of community engagement and design and design research are also part of that and all the communities also, of course, who we need to collaborate with to understand what is desirability. So to zoom out to a little bit of higher level the necessary ingredients for innovating successful products. It's a collaborative process so it's a lot of different people to make this happen, to make products. So like I said we need viability and viability is about traditionally this might be what's the business like if we are at a big corporation what's the business that we're trying to be successful in. And our business is our mission, our making the sum of human knowledge available and contributable to by everyone. So we need to make sure that what we do is viable towards that mission. And then oops go back. So also we need to make sure it's viable, it's feasible to do that. So we need to make sure that the technology can make that happen. So in proper proportion what can we build and then and then desirability. So all these three ingredients need to be in proper proportion to make successful products to innovate successful products. So it's also an iterative process. So this is a little framework just describing the innovation process in general and this process happens when you're trying to understand concepts like the concept of an encyclopedia like maybe before Wikipedia was here people were looking around it okay what encyclopes are existing what's possible what's what are people looking for what's missing in the encyclopedias of before Wikipedia. And then so you start in this understand section about you see what's existing what's concrete and then you can you start trying to understand what is existing now and what are the possibilities for the future where there are opportunities to provide something new where there are unmet needs where are areas that we can innovate and so you look for unmet needs you look at the big picture you learn about opportunities and then you kind of move around to trying to know more about that and you're getting kind of abstract now you're moving into you have all this data from the research you've done from looking at what's existing from trying to understand unmet needs and you look in all that data and all that research and find patterns there and you can find opportunities and you can extract insights so then things are kind of abstract but you know a lot more than you did before you started this process so then you're moving over from the abstract into you're moving you're still an abstract but now you have fodder for design you have fodder for either building that concept out building that product out like there's the concept of a wicked of an encyclopedia and there's the idea of editing the encyclopedia so then there's products you can decide to build to enable editing so we have wiki text editor we also have visual editor those are two products that afford the same they're productized a concept to make a concept possible so then and you can also do this process with a single feature like how do I add a sentence to an article within wiki text editor or within visual editor so you move around the cycle either really fast or really slow depending how bigger questions are so then once you you're hopping over now into the area of making something so previously we were trying to know like what's the current context and where the possibilities what are our capabilities what what can we move or can we change here how can we meet unmet needs and fulfill opportunities so then you in synthesis synthesis is all about taking what you just learned and understanding how does it apply to this project like what did we learn about editing how how did we learn what people want to do how they want to change the encyclopedia and all the insights and the unmet needs and the things that we found out in the previous stage we're now applying to what does it mean for these products of of an editing tool wiki text editor or a visual editor whatever products we're trying to build and what does it mean like at the very high level what is that what we learned about the current site encyclopedias know for what what we're going to build so what is all this that we learn no mean for this project so then you move over into delivery and this is like really concrete making stuff like what we learned and the concepts we built or that we've come up with how are we going to evaluate that those concepts are really informing the problems we've discovered or or solving the unmet needs so you do concept evaluation you iterate the concepts further you check it out with people who the users that you've you know the people you've investigated with in the beginning you want to go back to the data and see if the solutions that we've come up with the concepts are really solid to solve those problems and those unmet needs we found so that's really high-level like process so how does design research fit into all this you can kind of break it down into evaluative and generative and there's a lot other stuff that goes on to enabling this process which is this little box on the right is building design research at wikimedia foundation that's like scaling the team and things like that so for generative research what you get is what you get from design research is better understanding people's needs better understanding their mental models like for example when I applied to this job I did my task was to do an evaluation of the upload wizard and people's mental models of uploading an image to a website is really clear like it's been established from Microsoft and Google and email and Facebook over a long time but our process is a little more complex we have more steps that people need to consider so those mental models clash with what we're building there's reason for that so we want to slow people down to allow them to make the decisions about attributions and at the same time we don't want to stop them because we're putting a lot more process in their way than they're used to so understanding what people's mental models are existing helps us to build products that will work with those mental models even if we need to change them a little bit so we also can understand what are the needs people aren't getting met that that they need to like for example new editors when they go to try to edit is it can they can they actually edit their goal may be to create a new page but what is there anything that blocks them from that their need to like be able to create a new page how are we meeting that need or not and what are their what are people's motivations why do people want to create a new page and Wikipedia why do people want to go in and edit all those things are help we have we design research can help really define and inform and all this stuff is producing fodder for design to build and design and engineering and building the encyclopedia building the products that are making it go so that's generative and generative research can inform a big concept like the idea of Wikipedia of an encyclopedia or can inform the idea of something like an editing tool like Wikitext editor or visual editor or a way to look at images or whatever the product is so we do this by methodologies like design ethnography design ethnography is really condensed ethnography and you think of ethnography as going out and living in a village with people for like three years and really understanding their social conditions and their culture and all the things that make them a unique environment and population but design ethnography has to fit within design and building product cycles so we shrink it down to fit so what it is is going into people's context and learning about what their goals and motivations are observing them doing what they're doing going and you know kind of so we can we can do that remotely we can go and talk with people about why they're contributing to Wikipedia or why they're not and just spend in depth time with them because from analytics we can understand what people are doing and what they're not doing but there's big questions about why why are they not doing that or why are they doing that and by spending more in depth time with people we can understand those why as we can find patterns if we talk to like 15 or 20 people we can see a pattern build a really strong hypothesis and then go and check that out with the analytics again we can do exploratory research which is a little less intense than an ethnography it's you know you can explore and ask people to do their first edit while we're observing them and talking with them and we can see and observe what they do and what they don't do and ask them questions and ask them for feedback and dig in a little bit more we can do collaborative design where we're bringing people in and help working with them to design what what would an editing tool be like for them have 15 or 20 people come in and see if there are patterns in those contributions that people are bringing in and we can do another example is like a card sort which is a way to understand how people organize information it will inform our information architecture so I have a few examples of things I've done we've done since I've been here we did some user experience about media viewer we interviewed and this was really to do usability testing about the enable disable flow but we also asked people about what do you think what's their experience we watched them use it and we watched them use the page view and got feedback from them and we explored with them what is their experience so we could then you know share with the rest of the teams and find patterns in those experiences and then inform how to improve the experience and for editing tools we're doing a card sort which is in progress we need to get some more people to participate but it's going to help us understand how to best organize the tools in a way that makes sense to people's mental models of editing tools and editing tasks we're doing a survey about frequency of editing tasks and what people you know people do those tasks in wiki text editor or visual editor and also get give us some feedback is that an easy task or not what how does it work for you and then we can go and talk to those people and observe them and get more detailed understanding when we find some patterns for them from the bigger surveys and also we'd really like to talk with like I kind of mentioned earlier observing people who are doing their first editor first few edits and see what that experience is like have them share that experience with us in numbers of people like 15 or 20 people we're talking about education people who are teaching Wikipedia and talking with them and observing their students as they're learning and then also we want to learn from experienced editors like what what is it like for you to edit what do you need and how can you share your experience with us so we can better understand that experience and build for all the different types of users we're supporting so then there the evaluative type of research once you have a concept or an idea for a product I look at high-level concept or idea for a product like an editing tool or something you can evaluate that so you put that in front your assess if that concept works for people does that does is that going to solve their problems is that going to help them accomplish their goals and does it work with their motivations you can understand the usability of like if we have a prototype of an editing tool like we have prototypes we have beta of a visual editor so we can put that in front of people and understand how they're experiencing and where the bumps in the road with the visual editor we want to understand where those bumps are and try to smooth them out so and are these products concepts are they relevant to people are they useful are they something people are need and the way we we get that information we can do a heuristic evaluation or a cognitive walk through so user experience professionals can look at a prototype or a concept more like a prototype to see if there's there's heuristics we can assess a product with and then there are things that'll stand out like red flags yeah this is an issue maybe some things there's no issues but we'll see the issues and this saves time in recruiting people for usability studies and things like that so we can create efficiencies and doing heuristic evaluations and cognitive walk-throughs and then also it's very important to put things in front of people to like do a usability study study and concept evaluation is like getting the right people the people who we're building this for and seeing if this concept is that fits with them does it work with them so it's a conversation it's having them use it dive in we can do this in in beta features like we can get feedback from people who are using beta features over time so it's not just an hour with the person but we can talk with them over time and see how is it working as we evolve this things like that and usability testing also works with product and features we do gorilla testing sometimes which is really good quick feedback for just like a simple task or something we go out in the park and hand people a phone and say hey would you mind spending five minutes with us we get a little demographic information from them and then we ask them to do a task one person's facilitating and then really importantly one person is observing so because what people say they do what they say is easy or not many times is very different from what they actually do so memory is tricky and people sometimes want to please us but we can understand their experience in tandem with what they say by also seeing what they do and that observations really important so we make sure we write down those observations as well so we've done so an example of concept evaluation will be we're planning some research about talk pages and flow and like how do you know it's new users new contributors and maybe casual contributors people who haven't been around too long how are they experiencing talk pages can they accomplish basic tasks and talk pages how is that experience for them and then also how do they do the same tasks are the same things in flow and how does that work for them what what what is that like and we can kind of compare those two products and functionalities for new users and then we'll we'd like to do the same kind of a research with very experienced contributors to see what their experience of flow is versus talk pages that will really inform how we build these products to support the needs of various users we did some usability testing with media viewer we've been iterating pretty quick on media viewer recently and we did usability study of the enable and disable flow so our people able to disable media viewer and then re-enable it and how easy is it how where the bumps in the road we found some feedback we got some found a few bumps and we iterated those and then we followed up and did more testing and we found that those changes made it easy for people to iterate to enable and disable so okay and this is all looking at patterns so we're looking at not just one or two or three people but we'll test with maybe seven or eight people after five you're starting to see the same results over and over and over and this has been researched and studied for usability testing to see if a human of a certain group of people can accomplish a task seven people is plenty but when we're trying to understand people's needs and goals and more complex stuff like that are we still online okay when we need to talk to more people and we like to triangulate with analytics to understand who to talk to and things like that so this is a quick description of gorilla testing we recently went out in the field and showed some people a wiki page as is on on the app and then with a little bit of a redesign to see if we can make it easier to comprehend and that was just it's not like statistically significant but it's we talked to maybe seven people and we got a little bit more feedback than with no feedback for the designers to iterate this design towards and we'll do usability testing and so design research there's different methodologies for the different stages we are in product development and designing so the other big thing that I've been doing since I've been here and also Daisy who's our associate design researcher is really collaborating to do this is build our team so that we can support the product teams and and the community and and everybody involved so we're scaling our team so there's the two of us right now and we're hiring two more people we're having a senior design researcher we're looking for someone who's really really tried and true tested on usability studies being able to just jump in and do the do the research with us and we're also looking for a participant recruiter which is really important because recruiting is very important we need to talk to the right people for the for the right projects and that effort is a lot of time it's a lot of labor to schedule people to find the people and so we're we're gonna get a part-time person to do that scheduling which is going to enable us to do more research we're also defining our tools and our and our processes so design research is kind of a new field in the last 10 years I've seen it grown and get more recognized and needed in all kinds of product development whether it's manufacturing or software and so there's a whole bunch of new tools now that we didn't have before a lot of things were manual where now we can do scale our research like we can do benchmarking about usability and there's tools that afford us to do that we can do remote research a lot easier and be able to see people's face and reaction as well as what they're doing on the screen so we're using Google Hangouts as our as our usability lab right now and we're we're we were looking for an open source survey tool but there wasn't one that met the needs of robust survey tool that we needed for design research so we're using Qualtrics things like that we're like looking for the tools that we need to support our processes and the recruiting process of like creating efficiencies in our processes and how we set up usability how we do the recruiting and all that we're also educating about what is design research and what what can we what do we provide and who can we connect with and what are the values of this effort so also we're collaborating with many teams not only the product teams but we're embedded with the analytics team because we're really wanting to do triangulation with the analytics team they can help us find who are the right people help us segment all the people who are part of the communities and understand who to talk to when they can also tell us about what's happening and what's not happening and they do a lot of really they have superpowers and they do a lot of really important experimentation to help us understand things and so we can contribute by helping to understand why things might not be happening or why they're happening so we can go and talk to a representative sample of people that maybe are just dropping off and not and you know maybe they've tried to edit five times they've done five edits and then we never see them again they can tell us who are those people we can go talk to a sample and then get some insights and see some patterns and get hypotheses about why that might be happening and then try to confirm or refute those hypotheses again so and community engagement we want to reach out and reach out as far and wide as we can and collaborate with communities and collaborate with everyone and anyone and engineering of course did I forget anyone community members of course everybody so then this now so design research in agile product development I've worked at Cisco and Salesforce and their two varying degrees agile I guess Salesforce is pretty agile Cisco when I started was just becoming agile they were kind of mixed between waterfall and agile so always in my work it's been a little bit of performance art about how to fit research in to the development process because when I first was starting that there was not a lot of knowledge about the value of it I mean the designers and the product managers knew about it but engineers not necessarily in product other people didn't necessarily know so I would have to do a lot of talking about well if we want to put product out that is usable we need to do usability studies before it's put out and then the feedback that we get is going to be a lot more valuable once it's out there that the easy things to find the you know is the they can't find the button that kind of thing we can solve before it goes out into the wild so my point is understanding when to do what research within product development and making the room for that so we have a concept which let's think about the concept of editing and then we have the two products we have wiki wiki text editor and visual editor and those products are built by a whole bunch of features so there's adding links there's adding references contributing text editing text all these little features add up to build a product and those products then are the instantiation of a concept so and as we build product we are building feature feature feature and then we when is it time to release when when how many features makes functionality okay to put out in the in the in the in the wild when is that we have this concept of MVP and defining a minimum viable product and that means like it's a functionality that people can use it's a functionality that is usable it might have it's you know hopefully it meets a need or solves a problem yeah so and then you know there's the MVP and then you add more features on and then there's maybe a second MVP and then you have a fully done baked product over time so I have only been here for five months so I'm doing my best stab at what what what happens here so I and this is there's a question mark here because you guys all can correct me if I'm wrong and tell me you know if this is my assumption or this is you know kind of what I've seen so there's a concept and then we add some features and the designers cheers to the designers because they've already been doing usability testing before there was a design researcher here they're using user testing calm and iterating the designs and then it goes there's features built and then there's a MVP release and then more features are added MVP and then a product I don't this is like high level big gas from from what I can see and do we have sprint zeros here what in my experience sprint zero is like a mythical beast that everyone really wants to have happened but there's never time for it but I think there's great value in sprint zeros to really all get rallied behind this is a really important thing to put out or there's there's these are the reasons we're going to build this and planning like what is the minimum viable product one of the stages it should be released and what are the things that we need to do to make it a successful thing in the end so any sprint zero no yes sometimes okay so then I am advocating to have a little bit more room for design research in our process so I there here's just kind of like a high-level example of what it could look like so that generative research I talked about like going in talking to editors and really understanding the newbie editor experience and what are the things that block them and what are the things that are motivating them and their goals and and maybe we understand that here but I want to know that I want to know and I want to know what that is so this generative research can take a little time so that and and it's great if everybody participates of engineers and product matters and designers and community members collaborate on the stuff then we're all rallied and there's a lot of good alignment on what we're building so then that kind of thing and then benchmarking is really important so that you can start benchmarking on existing things you can like for example let's say there's no editing tools in Wikipedia we could go to look at analog products like how do people edit in WordPress or other analog products out there what are the experiences they're having what are the things that work and don't that's one kind of exploratory research you can start benching and you can start benchmarking on what are people's current experience and then you know we build a minimum viable product in a couple weeks after that's out in the wild we start benchmarking our own product and see how is this measuring up to the existing experience wow look it's a lot easier or whatever the outcome is and then over time as we add more functionality and more features we can continue to see if what we're doing is is progressing is making it easier for people to edit is it upping the numbers of contribution or or not and and if not we can understand why by doing you know concept validation early on so that we are sure that this concept is going to enable people to edit or enable new people to edit or enable really experienced editors to continue editing or whatever the problems we found are we solving those problems and then like I talked about here's stick evaluations of just quick looking at things to see is it usable is it where are there any red flags doing usability testing along the way and affording time for that we need to find the right people and we need to like spend a little time with people if we spend an hour with people we can do probably three usability tests and also gather some really good generative research about what who are those people and what are their motivations and goals and as we do those we add it all up and then we're understanding the people in the community and the people who are contributing over time and that will help us understand user segments personas the really the different types of people who were building for it so it looks like a lot it's like a lot of a lot more work but actually it can create efficiencies in product development so there's less refactoring we're not going back and we're not putting product out there finding out I mean there's good reason to put product out and test it and see you know doing load testing and doing beta features and we get big numbers but there are things we can do before that to make it a lot more efficient so that we don't have to oops whoa go back and start over again we don't want to do that that creates a lot of spin and churn so this is a really in my experience a valuable investment to do this type of research before we release and so there are some implications and that means we all need to collaborate and there's big conversations going on about improving our development process and how we're going to do that who's contributing how we're going to what are the things that we're going to do to improve our product development process we need to collaborate on that I kind of feel like we need to slow down a little bit not too much but just enough to afford some more research and maybe other things that we need to do and also broadening the collaboration more broadly because we have we have RFCs which are functional for a certain set of our users but then there's millions of other users who don't even know about RFCs or how do they how do they contribute how do they how do we collaborate with them and design research is one of those ways community engagement is one of those ways we need to broaden our collaboration in my mind also we agile is sometimes and I correct me if I'm wrong I think sometimes it's misconstrued as just a way to go faster and sometimes that can cause problems we just go faster put something out there see if it's broken and then fix it and there's like I said before there's things that we can do before we put it out there to get more productive feedback so that we can get we can find the really obvious things that we can fix before getting it out in the wild and then once it's out there we can get you know more productive feedback from people like yeah this is useful to me or I don't like the UI or whatever kind of feedback we'll get more productive feedback if we if we before release get the obvious stuff taken care of so there's those things so and then I just finally let's make a little pitch for everyone to participate anyone out there in the world we want to hear from everyone and so we have a little one of our things we're working on in the research department design research department is getting a big database of opted-in participants so we want people to be to say yes I want to contribute I want to participate and they give us a little bit of information about like how they interact with Wikipedia what wiki projects they work on geographically where they are located at least in time zone so we know how to schedule people in time a little bit of information about that how to contact them so we can schedule with them and what their interests are so then when we're trying to move fast and not take too much time out of product development but fit this research in we can have a really efficient way to gather participants and find the right participants so there's a little survey you can fill out to opt in and then if you ever want to opt out you can send an email to this and you're perfectly welcome to opt out at the same time so this will help us really be efficient in our gathering of people we have a lot of other ways to invite people to research too but this will really help us be fast so that's my presentation does anyone have questions here's a question from IRC does the design research team engage in blue sky research research for the sake of building understanding that doesn't have an obvious and immediate use for a product it's a great idea we haven't done it so far but I would love to do that that's kind of sprint zero stuff right like having time to do that I know that the I participate anytime I can with the designers and doing concept generation and blue sky kind of things like that yeah and the research that we do can contribute to that blue sky being kind of you know a little grounded in and in needs and things is it on is it on okay sorry so obviously I understand the value in product creation concept the conception of products and iterating on products and the need to sort of slow down to go fast right and where are you in that conversation in this organization and how could I help advance that conversation I mean I have no idea what I could do but it seems like the right conversation to be having and I'm kind of wondering what what you're hearing back yeah I mean I feel like I I think other people have been talking about slowing down a little bit too but I am not exactly sure what that all means so I'm kind of just just getting here getting the department organizing and trying to scale so I'm trying to see where how to fit how to fit in how we're everything is kind of changing and how are we going to all land so I want to be part of the conversations about our product development process and what's changing and I've that that you know include me in those conversations yeah how do you account for users who are hard to reach due to distance language time zone etc are we designing Wikipedia for Western English speakers because it is easy no we're designing Wikipedia for people all over the world right we want to reach everyone so part one of the questions in the survey is do you have what kind of technology are using so if we can communicate with you over the internet that's great but that's I'm totally happy to go out and talk to people too we are it is a challenge to do translation I'm still trying to figure out how to find people to help me do translation in things like surveys and card sorts and when we do moderated research I need people to help me translate that so I am totally aware of that need and trying to figure out how to make that happen that's not something I'm taking lightly did that answer okay do you see design research as based primarily on qualitative methodologies or can someone be a design researcher while specializing in quantitative methods I think the second is true quantitative and qualitative are both design research the the so it's a little bit different of methodologies but they're both informing design I think and concepts and products and and and also it's really powerful when quantitative researchers and qualitative researchers work together and triangulate because like I said there's the what and the why and we if we are collaborating together it's really powerful understanding what's the future future for ethnographic methods in design research do you guys plan to make time for becoming Wikipedia in order to more effectively understand your users yeah of course I'm I and learning to edit along with all the other newbies so that's so I'm diving in to learn these things yeah I think it would be really important for us to take the time and it wouldn't be just the design research team doing ethnographies it'd be people from all kinds of teams and community members coming along and doing ethnographies I would love to go out in the world and be you know out in schools out in anywhere there are people at editathons and I want to help bring some structure to that kind of those observations because I know it's already happening all over the place but I want us to be able to gather the information from those observations so adding a little structure to that and also bringing everybody along and then bringing that information back so that we can really bring that rich information into how we're operating here into making decisions and then building products any last questions from the room thank you so much Abby that was really great thanks everyone