 Thanks so much everyone, this is such a great crowd. So we're here to talk about designing inclusive products and as we get started I'm going to tell you a little bit about who I am so you know who's going to be talking at you for the next 40 minutes or so. So I am a senior product manager at Google, I've been working there for about 8 years. Before that I got my computer science degree at Stanford and while I've been at Google I have worked on Google search, Google analytics and I was also the first product manager on Project FI which is Google's wireless carrier service. For the last year I've been working as the CEO of a team called Grasshopper, thanks guys. And we're working within the area 120 incubator at Google working on an app that is aiming to teach adults to learn to code. In my spare time I'm also writing about topics around diversity and inclusion at fearofpoets.com and occasionally tweet at the fear of poets alias. So why am I here? So as a petite mixed-race woman I am very well experienced with products that may not be made for me. That being said I don't think that it's something that anyone is trying to do intentionally, they just maybe haven't really thought about what my use cases might be. So over the last several years I've been trying to kind of get a bunch of different best practices to think about how can we make products that work for as many people as possible. And that's what I'm here to share with you today. So a couple of disclaimers before we get started. First this shit is really hard to talk about. So my intention is not to offend anyone but if I do, apologies in advance. The second thing is that, one of the things that I wanted to say is that there is going to be some kind of topics in here that are kind of tough to talk about. So if there are things that are in here that are kind of offensive or whatever, it's because they're there to show you what can happen when you don't build inclusive products. It's there to build the story. And the last thing is that nothing is going to teach you how to build an inclusive product better than having a diverse and inclusive team. That being said we could talk for hours about that and that is not the purpose of today's talk. So just keeping that in mind and if you want to talk about diverse teams in the future then let's chat. So getting into it, why is building inclusive products a product manager's job? You could say well it's the engineer's job or it's the designers who should be making this all inclusive. Well as a product manager you are the one who is regularly the one who is deciding what the priorities are. You're the one who can sometimes say hey you know what let's not move on to that next stage yet because we're not ready. So I really do think that there's all these different phases in the product development life cycle and we're going to talk about each one of them today. But you have the opportunity to make a difference at every stage and add a little inclusive touch to make your products more inclusive. So getting into it, let's talk about getting buy in. So as a product manager one of your jobs is to manage stakeholders and those stakeholders may or may not really be thinking about how important it is to be building inclusive products. So ideally your team just gets it you know they're super on board with inclusion they're willing to take the time that it takes to build inclusive products. But in case you aren't living in that perfect world here's some talking points that you can use to talk about why inclusive products really make a difference. So the first one are some stories. In the 1970s airbags were first invented and unfortunately the people who were in the room when airbags were created were basically average sized men. This meant that when airbags hit the market that a lot of women and children actually died because the initial airbag design was not inclusive. So hopefully the products you build will not be causing deaths but this is actually one of those scenarios that I think is hopefully in our history and not in our future. But just kind of keep that in mind. Another case is the case of machine learning. So Google Photos a couple of years ago launched a feature that auto categorized a bunch of your different photos. And unfortunately they tagged two black people in a photo as gorillas. This was not the intention of the team that was doing this. That being said it does expose that they may not have had as much user testing as they might have wished they had had. So there's cases in which having non inclusive design can lead to unfortunate consequences like this. One more thing so I don't know if y'all are familiar with what Airbnb has been dealing with from a racial profiling perspective of the last year or so but there was cases where people would get denied their reservation because of the color of their skin. And in Airbnb's case I don't think that again that they were intentionally creating a platform that would encourage or reinforce racism. That being said the racism was already there in the population that it was serving and what they have been trying to do over the last several years has been trying to change their platform to make decisions that helped counteract some of that bias. So at kind of each stage whether it's intentional neglect or intentional or there's neglect there's cases where the machine learning just isn't working that well or you're creating a platform that many people use. There's places where non inclusive design can have really sad unintended consequences. That being said if the stakeholders don't quite like the stories but they like numbers let's talk through some numbers. So as far as the racial population of the United States here's a recent analysis by Pew Research about the distribution of race in the United States. As you can see it's actually and this is also a projection so you can see that the U.S. is getting more diverse so maybe you can tell your stakeholders that this is planning for the future if they're trying to plan for racial equity. Another way to think about is the spectrum of which people can actually use physically use the products that are being developed. So in 2003 Microsoft commissioned a research study that said well what is the spectrum of physical capabilities within our population. I think many times people think of products as hey there's a bunch of people over here that use products like everyone else and then there's the small portion off to the side that we have to do special things for and I think what I want you to take away from this particular analysis was that there's actually a huge spectrum that the majority of people actually have with regards to their ability to interact with the pieces of software or hardware that we create as product managers. So this could be differences in physical mobility, it could be differences in the way that people see but it is a spectrum and not necessarily a here is everyone and a small margin that needs special assistance. Age is also another area that we can be thinking about or should be thinking more about when it comes to building inclusive products. So Pew also did a research where they are projecting out that the overall world population is aging or you could look at it a different way which is that as product managers a lot of us think about you know millennials are really cool to talk about or Gen Z they're getting a lot of conversation but if you look at half of the adult population is over the age of 45 and so if you're not building products for the people on that other side of that graph then you're cutting out your half of your potential user population. So these are some numbers to maybe think about with your stakeholders. And then if the numbers don't get to them maybe costs will appeal to the bottom line. So when you don't build inclusive products there's also the cost of damage to the brand or business due to bad PR like some of the things I was just showing around Google Photos or Airbnb. There's also excessive support costs so I used to run support for Project FI from a product perspective and I can tell you you don't really want people calling in because it's really expensive and if you build a product where people are calling in all the time because the product doesn't work for them that's going to be a lot of money so if you build more inclusive products upfront you're actually spending less to support your larger population. There's also lawsuits I can't really predict what these things might be but they're expensive. Product returns also very similarly to excessive support costs if someone buys your product and it doesn't work for them they're going to return it and that's going to be really expensive. And there lastly is this post-launch reworking of the product. So say you've been working for the last several months or the past couple of years on the perfect product and you launch it and it turns out it wasn't very inclusive and so a lot of the people that you really wanted to use your product didn't actually end up using it. So now you have your active user population that you have to be actively supporting and also doing a bunch of new-stage stuff to work out why the people that you were hoping to use your product aren't using it that's really expensive so hopefully between the stories the numbers and the costs you can convince your stakeholders that building inclusive products from the very beginning is worth it. Next on to identifying your users and their needs so ideally in the kind of product design process you start with your users figure out what their needs are then you build your product so that's where we're starting here and the first part is trying to interview a bunch of potential users and then develop personas. So invest at the very beginning on having a diverse set of people that you're interviewing for your personas. Some of the things you can do here kind of glibly think of who your users might be and find them. However a more useful tactic is Craigslist so I use Craigslist a lot it's remarkable how how many different types of people use Craigslist and if you put out a link that says hey do you want to do a user study for you know 25 bucks for 30 minutes or something like that you get a lot of people from very different backgrounds. Also think through partnerships so if your product is partnering with anyone else they might have an existing user base that matches what you want your user base to be because they're your partner and then you can ask them for a diverse set of people from their existing user base that you might be able to interview. And lastly friends of friends are teammates you know if you're looking for a certain demographic that you want to be infusing into your persona pool then you know use a network effect and it's actually a surprise it's it's been really helpful for me in my own work. And then if you have an existing product two tips one is if you have an existing product and you want to interview your existing user set make sure that that you try to find as many diverse people in that user set as possible. Sometimes it's really easy to say hey there are user let's just talk to them but really try to find a bunch of different people it'll make your your interviews a lot richer. And the second piece around having existing product is that find people who aren't using your product and ask them why. So it might turn out that your product as it currently is is excluding a certain subpopulation. Go and find that group and talk to them and maybe figure out why they aren't using it and that can be really informative to your next stage of product development. So then once you've done hopefully a large set of interviews now it's time to distill that research down into personas that will then inform your product design. So there's a researcher named Indy Young and just so you know I have these links in my talk there at the bottom I have on the fear poets site that I have I've uploaded my site this deck there so you can like click on these things later. There's some really great research and very long form stuff here that I've tried to distill down. So for Indy Young she posted this particular tweet a while back that was kind of controversial was please remove age, gender, ethnicity, location from your personas. None of these things cause behavior slash thinking but they can cause a lot of assumptions. And so what she means by that is like as soon as you start to you know add things like age, gender, ethnicity it starts coming loaded with a bunch of different stereotypes and is that actually the thing that is going to help drive forward your your product design. Well her point is focus on motivations like that is so much richer than what someone's age is you want to know why they want to do something. So her she encourages focus on motivations in your personas leave kind of everything else out if you can. Another tip is if you do have names pick gender neutral names. So a name like Pat could be Patrick or Patricia. These types of things can then make people a little more less reliant on the stereotypes that might inform their thinking. And then lastly if you do just you can add demographics if people are not really empathizing with the persona but if you have the opportunity add counterintuitive characteristics or demographics to the persona. So for instance she gives the example of the user who is kind of a technophobe who has is uncomfortable technology and is worried about data privacy security stuff. And sometimes that can be a persona of someone who's kind of on on the older side but it could also be a 35 year old mother of two. And so kind of adding these counterintuitive demographics can cause people to think more about the motivations than say leaning into the stereotypes and be like oh well everyone in that category thinks that and that's not something that we really need to deal with. So to give an example I'm not going to read all this but this is just kind of an example of some personas work that she did as age for people who were checking or not checking their luggage. And so this protects stuff persona is a person who is worried about their stuff getting battered when it's checked and so they want to keep it with them it's because they have things like their guitar and it has sentimental value. So they're going to always bring things on the plane with them that's their motivation. As opposed to this won't go aboard persona which is they always have something they you know beer from their trip to Belgium or their scuba gear so they're always going to be checking their luggage because they can't otherwise bring the stuff that they want. So as you can see here no names no ages no demographics but strictly motivations and this can be a really solid foundation for doing product design work. All right now matching your business to your users. So we spend a lot of time talking about like or at least when I have talked with other people about building inclusive products spend a lot of time about you know how are we building it but I think there's a stage before that which is when you're you know pitching your business. What are some of the things that before you even start building you start deciding as a product manager that actually will impact the inclusiveness of your product. So the first thing is platform so I'm gonna talk about mobile versus desktop. So it turns out there's more people who have mobile phones than desktop computers. I think we are moving more and more to a mobile first world but it's worth looking at the numbers. More Americans own a smartphone than a desktop and then if you look at something like the Indian market then you start saying that 34% of Indians own a smartphone whereas only 10% own a desktop. There's also an increasing trend of people who are mobile dependent. These are people who can only access the internet through their mobile device. This population in the United States tends to skew towards the young underrepresented minorities and lower income and so if you want to kind of be a more equitable product try thinking about going mobile making your mobile experience as strong as possible because these are people who might not otherwise be able to access your product which also brings me to iPhone versus Android right? I've often heard that iPhone you know you develop for iPhone first because that's where the money is they're all really rich but if you cut out the Android population you're actually being exclusive so if you have the opportunity to make your product for both iPhone and Android why not? That's going to be more users it's gonna be more inclusive and when there's tools like say React Native or even better maybe a progressive mobile web app you can actually build these products to work on both platforms and make it available to everyone. I also want to talk about things like pricing so here right is a graph of the amount of disposable income Americans have in a given year. 70% of the United States has less than $2,000 in disposable income each month so if you think about that if you're building a product that's you know the same amount as say Spotify right it's like eight bucks ten bucks that is what you know 120th of a person's disposable income of you know 70% or even have that money right and so if you start thinking about pricing it's not just about like oh I think I could get this much money start thinking about who your users actually are and do they have the disposable income to actually be able to afford your product and then even beyond the United States start looking at oops there we go the rest of the world so that was a graph for United States we're looking pretty good with regards to the rest of the world with regards to disposable income right so one thing to keep in mind when you think about pricing is that free is the most inclusive price so not everyone can make products that are free but if you have the opportunity that is going to make it accessible to more people another thing to keep in mind for pricing is the ability to budget is important I'm saying this explicitly because of we're in Silicon Valley not everyone thinks that budgeting is as important as it actually is but the majority of the world budgets and the majority of the world budgets really well so when I was on project five we had this pricing model which is pay for a certain amount and then we'll credit you back or you pay over and a lot of people were asking why are you doing that why don't you just pay for it you use that just makes sense mathematically but the reason why we went that way and the reason why it tested so much better with real customers was because people wanted the ability to budget and so as you think about pricing do you think about the importance of budget and where Pete that your product will fit in a user's budget and the last thing is also think about the price based on the value to the customer not based off of cost so there's a lot of research around pricing best practices why you should not do cost-based pricing however there's a lot of really good reasons why you should price based on value to the customer because cost is a function of how many people are using your product and so if you can develop a product and then find its value for a larger number of people you will actually reduce your costs so just some tips to keep in mind as you're developing your business story around the product before you get into actually building it but now on to building it because it's the really fun part so this is where I think you know you start getting involved and much more with the engineers and the designers to build and bring your product to life so a couple of things there's there's a lot of different vectors upon which you can start thinking about inclusion and design when it comes to the building piece the first one I want to touch on is designing for privacy and user safety so I recognize it was laughter in the room but this is actually a really sad story so there is a class of people where I feel like they don't get talked often enough about when it comes to product design these are people who might be victims of abuse the LGBTQ community people kids who are accepted to bullying and for these users privacy and security are actually incredibly important and can often be the difference of you know physical safety your life and death so a couple of stories this particular article was written at the launch of the Google Buzz thing this is 2010 so it's a long time ago and we've learned a lot since then but what happened is that we created a social product that auto created connections between people or made them think that they were those connections based off of their most frequent contacts this person's most frequent contacts were her boyfriend her mother and her abusive ex-husband and that is something where we violated that person's trust and safety at that point in time there's another link here too about how Google outed someone where there was an auto lookup feature where someone was presenting their old and authentic gender at their place of business and then through an auto lookup feature their employer ended up finding out their new authentic gender and that was a thing that was not desirable as you can imagine from the person who had been going through this transition so another thing to think about with regards to privacy and safety this is an example of really great local safety features design this is a screenshot from a product called recovery record it is an app for people with eating disorders so when we think about phones we often think of them as being ours there are things that are private there are things that if someone was to take them that maybe they might find things that they you know might that might be compromising so people don't you don't give your phone out but not a lot of people or there is sorry there are some people who don't have that choice there's for instance lgbtq youth or people who have eating disorders where their their family will take their phone from them and try to figure out what's going on so in this particular case recovery record they have this feature called discrete reminders so these are reminders to eat however if someone was to take this person's phone and see a notification like buy gift for Cindy at lunch they wouldn't actually think this person has an eating disorder they think they have to you know buy a gift for a friend so these are kind of things that you can think about for people who need a little more privacy and safety and what they can do if the privacy of their phone is is compromised another thing to think about is also location sharing so for the majority of the world location sharing is kind of fun you can see where people are but again it only is as fun as transparency is actually there so as you're building safety feet or sorry location sharing features think about who can actually see the users location is it friends is it people that you have opted to share with or is it public and anyone could look at it or maybe think about what actions trigger the location sharing or does the user know when the location is being shared it could turn out that that person is at a location that might reveal something about them that maybe they don't want to be shared or it could also be the case that someone is getting bullied and now their location is being shared with the people who are bullying them so these are things just kind of think about as you're building in location sharing features there's also engineering for accessibility so for people who can't necessarily see the user interface like you know in all the vivid colors that your UX designers have carefully designed there are features to help those people use your technology so these are things like screen readers for web and then also talk back and voiceover for Android and iOS respectively these are things that allow a user to kind of tab through your user interface and read out what is in your user interface however these things only work as well as your ability to engineer them so if you have a so for instance the screen readers will follow a certain tab order and some of the engineering teams I've worked with when they first built a version of a user interface the tab thing was going like all over the place and it was very incoherent as to what was actually going on so you can work with your engineering team to actually create a coherent story as the the screen reader or talk back is is going through the user interface so that people who have site impairment can actually use your product there's also designing for color blindness so for most of the world the rainbow looks like this but one in 12 men and one in 200 women start to see this rainbow like this maybe there's some in the room and it's really important to kind of keep these things in mind as you're designing a color palette for a particular or designing color palette for your product there's some really great tips in this particular blog about how to design for color blindness but one of the things I thought I would pull out is so Facebook Mark Zuckerberg is is color blind and so he designed Facebook in blue so you can see that one the best and you can see here that there are some errors right and they're in red and that is a color that is sometimes hard for people to see so you don't want to have your user interface only show errors and colors also include iconography so if the person can't see that color contrast there's other indicators of maybe they need to be fixing something alright on to speed so this is a world map of how fast the internet is for most people us in America we're not as good as Australia but we do have some pretty good internet and you should be thinking about what is the experience like for people who have a slower connections in order to reach more people so I I've actually heard that Facebook they have a network on to G to force the people in the the Facebook Plex to make sure that their products work well on to G there's also I think some features in Chrome that allow you to slow down your connection speed to see what your website would look like and load at different connection speeds alright some other things to think about so forms forms can be inclusive they can also be exclusive I remember as a 10 year old trying to fill out some standardized tests that I had to pick one race and it was very confusing for me it's like which one do I choose but I think that race forms have gotten a little more inclusive it's much more of a check all that apply or fill things in but when it comes to gender it can sometimes still be a little narrower than it should be and so there's a great resource here about all the different things that you can do to make your kind of gender form more inclusive one of the things that it says is do you need to be asking for gender in the first place what what value do you have from that but then also think about the use case that you have for that particular data so this one it's asking for gender because it wants to be able to have a user interface that addresses their user with the correct pronoun so it's really tailored to that use case and also has this really nice feel that says custom pronouns one more thing is around screen size so I have tiny hands I have sometimes almost sprained my wrists using some of the very large phones that are in vogue right now but it's in both the small and large case it's really something to think about so I heard the story of a person who was working on an app for crane inspectors and no one was using their crane inspector app and they sent someone out to the field and said hey you know why aren't you using our app and apparently a lot of crane inspectors have large hands and they made all the buttons very small and so one way around that was they got in a stylus so they could you know use the stylus to navigate the UI but they didn't want to take a stylus with them up on the crane so as a result they just weren't using the app so testing out your app with a large variety of people with different hand sizes can actually be really helpful all right the last piece of build it is user test the shit out of everything you're only going to have an inclusive product if you have as many people using it as possible again here craigslist can be your friend just ask a bunch of people to come in use your product it's a good fast cheap way to get a bunch of different types of people using your product um maybe even better than this though because craigslist is pretty local you know we're in the silicon valley bubble uh maybe think about going for a road show so maybe take your product to the east coast to middle america some place rural um if you're trying to get international users maybe even go to those places and you know have some some great food while you're there um but that can also be pretty expensive um so a kind of cheap tool that i would recommend is user testing dot com this is a product that allows you to either give your app your website to a panel of people these people can be from a broad range of socioeconomic groups and also different levels of technical facility and they will screen record people as they do the different tasks that you ask them and it's pretty cost effective so if you can't afford a road show this is a pretty good way to get a broad set of users using your product and lastly create a network at your company so if you happen to be at a larger company you can actually recruit a set of people from diverse backgrounds and say hey before this goes out can you just kind of give it a run through and that might be a good way to bootstrap inclusive user testing all right now on to go to market so you've built this amazing product now you want people to start using it so before we go any further i'm going to say warning proceed with caution there are two things here one uh if you are building a product and it's not inclusive but you decide to do inclusive product marketing people will call you out on that it is not a way to put lipstick on a pig um the second thing to note here is proceeding with caution is even if your goal is inclusion if you don't have a diverse set of people vetting your marketing campaign it can really backfire and as an example dove did this a couple weeks ago where they had this awful commercial where they had a black woman turning into a white woman um really insensitive very much reflects the fact that there was not a black person in the room when they were pitching us completing it um and uh when they talked to dove dove was like well we were trying to be inclusive it was like well you were but it did not hit the mark um so as you're thinking about inclusive go-to-market strategy do keep some of these things in mind so you don't need to hit you know diversity right in the head what you can do is something a little more like what apple did with their ipod advertisements this is kind of old but is so good it's so iconic and the reason for this is because they have these profiles it's not you're not looking at a specific person but you can almost like imagine yourself as one of these silhouettes and that's i think i think a really cool thing about marketing is that the best marketing is an invitation to the people that you want to be using your product and so what ipod did apple did here is that they try to have people imagine themselves as an ipod user and i think there's a lot of different ways that you can start to do that and work that into your go-to-market strategy so for instance uh facebook has this inclusive hands resource so they've done this thing where they have all of these um photos of people using phones if you want to have a screenshot of your app you can have a bunch of different types of hands using it and that's really cool and people can see themselves maybe you using that a little bit more than say you know the standard you know white male hand um slack also did this really cool thing where when they introduced the add to slack button they had a brown hand um and it was a small thing but it's actually really big for a lot of people and so these are the types of things that you can start to do for your go-to-market that invites other people from different diverse backgrounds and see themselves as one of your customers um i'll end with some work here that's done by alice lee um alice lee um redid um the wordpress site to have these really cute illustrations but what she did as far as as part of her process is she created people with um different body shapes um a color palette uh with different skin tones and also included people in different scenarios so you know this person's in a workshop uh this woman is wearing a job um and what this does again is they're they're lower fidelity so they're not like images of people but because they're diverse uh they really allow people from a broad set of backgrounds to imagine themselves as invited to be using this product um and she has a really good analysis of her work there all right so you've launched your product one last quick thing about the numbers so now you've launched you're getting all this really really great data just be cautious of how you interpret that data you could say like everyone loves our product like you know our c-sat scores are out you know out out the door or out the roof or whatever um but that may not only be a subset of people you know it may turn out that your product is actually excluding a sub group of people and they're not going to use your product you're not even hearing from them and so as you look at your data that's coming in uh try to look at um different subsets of people um and see if you can try to find insights and also figure out who isn't you're using your product uh because they're not going to be part of your numbers so with that I'll just kind of remind you all that at every single stage of the product development process you can make a difference and add inclusion and with that I'll open it up for q and a uh question in the back oh thanks yeah How do you strike the balance between inclusion as well as perfection in some sense, and then especially, you're saying cover girl, I'm actually less familiar with what cover girl has been doing, but the, ah, yeah, yeah, yeah, right, right, yeah, so the example being given is on a toothpaste commercial, they're showing, you know, pearly white teeth because that's the goal, but in reality that's not really what most people's teeth look like, and how do you kind of counterbalance those two things. And so I'm a little biased here, like I really love the fact that people are starting to do advertising campaigns that are no Photoshop. So there's a couple of fashion companies like ModCloth, I think, love the charge, where they're saying we don't do Photoshop, everyone is super real. I think also Victoria's Secret had a really interesting, I think it was the summer or the spring, where one of their lingerie models, they put a post on Instagram, and either it was a mistake or not, but the model had stretch marks, and everyone was like, holy shit, Victoria's Secret models have stretch marks, yeah, that's truth. And I think that, I think people do really want or crave almost that more authentic stuff, but I don't know if it sells. I think that's really the crux of it. And I honestly don't know, but I think that if we kind of move more of what we're looking for and reinforce those messages of authenticity, I think we're going to see more of it in our advertising. At least that's my hope. So you mentioned making arguments to stakeholders for inclusiveness. Yeah. What if the data were to overwhelmingly show that a certain factor just isn't worth being inclusive about? So for example, you want to make your projects inclusive to autistic users, and there might be some moral illness to make it inclusive, but the data just overwhelmingly doesn't financially support it. Would you still push for that, generally, or not really? Yeah, so I think the question is, so if it turns out that the data counteracts the investment inclusion, so maybe it costs too much to be inclusive to that particular subgroup, what do you do? And first off, I would kind of almost applaud the team for at least having the discussion, the trade-off, and like looking really, really deep. If it gets to that point, it's really hard because at the end of the day, many of us are building businesses. I think the hope would be that you can try to make tweaks around the fringes, maybe, that will make it more inclusive, or maybe the things that are needed to support that group are also needed by other groups. I think that actually tends to be more often the case than not, is that if you look at any particular subgroup, there's going to be another subgroup that needs something similar. So maybe you can tie it to other, or you can tie the features needed for, say, an autistic user to the needs of other users and see if you can justify it that way. I'm going to, I forget who did first, but, okay, yeah. My question is for Google and Facebook, sometimes it creates a filter above all. People just see what they want to see, and sometimes people that they cannot reach the information that they, maybe they don't know yet they want to see or they need to see something. So is it inclusive or exclusive? Alright, that's a tough question. Basically, so people who are using Facebook or Google, we have a lot of personalization features in each one of those, and it can create a bubble around what types of information that person can see, and whether or not that's inclusive or not. And in some ways it's like the personalization features are incredibly inclusive, let's say, because it's exactly what the person wants to see. But I think that the recent political dialogue that we've been seeing or the notes about fake news, I think that is causing a lot of these platforms to rethink some of these policies. I don't know if I would necessarily categorize that as inclusive or exclusive, but more of a larger political and product discussion around how do we decide what people see and don't see, based off of their choice. I'll do you and then you. I'm trying to see how you can make this pragmatic what you're talking about. So let's say I'm a product manager and I want this message of inclusion to really go down the line all the way. Design, engineering, everybody else. So if I don't put it in the persona of the vehicle to really move it, how do you kind of apply one of the things I talked about and put them into practice kind of at every stage or maybe only at one of the stages if that's all that you can really access. The baby example, right? If there's no persona for the baby, how does that get? So I think one of the things that's probably the biggest lever you can pull here is the user test everything. So and that's just like a, it's a standard process in the product development cycle, and then you can just make sure that a lot of people are being brought in. So if you've kind of skipped the persona stage and the engineering phase, you can actually kind of circumvent it by planting certain people. Like, oh, we haven't had this type of user yet use our product. Let's see how it is. And then to have the user researchers be like, oh, God, that was awful. So and that can cause some repercussions to go kind of back to the drawing board. I think that all of these tips here, it's going to be kind of soul crushing probably to do all of them at once. But what I'm trying to do is give you a bunch of toolkits to think about at every stage and you can probably start kind of chipping away to make things more inclusive. But I think the user testing is going to be the place to really put the lever on if you had to pick someplace to really have people feel the pain of a product being non-inclusive. Oh, yeah. No, it's definitely not the right order of things. But it is the type of thing that if that is the only lever you have, that's probably going to be the most impactful one. If you do have the opportunity, though, to start earlier up in the final, do it. So the question is kind of how do you think about competition with products as you might have to expand the overall set of use cases that you need to support in order to be more competitive in the market? So I think there's kind of like two phases here that I didn't really quite talk about. There's the startup phase and then there's the big company phase. And when you're doing a startup, you almost kind of want to figure out who your very narrow target audience is and try to build a perfect product for them and then move further out. And then these are the tips and tricks that you can use to broaden your scope with a large company like Google where I've been at. It's like you should be thinking about the stuff all from the beginning because on day one that you launch, it's going to be millions of people who include lots of the people impacted here. So as far as the case of, say, Strava, they're in that kind of startup phase where they're finding a target audience, being really successful in them and then starting to move into more markets. I think that as you're scaling and as a startup in general, you just have to be very conscious of who you're targeting and not to go too far too soon. But if you do have, say, like, all right, now I'm going to target. So I have the pro runners and now I have the amateur runners. Can I get some of those amateur runners? Can I get a diverse set of people in that space? So you don't want to go, like, all right, in order for us to be inclusive, now we're doing swimmers and bikers and people who are doing synchronized swimming. Like, it's going to be probably a little too much to do, but within each target set that you have, if you have a business case for it, try to make that user base as inclusive or diverse as possible so you can make your product inclusive within that space. That was a really good talk. How would you approach B2B products rather than when you're trying to build inclusive products? Okay. So how would I approach B2B product management with regards to inclusion? So I worked on Google Analytics for a number of years and it was kind of remarkable how B2B versus consumer product development is incredibly different. The way I like to characterize it is that B2B, people just really want to tell you what they want, whereas consumers are like, I don't know what I want. Everything's great. And then you have to kind of narrow in to figure out what it is that they actually need. For B2B, I think again kind of going back to the question of who's your target audience, and then trying to figure out are there different flavors that you want within that target audience? So if you were to go after, gosh, okay. Lots of different thoughts. So one is just like making sure that the businesses that you're servicing have a lot of different characteristics. So you might find, say, I'm thinking in my mind that there was one particular company that worked with us all the time on Google Analytics and they ran a small like home rental in North Carolina. And they were fantastic. They were always helping us with everything. But they were this small company that really loved to use Google Analytics. But we needed to find other people in that kind of portfolio of roughly the same number of users, same amount of needs, same amount of spend that was happening. So you want to try to find other use cases or other companies that kind of fit the same profile to create a persona. So that's kind of like in the business case. But then also to kind of come back to some of these things. So you're still going to run into things like the accessibility engineering, the color blindness stuff. I think there's things that span both consumer and B2B when it comes to the actual experience of using the product as far as the usability that you can also kind of suss out with user testing. So it's based off of user testing, if a bunch of people give you the same feedback or different feedback. Totally different feedback. Yeah. So if you have, so just my rule of thumb with user testing in general is that you try to, like you want to have a certain end of people and beyond a certain end it ends up being a little bit repetitive. And so if it turns out that everyone is giving you very, very different feedback, then there might be something actually more wrong at the core than it being that there's a bunch of different user issues. So you'll find that if you have a diverse set of users they'll still find the same usability issues. If you do like eight interviews probably like five of them will find the same thing if it's like really a usability issue. So I think what I would kind of do there is that if everyone is giving you feedback and they're struggling there might be something a little more core happening at the base than it being a particular usability issue. Thanks for this is really meaningful discussion. I'm wondering about in terms of like how would you best like measure the ROI for more inclusive products. Yeah. Yeah. So how do you measure the ROI of inclusiveness and report back to stakeholders. That one's really hard and I think the part of the reason why building for inclusiveness is really hard is because sometimes it's this kind of fuzzy thing that you can't put a number to. Whereas like if you put a number to dollars that speaks really, really well. I think that stories of inclusion so a lot of the brand stuff or the qualitative stuff ends up being slightly more impactful here. But you can also if you have the opportunity to get data if you have a lot of data and you can actually ask people maybe for the demographics in a user study or an opt in or something you don't want to as I was saying earlier you don't necessarily want to have to force people to ask questions but you can mine around that data and say like well this particular user base with these demographics is not spending as much money as this other one. There's a problem here like let's fix it because they could be spending as much as this. You can kind of if you have access to that data it can be hard to do at scale. So I do find that the inclusive stuff is a little bit easier to sell as stories. And I'll get you in the back in a second. So thanks for coming Laura. So with all these factors you have to consider for building inclusive products. Do you ever find yourself having a problem if you want to address it first? Oh yeah. So how do you determine what the top priority is? Yeah so I think so the question is how do you prioritize across all these different things and I'm like smiling at my team back there because you know we're working on some of these things but it's hard to tackle all of them. And I think part of it is figuring out like what's a quick win. You know if you can have the opportunity to a quick win so we had a progress bar in our app and we actually have someone on our team who's colorblind he's like I can't tell the difference here and you're like okay let's just fix that it's a hex value it's super easy. So let's do the quick wins. I also think that if you're able to do some of the higher up funnel stuff it actually makes it inclusive from the beginning. So if you are able to actually find diverse personas you're doing a lot more of the work up front to make things inclusive. But as far as all the features around the accessibility as far as the you know yeah there's it's like a lot I think there's like a longer tail of engineering work that you can put in here. My philosophy around this so far has been you don't want to be reinvesting on things that are that you're just going to have to redo over and over again. So on our team we haven't quite done some of the accessibility engineering yet because we're still moving things around so much. But if you have the opportunity that you know that the investment that you're making is going to be worth it and you're not going to have to dig it back up again go ahead and throw that time in and it'll be worth it. You in the back. So there's a comment earlier about the way you put in the industry in terms of finding certain things. Yeah. So are there any like exercises or things you might need to kind of check the functional bio challenges in the initial development phase. For example if you try to build a product that serves a wide range of technical ability. Yeah. Like when you look at the kingdom your designers and engineers are all fairly tech savvy and you might be making assumptions that you don't even realize you're doing like oh people are going to know this is the manual. How does it navigate. Yeah. Until what you need to do. Yeah. So are there certain things that you might recommend to you in the initial phase that are kind of counter-attacked internal bias. Yeah. So the question is so if how do you counteract the kind of philosophy of all your user testing at the end because if you catch stuff at the end it might be a little too late. So what can you do to earlier on in the process to catch biases that might be inherent in your design. And I think for that it's you don't have to wait until the end for user testing. For the products that I've worked on at Google we've actually done user testing even before we started building. So if you have the opportunity now to test something you're going to get valuable information. Another just kind of product manager tip trick I think a lot of user I learned this from user researchers which is if you can do lower fidelity mocks it actually produces better feedback on the more core concepts. So you don't have to wait for things to be in prototype or even in high fidelity mocks in order for you to start testing. You can do like Balsamic is one of my favorite tools where it's just they intentionally use a variant of comic stands almost for all the UI. But it's because you can start people's feedback. They're like oh this is something you're just testing out so I'll just kind of go a little bit crazier and give you better feedback. So we on our project actually did wire frame testing before we even started building anything. And I would strongly encourage you to be doing these types of check-ins regularly with real users and also to back to the kind of B2B question. When I was working on a project called Google Tag Manager one of the things we did was we started kind of building but then we took made a PowerPoint presentation and about the product and did a road show across the country with a couple of different advertising agencies and they were able to say like hey you know what this really sucks and you totally miss the mark on what the important feature is here. And so we were able to do that before we started building and actually ended up building in an account management system that was much more on point with what the user wanted before we built it which was great. So it's never too early for user testing. You can't do too much. I wish we had bandwidth for more honestly. Two more questions. Okay, you two. I'll do you first and then you. How much time do you invest in user testing? And I think there's a spectrum here. What I've worked with the majority of user testing that I do is bringing people in for half an hour and just getting their eyeballs on it. And no more than two days, probably no more than eight participants. And that'll just kind of give you some feedback. You can do these kinds of road show things and that can take like a week or so. There is a type of user research that happens at Google. If you have a full-time UX researcher, they can go and like do field research. They can like go and do these like crazy expensive surveys and do survey 2,000 people and come up with all these like really awesome insights. That's really intensive and you may not have time for it. I find these kind of ad hoc, you know, eight participants max type and user studies to be really helpful. So, yeah. The question comes out of a personal experience. I'm traveling today. Do you have a moment? I'm from Germany. Cool. Welcome. I'm with my family and my little daughter. I'm the perfect home user for a bit with a car seat. Okay. There is a product out there. And I can't download it because I don't have access to an US app store. Oh, yeah. It's kind of confusing. There's no way to get to this product. The app store has been tailored for me. I'm a colleague. I'm spending time with this my baby. And the designs of the product obviously don't do this. I can't inform them of that. So when you figure that out, have you managed to optimize whether they've gone and analyzed passive feedback as opposed to user testing work where you can get blinds out of it because you're looking for the people and you can look and you think you're right. But the unknown unknowns, those things that you don't know, you're missing, you're never going to look in that particular corner. Yeah. But that message will find you if you have to check the other channel. Yeah. So the question is like how do you, so user testing isn't really the best way to get everything. There's a bias in user testing. And how do you actually create channels for getting passive user feedback or things that may not involve having an in-person thing. And I think to the extent that you can make those channels available as much as possible. So we're working on like different ways in which there's a product called Instabug. We can just like shake stuff and then user feedback pops up and that's a way for people to very easily give you feedback. You can also, like we look at all of the reviews on the App Store for our app and look at that. We have an email address. We, you know, you just try to make it possible for people to contact you in as many ways as possible and try to look at that feedback. If it turns out there's too much, then you try to figure out a way to scale that. But for early stage projects that usually have a smaller team, you probably won't be overwhelmed by how much people are contacting you. And just make sure that you can plaster your email address or whatever as a product in as many places as possible and help people write to you. And also survey people too. So one of the things we also do is if people aren't giving us feedback, we'll proactively send out a survey to try to solicit user feedback.