 I'm Erica Hall from San Francisco, founder of Mule Design. I'm excited to be with you. We're going to go on a journey of a lot of information. A lot of ideas stuff. Yeah, this is cool. But because my talk is about asking questions, I'm going to start with a question for all of you. And I want you to look in your hearts and answer it really honestly, very candidly. Do you enjoy being right? Raise your hands. You like having the right answer. OK, it looks like most of you are truthful people. There. You are right. Hold on to that feeling. Hold on. It's a good feeling. And I'll tell you something about me. I love being right. Ah, it's the best feeling. I was that kid in class, the one like, call on me. Call on me. Jumping in there. Right answer. Correcting people's jokes. Very popular child on the playground. And so then I always wanted to have the right answer. And so then when I went to college, I thought, you know what, I'm going to study philosophy. Because you know what philosophy is? It's the art of winning arguments. So it essentially weaponized this tendency. And I learned how to create the conditions in which I would always be right. Because you set the terms of the argument. You define like, oh, we're going to be talking about ethics. And for the purpose of this conversation, I'm going to define ethics as this. I'm going to define free will as that. And you can set the terms and create an argument and dominate in your face socrates. But then I graduated. I went out into the real world. And I'm the rightest person. I have a degree in right. Praise me. I'm so awesome. But I learned that this desire I had to be right, this good feeling I wanted to hang on to, had a lot of really unfortunate effects. One, it made me really annoying. It made me a very bad listener. Because I was that person, you ask them a question, and they jump in with the answer before you're even done. And you're like, that wasn't even my question. And the worst effect it had because of that is my desire to be right, to have the right answer, increased the chance that I was actually wrong. Because I was not listening. I was just trying to prove how smart I was. So because of the intervention of many kind friends and colleagues, eventually I got over that. And there's nothing wrong with wanting to find the right answer. That's a good thing. I'm not telling you, go out and be wrong and love it. What I'm saying is that it's important to want to find the right answer. But that is not the same as wanting to have the right answer inside and to prove that you're right. But the problem is that throughout our educations and a lot of times in our careers, we are not rewarded for having questions. We're rewarded for having answers. But the truth is that a good question is much more powerful and much more effective than a good answer because answers have a really short shelf life. Like say you were designing something for high school students and you had a question like, oh, how do high school students spend their time? Like maybe you do a little research and you get some answers and you're like, sweet. We have the answers. But you can't stop there. You have to keep asking that same question because otherwise the world will change. Things will move on. High school students will spend their free time in a different way. But you'll still be holding on to that answer. You're like, no, I was done learning. I learned, I did it, I'm done. But this is so often how people in design and business and technology treat asking questions. You're just trying to find your answer as quickly as possible so you can move on. And when we're coding or writing or designing, like when you're sitting at your keyboard, you feel like you have so much control. Every pixel goes where you want it to go. Every word goes where you want it to go. It's like, ah, I'm creating this new thing. I'm so excited about this active creativity. But then what happens is you take whatever you've developed at your keyboard in front of your screen and you send it out into the real world, where you don't have any control. And people don't necessarily care about what you have to put out there. And the only way to increase your chances of success is to learn about what's going on out in the world because you can't control it, but you can influence it. And unlike a philosophical discussion, you cannot set the terms. You can't define what's gonna happen for people. You can only make assumptions. And those assumptions are risks unless those assumptions are well-informed. Because you're like, oh, we're gonna assume that people really value what we're putting out there. But if you don't do your research and you don't ask the right questions, you're not going to know if that's really true. So I'll bring it back to a little philosophy. Uncertainty is an uncomfortable position, but certainty is an absurd one. And the problem is asking questions is really, really uncomfortable. And a lot of people who advocate for research, I think, don't emphasize that enough. They just say, of course, of course everybody wants to do research, but you have to recognize that as people who are often really rewarded for having the right answer, asking a question or admitting that you're uncertain is deeply, deeply, deeply uncomfortable. Because being right, that is a good feeling. Not knowing is a really, really bad feeling. And that's at the heart of a lot of the objections to doing the research in the way that we should be doing it. The good news is that you can start wherever you are. Just like right now, if you shift your mindset from thinking, oh, having the answer is really, really good and really powerful to asking questions is really good and I wanna be a curious person and I wanna ask a lot of questions. I don't wanna just hang on to that really good feeling. I wanna live in that place of uncertainty. All you have to do, all, is make that shift. And you will be able to start gathering more and more information that increases the chance that you actually have the right answer, but it's going to be uncomfortable. And that discomfort is the greatest barrier to learning. So the place I like to start is asking the question, why is design so hard? And this manifests in terms of all of the bad design out in the world. I look around in the world as a designer and I'm very curious and whenever I interact with something that seems hard to use or awkward or not very aesthetically pleasing, I ask, how did bad design happen? Because like you look on Amazon, there are like 10 million books about design. The principles of good design are really, really well known. They're out there. So how does bad design happen? Like this is a question, I have it so curious. Because there's so many kinds of bad design. I look at my local newspaper website and I see some less than optimal design. Like maybe it looks a little cluttered, but you don't understand. That is all the article content on that page. That's it. That's it. Then there's stuff. I'm not even showing you. There's like four more feet of scroll. And I look at that and I think, okay, I'm sure that the people who worked on this know how to design a really effective and easy to read article template. We've designed a lot of editorial sites, but it's like, how did that happen? That's so interesting to me. Or what I call the Jared Spool commemorative example. United Airlines. I am sure that a lot of very intelligent, skilled people worked on the site. And yet this was the result. And this is a little unfair because they've redesigned recently. So, I guess there's an area of focus now, but then there's some other wacky areas with just a lot of stuff. They like post-its. I think they took post-its a little to literally in their design process. Like, how did this happen? And then you think, well, you know, some companies don't have design in their DNA. You know, some companies are just, they're full of design geniuses who are fantastic and always make good choices. And if you've used iTunes lately, I've been running, listening to my phone, trying to change a song. And I've like thrown my phone across the park because it's like, how does this even work? Marco Armit, the CT of Tumblr and founder of Instapaper, has great phrase. It's a toxic hell stew. So, this is an amazing mystery. How did the most valuable company on the planet who everybody else copies because their design is so genius, how did they do this? How? How is that possible? And their companies that have really, really good design and really, really good customer experience. Virgin America, I love Virgin America. Everything about it was so well thought out. I loved flying on their planes. I loved ordering my food at my seat. You know, I love the booking experience. They really got it right. They totally focused on what the customer wanted. They didn't exist anymore. So, I'm here to bring you the bad news that there is no ROI of customer-centered design. There's a business and there's a set of choices you make and we've labeled some of those choices design but just because something is very pleasing to the customer and beautiful to look at does not necessarily mean it's going to be good for your business. And then some things are just mysterious. This looks like a thing that was created by people who knew what they were doing but then ask what were they doing? Video bills. Okay, I'll explain this. Time Warner Cable, company that provides access to a lot of video content, thought, huh, maybe while you're waiting for the next season of Game of Thrones, maybe you've run out of things to watch so you'd like to watch your bill. He's so happy. Oh, it's such an interesting good, oh, oh no, everybody just got killed in my bill. The red wedding of my bill. And this is bad design because of the opportunity cost. Anybody who's working on something like this is not doing something more useful for the world. And sure, you could make a case that, well, video bills are good for accessibility. But hey, why is your bill so complicated that it provides enough material for a video? That's a good question to ask. So, and then my favorite in the last couple of years, many, many millions of dollars went into the innovation and creation and engineering of a device that squeezes bags of frozen fruit. Looks nice, but why, how did this happen, right? So it's sort of mysterious. And I'll tell you this exercise, this like, how is the meeting, how did this happen? This automatically makes anything really annoying you're dealing with kind of interesting and fun, because you start thinking, how did this happen? Why is there bad design in the world? And there are new ways of having bad design. There are new technologies. A Bank of America, I take this one real personally, has developed an AI assistant named Erica. Yeah, not displeased. Displeased. So you're like, why is it gendered? Why are they calling it she? What's wrong with online banking? There's another sort of marketing campaign that says tap on Erica. No, do not tap on Erica. Do not tap on Erica. So, what do all of these things have in common? So this is a range of less than ideal design across many media. The thing that all have in common is people. And the reason design goes wrong is because people want to have the right answer. Because somebody had an answer, somebody said, you know what, I wanna have a really fancy machine that squeezes bags of frozen juice. And I'm gonna put that out in the world. I wanna create, like everybody else is doing AI, chat bot things with names and genders. Let's just do one, I wanna have it. So the reason why there's bad design out in the world is because people were more interested in defending their answer throughout the process than really asking questions, asking like, is this something people need? Do we have all the resources to do it right? Is designing something like this and really serving the customer need, is that really gonna help us make money? Or is there something else we should be focusing on in addition that will help our business be successful? And the problem with people is that we are riddled with cognitive biases. You're not meant to read things on this chart that's based on the Wikipedia page for cognitive biases that I really recommend looking at. The point is that we have so many, I can't even fit them all in one slide. So we're riddled with cognitive biases. We wanna be right and we go into these processes where we're trying to design new, amazing things in the world because that's all design is. Anytime you wanna go from what currently exists to creating something new and putting it out in the world, and this is large or small. This could be a cool new chair or this could be a whole new system that helps people pay their taxes in a simpler and easier way. Any level of thing you wanna get from the current state what actually exists to some better future state. And in order to do that, you have to understand the current state. We've actually had fights with clients when they say, oh no, no, we don't want you to think about the way the world is now. We have blue sky, blue sky. Just imagine the best case scenario. And they think they're making our jobs easier by doing that. But they're not because any innovation that changes the world fits into the world. But that's uncomfortable because then we have to ask questions about the world and we have to think about a lot of really suboptimal things. So a design project, you know, I construed that very, very broadly. All it is is a series of decisions. Like we really like to focus on the artifacts. Like, oh, design is creating things and making things. But what it really is is making decisions about things. It's making choices. And the reason we need to ask questions and do research is because research leads to evidence-based decisions. And this sounds like it should be obvious because if you're not basing your decisions on evidence, what are you basing them on? So anytime somebody goes into a design process without doing research, ask what are their decisions based on if not evidence? Fear, confirmation bias, having an answer they found a long time ago. People like to feel in control, right? Because uncertainty is so uncomfortable. And so they even like to, when they talk about like different ways of gathering information, how often is there a diagram like that where it's like, oh, let's just put it in little hexagons. Let's make it tidy, let's control it. This is just a natural impulse to deal with how messy the world is. But the world is messy. And a key to this, a really important part of doing design research is collaboration. And we'll talk about it for a moment because like, you know, open source software, that's pretty collaborative. But then maybe you go out into the world and you're dealing with clients who are in a business context and all of a sudden you feel like, oh, we're not collaborating so well. What's going on? We all, those are all nice people here. Collaboration in business is so rare. There have never been any actual photos of it. That's why you see this. That's why stock photographers are like, oh, I've never seen people collaborate. Yeah, put all your hands in. Sure. People can work alongside each other for a decade and never truly collaborate. Because all collaboration means is working together towards a shared goal. But a lot of times people never check to make sure all the people working together actually have the same goal. And collaboration is hard for some of the same reasons. Design is hard, right? It's hard because of ego, because people like to have the right idea. They like to have the winning idea. But I don't really blame people for that, right? Some designers have a bad reputation like, oh, they're so egotistical they want everybody to listen to their designs. But what we call ego is really a problem of bad incentives. A lot of organizations reward the person who confidently says an idea. And that's why we run into issues with bias and discrimination in the workplace. And as a company you hear about people stealing other people's ideas. Why would that even happen unless the organization was rewarding an individual contribution more than rewarding a team achieving a goal? So I don't blame the individual I kind of blame the system around them. Organizations put people in silos, right? It's like, oh, visual designers here researchers here developers here backend engineers over here. That's not how anybody experiences a product service software out in the world. It's not like, oh, now I will look at the code. Oh, now I will look at the words. But people aren't put in a cross disciplinary collaborative situation. And a lot of times there's competition for internal resources. Oh, we're all supposed to be working together but no, you get the researcher I get the researcher you get the writer, right? And so then you're not in a really good position to be collaborating with each other. Or people are far apart either physically or conceptually with no facilitation. It doesn't just happen on its own. Like sometimes you do get an organization where you can sort of walk by somebody in a hallway and like, oh, yeah, we have to resolve a small conflict but without good facilitation collaboration does not happen on its own. And academic research, right? Is not traditionally collaborative. So when you just bring research into an organization a lot of times what that organization will do is just put the researchers over in a corner. Like I talked to some people who worked for a large technology company who said the researchers literally had a different building with like a different air circulation system. And so you'll get the people learning the knowledge over here and then the people solving the problem making the decision over here. And if that's not being brought together you don't have well-informed collaborative decision-making you don't have the conditions for good design. Collaboration requires both intent people have to want to do it and they have to be rewarded to do it or people will do the thing that they're rewarded for. And a lot of times people are more rewarded for coming up with an answer that sounds good. And Dan Brown, my friend Dan Brown with the worst SEO has written several very good books. I feel that, like whatever I look at them up I have to go through like 10,000 different versions of the Da Vinci Code. And he read this great book called Designing Together which is about design collaboration that's really good and he said collaboration requires behavior change and that's why it's so rare because as we know if you're designing products for people out in the world you know like changing people's behavior is really really hard. It's the same in an organization. So it's really hard and you have to be intentional about it the same way you have to be intentional about design. You have to have a plan, provide a rationale, define roles and responsibilities, set expectations, communicate progress, reflect on performance. You have to do every single one of these things or you're not being collaborative. If this isn't part of your process you're not gonna be working collaboratively but this feels like extra work and it's like it's so much easier to just like keep delivering things right to keep making things without worrying about being collaborative and you have to embrace conflict. This is the counterintuitive part is that being collaborative involves conflict because the difference between a collaborative process and a consensus-driven process like we've all heard like oh, consensus-driven design is really really bad. That's because if you're trying to achieve consensus what you're trying to achieve is everyone agreeing. If you're trying to collaborate what you're doing is working towards a shared goal and those two things are very very different. As long as you agree on the goal you can have a really productive exchange of strongly held ideas in the service of that goal but the problem is a lot of times this sort of interaction is defined as soft skills and people don't really learn how to have good conflict so it gets personal, it gets toxic and then people just avoid it and they optimize for agreeing and that's where you get the consensus-driven design process because you're like oh, but we all agreed why is the design bad? Well because you were aiming to agree you weren't aiming to achieve your goal together and this stuff's all really really hard but not for the reasons people think it's hard. It's not hard because the technique to learn is hard like the research process is hard or the software is hard, it's hard because we have to overcome something very deep inside ourselves that isn't often recognized. And then the really hard part so once you embrace like yes I have intellectual curiosity I will be in the place of uncertainty I will gather the data, I will ask the questions and then you bring the data to somebody else and you're like I found the data and they just reject it and you're like wait a second I did everything right but the problem is when you're trying to influence decision-making you're up against people who wanna have the right idea and you're up against the fact that new data doesn't change minds like there are plenty of social science studies that show this like if you come to somebody with new information that challenges their beliefs they'll just reject it and they'll dig in harder and so when researchers come into this they're often not equipped to deal with that political reality and I've heard some really really unfortunate stories about that. This is why we get so many objections to doing research like I've heard all of these again and again like oh we don't have the time oh we've got the time to build something but we don't have the time to determine whether we're building the right thing same thing for money right we're totally willing to like set fire to money so that we don't have to do research oh we don't have the expertise same thing how do you have the expertise to design and build something and you don't have the expertise to determine whether you're designing and building the right thing for the right reasons we have specialists this is what I mentioned they're often in another building generating reports for people to ignore oh we'll just AB test we'll just have we have analytics that's more concrete right that's more certain you know what talking to people gross messy let's make a prototype way more fun right and making a prototype is a type of research but it is often a very expensive way to learn but because a prototype is an answer it's much more comfortable and that's really the reason people do it and of course even to this day billionaire geniuses who change the planet didn't do research so why should I do research geniuses don't have to do research genius I have the answer so we hear these objections but there are so many advantages it actually makes decisions go faster like when I work with clients I tell them a design project moves at the speed of decision making if you don't have your research if you don't have people working from the same information then you'll take a really really long time to make a decision right if you've done your research then decisions can go really really quickly if people are working from the same information you decrease your risk you save money you increase the value of everything you're making something more valuable every member of the team is more valuable you get continuing returns because this is the thing about asking questions and learning if everybody's asking questions and learning then people are collaborating better and better and getting more and more well informed and making better decisions it's not like every time you have to start from scratch an organization, a team can develop this knowledge base over time and develop this curiosity and keep asking more and better questions everyone is more effective and more efficient but the reason that you hear these objections it's all a smoke screen it's all because people want to preserve that really really good feeling of being right and don't want to go to the place of being uncertain and you have to make sure that everybody on a team isn't just working from their personal view because you could say, yeah yeah yeah we all have a shared goal we all know what we're supposed to be doing but unless you check you don't know because the goal of doing research together of asking questions and finding new information and answers together is to work in a shared reality and there's often a misconception that any sort of research is unsuccessful unless you discover something like wildly new and a lot of times decision makers use this as an excuse to not do research to say oh we already know everything but it's like does everybody working together to make decisions to do the work do they all know the same things? Have you checked? And by working together as a team and asking questions collaboratively what you do is ensure that you're all working from the same shared reality because otherwise you'll find out way too late that you have very very different ideas about what the business priorities are what the customer needs how you're gonna value success and the place to start because you can't start by talking somebody into research like I said you can't go to somebody and say oh you think you have an answer let me actually tell you so when you go to somebody who thinks they have the answer and you tell them they actually don't and they need to do research you are telling them that they're ignorant and wrong and that is not a way to get somebody on your side right this is why if you've ever gone to somebody with authority and said we really need to do research what you're telling them is that they're wrong that is not a place to start collaboration you have to start from point of agreement and the point of agreement I found the one common point of agreement is generally everyone wants to develop better products faster if you're in an organization or a working environment where that is controversial take a moment maybe find a different one like oh I wanna do worse things slower I mean sometimes it feels like organizations do wanna do that and also no one wants to read a research report the goal of design research is not to generate reports not even researchers like to read research reports but for some reason there's this idea about oh that's what research is for it's for making a really good professional looking report but it's not you're just asking questions and getting new information so you can make better products faster you know you're successful if you're making better products faster and working more collaboratively here I have a historic video to show you in case anybody hits this particular objection Hi for those of you that don't know me my name's Steve Jobs and this is the first of one of many chalk talks we're gonna have this year together the subject of this one is really important which is who is our target customer why are they selecting our products over our competitions and what distribution channels are we going to use to reach these customers a lot of light bulbs have come on over the last 90 days I've had the good fortune to be with a lot of you out in the field meeting customers getting first-hand information as to what they're doing with our products you have fed a lot of information to the management of this company we've done a lot of thinking and looked at the data and all of a sudden out of this data some very very important things have come to light I want to share them with you today even geniuses do research and then because they're geniuses their marketing tells you there's no research they just know what they're doing so a lot of times design research gets conflated with usability testing or user research but you kind of have to understand the whole burrito there's your organization like you and your client or your internal team there's your uses and customers and then there's the whole real world context and these are all the things that you don't control but need to know about but the good thing is the actual research process is really simple the most important thing is not to be dogmatic about it there's so many different things you could do and everybody's looking for what's the method what's the right method so I can again find my answer and not have to ask questions anymore and then they argue about artifacts and ways to like oh is it personas is it jobs to be done meh meh meh meh designers are always finding ways to like carve out territory and have these arguments about methods in order to avoid I think getting down to that really really scary place but all you have to do is make sure that you're being goal driven and being very skeptical this means always have a goal and always be willing to question your methods and never find something and just like hang on to it and say no this is the one right method because it's not because the process the simple process is just forming questions gathering data and analyzing the data that's it if you do all of these steps you're doing research correctly if you skip a step plenty of people just do the gather data part if you just do the gather data part you're not doing research right and then you get insights those feed into any decision making process you have and then hopefully you're more successful because of it but again you also have to have a really good decision making process as well you can't go out and find all this data and feed it into a process where people are too focused on proving they have the right answer research activities are only ways to answer questions that's it and you need goals in order to give the research purpose if you don't have a goal you can't start down this path at all and you need questions to give data meaning so forming questions is the most important part of the process and the most overlooked only after you have goals can you form questions because all you're trying to do is place better bets on human behavior that's all business is you just want to have more well informed bets because you can never have certainty right you can have higher levels of confidence so you have to ask the right questions and there's a lot of confusion around this there's so much more talk about what method than how to form questions but if you don't ask the right questions it doesn't matter how good your data is a good question is specific it's actionable, it's practical a really bad common question how do we get millennials to like us? Businesses ask this constantly it's meaningless it's not specific millennials aren't really they don't have behavioral differences but it's like ah we've come up with a way to avoid asking meaningful questions a better question is more specific you might ask how do recent college graduates decide what to have for dinner a really good question is asking your team your organization what you really know get in a room like this could be a really scary question to ask you don't even have to look outside your team to really say what do we know and how do we know we know it the best question is the unknown that carries the most risk and examples of risks are people don't value what you're making maybe you're listening to the wrong customers maybe your business model doesn't support it you know maybe somebody else is doing it better or you know maybe if you do succeed you will destroy the planet that's a good thing to know before you go down a path and another important thing to think about is what you need to know and what you can ask people are very very different research questions are not interview questions and an interview might not even be the best way to get the information you need to design for example if you want to know how people spend money on vacation you can't just ask them that because people start lying to you, right? but you might ask them oh walk me through your last vacation you might want to know how likely people are to adopt your product like you see this and like how likely are you you see this as a literal survey question all the time nobody can answer this question but you have to see what products people already use and then make the judgment to figure out how likely they are so there are many stages in the process when you can ask questions like you don't know what to ask what's going on, how does it happen how well does our solution work to solve the problem people jump to this too quickly why is this happening you can't just look at measurements you can't look at quantitative data and get causation out of it but if that's what you need to know you have to know that that's what you need to know before you can ask the right questions so then the gathering data part you have to start with your goal, your topic, how much time the method you use is totally up to these factors, right you have to start with your goal you can't start with a method I've talked to people in so many organizations who say we talk to customers every Friday I'm like why and it's like because we feel like we have to oh we run surveys, why oh we go out and hang out people in their homes really is that the best use or time and money you know it all depends there's no one right method it's whichever one answers your question and is the most practical and useful and you have to think about whether you need to know something quantitative or qualitative so the nice thing again about dealing with research and data it sounds so complicated it's really not when you get down to kind of the base principles the only types of data you have to deal with are data you can measure and data you can't measure and you've got to be really clear on which answers your question this is from the Manga Guide to Statistics which I highly recommend you need both you need descriptions and measurements one is not better than the other you need to know whether you're asking a question that demands a measurement or a question that demands a description measurements will never tell you why and then you choose the activities that will give you the kind of answers that answer the type of question you have and one of the problems is that because of this sort of anxiety about research and asking questions is that people doing qualitative research try to apply a quantitative standard to it because it feels more science-y and so the idea that you have a fixed and measurable reality that you need to do a quantitative study people misapply that to qualitative research and that's why you get things like people in labs whether they should be out in the field that's why you get eye tracking totally unnecessarily when you can just do a usability test because there's a sense of if it's more controlled it's more science-y and it's more accurate but that is not right quantitative data doesn't capture the context and the most important thing is not to just go on fishing trips you can't just look at data and expect to learn anything that's just going to be a way for your biases to creep in because you'll be looking to be proven right you won't be looking for the right answer out in the world and why not just make a prototype? Yeah, that's a way of learning prototyping is cool, prototype and test prototypes are comfortable because it's an answer you don't want to go straight to the answer and oh, let's just see how right I am that's what a prototype is really asking and Victor Lombardi had a great, great quote for this if we only test bottle openers we never realize customers prefer screw top bottles and I can tell you when I get home from a long day working with clients I will pick the bottle I can just crack it right open I don't want a fancy I don't want like a $300 fancy rabbit wine opener but this is actually because people are so irrational this has actually led to another design problem which is sommeliers in high-end restaurants they can't bring out like oh oh you ordered a $500 wine let me crack that open for you so there's now a new tool which opens a screw top bottle with a certain amount of flourish it's true because people not rational not rational, nobody makes rational decisions and it's so easy to mistake the polish of a prototype for the quality of an idea right you're like oh and then people get precious about it nobody wants to critique a prototype that looks beautiful if the idea at the core is not a good idea and surveys I'll take a moment for surveys the problem with surveys is they're too easy to run and they generate data that's so measurable and countable and feels so certain that they're just comfortable it's like look at all these answers oh I got bad answers at scale that's cool that makes me feel good and you see like man I see these all the time somebody wrote to me about 4C surveys and these pop up on every single site and 4C is very popular because it purports to do the alchemy of turning qualitative into quantitative right so if you look at these questions rate how well the site's organized rate how well the site who thinks like this no human thinks like this and my favorite rate the number of clicks so that people are gonna like pick two yeah it only took me two clicks cool these are terrible but they're so popular because they give answers that feel good and so if you're thinking about running a survey don't do it cause running a survey is easy do it because it's the best way to answer a question and if you do run the survey focus on asking things that people can actually answer and ask it at the right time just to send out a blanket survey to people you will not get good data but it will feel really good you'll be like oh I didn't have to talk to people people are terrifying and then you have to analyze you have to go to this step you can't just ask questions and look at data you then have to ask what the data means and you have to do this with your team so everybody understands what it means and you don't have people drawing different conclusions and then you end up doing all this research and fighting about personal opinion anyway and wasting all this time and then saying like oh it's not even do research it was a total waste of time knowledge is not useful unless it's shared if knowledge is just in one person's head it will not benefit the team it will not save time it's not about having an answer it's about having everybody working in a shared reality and the best way to analyze especially qualitative data or any data is to talk it through with everyone who has to make choices and decisions based on that data but sometimes you know sometimes you work with your team and then you have to report it to other people and you've got to remember that data doesn't change minds so you can't just be collaborative with your team and say oh we're totally on board we're asking questions we're being uncomfortable we're being open to new information and then we're going to take it to a decision maker and they go like like I talked to a researcher once who was working at a big tech company and she was talking to this VP of product and she saw his roadmap and she's like oh wow I have data that totally contradicts that so she made an appointment with him for the following Monday morning went back over her data made a big report came in Monday to find out that he'd canceled the meeting power Trump's data it does and she was blindsided because she didn't have anything she's like all I have is getting the best quality data I don't have the tools to overcome politics and overcome a fear of being open to asking questions and being open to not knowing things so if you have to go in and talk somebody into your findings always start from common ground always start from agreeing and then introduce new information because you cannot win arguments with data you can only win arguments with questions and by starting with common ground start with shared goals tell a story like be fun I've been to so many boring research reports right they come in they do a presentation they talk to their methodology don't be afraid of putting on a show people like shows and then keep it brief keep it to the point and create a habit of collaboration because again it's too easy to do the things that you're rewarded to do for yourself and not collaborate with your team because a lot of times there are not incentives in place for collaboration so it's a discipline that gets easier over time the same way that asking questions gets easy so you always have to keep why you're doing what you're doing at the center of everything you do and if you create this as a practice and you develop this intellectual curiosity and you think like wow it's not just creativity it's curiosity that everybody needs to cultivate that's an innate thing that can get you know smushed down by wrong incentives you think okay it's fun it's more fun to do things to design things to put things out in the world when we know why we're doing it and then we have a better chance of conveying to other people out in the world why they should care and this will be better for everything thank you Erica thank you so much that was wonderful does anyone have any questions for Erica just go ahead and raise your hand and I will bring you a microphone hi so first I want to say I really enjoyed your talk thank you and what you hit on were a lot of things that were very useful that I can take away with the way I deal with my team but sometimes and more common than I like to admit a client says no I have this terrible idea and I know you're making a great thing but I don't want a great thing I want this terrible idea and that becomes a situation where okay disagreement and pushback can lead to chargeback and losing client right how do you deal with that and stay honest to these principles mm-hmm the way we've always done it is organizational research is non-negotiable in our process we go right away and we tell them because we explain like the what slows things down is the speed of decision making and we need to be well informed and in order to serve you we need to understand you because what's a good solution for one organization might not be a good one for another and so we start by understanding them and understand really understanding what their goal is because if the goal is not more important than that person that decision maker's solution you will never win you just won't and you have to establish that in the beginning you have to lay the groundwork we talk about creating a framework for decision making and so we go in and we talk to the client and we talk about the decision this is why uh... my the message I'm trying to hammer home with designers as much as possible is take the emphasis off the deliverables onto the decision making so we talk to the client we're like here's the goal what are all the decisions we need to reach that goal what are all the all the artifacts we're gonna be creating to support the decisions so we take the artifacts and knock their importance back a little bit and say you need to make a good decision we're gonna show you things that help you make that decision okay what evidence are we gonna base that decision on and so we have to explicitly know how a client team makes decisions before we insert any recommendations into that process because if they're not making decisions based on evidence there is nothing you can do you have to understand what the standard is for decision making and it's hard and once that once uh... you know the horse is out of the barn or whatever you can't put it back you can't say oh we're shifting the basis of truth you have to start at the beginning and say let's agree on what our basis of truth is before you show them any artifact or else it'll be all on polish and not on concept thank you so much can we get another round of applause for Erica and uh... I'm hanging out too because I know we're on time now and so if you just want to ask questions I'll be here in whatever location they tell me is most convenient for everybody thank you so much