 Hi everyone. Thanks for having me and welcome to the session on qualitative research 101. My name is Pam Brown and I lead the product management team for AWS certification. Previously, I was director of product management at the graduate management and mission council. I've been in product management for over a decade and I have experienced managing all parts of the product development lifecycle. My path to product management is a meandering one. I have a master's in public policy and I used to work in international development. My first job out of grad school was as a researcher looking at the impact of microfinance programs on the poor. It was actually in that role that I really got my feet wet in qualitative research. I use a lot of what I learned then in my current work as a product manager. I'm a big fan of qualitative research and I believe all product managers could benefit from understanding how to use qualitative data. Now, how did I become such a fan of qualitative research? I'll take you back to 2004 when I was living in a rural village in India working at a microfinance bank. The woman in the middle of this photo was my mentor and she remains a close family friend to this day. The women around her are some of the bank's customers. I was there trying to understand how these women managed risks, specifically health risks, as in how did they protect themselves against the financial impact of getting sick? Now, on the surface, it would be easy to assume that they simply didn't. At the time in India, there was no such thing as health insurance for the rural poor and even for the middle class in India, having health insurance was rare. But I knew from having studied about the financial lives of the poor that there's more to this than meets the eye. I knew that they had ways of saving, borrowing, lending, and all of this took place outside of the formal financial system. So, were there ways that they insured against risk without the proper health insurance, as we know it? Now, I spent a lot of time that year observing, listening, and asking questions. I visited health clinics, attended meetings of groups of women called self-help groups, and I talked to women just like those who are in this photo. And what I learned was that even though they didn't have or didn't even understand the concept of health insurance, they had ways of protecting themselves against the financial impact of sudden illness. But you wouldn't see it if you didn't know where to look. That is, if you were looking for, say, financial records or written policies, you wouldn't find anything because they didn't exist. I had to use qualitative research methods. Now, you may be thinking, okay, Pam, I get it. Use qualitative research where there is no quantitative data. But what if you manage an application, and you've got lots of quantitative data, you can see all sorts of metrics like number of active users or activation rates. You can even run AB tests and get hard data that way. Would you still need to do qualitative research? I would argue enthusiastically that yes, you do. To make this argument, I'll start with what is qualitative research. I'll then talk about why use qualitative research in product management. Then I'll discuss when to use qualitative research, and I'll move on to how to conduct qualitative research and end with some key takeaways. Let's get started. So what is qualitative research? Qualitative research involves the collection of non-numerical data. They include customer interviews, focus groups, and open-ended questions and surveys. These are all examples of qualitative research. So our product reviews, customer service feedback, and even direct observation of your users. Now, this is different from quantitative research, which relies on quantitative and numerical data. This includes everything from sales and usage reports to AB test results. Quantitative research analyzes numbers. Qualitative research gets a bad rap sometimes because some people see it as subjective or lacking in rigor. For example, if you sift through your customer service tickets for your product, you're likely going to find a lot of negative feedback and complaints about your product. That's because people who bother with contacting customer service are typically dissatisfied with your product in some way, and they don't necessarily represent the general population of your users. So there's only so much that you could extract from that. Maybe you talk to a customer, and that one customer raves about a specific product feature that you're thinking about, but that's just one customer. Again, what weight do you give that feedback? So it's true that qualitative research has limitations. It involves small sample sizes, and you can't really extrapolate it to represent the entire population of your user base. Interpretation of data can be subjective and hard to replicate, but quantitative research also has limitations. And as a product manager, you should understand the limitations of both and figure out when it is appropriate to use which type of research or data. Let's talk more about the reasons for using qualitative research. So why use qualitative research methods in product management? Well, to put simply, numbers don't tell the whole story. Numbers can tell you what is happening and when it's happening, but they can't really tell you why something is happening. Here's an example, and I've made this up. The numbers are fake, but you'll get the idea, right? Let's suppose that you manage a product that, compared to historical performance, is just not doing so well. This is a chart of sales that is going down year over year, but let's say you've got other metrics, say activations or average time of usage, and everything is sort of just going down. Let's say that is a fairly mature product. The product has been in the market for several years, so you hypothesize that maybe part of the reason for the so-so performance is because, well, the product is old and it needs some sort of updating, maybe the content to stale. Now, before diving in and just figuring out a plan for updating the product, you figure it'd be wise to do some qualitative research and get the voice of the customer. Now, through this voice of the customer, you learn that actually the users you do have are quite satisfied with the product. And sure, there can be improvements, but it doesn't seem like updating the content will necessarily have a big impact. So instead, what you find is that people are actually having trouble discovering the product or understanding its value, that these are the bigger challenges. You find people who didn't use the product at all because they just didn't understand it. They either were confused about what its value was or they confused it with something else that you offer. Based on that insight, you change your hypothesis and take a different approach. This is actually something that happened with the product that I managed. Based on the voice of customer work that I did, I actually changed my approach and focused on developing a new name and positioning for this product. I then tested it through one of our sales channels to see if I was right. The test went well and so then I moved forward with rolling out a brand new name, a new look and positioning for the product across all of our channels. The result was increased product awareness, which translated into higher sales. Had I relied on just the numbers and had I just been guessing and what the underlying problem was, I would have wasted resources on something that didn't address the core issues. Instead, I used qualitative research to understand the numbers, formed a hypothesis, conducted a test to gather more quantitative data, and then I used the quantitative data to validate my hypothesis. I'm actually a big fan of quantitative analysis. I was even a statistics TA in grad school, but I'm careful not to use it for everything. I advocate for using both quantitative and qualitative research methods to generate customer insights and drive product development. Let's talk more about when to use qualitative research methods. So when do we use it? I use it in several ways. One, I use it at the beginning of the product development process when I want to understand the customer problem. That's what I was doing in India, using qualitative research to just understand what is. I then use this information in combination with any quantitative data that I might have or maybe relevant to build my personas and customer journeys. I'll give you an example. This is a side project that I worked on a few years ago. It wasn't directly related to my job at the time, but I was asked to look at ways of improving the public bathroom experience. I focused specifically on parents of young children. I was the parent of two young children at the time who are now a little bit older, and I knew firsthand how awful the public bathroom experience can be for parents of young children. I interviewed eight parents of young children, both boys and girls, in Washington, DC, to understand their experiences at the playground, rest stops, restaurants, basically anywhere that a child may need to use a public bathroom. The findings were fascinating. I heard about all sorts of workarounds that parents have to come up with to deal with the challenges of public bathrooms. For example, I learned that the handicap bathroom stall is quite popular with parents because it's bigger and it has a bar that kids can hang on to so that they don't fall into the bowl. I also learned that parents of boys have come up with all types of creative ways to get their kids to aim properly. The most creative one that I heard was from a parent of a boy who needed to use a bathroom on a plane. Because the space was so tight, the parent ended up having to lift the child horizontally above the toilet like he was Superman so that the child could tinkle into the toilet from a prone position. Now, for this project, I used a qualitative data I collected to build a persona. I gave her a name, Annie, and described in a few sentences the key behaviors and motivations that I observed in my interviews. The persona represents a composite of the parents I spoke with. Annie is 35 and mother of a three-year-old. She's relieved that her son is finally potty-trained but dreads the fact that he often waits until the last minute to go potty. She prefers not to use public bathrooms at all, but is resigned to the fact that she will because of her son, and she always likes to feel prepared, even though her son is potty-trained, she still carries around a diaper bag. I then built a journey map to document the highlights and lowlights of the simple experience of using a public bathroom. I broke it down into multiple steps and identified the opportunities for greatest impact. Now, what you see here on this slide is an abbreviated version of the journey map that I created based on the qualitative research that I did. The gray solid line represents what most parents experience when taking their kids to public bathrooms. The orange dashed line identifies opportunities for delighting them. Now, take a look at the last few steps. Something as simple as washing hands can be very challenging for parents of young kids because most things are at an adult height. Parents have to lift up their kid and hold them with one arm while turning on the faucet with the other so that their kids can wash their hands. Now, this can be even more challenging when the soap dispensers are placed far away from the sink. From a design and product perspective, something as simple as creation of a step stool beneath the sink and a soap dispenser that's closer to the sink can be a huge relief to parents. Now, this is not stuff where you can find quantitative data on. You may be able to get a quick pulse check on satisfaction. I've seen public bathrooms asking me to rate my experience with an emoji as I'm walking out, right? I just kind of hit a button. But that is not going to give you the depth of insight that good qualitative research can yield. Even asking someone in a survey, how would you rate your experience will provide just limited data. And that's because most people have come to accept their reality as is. They may think in this case, well, the bathroom was clean, so I was satisfied. But that's a very limited view of customer satisfaction. The fact is, most people wouldn't be able to articulate their pain points because they're not even aware of their own pain points. Only when you're diving deep and really listening to the customer, can you uncover them. And that's really when you as a product manager know that you've hit upon something. When you can identify a pain point that even your customer doesn't know that they have. Qualitative research is great at helping you identify unarticulated customer problems. And these are the problems that I as a product manager find the most interesting to solve when customers don't even know that they had a problem in the first place. I'll give you two other times when I use qualitative research. One is to test out product concepts. Let's say I uncover some unmet customer need, and I have an idea for a solution. I'll then work on a prototype. The prototype is usually pretty low fidelity. Sometimes it's a clickable PowerPoint where you simulate an online experience using PowerPoint. I've also done this using drawings and video so that you can show someone what the end to end experience would feel like. I'll take this back to customers and just see how they react. Does it make intuitive sense to them? What questions do they ask? Do they get stuck? Are they confused? All of this is incredibly helpful feedback as you, the product manager, continue to refine the product concept. I'll also use qualitative research when I want to dive deeper into the quantitative data. I gave the example of the sales going down and you're just trying to figure out why this product has not been performing. I'll give you another example. I was once looking at sales and usage data for a product that I managed and there were stark differences in patterns across regions. Specifically in this case, China just stuck out. It just seemed like an anomaly. My colleagues and I had theories for why China stuck out. One of those theories was that our product was all in English and maybe the Chinese wanted something in their native language. I wanted to hear directly from the customers though. These were just hypotheses. I conducted interviews and focus groups with Chinese users of our product, as well as non-users. While it's true, what I found was that some users did want it in Chinese, there were also a fair number that just didn't. They actually insisted on it being in English for a variety of reasons. I won't go into what happened with that particular product, but the point is that if we simply relied on the quantitative data and made guesses, we would have been guessing at the reasons for why certain users behave differently. Let's talk about how to conduct qualitative research. Now that we've gone through reasons and situations for conducting qualitative research, let's talk about the how. How do you go about doing it? I recommend starting with a research brief because it walks you through all of the key steps. I always insist on a plan because I find that without it, it's really easy to get sidetracked or lost. Before you know it, you've lost sight of the original objective and you get back a bunch of data that either doesn't make sense or doesn't matter anymore. Or you may think that you need to do qualitative research when really based on your research objective and key questions, numbers are the way to go. Articulating a plan up front keeps you honest and align stakeholders to what you're researching and why. If you Google research brief templates, you'll find different ones out there. This outline is based on one that I use the most, but really you can use whatever you find to be the most helpful to you. The important thing is that you should use one so that you put a plan down on paper. Let's talk through the key sections. The first is the research objective. This is a statement of why you're undertaking this piece of research in the first place. It can be broad like trying to just understand how parents with little kids use public bathrooms or more specific, like what users think of a specific product that you just launched. It should be succinct and communicates the purpose of the research. Let's say I'm product manager of an e-commerce platform and I want to know whether people like using it. I would probably consider doing a survey and asking, well, how would you rate our platform and would you recommend it to others? But if I want to dive deeper to understand how people are using our platform and why they use it or don't use it, I'll also want to undertake some qualitative research. The next section contains hypotheses and the key research questions. It states the assumptions going into the research. It also states the specific research questions that we're trying to answer. I recommend for the number of questions, keeping it to one to maybe three questions at most, keep your research questions focused. The questions should flow from the research objective and articulate exactly what it is that we're trying to answer. Let's say that your objective is to understand how people are using your product so that you can increase product usage. You may want to ask, one, how are users using your product? And two, how are users using your product alongside other tools that they have? The second question is an interesting one because you'll see if users are relying on other tools in order to compensate for features or functionality that may be lacking in your product. The third section is a summary of background information that relates to the research objective. It includes any previous research or data analysis that was done on this topic. I recommend having it because it grounds your research with context and data. The fourth section is the target audience. Who do we want to talk to? Are they current customers? Are they lapsed customers? Are they potential customers? It's important to think through who you want to speak with. Don't assume that you always have to start with your current customers. Depending on your research objective, you may want to talk to people who used to be your customers or who aren't your customers at all, but meet some other criteria that you care about. In this section, you should also articulate a sampling plan. That is, how are you going to find the people that you want to speak with and how many do you want to speak with? There's no single right answer, but there are some guidelines to think about. How specific is your target audience? If it's pretty specific, then six, maybe eight, interviews is generally sufficient. If you're targeting a broader group of users that spans, say, age, gender, geographic location, you may want to consider grouping these into sub-segments and structuring your research based on these sub-segments. Building on that bathroom example that I gave earlier, if I were interested in talking to not just parents of young children in Washington, D.C., but parents of young children globally, well, then I would probably create a separate sampling plan for each target sub-segment based on geography. As I mentioned, you should also think about how you're going to find these participants. Will you draw from a customer list? Recruit participants through social media, would you hire a firm to create a panel? Also, be aware of customer consent and data privacy laws. In some places, you can't just reach out and ask if someone would like to participate in your market research. If your company has a legal department, I'd recommend consulting with them and making sure that you're in compliance with these laws. All right, on to the next big section. This is methodology. This is where you're going to articulate how you will go about collecting and analyzing the data. Will it be through a survey, focus group, customer interviews, or some other means? It doesn't have to be just one. Depending on your research objective, you may want to combine, say, interviews with a survey. I do this quite a bit, actually. For example, if you're interested in speaking with a very specific type of user, but you have no way to actually filter out for this type of user based on the data that you have, you can use a survey as a screener and ask questions that will help you identify people who will meet your criteria. You can then follow up with a subset of users through interviews. Bottom line, this section should spell out clearly what you're going to do to collect and analyze the data. Once you've articulated how you're going to collect and analyze the data, you should spell out what you're going to do with it. What's going to be the output? Will it be a paper, a presentation? Who's going to receive it? By spelling this out ahead of time and circulating it with your stakeholders, you can identify any missing pieces up front. So for example, maybe in your mind, you think you'd be fine with an email summary of your findings to your manager, but she thinks that you need a full presentation ready for the CEO. Well, that's probably good to know ahead of time. The last two sections are timeline and budget. Think through how much time you'll need to gather the data. Depending on the target audience and methodology, this may vary between a couple of weeks to a couple of months or longer. If you need to hire a firm or resources ahead of time, we'll build that into your timeline. Also, figure out the budget. What do you need supplies? Travel, incentives for participants. Think through what you need and create a budget. Now, with all of these sections, it should cover 95% to 100% of what you need. But again, there are other templates out there. Take a look and see what works for you. After you've developed your research brief, it's time to carry out your research. I'm not going to go into how to conduct specific types of qualitative research in detail, but I do have a few tips for you. One, always buddy up. If you've never conducted customer interviews or focus groups, I'd recommend shadowing someone who has so that you can focus on listening and taking notes. If you're comfortable running interviews and focus groups yourself, still find someone to take notes for you. Don't take these tasks lightly. Finding unarticulated customer problems depends on your ability to really listen and empathize with your customer. It's easier to do this when there's someone else to help you along. Also, practice writing and asking open-ended questions. For interviews and focus groups, I like to create interview guides that contain my main questions alongside a set of probing questions. This way I'm prepared with follow-up questions regardless of what answers I get. I find this to be a better way to prepare for an interview or focus group than having just a sequential list of questions. With many questions and a set of probing questions, I can be more flexible and react to or follow up depending on what answers I get. Be comfortable with the fact that you may not get through your full list of questions and instead focus on getting through the insights that you want to tease out from your interviewee or focus group participants. Three, take the time to reflect on what you heard. Don't simply take what you heard verbatim. Remember that customers don't always know what their pain points are. Consider the body language gestures, tone of what you heard, and build upon that. Now, some people would argue that that's too subjective, but I find that these are valuable data points when I'm trying to figure out not simply what works, but what really wows a customer. Now, I'm going to end with my key takeaways. One, qualitative research is great for customer discovery and understanding the whys in the house. Two, qualitative data can complement quantitative data to generate really powerful customer insights. Get comfortable using both. Three, use your research objective and key questions will determine how you conduct your qualitative research. Think through these carefully. And lastly, I encourage you to dive in and just start talking to customers. It may feel odd or uncomfortable at first, but I guarantee that once you start doing it, you'll begin to understand your customers in a deep and more meaningful way. Thank you.