 Everybody, my name is Chris Morrill. I'm a Senior Product Manager of Technical Products at Amazon Alexa, focused on the books category of shopping and voice commerce, that intersection, and I'm here to talk to you today about a suggestive topic. When in doubt, seek them out and what the teaser is. We're going to talk about customers. We're talking about customer feedback mechanisms today. Brief introduction. Well, great news is that we can already check that off. We're going to do a little bit of contact setting. Then we're going to talk a little bit about a specific feedback mechanism that I've used in the past called advisory panels, and then we'll end with a little bit of recap. Now, just before we get started, the focus of my discussion is going to be on application. We're going to go through a little case study that's not fictional. I'm just drawing on examples from my product management every day life and the situations that I've run into so that I can share some of the things that I learned along the way, things that have worked for me, and honestly, we're going to talk about some of the things that didn't work for me. The point is that after this talk, I want you to be able to take things away from this and go out and try them. I love applied classes. My discussions are always going to be around applied classes. There's a key warning here, which is that this is neither mutually exclusive nor is it collectively exhaustive. There are many, many different ways of doing this. The purpose of this discussion is not to boil the ocean or present a theoretical examination of many possible frameworks for collecting customer feedback. I'm just going to talk to you about one that's worked for me. Customer feedback and you can start to read through on the left is an app called Discord. During a beta program that I ran, this was the bugs channel was always a popular place for people to go in and report problems. I use that to illustrate, as a PM, I want to maximize the value of every feature. I'm constantly making trade-off decisions between customer value, developer velocity, and time. By developer velocity, I might mean number of resources because resources is a lever that we can pull. But I'm always thinking about what is that customer value? As I think about that, customer feedback alongside some quantitative research data that I have, having feedback from the actual customer I find has been a way to help crystallize some of what the customer value is. For new product development, especially in the spaces that I've worked in where, yes, there are similar use cases that I could look to. There's innovation in adjacent spaces where I could take a use case. But when it comes down to it, finding the right product market fit in these new product launches, new technology innovation, I found that that's where going and just directly speaking to a customer when I was in doubt was the fastest way to be a tiebreaker in a situation when maybe two data points disagreed or maybe our design team was convinced that one specific approach was better than another. Customer feedback has been that way direct. Customer feedback has been that way that I've found, really helps. I've done this a couple of different ways throughout my product career. We're going to talk today about advisory panels, but as I alluded to in the image in the previous slide, there's also beta communities, which can be really useful ways of gathering a small subset, a representative sample of your customer base, your target launch customer base, and giving them access in a programmatic way, giving them access to early builds. You run into beta communities all the time, you're probably constantly opting into them. I know I always opt into the best ones with the best of intentions, and then probably never follow up on half of them. I'm a terrible customer for being a product manager. Then the other one that I would say is post-launch engagement, which has also been useful to me just monitoring, what are the channels that customers might use to communicate with the company, the product team, monitoring those, and then just not being afraid to, if somebody trashes your product on Twitter, I mean, using judgment, but that's a place where I've gone and said, it sounds like you care a lot about this product. What's making you so angry? Sometimes some of those conversations haven't been as useful, but most of the time I take away at least an understanding of a customer's deep frustration point, and I tried to compare that to, did I miss something in my prioritization framework? But let's not get crazy. We're just going to talk about one of those today. Let's talk about a couple of prerequisites. Before we get anywhere down the road of talking about putting together an advisory panel, I'd be very clear on the target customer, because at the end of the day, when I'm building an advisory panel, it means that I've got a product in mind. I have a collection of resources and a timeline to deliver value with those resources that's in some way higher than the opportunity cost of whatever those resources could have been doing otherwise. So to get the most value out of an advisory panel, I have to be really clear on the target customer. Who are they? What do they look like? What do they like to do? What do they hang out? How do they talk to each other? How do they solve problems? Being really clear on the different dimensions of the customer, one, even earlier on, helps me identify what's the customer value, but then I can use those dimensions to start to pinpoint who are my early adopters? Who's feeling this pain point that I'm solving? Who's feeling it most acutely? Who's gonna be most excited that I'm solving this problem? And then here's where we get into some of the soft skills, reaching out and pitching them on participating in an advisory panel. I'll talk about that a little bit in a second. About the how, but then just sort of it, my tip at the bottom of pitching an early adopter or several is a great indicator for product market fit. Especially, depending on in the, actually I think I've found in both B2B and B2C use cases, early adopters are always happy to tell you that you're wasting their time. And the degree to which my target early adopters are receptive to a pitch asking them if they'd like to be a part of my advisory committee or advisory panel, helps me sort of gauge how close I am to product market fit. If in my email outreach or LinkedIn or talking to somebody on the floor of major conference or whatever channel I decide is the right way to approach early adopters. If in that first couple of minutes, they're interested, they understand the problem that I'm solving, they agree that there's a value to be solved. Then in that case, then I know that I'm closer to product market fit. If I'm trying to explain the product to them or trying to explain the feature, the value, the pricing strategy, and they're just glazing over, I'm not getting much traction, it can signal me to go back and relook at my value proposition and relook at the customer problem. So just a really great way to sort of assess risk early. A picture in the background is me and two other gentlemen pitching some folks in a, this is just a pitch competition in business school. Why did I do a pitch competition? Cause I had no idea how to pitch anything. I had done some persuasive speaking, but not a lot. So I just sort of use this picture as an example of, it was very uncomfortable, but I think it was just an awkward, when I was dressed up in a suit, I'm like tech, why? But it taught me how to present things and that I didn't realize at the time, but doing things like that subsequently helped me. So we got a little bit of, so what has this help aside from what I've talked about already in terms of faster achievement of product market fit and giving me some benefits in terms of feedback, which we're gonna dive more into. Another critical element of this, even if you didn't care about the product market fit is then thinking about feature launch. So establishing an advisory panel helps me enable the measurement of customer satisfaction over time, whether I'm measuring that in a survey or qualitative interview or click stream data, it enables me to start to measure a benchmark, which then helps me start to think about, well, what is the, how satisfied do these customers need to be to be willing to pay for it, ultimately? And how satisfied do they, or whatever the KPI is. I've worked on products where we, what I established was correlation between customer satisfaction and engagement over time or some measure of value over time. And then the third thing that I think is sometimes, sometimes overlooked or has been in my experience is that if you crush it, which is a completely technical term, but if you rock it in a beta and the folks in your beta product launches and they've got a sense of ownership, they feel like they've been heard during this beta, they walk out into the world telling everybody about it because they feel like it's not, it's not your, it's not my product, it's our product, that they fed into, that they, hey, check out this little feature. I gave the feedback, it's blue, because I said, red's an offensive color. And just sort of making that up, but having those early adopters turn into evangelists is very, very useful. And I'll talk about that more. It's very useful at launch and beyond when those early adopters turn into, in some way drive adoption of additional customers, whether it's through network effects or through sort of a head torso tail business where the head of the business defines what the torso of the business will do. We were to talk about this situation called Skill Flow Builder, this product that I shipped, but I realized that I would be amiss to proceed without describing what's an advisory panel, an advisory panel, simply put is like a small scale beta community. It is in a situation in which I'm looking for consistent feedback from a regular group of customers that represents the target segment that I'm gonna go after at product launch. I am looking for between three and five customers. In this case, I've used advisory committees in times in B2B situations where it's not feasible to have 200 companies participating in a beta, or situations where the customer's time is expensive or the customer is not easily accessible. And so representing finding three to five customers who are accessible or I'm willing to pay the cost of accessing them because they represent a broader market is when I look to assemble an advisory panel over a beta community. An advisory panel can be, sometimes I may be looking for more strategic feedback versus tactical feedback as in the, usually when I have executed beta panels, those are, it's more tactical feedback. It's a AB study, it's a, or it's a, you know, it's an AB test, it's a pricing study, it's a, you know, comparing one feature to another. And with an advisory panel, sometimes I won't, the, with an advisory panel, the trade-off is, most of the feedback is going to be qualitative, so it's going to be more directional in nature. Broad take away, advisory panel, three to five, three to five customers representative of the target customer that I'm going to go after. And they will assist in from the, you know, can be helpful as soon as I start contemplating a feature or solving a customer problem all the way through product launch and beyond. So let's talk a little bit about Skillflow Builder. So the goal was, notice this problem of developing voice interactive entertainment experiences was very difficult and was especially frustrating to professional developers who maybe come from industries where tooling is, industry tooling is mature and fully featured and usually available as a service coming into developing for voice interactive experiences where there was less tooling and they, and the existing tools were less mature. So the goal was, at the end of the day to enable development, faster development of these experiences for professional developers. And there was a key there of professional developers in that we were not for the purposes of this product, we're not going to tailor towards hobbyists or developers, a single person developing a small shop, somebody, a hobbyist, somebody not doing it for a living. And so our target customers were these early to mid-stage studios, some agencies and a key insight that we guided much of the early product development was discovering through some initial customer interviews that this is potentially obvious, this is potentially obvious, but that these customers have very low risk for unproven technology because they are in a content production flow where they want that production process to be as deterministic as possible. And when I say deterministic, I mean that the inputs and the amount of time and the output of a process is as there is as little variance as possible in differences between running the process, whereas non-deterministic process, I don't know, I can put it an apple and it comes out a carrot with legs, that's a non-deterministic process. And a horrific example, you can, a better example of a non-deterministic process might be if we're doing creative exploration. So in a creative exploration where we are brainstorming for a new, we're doing a brainstorm, that's non-deterministic, the inputs and outputs may not match from one brainstorming session to another and that's by design, great, excellent segue. So just this recaps a little bit, but so the advisory committee, we built relationships within Skillflow Builder, built relationships with three target companies. Important here that they were representative of the segment that I was pursuing because what I was looking for was fidelity of signal that at the end of the day, when I'm talking to the advisory committee, I want feedback from them that is representative of the broader market. And what we did is gave them a seat at the table. So involve them in early, early feature design meetings, early wireframing, we even involve some of them in some early brainstorming, helping them, giving them a sense of ownership as much as possible was key to them feeling like they could participate and that their feedback was heard and that they generally speaking understood some of the trade-off decisions that we were making which helped lower the perceived risk of them adopting the products that we were developing. We also, I found a great use of time with them was problem solving. If we identified a problem with really difficult trade-offs, one of the first places I would go is just reach out to them. In a previous talk, I discussed a sort of difference between there are scrappy and there are spendy ways to get things done as a product manager. And sometimes it's just as scrappy as reaching out to a guy on Slack or talking to a designer on LinkedIn and saying, hey, we've got this problem, this really thorny issue, hear the trade-offs. I mean, what do you think? And getting that additional feedback of someone outside the team who potentially would have to adopt this or use this in the future, gained some really great perspective. And then we couldn't always give them everything that they wanted. The backlog was always much, much longer than what they were, what we were able to deliver. But we were always explicit with them with why we were making, what was the prioritization framework? Why were we making the trade-off decisions that we were making? And that came to be very helpful, especially as we ended up expanding into a broader beta. What we found is that they would then answer questions for other beta members. So when another beta member might say, well, I really wanted feature X before I could even go answer that question or comment, someone from the advisory committee had already answered it, which was tremendously helpful. So let's just move out of the theoretical and into the actual, let's talk a little bit of a day in the life. And this is absolutely real. So on the left hand side, what you see is a syntax, a simplified syntax that's based around narrative and the expression of narrative interaction with an Alexa device where the interaction pattern is the Alexa device says something, the customer says something, Alexa device says something, the customer device says something. This syntax is built around facilitating that interaction. Early in this product development, the team said designers and writers will learn to use the SFB in this case, that skill flow builder syntax using code editors like VS code and exhibit A to your left. The product manager, who happens to have a bachelor of arts degree boldly states, no creative will ever write in this, never. The fervor of that statement, the passion, if you will, falls upon deaf ears. The team is completely unconvinced and is believes that the best way, the best path to market is to ship only modules and a state machine that process this syntax without any visual affordances whatsoever. Challenge accepted. So again, real life, said product manager, emails one of our advisory committee, advisory panel members and sets up a meeting with two of their designers and a writer to get some feedback on this syntax. That PM also invites the entire team to attend and has the designer, our team's designer, walk the members from the advisory panel, walk them through this syntax and get their feedback, which generates this verbatim. We will never, ever period, ever exclamation point write story in a code editor, which is why eight months later when we shipped the product, there was a React based desktop application that was an abstraction on top of that syntax. We didn't get rid of the syntax, the syntax was useful. And I will say, as much as it pains me to say this, truly it was the engineering team that proposed that. They did have the last laugh because not 18 months later after we had launched this product, a designer from that advisory panel company ended up saying that what she found so great about the product was the syntax. And she loved that she had learned the syntax. And once she learned the syntax, it gave her so much flexibility and freedom and she could work so much faster. And so it was on that day that I actually went back to the team and said, you know what, you were right. And if we could have just gotten more designers and writers to ramp on it, maybe we could have shipped without a visual editor, but I'm not sorry. I think some other things that the advisory panel did for us was certainly around beta testing. So once moving out of sort of an early alpha phase into sort of first requirements and the more formal beta testing, once we had collections of features available, they were very helpful as super users in those beta tests. And I had to be careful to not overindex on their feedback because they were already familiar with the product and its use cases. And so had to make sure that the betas were consisted of both members of our advisory panel, but also other companies that we'd solicited to make sure to balance that feedback out. But they're also phenomenal at pointing out edge cases that just aren't obvious to the team as we are building, you know, day to day, especially around things around different operating systems, edge cases in tech documentation or blind spots in our tech documentation. Open questions about, well, will you support this SDK or this SDK? What will you do when they say SDK is end of life? Those types of helping the team look ahead and come out of the sort of day to day feature development that can be very focused in the nitty gritty. That was really useful. And then also establishing that exit criteria, which I discussed a little bit as part of the so what, you know, the what is that CSAT threshold for not only what is the CSAT threshold, that's sort of the launch criteria, but by exit criteria, in this case, I'm talking about confirming my hypothesis on the mix of features that collectively represent enough customer value to overcome barriers to adoption. So an exit criteria saying we are, the product is we are ready to exit beta and to move into a launch, once we have X number of features and have met Y customer satisfaction, because those two things I am confident that if we have both of those, we will achieve product market fit and have a successful product on our hands. So this is a sort of the result of what happens at launch when you launch with an advisory panel. So this is a small company called Electronic Arts and they were members of the advisory panel. And as you can see in the verbatim, there's some discussion about around using tools for developing, in this case, they reference skill flow builder. And the results of sort of forming this committee and making sure that they were, they were with us all the way from pre-technical alpha, all the way through post-product launch was we went and instead of a member of the team or a member from management announcing the product at a industry conference, we had someone from the advisory committee or someone from the advisory panel announce and it was some folks from electronic arts that ended up announcing that product on stage and it was, the product was adopted and it was successful. And if as I looked back over the hundreds of decisions that we as product managers make over the course of a product's life cycle, I think about all of the small decisions that later proved to be impactful and the difference between the successful product and a not successful product was the times where I remembered a verbatim or stopped long enough to give somebody a call or send a text message and ask, hey, you know, what do you think about this? You know, how would you think about this problem? Ultimately what it did is it earned trust with the target customers because instead of trying to drive adoption from a perspective of you should try this, the adoption message was show not tell. Take it from these three customers who participated in this phase of product development, ask them. And having those verbatims helped tremendously. And this is actually a sort of an overview of the stage at the, that was at the industry event where it was announced. I think this was right before, right before it was announced. Just recapping in general what we have talked about. So just want to honor that talking to customers is hard, but it's worth it. I completely acknowledge and have experienced multiple times that finding those initial customers, zeroing in on them, walking up to somebody and saying, hey, I have this idea or, hey, I think this is a problem. You know, or, hey, talk to me about your experience with a problem, it can be a little, it can be a bit intimidating, it can be, it can feel unnatural. What I've found is that it's very worth it. And that there are many different ways to ensure that I've got the right amount of customer feedback as I go through the product development lifecycle. But advisory panels are one that we've talked about today. Again, about three to five participants who can talk both strategically and tactically. They're more valuable at the outset of product development as I'm, well, I'm still trying to refine feature requirements while I'm still trying to determine product market fit. And in fact, one technique to remember is that pitching early adopters can be a great way to get a indicator as to if you've achieved product market fit. Because if you have, you'll know, because you'll have really quickly, you'll have a short list of early adopters who want to be on your advisory panel. And then there's other ways of generating sustained customer feedback like beta panels and also monitoring post launch channels of communication. And then ultimately advisory panels have helped me launch many products over my product management career. And it is always humbling as a product manager to see where others have come together and contributed towards a feature or product that I'm launching because they leave their sort of their fingerprints on the product which to me is always inspiring. I'd always rather build products together when I just enjoy it more. So that's it. Thank you very much. Again, Chris Morrill, feel free to reach out to me on LinkedIn, always happy to connect and continue the conversation. And thank you so much to Product School for having me. I always enjoy these talks.