 All right. Hey, everyone. I'm Hershey pronounced like the Hershey's chocolate. So excited to be back here at product school. And today I'm going to talk about creating closed loop feedback with data driven decisions. So as PMs every day, every hour of the day, literally, we are using data to drive decisions for our roadmaps. However, customer feedback is so important and having a closed loop feedback in processes even more important. So in the next 25 to 30 minutes, I'm going to share some strategic and tactical concepts that we have seen work with our customers and you know, we applied here at Gainside. And hopefully you'll find some actionable takeaways from from the session. And a little bit about me. I'm I've been at Gainside for about a little over two and a half years heading up the customer success team for our product line Gainside PX, which is centered around capturing product usage analytics and driving in app user messaging and service again to drive data driven decisions, but also orchestrating entire user journeys. So day to day, I'm dealing with all the product managers product leaders from companies of all size and segmentation. So lots of learning and exchange of information there. Super excited to be here. And please put in your questions that you may have in the chat and I'm happy to take it as the session ends. So what are really the goals of creating a closed loop feedback. Essentially, it's for our teams to have actionable insights. So if you break it down in sort of two categories. First, I would think is primarily for our product team. Second is primarily for our customer facing teams. So for our product teams, you really want to have a voice of your customer their sentiment in the roadmap decisions that you're making for your upcoming sprints and overall strategy of your roadmap. So you want to get their sentiment or their voice around any new features that you're planning, or also particularly identifying friction points so you can improve existing workflows and effort scores that you may have in place and ultimately help them to get the time to value and outcomes very quickly within your product. The second goal essentially is identifying potential advocates for your in your customer base so they can talk to your references or give you nice GTC reviews and whatnot and do webinars for you. And for your customer facing teams, it is also essential to identify at risk customers and get ahead of it before your renewal discussions and have a pat to green. So these are essentially the high level goals of creating a closed loop feedback and how you can leverage all of this data in your roadmap decisions. Now what could be typical challenges I mean we all know about closed loop feedback you know we all know what the goals are, but what are some day to day challenges you may be running into in creating a program around closed loop feedback. So number one is low survey response rates with emails the typical channel that we're using today are emails, and we all know like the survey response rates about are about two to 3%. So that is not really good enough data set for us to make roadmap decisions just based on two to 3% response rates of our customer basis. The second challenge is having an effective plan in place. So many a time we would collect feedback but we are not really closing the loop or don't have a nice plan in place to close the loop with our customers. And thirdly, scalability to run a feedback program. You know, many a time we're trying to close the loop by getting one on one interviews or as a customer basis growing, we may not always have the means resources time and the bandwidth to scale our feedback program. In my next few slides I'm going to share some strategies tactics and tips to overcome all of these challenges and help you build out a scale close loop feedback program. So here are four strategies. Number one, it is very important to identify your goal and the eligibility criteria. So defining your objectives of the survey because they're different types of surveys. Identifying that goal is super important so we can choose the right type of survey and I'm going to talk about this in one of my next slides. Also defining whose opinion matters that are adoption champions in your product. There are users who are loving for the first time the users who are, you know, repeat users. So really identifying whose opinion matters which persona matters. So you can define an eligibility criteria for your surveys. Secondly, having an action plan is super important. So a few examples are once you identify a potential at risk customer to reduce churn. What are your what is your playbook what are your steps that you're going to take to turn that sentiment around Incorporating product risk in roadmap decisions now we all have those customers in our customer file where they're like hey if this feature doesn't come out. I'm not going to renew so identifying those situations incorporating that in your product risk, which you can ultimately make a decision in your roadmap. Now again, marketing teams love this. So having that action plan in place is great to close the loop around the feedback that you're collecting. Thirdly, leveraging technology to scale. You know we all doing email responses but how do we follow up how do we close the close the loop in a scalable way is super critical, especially as the B2B space is booming. So how do we scale our programs in place so thinking of automated email follow ups right to in product responses, creating CTAs and slack updates for your customer facing teams to follow up with with your customers survey response analytics for your pms and pmm so that they can look by each feedback one by one, but also they can review overall trends and comments that are coming in that can help them to categorize their overall service. And fourthly, standardizing or centralizing your efforts. There's so many different customer facing teams that touch our customers lifecycle so for example, once you close the deal there is services team who will come and on board the customers. Then there'll be CSM team then there is support team then you know there'll be product teams talking to them. So really centralizing the multiple feedback programs with the multiple teams in place, putting them on timeline so to avoid fatigue and also inconsistencies around capturing feedback in a timely manner with your customers. So these are four strategies that can help you define what that closed loop feedback looks like and developing a nice and scalable program around this. So let's go into now how we can implement these strategies. So if you think about this three step this is a good three step framework to implement the strategy for closing the loop. Simple. This might look like oh hey I already know this right. But there's some points here which from our experience and working with product managers that we found very useful to implement. So when you're listening or listening to both the qualitative data and the quantitative data. So what is qualitative data it is pretty much you know what you're hearing. First hand from your customers directly or from your customer facing teams your CSM your support teams your sales team. So that's all that is qualitative data and it's called very unstructured way of collecting feedback. So as much as that is important quantitative data is important as well. So that is the surveys that we want to trigger to our customers and hear from them firsthand, but in a quantitative way so really looking at you know what the survey data is showing, comparing it with the product usage analytics as well. So let's say if my enterprise customers specifically admin role give me this particular feedback that helps me put things in perspective and compare it with other similar feedback that I may have gotten based on their usage of the product where they are in the journey what's their role what's their role. So again combining that qualitative and quantitative feedback, and then that quantitative data further thinking of two things one is the feedback and the survey that we collect survey responses that we collect from our customers. And secondly the product usage analytics that we have to sort of marry that data and see what are the trends coming out and validating it with the hypothesis and qualitative data feedback that you're hearing from your customer facing teams. From there, analyzing the data like I mentioned enterprise customer or admin user role. So really segmenting cohortizing your customers slicing the data is super important. So you know the three things typically want to think about when you're analyzing the data and how you can slice it up or cohortize it. One is your customer segment. So always I'm sure you all relate to this that the enterprise goals of customer the enterprise enterprise customers their goals are very different than the customers when the SMB segments. The way each of these segments utilize your product the feedback that they have enterprise always thinking of enterprise great features scaling multi product and whatnot. So always thinking of okay, what are my customer segments here breaking your data when you're analyzing based on that. Secondly, your life cycle of your customer customers who are in onboarding versus who are in adoption customers who are in year one versus year two customers who are in first seven to 15 days versus customers who are in 30 to 90 days. Very different goals, very different experience. So again, when you're analyzing the data breaking it out my life cycle. And then of course you want to look at their usage as well. So when you're capturing that feedback and analyzing the data, look at how their usage has been somebody who has not used the product. Now, you know how much would you give weightage to their feedback and services responses. So these are some things to consider and attributes that you want to add in your tools where you're analyzing the data of course in the inside px you know these are out of the box options that are available but any other tool that you're using I'm sure they offer these kind of functionality so it's always good to take these attributes into into consideration. And lastly, acting on the feedback right now there are two ways broadly I think how we can act on the feedback is number one. We're not really reviewing the feedback and seeing and categorizing hey is it really a roadmap request or enhancement, or this is actually a user knowledge or training gap that we can overcome by providing guidance to the customer, either why our customer facing team or through in app guidance and messages. So providing guidance or making changes in your roadmap to improve workflows and improve customer sentiment. These are typically two ways you how you want to close the loop with your customers. So going double clicking into number one little bit here. So how do you listen right so map your feedback to a timeline. And for that what is important is identifying your goal of what you're trying to achieve with the feedback as you can see there are at least five types of surveys today that we have in hand. You don't want to overwhelm your users. And at the same time you want to have a very clear objective very clear goal of why you want to collect feedback. So once you are looking at the feedback goal so for example, your onboarding is finished and you want to get feedback how the onboarding experience was. You use a CSAT survey for that or how the support experience was, or if you're trying to do roadmap decisions, then you know for you want your goal is okay how was the intuitiveness of my product for my first time users, or how powerful is my product so that you know to target your frequent users. You want to you know your goal is how was my, how was my new release and product enhancements. So for these you typically want to do customer efforts score. So really identifying your feedback goal will be number one that we super important. And from there, like I mentioned in the strategy. That was the number one strategy was identifying your goals and whose opinion matters so so the responder column helps you define is this feedback targeted at end users or buyer and then further breaking it down by user roles or personas. And then then you define what is the frequency and so here's a typical you know the suggestion that we have put that we've seen works very well. These are the types of feedback goals for that you know quarterly by annually feedback frequency that works and here's a typical month this month basically talks about once your customer your prospect becomes a customer it is their month one and six or month five and 11. So this is a good framework to think about when you're applying a closed loop feedback program. So once you identify your goal responders, map it to a timeline. Right. So here's the month one post contract of my customer. They're in onboarding phase I typically want to trigger CSAT in month to week one from there in month to week week two or three because they've spent about three months two months in my product. Maybe I want to capture customer effort score and understand how was the user intuitiveness for my first time users. Then in month four you want to capture the buyer NPS month five you want to capture quarterly NPS from your users as well because they've spent four to five months enough time in your product. So once you've mapped it to a timeline, you know, map it to the channels where you want to which you want to leverage to capture these feedbacks. So in product is a great way than the other channels like email one on once your community online community that you may be having. So once you've identified your goals and objectives your user roles, map it to a timeline to find the channels you want to use and then from there go on and implement this program. Now once we have put this program in place you started collecting data, how do you analyze all of this rich feedback that your customers are taking the time to provide. So analyzing the feedback as I was mentioning slicing the data by various cohorts as you can see these are screenshots taken from game site PX these are some out of the box reporting available the filters as you can see on the right. This is an example of using different filters and co-hortizing my survey data right so I'm really looking at folks who are product managers who are in the game site PX product. And they belong to the enterprise segment of customers and also are in adopting stage they are not first time users or they're not an onboarding stage. So analyzing and co-hortizing slicing your data is super important so that you can really focus on the right persona right cohorts as you're driving the decisions but also passing on the feedback to your customer facing teams for example enterprise. We are very high touch so it is great to bring in your CSM so that you know they know about this and they can have those one on one conversations interviews with your customers. So analyzing the data by various cohorts super important. Then is the onboarding experience is another example onboarding is super critical in a customer's life cycle and journey. So you want to you know put a probably a multi-question as survey and understand how their onboarding experience was so that you can optimize their onboarding experience their journey with you. And also in future how can you reduce the time to value for your customers so that you know your product becomes more sticky in the first initial days what was their initial experience with configuring and setting up your product. So those are you know good questions to ask and to be curious about how their onboarding experience was not just with your team but also within your product. Did they have access to the right documentation that they needed how easy or how hard it was was the time something similar to what was promised to them and sales or what they were expecting. These are great questions to ask and from there gives us good data points to make decisions as to in what parts of the organizations we want to change is it. Is it the you know changes the services team that we need to make is it to change in the product flow that I need to make maybe improve my documentation. So really helps you understand area of improving the customer experience and where you know there may be delays and how can we reduce that time to value for our customers. Thirdly you can use this data to learn about user priorities and again these are all screenshots from Ginseng PX these are like out of the box supporting this is a screenshot of multi-question as survey so you can really learn about your user priorities as you are planning your roadmap. And understand what do different cohorts care about and you know before your release and then once you made the changes in your product then capturing their sentiment after the release. So again very powerful tool surveys and feedback to get your voice of the customer in your roadmap and also getting ahead of any potential at risk or you know turn customers. This is another example live that we're right now running in Ginseng PX gotten feedback around hey the editor experience is not super intuitive. So what we did was we we triggered a customer efforts code triggered it very contextually so we identified our audience criteria is now again in Ginseng PX it gives us a lot of flexibility to identify the right cohort target audience that we want to go after so. You know users who've been in the system greater than 30 days and segmenting it by customer lifecycle your ARR but also most importantly when they have taken the specific action where you know we want to capture their deeper feedback. Going through those four or five steps at that point triggering the survey in the product and capturing there in the moment sentiment that is super key here and also collecting comments of you know understanding why they gave a seven or two or a three and then you know you can prioritize the feedback based on the the stage their persona the revenue impact it will have and your if it goes overall with your long term strategy of the product. These are some experiments and you know feedback loops that we are running within insight as well. Lastly, once you've collected you know implemented the program collected all this rich set of data you've analyzed it. Now how do you act on it. So to close the loop to close the loop very effectively you need to be sure that it is timely. You know you need to act after the feedback shortly after it's provided not to you don't want to too many days to pass by otherwise it loses its relevance and the customers already lost the trust in you. Also having you know that accuracy having a very clear and specific idea about the customer's relationship with you to date the the events that have taken place the usage that actually led to their feedback so having that you know accuracy and clarity about those customer relationships again if you're using products like inside PXES you have all of that rich context along with the feedback when you're collecting it in the product. And then having you know having an action plan proportionate to your segment of the customers of different customers will have different needs. Different personas will have different needs you'll have different segmentations based on the ARR so how can you also tailor your responses or thinking of these three things, timely accurate proportionate when you're defining the program around how to take action. So some examples and ideas that we've seen work with you know hundreds of customers that we work with or some things that has worked for us as well here at Gainsight is set response and cohort based on automated email follow ups so for example, if a customer has responded with a promoter or a passive or a detractor having your three different templates around that. And also having templates around like based on hey these are my enterprise customers versus these are my SMB customers so having that automated email follow ups that will help you scale as a company now of course if you're using a tool like Gainsight PX it lets you close the loop in an automated way. If you're using Gainsight CS as well you know it creates CTAs for your CSMs especially enterprise customers to tell them that hey you know this user from this customer base gave this response to CSM go and follow up with them. So again this is recommended in enterprise customers otherwise the program doesn't become very scalable if you have your customer teams follow up. Also identifying your customers where you want to conduct one on one interviews. So again identifying whose opinion matters going over the comments and really learning about their new challenges and sometimes again it may come out as hey this was just a user knowledge gap so providing them that guidance over a call or via documentation or via in product guidance because if one user has that comment for you or feedback for you it very well might be that there are other users having similar journey in your product so then how do I scale that experience of providing guidance to my users. So in product guidances and walkthroughs you know which we very regularly do using our Gainsight PX platform is super critical in improving these experiences for your customers. Also you know you can improve your documentation use knowledge center bot like we have a checklist like a bot which sits in the bottom of the of the product and we place our most frequently asked questions in documentation there as well. And also then you know when you're conducting one on one interviews you've learned about new challenges and they're really like hey these are good things to be added to the product you review with the product team for specific roadmap changes. These are some like simple basic ideas to start with as you're you know putting an actual action plan in place. And then there are some advanced ideas as well so you can you know collect feedback from beta programs and early access users before it is you know G8 to your rest of the rest of the team so again closing the loop there is is a very scalable program there to put in product ways for your beta program for your early access users so not always having to do one on one interviews but you can very quickly collect the feedback make changes to your roadmap before you G8. Another advanced way is you know when you're doing release surveys you can do survey check sentiment checks before and after the release to learn what works and what doesn't work great example is like a CS score. Let's say if you're targeting on improving a workflow you know this is a common challenge or pain point across our customer base. So collecting that CS survey before in completion of that workflow making your roadmap changes you know released it and then collecting the feedback after and seeing what worked for them what did not work how the you know ratings differ. And then you know message any upcoming roadmap items as well that is another way you can close the feedback because you know hey these already on my roadmap customer thank you for your feedback really appreciate it. So these are some advanced ways of you know programs that you can put in place to close the action is super simple not very high effort, but definitely high ROI. And then thinking about your strategic accounts which have the biggest impact on your retention numbers on your expansion metrics dramatically which can, you know, really swing the numbers they're not some metrics for your overall company. So tracking product risk against IPX we actually like you know, when we capture feedback from the customer if we determine there's a potential risk to renewal, which is specific to the product roadmap item. We actually create a product risk, you know the CSMs can provide that invaluable product feedback and really understanding the use case and what what solution the customer is looking for. And the product teams uses that product risk and prioritizes deep prioritizes based on the revenue impact it might have and how it sits well with their roadmap strategy overall that they're thinking. It's important for pms to balance out features and you know, held accountable for revenue attribution as well so so increasingly you all may be seeing that the jobs or the expectations of the pms are evolving and going beyond just shipping features but balancing feature requests and being held accountable for really attributing your features and roadmap to the revenue attribution and retention numbers. So, these are some ideas you know can work cross functionally with your teams to close the loop and feedback with your customers. So here's an example of one of our customers Autodesk who's using PX to actually go from a program of insights to action this is the program that they call it insights to action. If you're interested you can actually go to pulse everywhere from from June where Autodesk actually also did a 25 minute deep dive presentation on how they rolled out this program what was their challenge and what was their pain point. But here's a summary of you know what they were trying to achieve is they really want to you know measure user sentiment so they can very early on get visibility into the customer journey that can help their product teams to use that feedback for the roadmap items and also for CSM teams to get early insights into new risks and Autodesk is a desktop application. So you know there are not many tools out there actually very limited a couple of tools that allow for desktop in product surveys and messaging and you know offline tracking. So they use Gainsight PX to do that. And here are the results. 30% improved response rate from the 2% email response rate that they used to give can you look at the look at that number massive increase right 28% increase right there. And they you know because they're a global company they also localized the languages of the surveys into 16 different languages with that survey and got a 30% response rate. So get super efficient for them super scalable for them and super valuable feedback for their product teams and and data science teams. And from here you may be wondering okay hey great Autodesk got a 30% response but what is the really the benchmark what is the response rate I should aim for my size of the company. So here are some survey benchmarks know that we have looked at our data sets in Gainsight PX and and we were able to come up with some of these benchmarks from the surveys hundreds and thousands of surveys that our customers have been running using Gainsight PX. So the average response rate as you can see from this graph is about 27% for in in product survey types right now in that as you can see based on the survey type. They're the benchmarks differ but on an average about 27% for NPS as you can see is about 25% bullion is the top one almost close to 40% because it's a simple thumbs up thumbs down customer effort score also typically it's 127 doesn't take from the customers again close to 40% 37% response rate. As you would have expected multi questionnaire has the lowest response rates from a benchmarking point of view so about 20% but still not bad compared to email response rates. NPS about 25% rating about 32% because again ratings also a simple one. So here are some great benchmarks as you're thinking of launching your in app or surveys programs. These are good benchmarks to refer to. And also doubling down on multi questionnaire survey you may think hey, how what is the right number of questions I should put in. So you can see the average response rate is about 20% for MCQ surveys from one to 10 questions questions now 10 is of DC as we increase the number of questions the graph is very obvious here that the response rates are going down. So typically, you know, of course the one question highest response rate, you know, more than 40%, 223 dramatically drops but still hovering around 25%. So you if again determining your objective is super important, and from there identifying what are the right questions and most important questions I want to put out there, and based on that defining your multi questionnaire survey. So yeah, rounding up our discussion here how do you close the loop on the feedback and make data driven road map decisions listen, use your qualitative and quantitative data look at the product usage data look at the survey data and talk to your customer facing teams, and you know combine all of that data and and make your decisions to going into how do I now analyze this data and I've got all this rich set of data, analyzing the data based on customer segment lifecycle usage. And from there, you know, deciding an action plan, either providing guidance to the feedback your customers, or making road map decisions and changes. So yeah, look forward to hearing any questions that the group may have anything that resonated or did not resonate, you know, some of these tactics that you may have already applied, worked for you did not work for you would love to hear so you know we I love to learn from the community and see what are the changes, you know, I can make an intern help out the you know product leaders and managers that we are working with.