 Good morning, everybody. My name is Nissan Jarrett, CEO of VARY. Very excited to have with us today. Thank you so much. Okay. Thought leaders in the space of data-driven customer growth. With us today, Debbie from Glassbox, Heather from Heap, and Kyle from Zora. So, none on our panel today is Einstein. But he said a very well-quoted statement that if I had an hour to solve a problem, I would spend 55 minutes understanding the problem in five minutes coming up to the solution. Clearly, he wasn't a product manager, otherwise he would have said it takes me five minutes to come up with the solution in 55 minutes to convince the organization of my solution. But today, we're going to be working on the getting to the core of how we use data to better understand the problems our users have. Any good product manager knows that the core, as Einstein said, of creating great products and great solutions is to truly understand what our customer problems are. Why is it that we don't? Why is it that there's so many products out there with one and two star reviews? Why is it that so many products with multi-million dollar budgets fail? We're going to try and answer some of that. We're going to talk about how can data be used? We're going to talk about some of the pitfalls. We're going to be talking about the culture. Of course, we're going to be talking about AI, because how can we not? If you walk out of here today with one idea to take back home and to implement in your organization, leveraging data in a better way to understand your customers, this will be successful. One last thing before we start asking the panelists question, I'd like to ask a question to the audience. Please raise your hand if in the past year you have discovered a customer problem. Raise your hand if you've discovered a customer problem there. Keep your hand up if you have discovered a customer problem once a month. Once a month, you've discovered a customer problem. How about once a week? You've discovered a customer problem once a week. Anybody keep their hands up after one day? Every day you've discovered a? Let's agree that either our customers don't have problems or there is way for us to continue to discover them and we have to do it quickly efficiently, especially in the age of AI. With that, I'd like to pass it on to our panelists and I'll start off with Debbie. Debbie, tell us a bit more about what you're currently focused on in the realm of data for customer growth. Good morning. Thank you for having me. I'm Debbie Braini, Senior Vice President of Marketing at Glassbox. Quickly, Glassbox is a digital experience intelligence platform that aggregates all of the data about all of your digital customer interactions, and that allows you to essentially uncover four key areas inside about your product experience. User insights, which tell you about the actions that users are taking, the experience insights, which are educating you about how they interact, as well as the struggles they face with your product. There's the technical insights, that you find, prioritize, and fix issues faster, and last but not least, the customer feedback, that quantitative data that can then be correlated with all of that, sorry, I said that backwards, qualitative data that can be correlated with all of your quantitative data. It's extremely powerful having all of those data sets together. I know that not just because they pay me to say it, but also because we use Glassbox to optimize our own product and digital journeys as well. Really excited to be here. Thank you, Debbie. We look forward to hearing more about that and more about the insights you've gathered at Glassbox. Rachel? Hello. I'm Rachel Opsler, CEO at Heap. Heap is a digital analytics platform, so similar, but slightly different. We really focus on providing complete understanding of the customer. We do that by allowing you to really easily capture all sorts of information about activity to be able to string this together, whether the journey is across platforms, across products, across, is that better? Across platforms, products, devices, sessions, so really looking across a whole journey. Also by allowing you to do this at scale, so really understanding journeys at scale, but then also be able to dive into the specifics of an individual user's journey and really understand not just what they're doing, but why. Great to be here. Thank you. Kyle? Yes. Can you hear me? Yes. I can hear you well. Again, I'm sitting next to you. Go ahead. Good morning, everyone. My name is Kyle Colich. I figured that was going to happen. My name is Kyle Colich, VP of product in Zora. I'm also the co-host of the wildly popular podcast called Can I Speak to Your Product Manager? We definitely want to, with all the product managers here, we'd love to have you as a guest. I work in Zora. I run the billing product, which makes up our CPQ, so our Configure Price Quote tool, as well as the core billing product that handles all subscription management and consumption-based pricing. I think this talk track is a perfect thing to talk about because data-driven customer growth is very key to my heart. In fact, we just launched a project last month on Zora Mediation, which is a tool that sucks in all the data from all your products and can be of a crate like meter. So imagine your house, your electric meter, and it gives you what's going on, how much kilowatts you consumed. Same thing, we can do that for all your products. We can then see how many users signed in, how many API calls were made, how many bill runs were made, whatever it is we can create a meter for. And on this journey, we launched it last month, we're getting good traction for our customers, but we noticed they were doing something very unusual with it. They were kind of deviating it and not tying that meter to a subscription or to an agreement or contract. We thought, this is unusual, what's going on here? And so we talked to the customer, we got that data back, and we realized that they didn't need that contract. They just wanted a quick price and then meter that right away and put it on an invoice. I'm like, well, that's great feedback. And we're learning that. We changed the product, we've modified it, and then we released that change last week. So now a customer can choose between, do I couple this usage data to my subscription or do I just make it go right to the invoice based on the charts I want to put in place? So a great experience of learning from our customer and being able to handle that change. Kyle, thank you starting us off with real world examples. We're going to be going through a bunch more of those real world examples so we can have real world takeaways. To finalize the introductions, at Weeville, we work on making the product manager's life easier. We know that product managers, for some reason, tend to be really busy and have a lot of responsibilities. So at Weeville, we focused on helping them make user research much, much faster than typical tools out there in the market. Within just a few minutes, you can get feedback from 120 users. And that could be at the prototype level when you're designing concept level or even for live websites. Going to talk about this a little bit later, but for those that read even faster feedback, there is an AI tool based on generative AI that we developed that within five minutes, you can get feedback, no users involved. We've helped influence that tool with over a million people that have completed user studies before, which enables you to get feedback within five minutes. So we'll be going to talk about the role of AI just now. So let's get started with the first question. And Debbie, we'll start with you on data-driven customer growth. How do you define that? What does that mean for you? And if you can share an example to get us started. Bear with us. Data-driven customer growth to me is about really always seeking as much data as you can find. I know that sometimes you're early on, you don't have a lot of data, you don't have a lot of users. So how do you essentially build data, establish data into your prototyping process? How do you establish data into your early adopter or pilot type programs? And then as you progress, you will get more and more data, you'll become more and more sophisticated. So to me, data-driven growth is about a journey to find and use more and more data as you mature in your product's evolution. I love that, at every step of the way in thinking of the methods and the tools that we can collect it early on when we're in the prototype stage, as well as when the live website, live product evolving and advancing it further. Rachel? Yeah, I think about it as an especially important for product management. I think in product, we have a tendency to want to set goals around outputs because they're easy to measure, right? So we got this thing done that we were developing. But what I think is much more important is that you drive to outcomes. And so that is a way of thinking about being data-driven, right? That you're not trying to launch this feature, you're trying to drive adoption of your product or move the number of weekly average users that you have. And so to the extent that you can really focus on those outcomes versus outputs, it allows you to be much more data-driven in your decision-making. The other thing I think about is how do you use behavioral data to be an indicator of how you grow better, right? So what a user does is typically a much better indicator of what they want or what's important to them than what they might say, right? Like you can ask someone, would you buy this thing? And they say, yes, but that's not always the answer, right? It's hard to get answers to questions like that. But if people are using something that you put out in the product, that's a very good indicator that they find it of value and they're coming back to it. It's an indicator you find it of value. So it's really looking at usage data. And as an example, like finding those signals in the product that tell you this user or customer is ready to buy and out on SKU, is ready to try something else and taking that data and making use of it to grow. Your business. Rachel, I love that. You touched on two such important points. The first one being that the KPIs, understanding what the results that we're trying to drive with this data is the place to start rather than drowning in an overflow of information. And the second one, really about driving that to other parts of the, I'd love to hear your experience in driving this to other parts of the organization. Once we have that data, how do we drive change within our organization? Yeah, usage data or behavioral data can be useful to many parts of the org. I mean, one example is we've all worked with CSMs that are building health scores. Well, these health scores are typically based on usage data. Also sales. So when you find those indicators in your product that someone might be ready for an upsell or a cross-sell or to try something else, I love self-service and I think you should do self-service wherever you can. But you can also, if you're not ready for that yet, take those indicators and just make them a sales lead. And if you're worried about whether it's a good sales lead or not, we did this at HEAP. And what we found is just create the leads, start small, start with something you think is pretty like simple and straightforward. Like when we see people go to the plan page and click on the higher plan, that probably means they want the higher features and that higher plan. And so you throw that to sales. And what I've noticed is that when you're experimenting with this, the salespeople typically will pick it up very quickly. As in they get a lead type and it turns into a conversion, they will remember that lead type and they will tell you, give me more of those. So it's almost like a self-correcting system. So I don't think you have to worry too much about getting it right, send it out, and it'll self-correct if they're not valuable. People won't work them. If they're valuable, they will work them very quickly. I love that. Tying the results into the actions that people in the organizations will take and drive their action. Kyle, your thoughts on what is data-driven customer growth? Yeah, I think that we talked a little bit about the ROI and making sure that's clear for that. And one thing you forget is when you have, we have a product called Zora Payments that shows how we can do high payment acceptance. And the points that keep kind of validating that and showing the customer that we've got you at 98% payment, 99% payment. And that continued kind of reminder of the value you're giving it really strengthens the relationship with the customer and they can kind of remind, oh, I am getting a lot of value from this product. I'm seeing it daily. And going back to the sales team, they love those kind of things. Because you can show, look, look, we did this customer, they went from 65% payment acceptance to 98% payment acceptance. There's a clear ROI associated to it. There's a clear revenue associated to it. You're saving money just buying this product. This probably pays for itself. So the sales team loves that and the marketing team can elevate that message and blast across all your customers. Beautiful. Just building on that, it's the combination of quant and qual and we hear the expression of qualt, just the combination of quant and qual coming together. The quant leading us to identify where the biggest problems are, but it's so important to collect that qual to understand why. Because that holds the answer to what is the next biggest opportunity I have. I can do an A, B test and find that A is better than B, but it could be that option C that I didn't even create yet is gonna be 10 times better than both A and B. So I really wanna understand why it is and understand the deeper why the customer is facing. So Kyle, just building on what you just said, how do you then, what are some of the obstacles, the cultural obstacles to adopt data-driven systems in the organization that you're seeing? Anything new is always kind of a hurdle for that. And again, with product, being a product manager, there's a bit of like this art and science, right? And there's intuitions you have of where you think things to go. And especially when you're starting off, you're like, I have a feeling the product's gonna go this way. And then when you back it up with the data, you strengthen your hypothesis. And so you really then kind of validate that this is where we should go with the product, this is the vision. I'm not just saying, here's the data driving it, here's the marketing analysis we're doing. Here's a really thoughtful TAM that's just not a made-up number. It's really thoughtful, what the market is. Here's what the analysts are saying. So you kind of contribute all that pool and that will help kind of change the behavior or any kind of barrier you're running into. There's nothing like is strong enough with a strong hypothesis and you have a data to back it up. It's pretty powerful. Rachel, I would love to hear your thoughts on cultural challenges that you've seen in the adoption of data-driven systems. Yeah, I think it's like most things that you're trying to adopt. It comes down to you need to attack not just the tooling, so not just having the data but the people in the process as well. And so just thinking about those things, people, it's a cultural change, right? So if your team is not used to using data, they may not have the skills to do it. It may just not be embedded as part of the values of the organization. And so one of the things that is really important is to both create processes that support using it and also as like a leader, continually reinforce it, right? Like you have to set the goals and the stage and the tone for that data is important. And an easy way to do that is anytime you're having product reviews or roadmap reviews and someone says we wanna work on X, you ask for the data, right? Like what data did you use to come up with decision? How did you come up with this? And then the other way to do it is to reinforce the process. So like one example that we did at Heap is I felt like we were making product decisions based on things that were really valuable but too far down in the funnel to really drive a lot of the KPIs that were important to us. And so I asked our lead analytics, our analytics leader to put together what we miss kind of aptly call a work back plan which is basically a spreadsheet that says like if we build this thing, this is how much it's gonna move this top level KPI. So this many people will even get to the thing, right? And this many people will use the thing effectively and so this is the result that it's gonna have. And just doing that forced everyone to start really thinking about top of the funnel because if you're adding a feature that is three layers in and a nav and you don't have a plan as to how you're gonna get people there, it's never gonna have a very big impact even if it's this really valuable thing that people are asking about. So it got them thinking about top of funnel things, it got them thinking about discoverability of things that they were building and it got them to much better be able to evaluate different opportunities where on the face, one may look like great but when you do the calculation, it's not good at all. So, yeah. I love it. Please Debbie. I was just gonna add on because we've heard several people say let's think about outcomes, not outputs but every function, not just product team gravitates towards activity-based metrics and so they do different activities, they tend to have diversion metrics and so having this kind of common data set or common tools is helpful but then you also wanna look at how do we align common metrics cross-functionally to drive more focus and kind of decision making around customer-centric growth and I have a great example from one of our local customers here, Discover, who've oriented all of their digital and customer experience teams around one metric called Struggle Score which is part of Glassbox's platform but the Struggle Score is something that they're mutually committed to improving. The engineering team needs to improve the Struggle Score by fixing bugs and errors in the system. The product team or the UX team need to do it by improving the actual user experience so everybody's working to achieve the same goal and that helps exponentially in terms of driving then that customer and data-driven growth. I absolutely love that example. We heard from Kyle about bottom-up approaches on how to get a data-driven customer growth mindset in the organization. Rachel shared with us the top-down how leaders should think about it and implement an organization and Debbie really shared with us a tool that cuts across the organization using a metric. I'll just, building on Debbie's example, I wanna share one of our client's MasterCard has now instituted that every product in the product development process before it goes from concept to design or design to engineering, it needs to get above an 80 on the Wevo that they run. Wevo is the metric that they use and it needs to get above an 80. If it doesn't get above an 80, it's back to the drawing board. Now, this is a top-down organization. This is very thoughtful leadership that put this in place. But as Debbie said, it's a metric the whole organization can rally around and know what do we need to do in order to make our products good enough for engineering because the last thing we want in the world is for our engineers to work on the wrong things. Our job as product managers is to make sure there are any minute of the day that engineers are working on the best thing they could work on. So with that in mind, I want to go Debbie back to you. How do you see, as we start to think about best practices, how do you think about segmenting the audience? How do you think about B to C, B to B and what opportunities are there to really think segment by segment in our target audience? Yeah, I think most people naturally gravitate towards, let's call it the activity-based or status-based segmentation. First-time users versus recurring users or folks that are in a certain status or subscribe to a certain product but there's more and more layers that you can layer onto that when you start to think about behavioral segments or customers who are satisfied versus not satisfied. So can you use NPS score or customer satisfaction score to look at how the customers that are either happy or dissatisfied or those who face a struggle, those who spend more than a certain amount of time on a page, how do they differ from the rest of the overall group? I think another one is technology. Some things are not, I don't wanna say they're not your fault, but there's different operating systems, devices, networks that people are using. People might be facing issues using your digital product that may or may not even be something that you can ultimately control. So I mean, I think you need to really break it down into several different categories of segments and then you will get a much more complete picture over time. Absolutely love that. The tendency so much is to go into those simple demographic segmentations, but as Debbie says, the real insight is the behavioral segmentation. What do these people do? What do they want? Where are they going? What technologies are they using that delivers such much better insights into how to help them grow and how to satisfy them? Rachel, additional thoughts on segmentation, B2C, B2B, others segmentation types? Yeah, maybe I'll just build a bit on that notion of behavioral segmentation. And so I mentioned a bit earlier, but the idea that you can segment your whole user base based on health scores, for instance, right? Like customers that are healthy versus not healthy because that really helps your plan for how you're going to engage with them. And there's many other ways to do behavioral segmentation. So that's the nice thing about behavioral data, which is a lot of what we're talking about today is that you have this huge wealth of data and you can basically segment anyone by has done, has not done, right? Or did three times, did two times. You can figure out how to correlate renewals to the amount of activity that users have done or accounts have done. So there's just so many opportunities and ways to be able to dig into your data and figure out what usage is telling you about whether or not your users are going to buy more, are going to be retained and renew or they're going to churn. And it's, we've never had such a wealth of data to be able to understand what's going on with our customer base and prospects too. It extends the prospects, trials, all of it. Love it. Kyle, any examples you can give to us or situations that you saw that the segment really made a significant difference? Yeah, I think when you were talking about the B2C, B2B side of it, it's very, very important because if you look at the B2C, that those kind of customers are looking for what's in it for the individual, right? So if you look at like New York Times as a customer of ours and they're very concerned about of, how long did they log in? How long we looked at the content? Did they play Wardle? All those kind of marks for that. But a B2B side, we have a revenue recognition product and that metrics for the organization. So it's more like, what is that organization is more important than kind of the, no, not more important than the individual but the organization metric is driving what the business is gonna do. And so that time that closes is more valuable. So we don't really care if they log in and see that every day as long as they can go and log in, close their books in six hours, that's a better metric which is vastly different than what a B2C one, dealing with content in New York Times is experiencing. So that's important. Just to, as we haven't mentioned AI for at least five minutes over here, so we can't get away with that. One of the things that we observed on the generative AI product that we've developed is the insights that you get, feedback on a website, on a product, completely change if you change the target audience that you're telling the generative AI that you're testing with. So generative AI is at the level now and we've trained, we've helped influence it by over a million user studies that we've run, participants and user studies that we've run in the past, but it truly identifies that different people respond differently and you cannot assume monolith, right? We are all assume, we all have our biases and each person has different biases, our customers have different biases. So in order to get into our customers shoes, we truly need to focus on the different segment. Super helpful. So taking a step forward on some of the pitfalls, and I'd love anybody that wants to answer this, what do you do when you don't have tons of traffic? If you're not Amazon, you're not Facebook, you don't have tons of Netflix, you don't have tons of traffic coming to your website, how can you still collect data? Well, there's a couple of different ways. I mean, I think based on the volume of your data, the type of analysis that you can do is different, right? So let's say you're in a trial. Well, that means that you're not gonna have enough data to aggregate and really look at things at scale, but then you can just look at every single user, right? You can just, you have the time then to look at like exactly what they're doing, watch replays, really understand them deeply. So I think it's not a problem if you don't have a lot of data, it just means that the methods you use to gain insight, get weighed more towards the qualitative that Debbie was talking about earlier than maybe the quantitative. Yeah, I mean, we also, we do what they call a Zora advisory group and we get our top customers in a room with us and if either before it launched, before you get people using the product, they get their hands on it, maybe we go prototype, we don't go full, build more Figma just to get initial feedback from it before we release it. So that's always helpful. Grab on to your important customers or the customers that can help really get good feedback from it, get them in the room with you, get that feedback before you launch it. Our advisory board, it's huge, huge help. Debbie, from your experience. Yeah, I would just say not having a lot of data is not the problem to kind of what Rachel was saying. It's having the mindset that you're always gonna find the data or create data at every step of the way along your journey is what makes you a data-driven organization. So I think don't think of not having a lot of data as being detrimental, it doesn't need to be as long as you have the mindset to seek and generate data in different ways throughout your process. I absolutely love that. It's a mindset, it's not necessarily that you need to seek for having tons of data. Mindset of what you can do with little amounts of data can take you a long way. We find that in our own work that the qualitative data that you can collect from 100 people, and Weevo gives you access to about 60 million people, can be really powerful onto itself even if you don't have people coming to your website, just passing people that are in your scene target audience through that experience and seeing what they do even when you're at the prototype level before you have zero people coming to your website, even if you're a startup or a mid-sized company or a large company launching a new product, can be super insightful and again, helping ensure that your dev team isn't working on the wrong problems. Okay, so we talked about segmentation, we talked about some of the best practices. I'd love to move to any of the big pitfalls. You can do with data honestly, if you press on it and twist its arm enough you can get whatever you want out of it. What of some of the best practices you saw in order to ensure you're getting the best conclusions and the best insights out of the data? Debbie, why don't we start with you? Yeah, I think you did this up earlier which is to focus on not what is happening because you can jump to a lot of conclusions if your data is only telling you what is happening. They may be right, but they may also be very, very wrong. So I think what you wanna prioritize is data and metrics that are gonna tell you why something's happening. So again, you need to be able to quickly look across both user behavior, technical data, and as well as experience session, session replay is super powerful. So there's multiple different ways that you can get to a root cause of that tells you again, not just where you have a problem, but why. Love it. Rachel? Yeah, I totally agree with that. I also think that data and insight is not only pulling together essentially triangulating to validate, it's also a team sport. I think that someone as an individual wants to go in and analyze things and try to figure out stuff and that's a very natural thing to do and that makes sense. But at the point where you have an insight or something that you think is gonna drive a decision, talk to someone else, validate it with people. Many of us as product leaders also have analytics teams that we can go to and just get their thoughts on something. It does not take away, like I'm not suggesting it anyway, means you throw it over the wall, ask your analytics team to do the analysis for you. I think we as product people have to dig into the data but it doesn't mean that we have to do it in isolation. Love it. Absolutely, the cross-team collaboration is so powerful. Kyle? Yeah, one thing I like is, follow the money and see where it goes, type of data too. One thing about, nice thing about consumption type of products is that most people come in and they buy like product one, product two, product three and then you have the other 30 products and they're trying it out. With consumption, you can have a customer buy $10,000 worth of credits or dollars and then you can see it apply against all the portfolio of your products and you kind of routes to where that money's going and say, that's a pretty good effective data if they're willing to spend money on product seven, product 20 and you get a view of like, that money's going that way. There's maybe something there and it also helps with free trials too. So if you launched a new product and you want them to try it out, using that, be able to take that bucket in that wallet and be able to route it to a new product for a product manager is great because you can see if it's validating and if you were spending money on it and do it for the customer, they can try a whole new portfolio, a whole new product on your portfolio, which is great. I love that, follow where the money is going and going away from this model of seat-based charging to just charging on what they're actually consuming. So as we're starting to wrap up over here, we'd love to ask you, our panelists over here, where do you see the industry going? Where do you see us going in understanding data in better ways to deliver continued customer growth, customer delight? Debbie? Yeah, I think one of the really exciting things we'll see in the next year is more and more of the data becoming more accessible to business users. So when we apply AI, especially generative AI and natural language interfaces into analytics tools, all of the sudden you can get to very rich insights anybody can without any reliance on technical nor data teams and also in very, very fast speed. So I think that's one of the most exciting things. You're no longer forming a hypothesis and then waiting to get the data to back it up. You can ask a question, you'll get that answer almost instantaneously. And then the iteration on the optimizations of your product can become much, much faster. I love that, the democratization of data in the organization, you're not reliable just on those sources of the data teams but anybody can access it and fast. Rachel? Yeah, I think to build on that along with the greater access comes and great responsibility, great screw ups. Like one of the things that is interesting about data and I know we've all done it is you run an analysis and it tells you something but then you realize, oh, I forgot to like take out our internal people. I forgot to segment this correctly. I forgot to do something, right? And so I think that that Gen AI is both going to accelerate a lot of learning and also accelerate a lot of misunderstandings about what is actually happening. And so the trick and the work for all of us over the next year is to figure out how to use Gen AI not just to accelerate the learning but to be smart enough to say, like, did you mean to exclude this group or did you know that like this breakdown looks like this? Like so to give the clues that can help you get to the right answer without you even needing to know that you need to do some of this. I love that. That's a whole meta deeper level of thinking about it. Thank you. Kyle? Yeah, and I said this earlier that I think product management is very like art and science, right? And right now we're seeing how AI is really helping with the data side, the science side of it. But I think we're going to start seeing how AI is going to help us with the ideas side, how we can iterate with them, how we can bounce ideas off AI and then give us feedback and we can accelerate that gut instinct we have to really make a really compelling product. Fantastic. I think one of the biggest takeaways we've heard here is that the role of the product manager is changing. Product managers are going to have to develop products faster than before because technology is accelerating faster than before, which means we need to get data and the right data faster than we did ever before. Thanks to our panelists today, we heard some of the opportunities, some of the challenges that faces that, the cultural challenges to make that happen throughout the organization, as well as I think a fairly rosy picture on how AI is going to help us do that better in the future. I would like to encourage you as we're closing up over here to stop by the booths. I know Heap and Glassbox and Sora all have booths on this table, on this floor, sorry, tables on this floor. If you want to check out the Wevo AI tool that gives you instant feedback, you can go downstairs. There's a booth for that there. And on behalf of our panelists, thank you for taking the time to listen today. Really appreciate your help today and your thought leadership. Thanks.