 Hi, everyone. My name is Rhea Guesin. And today I'm going to be talking about how to use AI to build a product roadmap. This is going to be part one of a two-part series. The next part is going to be all about how to think of adding an AI feature in your product. So let's dive deeper into the topic. But before we do that, a little bit about me. I graduated with an MBA from Stanford University. I'm currently working at PayPal as a senior product manager. And before this, I was a product manager at Oracle. And before that, I was an Android app developer. So my career has been all about being an engineer and then transitioning to product management during my career, especially when I was at Oracle before joining PayPal. I shipped four separate AI-related products. And when I was a student at Stanford, I actually spent a lot of time figuring out how to use AI and statistics to make product decisions. And that's exactly what I'm going to talk about today. So today's presentation is going to have four parts. Number one is, what exactly is a product roadmap? Now, a lot of you are product managers already, so you know what a product roadmap is. But it would be good to define it just to make sure that we are all on the same page. So that's number one. Number two is, why do you use AI for product roadmapping? And then number four is the meat of this presentation. How do you use AI to build a product roadmap? Finally, I'm going to touch upon a few things to be careful of, some caveats. So what exactly is a product roadmap? A product roadmap is basically a strategic plan that documents not just the vision of a product, but also the direction and the timeline of the product. In fact, as a document, it not only serves as a guide for product managers and engineers, but it also gives a proper plan to the marketing team and also a proper plan to the sales team. It's extremely useful for literally every team that is involved in any part of the product lifecycle. But then to actually build it, why do you use AI? Now for that, I'm actually going to take a brief example. So imagine that you are trying to decide what activity you want to do. Sure, one way to do that is to think about what activity you'd like to do, as in, do you like going bowling? Or do you like going to picnics? And what do your friends like? So one way to decide is to just ask yourselves what you'd like to do. But then another way to do it would be actually through statistics, past data, as in looking at how many times did it rain in the past week, how many days did it rain? Did it rain more than 50% of the days? Did it rain less than 50% of the days? And depending upon the answer you get, you can actually decide. And then once you decide between an indoor and an outdoor activity, and then you can again ask yourself that, hey, if I look at the past, which decision actually made more people happy? So in this manner, you're actually really objectively maximizing yours and your friend's satisfaction using past data. And that's exactly what companies can do with AI as well. And this is an example of a decision tree. So a company can actually use AI to actually make decisions. And the companies that do that are more agile and the adaptive market change is better. Now, how does that happen? Because whenever you're using AI, you're basically applying an algorithm to the data that you have. Now, the data itself, there might be biases there. I'll come to that later. But a lot of times, the algorithm that you're going to apply to the data that you have, it's not going to have, it's going to be precise. It's not going to be a victim of any kind of biases. Number second, it's going to be any AI algorithm. It's also efficient. And it can actually begin a lot of data. And then number three, AI, like artificial intelligence, can be proactive as well. It can actually keep looking at real-time data to keep updating its decision as well. So in the case of all this, a company that uses AI for its product load mapping process tends to do better. But how to do it is how to actually use AI to build a product load map. Now, in this section, I'm going to dive deeper into one of the ways I learned when I was a student. And then I'm also going to dive deeper into an example that I thought of, again, when I was a student. So whenever you're using AI for any kind of decision making, and probably this is something a lot of you already know, first, of course, you gather data. You process it. You binaryize it. You make it more consumable for the algorithm that you have. And then you try to fit a mathematical function to that data. Sure, there are a lot of caveats. You don't want to overfit the function. But once you get a function that actually fits in a decent enough manner, then you start using it for prediction and decision. So that's what using data to make decisions is all about. And that's exactly what AI is all about as well. So the example that I'm going to dive deeper into is called Customer NPS. What exactly is Customer NPS? It's actually the net promoter score. How likely is a customer? How likely is a customer to recommend my product to more people? How likely are they to bring more people to my product? So whenever you're collecting a net promoter score data, you basically can divide your customers into three buckets. Bucket number one are neutral. As in, these customers, they're not going to promote your product, but they're not going to detract other people away from it either. Number second are actually the detractors. And people who are so displeased with your product that they're going to tell other people to stay away from your product. And then finally, the best people, they're the promoters. They are the ones who are so happy with your product that they're going to bring in more people to your product. So imagine a case study. So imagine that you are a product manager for a business to business product. What that means is that your product is normally used by businesses instead of consumers. A great example of that is salesforce.com. Now, Salesforce is a company that actually sells its software to other companies. So it's a B2B company. It's a business to business company, as opposed to Apple, which makes iPhones and MacBooks, which are used by consumers like you and me. So Apple primarily is a B2C company, but we are looking at a B2B company. And imagine that you're a product manager for this company. Imagine that you've also built a minimum viable product with four features and your goal is to actually build a future roadmap. As in, you actually want to decide what to do. Do you want to add new features or do you want to actually develop upon the existing features? So how do you decide that? How do you actually make this decision? One way again to do that is instinctive. As in, you ask yourself, as in if you were a customer of your product, what would you want? But that's probably not the best way to do it. So what can you do? Now, also imagine that you already have data on which customer is in which company that is your customer is using which feature. So let's say that customer A is frequently using feature one, is not frequently using feature two, and is frequently using features three and four. Similarly, customer B is frequently using feature two, three, but not four, and so on. Now, a lot of times, you can actually get data like this. If you have the telemetry for your product, so it's possible to have data like this, and it's definitely possible to define the usage of saying like using feature one could be defined as using it at least once a day or using it at least once a week. So as a product manager, you'll probably define that, but let's imagine that you've done all of that and you have this table. So now, what we can do is we can superimpose customer NPS data from each of these customers. Imagine that you've already done the survey and imagine that you've already gotten your net promoter scores from each of these customers and the scores are these. Now, let's try to come up with a mathematical function that can actually predict the NPS values that we are seeing here. So we have already defined which customer is using which feature in a really mathematical term. So for each cell, we have a mathematical value. So imagine a linear function NPS equals A which is a variable times the value of feature one. So for customer A, the value of feature one was one, the value of feature two was zero, and so on. So imagine a function like that. So A times feature one, B times feature two, C times feature three, et cetera. And then you can actually look at some combinations too. F times feature two, times feature four. And what you wanna really do is find the value of these variables, E, B, C, and so on. So that the predicted NPS values are the closest possible to the observed NPS value. This is actually known as linear regression. And what we have just done, you've actually turned the data that you already have into training data and the algorithm that you're actually, the algorithm that you're actually training here is this. Linear function when you're actually working out the values of A, B, and C, you're basically training your AI system. And once you have done that, what you're gonna end up with is a proper mathematical function that can not only just predict NPS for a new customer, if you have the data for what features they're using, of course, but it also tells you which features or which feature combinations are highly positively correlated with NPS and which features or which feature combinations are negatively correlated with NPS. Now that is super useful. So, thanks. So this is a kind of an observation that you can actually make once you work out the function. And let's say that this three combination four, feature one, three, and four are highly positively correlated with NPS. Now, how is that useful? Now, for a product manager like yourself, you can actually think, okay, feature one, three, and four work really well together. Why is that? Is it the case one, three, and four who really click complement each other or is it something else? Now, it actually helps you make a product, make a data-driven decision for your product. It can also help your marketing team because now they know that one, three, and four work really well together. So now they can actually market features one, three, and four together. And similarly, it can also help the salespeople. So having a correlation like this can not only just help you actually make decisions for your product roadmap, but also really help other teams as well. So in summary, leaner regression, it's all about showcasing the relationship between two variables. It can actually be more variables, but in this case, it was NPS and a whole lot of different features. And then one variable is in the predictor. So what the predictor and it can actually be used to predict the outcome of the other. So for example, in this case, once we actually found the value of A, B, C, and P, et cetera, then whether or not a company was using feature one, that could predict it's NPS. So that's what this is all about. You can actually, of course, I use pass data to understand how different product features would impact NPS. And then, of course, we evaluated the value of A, B, C, and D, the coefficients of the model. So this was leaner regression. And then, of course, as a product manager, you can actually prioritize features or feature groups which are highly positively correlated with NPS. Now, let's dive deeper into another example. So going back to the decision that we actually saw at the very beginning of this presentation, you're someone who's trying to decide which activity to do. You have looked at the chances of rain and to actually determine the chances of rain. You have looked at pass data and you're trying to see whether 50% of the times over the past week, whether it rained more than 50% of the days over the past week or not. And then once you have arrived at the decision at the top level, then you actually look at, then you actually ask again, like, hey, what's the pass data on which decision actually makes more people happy? So imagine that this is exactly what you're trying to do. So in this case, this is actually called decision tree. You're basically creating a flow chart and in this flow chart, you can actually change the questions that you're asking. Here threshold that we used was 50%. This could change as well. And changing this decision threshold would actually constitute training of this decision tree model. So that's number one. Now, why am I telling you this? Because a company can actually use a similar approach for product decision-making as well. So imagine, right, now imagine that you're a product manager for an application. So you are making an app that a consumer would use on their phone and imagine that this application is a social media application. In order to make a decision, you would ask yourself, okay, in the past, what percentage of users have said that they prefer a dark mode? If it's more than 50%, then let's decide to add one. If it's less than 50%, then let's decide not to add one. And then once you have decided to add a dark mode, then what do you do next? Okay, then you ask yourself again, with past data, what is the percentage of people who prefer stories instead of ad hoc updates from friends? Okay, if it's more than 50% again, we enhance, like we actually improve the stories feature or if it's less, then we actually enhance the updates feature and the same thing happens for the other branch as well. And if you want to train this model, you can actually play around with these thresholds to actually arrive at a good satisfaction. So that's exactly what we can do here, right? And this is also another example of using past data and essentially artificial intelligence to arrive at a product decision. Decision trees are actually really common in the world of AI. And that's one example of how you can use it. So this was a really simple model, but then when you look at real companies, like bigger companies, this tree would be like super complicated and they would probably train this tree like price every day or something like that. So that happens a lot. So that's another way of using AI for product road mapping. But there are a lot of caveats. Not everything that we discussed here is completely rosy. So there are a few things we need to keep in mind. Number one is historical data bias. You might remember that when I was saying that an AI algorithm is precise, sure, an algorithm might be precise, but the data itself can be sometimes biased. Like for example, let's say your historical data is only for a certain demographic. Like, okay, imagine Facebook's beginning, like early days when it only started, when it was only kind of like rolled out for college students. What if Mark Zuckerberg had decided to use AI for product road mapping then, then he would have had data only from students at Harvard who are like, let's face it, they're really smart. And the way like, so their behavior on social media is probably not representative of other people. So a lot of times data itself might actually be biased, just because it actually focuses on a certain demographic. So how do you mitigate this? Okay, so to mitigate this, you actually have to understand the cultural shifts and then you actually kind of like have the course correct for the data itself. To make sure that such biases are corrected for. Similarly, you also have a feedback loop bias. As in a lot of times, what happens is that AI keeps making similar decisions again and again and again, because let's imagine that it recommended a certain feature and it was well received, right? Now, when it actually kind of like ingests this new data that this feature was well received, then that becomes a data point. Because of that data point, AI is probably gonna be recommending a similar feature all over again, just because it's part, like just because it sees that as a positive. So a lot of times you may not actually innovate if you're using AI to build your product roadmap. So a lot of times you have to have a human to actually mitigate this, to actually look at AI and to be able to ascertain whether AI is actually going in circles. Finally, number three is overfitting. Now overfitting is a really common problem and it's quite famous as well for any AI algorithm. So you might think back to one of the things that I'd said that when you want to actually create an AI algorithm, what you're actually doing, you're actually taking in data and then you're trying to fit in a mathematical function. Now the mathematical function that you part of that can actually fit really well with the training data with the data that you're ingesting. And that's not completely good because what happens is that a lot of times that overfitting, like overreliance on the training data itself, might be detrimental. An example here is that, hey, AI might think that people who buy winter clothes also buy gloves, but what would happen like, what would happen is that this AI algorithm would actually recommend such winter gloves only where such jackets are bought. So that would totally happen. So again, to mitigate things like this, what you need to do is to actually broaden your criteria and to understand the broader context of what's going on in the market. So overall, the solution for this is that, hey, don't think of AI as something that can completely replace a human product manager. That's never gonna happen. It can definitely assist us in making a product roadmap. And finally, only a human being can empathize with another human being and define pain points and solutions that are completely driven in an empathetic manner. Thank you for listening to me and yeah, thanks.