 Hello, everyone. Welcome to my talk. I'm Carlos Paz and today I'm going to be speaking about how to add value as a product manager in an ML team. But first, just a quick introduction. So as I said, I'm Carlos Paz, born and raised in Peru, but now I've been living in Europe for the last almost 10 years, working at different companies as a product manager, as a senior product manager. You can find me on LinkedIn. Also, if you have any questions, maybe after the talk, you can find all the links on my website as well. And please also check the productbookclub.com, which is a monthly book club that we do with all the product managers and authors of the books. So you can also join us and discuss online. Maybe a bit of background about the talk. So this happened to me when I joined a new team and I was introduced as the new product manager for this new machine learning team, right? And the imposter syndrome, of course, kicked in, right? I knew a bit about the technology. I had read about it. I have seen some videos, but I was okay. So what can I actually do? How can I actually add value to the team, right? So that is what made me realize that this is something that maybe I can share my experiences with my learnings because I did that with all the colleagues that kept joining other ML teams. And there were always some recurring questions and things that I was able to share. So that's what I want to share on the talk today. So the agenda, just some quick definitions to make sure that we all understand the same things. What are the differences between the normal product teams and the ML teams? Highlighting these differences also might help you already to realize how it is that you can add value to these teams. And then finally some small recap of what you can do tomorrow and every day as a PM in these kind of teams. So in definition, I already assume you know what machine learning is, what AI is, right? And what product management is, of course. If not, well, the internet is your best friend. There's a lot of definition and a lot of content about it. So on the kind of definition that I wanted to focus on today was on what am I actually calling an ML team. And from experience, what I see is that mainly there are two types of these ML teams. What I call an ML platform team. So for example, Uber will have this ML platform that they call Michelangelo. And then there will be the applied ML teams, right? So they use the technology of machine learning to solve a user problem, but they don't develop the platform itself, right? So another example is Spotify. I think their ML platform, they call it ML home. So there's a team in charge of that platform. And then there's a team that uses the technology to make some recommendations, for example, right? So this is key to differentiate. So today I'm going to focus on applied ML teams, which I think are the more common ones now. So to begin, what are the differences with ML teams, right? And of course, there are a lot of difference. There are a lot of similarities with the classical product teams, let's say. But I want to focus on the main ones. So are the things that also impacted how I then ended up focusing my time when I was part of these teams. The first one is that there is, I believe, a bit more complex product specification. Defining a product is always a bit difficult to define, of course, with your team. But in traditional software, you can sort of do a wireframe and then discuss it with a designer, with a developer. And it's easier to convey your idea, right? But when you're in charge of an ML product, the wireframe is not going to be enough, right? There are a lot more things that are more abstract that you need to define and that will really influence how your product performs. So you need to start defining metrics. You need to start defining trade-offs. You need to start defining what kind of data do you really need, what predictions you think are really going to be valuable. So this is the first difference that I see, right? The product specification is a bit more complex when it's an ML product. The second, maybe quite obvious, right, is that ML product or an ML team, it's all about the data. From starting on, do you have the right data? If you want to make predictions, you will need a lot of historical data so you can train your models. If you have the models, you have the data, but it's like garbage data. You can only expect to get garbage in, garbage out, right? So you need to not only worry about how the UX is, but you also need to worry how you are collecting this data. How are you validating it? Yeah, you need to also not only validate your user journey or the UX, but you also need to validate the predictions that you're doing online, right? And I will talk about this a bit more later. And then, of course, product metrics we always have them, but I think here they are even a bit more important or critical. And you also have a lot more or a different set of product metrics, right? As I mentioned, all these prediction ones are something maybe that you won't have if you don't have ML on your team, right? So discussing the trade-off, how are you collecting the data? Can you even start maybe without any data, without ML? And then also, of course, very important, how is this data being stored? Do you have any dependencies with other platforms maybe? The other difference is that you will go from what I call one-to-end user journeys, right? So in a classic development, well, you might have different scenarios or different journeys with some if statements or some for whatever. And you will always know, right, where the user is going to end up. But once you have machine learning baked into the product, the decisions stop being binary, right? But you have a big frame. So what do I mean? Maybe with an example, it's more clear. Let's say that you make a prediction for the best movie to watch next, right? On Netflix, you might decide to show it as like a recommended movie when the model is 99% sure of the prediction. But what happens if the model is 62%? Would you still present it as a good movie to watch next, right? What if the model only predicts with 15% accuracy? How do you phrase this? Do you present it? Do you don't present it? Maybe you want to use different copy depending on the certainty that the model has, right? So this is what I think that it becomes a bit more difficult to map and differentiate all the different journeys that your product will have. And of course, there is no easy or no one way of finding what that right number is to start saying, okay, after this, this is the best movie before this, it is not, right? So it's a bit more complex. And you can see also how this influences you. I think a classical example is on the spams, right? You develop a model and then the model will detect if an email is a spam or not. At what point do you decide to phrase it as a spam, right? Maybe if the model is 90% sure that it's a spam, then you put it in a spam folder and you label it spam. What if it's 50% sure that it's a spam, right? Maybe you decide to still put it in the inbox, but only with a warning. But you still let the user give you that feedback that actually this is a spam, right? So this is what I mean. You have to consider a lot more different scenarios once you have a model making a prediction and that prediction triggering an action. The next one is about unclear technical feasibility and uncertainty. Again, product development in general has a lot of uncertainty, of course. But I think it's maybe at this point a bit more clear what we can do with development, right? And I think a lot of the applications will be things that have been already tried. Whereas we're a male, you might still face a question on can we actually predict what we think we need to predict, right? So you need, again, a lot more data and experimentation to know how well your model is going to perform on life, right? And of course, I think there is also a bit more extra uncertainty because when you test your new product, you're not only validating if the value proposition is the right one, or if the cop is clear or if the UX is on point, but you're adding basically a new dimension, right? You're also adding, are my predictions well? So again, if I'm trying to predict what movies best to watch, maybe the copy is great, the UX is perfect, but maybe the predictions that I'm making are horrible, right? So then I will see that the test fail, but how do you really know that it was the predictions and it was not the UX, right? And there are a lot of different things that you can do, like testing this differently. For example, at booking, we could test ranking of hotels and maybe it was just a random order. And then you test it again, ranking organized by a model, right? So that you can really only measure what the impact is of a new ML model. Here also you need to discuss with the team on what kind of trade-offs you will make, right? So if you make a specific prediction, what are the implications of it? So as I mentioned before as well, right? When do you make that cut-off on the predictions that the models are making? So now after highlighting these differences with an ML team and a classic software development team, how can you add value to the team and of course to the ML team? First and foremost, you are still the user voice in the team, right? So remember that there are no AI-first products, right? If you want to build a successful product, it will always be a user-first product, right? So even though you might find yourself discussing a lot more about what technology to use and what model you are using and so on, you have to remember that you're solving a user problem, right? So that is, again, compared to maybe the data scientists, developers that will be on your team, you should be the user voice. So, you know, what I always show is that even though of course a lot of companies now mention that they are AI-first, again, you as a product manager should focus on your product being customer-first. And related to this, right? What you can start asking and making sure in the team is that you don't only work until putting a model live, but that you also make sure that you collect this feedback on your predictions, right? Are we making the right ones? Are we explaining them correctly, right? And making sure that you prepare also the product for it to be able to collect this feedback from real users. Another very important point is also to discuss with the team and define together what the actual goal is of the team. So this is, I mean, you have to do it in any kind of product team anyways. But again, I think it's more important to define the goal and the metrics, right? So what are you really trying to achieve with the product? And what are the models actually predicting, right? Which can be something super different and that maybe you don't find out because you just don't ask, right? So an example can be you are working at booking.com, right? And the real goal of the team is to increase the amount of people that end up staying at the hotel, right? So not only booking it, but staying there, right? So without cancellations and everything. So that's your business goal. And now you're working on a new email campaign. So you need, you want to build a model to decide what hotels to recommend on that email. You can train the model to predict which hotels are more likely to be click, right? That is one prediction. You can train the model to display what hotels are more likely to be booked, but maybe not stay, right? What hotels are users more likely or less likely to cancel, for example. You can also predict what kind of traveler this user is, right? Maybe a family or a couple. So then you also show a different kind of hotels. So you need to make sure that what the model is actually predicting is really going to help the goal that your product has, right? That again, they might not always be aligned and there might be a compromise because it's not easy maybe to predict something so close to your product goal. But at least make sure that that prediction is really aligned with what you want to deliver at the end of the day, right? How are the models performing, right? Discussing this with your team, with the data scientists, like how many times are our predictions on point, right? What is the distribution of the probabilities that we have? What happens when there is new data or when some predictions are not right? Are we capturing that? How often does that happen? And so all of these should be things that you have a metric and you have defined and agreed with the team, right? And very importantly, right? Of course, you need to define this before putting the models on production, but also after going live, right? So you might see that during the test that you do while launching the product, let's say 20% of people click the movie that you are recommending to watch next. But this might change over time, right? If what your model was using were some parameters that maybe now the product is not collecting anymore or maybe, you know, now with the changes in how we collect data with cookies and so on. Maybe the models don't have all the data that they needed anymore. So these can also change over time. And that's something that you always need to keep an eye on, right? To make sure that the model at the time that you introduce it delivers value, but that it also keeps adding value as time goes. And then of course, very important also to make sure of the old edge cases, right? So again, if you're working on making movie recommendations, how many times do you get to have zero movie recommendations, right? How many times does a user has seen these movie recommendations more than 20 times? I've never clicked on one. Like, is that maybe a signal for you that those predictions are not right? Or maybe you shouldn't even show it to them or that the UX is not right, right? So you need to also identify this and discuss and ask with the team on how you want to handle these edge cases, right? Because what I saw that will usually happen is that the data scientists will just focus on like how the model is performing, how it went on life, right? The designer of course will focus on the UX, the copywriter on the copy, but you need to be there to make sure that you connect all these dots. And what happens, right? When there is no movie recommendation, do we still show something on the design? What do we say with copy? Do we say we found no movie recommendations or we don't show the block at all? And again, how often that happens, for example, right? So you need to make sure that even though now you have this extra dimension of predictions, that all the dots still connect and that you have also a proper user journey in place when that happens, right? The next way of how you can add value and I think this is very important is on expectation setting with stakeholders, right? Again, you are using now a new technology. So this new technology also will represent or will bring a new kind of uncertainty and a new kind of blockers, right? You might now depend on people collecting certain data for your model or maybe on how to deploy those models and so on. So you always need to communicate these to your stakeholders, right? And it might be that sometimes your manager is not that knowledgeable of machine learning and what this means, right? So you also need to, that's what I put there, educate as needed because maybe they will keep pushing you to deliver things faster. But just like when you work with product development at some point, you need to invest time in refactoring the code. With ML, there will be a new set of tasks that you need to invest time, right? So maybe you decide on next quarter to focus more on cleaning the data, right? Or on collecting some kind of new data and that will influence on what other tasks you can deliver. So you need to discuss this and educate a bit your manager so that they also understand how you're investing your time. And then I think also very importantly, align on the trade-offs with your manager and in the company, right? So one example can be, for example, I was working on a product where we were doing some changes to... We were in charge of getting payments accepted, right? So you can do a model and then you can see that more payments were being accepted, right? But there might also be an implication, right, that these payments later on get cancelled or get like a charge back, right? Or maybe now you accept more payments but they were actually fraud, right? So there is a balance that you need to put in place. And these are goals and metrics that, of course, you should be aligned with the company beforehand, right? If you introduce a model, you're introducing a change, it hopefully has a good impact on your metrics. But there will always be another metric that you also need to keep an eye on, right? For example, again, if you're making hotel recommendations, you might see that you increase the amount of bookings. But then also you might also increase the amount of cancellations that happen, right? So how do you balance those two? And again, how do you measure then and how do you have the mechanisms in place to make sure that when one of those metrics goes out of what you expect, that you know it beforehand, right? And then, of course, be aware of bias and think ahead of where it can go wrong, right? We're working with data, there will always be bias on data. So again, discuss it with your team, right? How are we dealing with it? What mechanisms we have in place to make sure that we avoid action on those bias that we might have on the data? Is everyone aware of maybe aware of what the edge cases are? And again, how are we going to handle them? And then depending on what are you doing with these kind of predictions, what kind of mechanisms do you have in place to make sure that some predictions don't go wrong, right? Or what processes do you have in place to make sure that there is the necessary checks before maybe acting on something, right? So if you are working on something that is maybe a bit more complicated or more delicate, for example, you might decide that you need some manual intervention. And that is, of course, something that maybe the data scientist is not going to do. But again, you as a product manager might need to highlight to the company. And last, I think is, of course, again, to pressure on the team on making sure that you focus on the added value, right? Don't focus on the state-of-the-art machine learning, right? And this is, I've seen a bit more critical when you maybe start working on a new product and you have the full team, right? So let's say you're working on a new product and you have three data scientists, two data engineers, right? Of course, everyone will want to start putting models out in production and everyone keeps learning about new technologies. And we might be a bit excited to test that latest deep neural network technology. But again, right? The purpose of your product team is to deliver value. It's not to have the latest technology on production, right? So one way how you can handle this and how you can make this also more transparent when discussing with your team is to really define upfront and agree with them on, again, how are you going to measure that you are delivering value to the user and how are you also going to define that a new model or a new technology is doing better than the existing one, right? So an example, right? If you're working on making movie recommendations, are you going to accept that a new model is doing better if more users click on it or if more users end up watching the whole movie or if more users maybe stay longer watching the movie, right? These are all the metrics that can trigger different decisions and that can also decide what technology or what model you end up using, right? So the best thing that you can do is to define this upfront, to define this together with your team and then let this be the factor that decides what technology you end up using in your product, right? Not just going straight to use the latest one. So finally, I just want to give some maybe tips or also as a way of a recap on the talk on what things you can go and apply tomorrow and then keep an eye on every day with your ML team. Of course, first and foremost, I tell everyone, go and talk with your data scientists, right? Depending on the setup of the product, you might be working with them. Maybe this might be like a data science agency that is working closely with the team, but talking with them I think is the most important thing, right? So talking about what, right? Making sure that you understand the trade-offs, as I said, right? So if we're making a movie recommendation only when the model is what number are we actually labeling a movie as a good recommendation, right? This you need to be super clear and align for them. Understanding what the models are really predicting. As I said, right? You might get different results if the models are predicting what hotel is more likely to be click compared to what hotel is less likely to be canceled, right? So really align this. And how are all these predictions and all these trade-offs, how are these aligned to the delivering value to the user, right? Check your metrics. Do you have metrics, numbers, mechanisms to store this data so you know how often you're making good predictions, how often the predictions go to these edge cases, right? And then also I think it's super important for you to understand the pipelines, how the data is stored, how do we measure quality of the data, are there new kind of dependencies because maybe there is another ML platform team, right? So trying to understand all this technology landscape is also going to help you understand through, you know, the effort that the team puts, maybe a new blockers or configurations that you should keep in mind when planning an x-print, for example, right? And then I think of course super important and something that I've seen happen regularly. We always focus on the model does 80% of good predictions, right? So 80% of users see something very nice, but what happens when it goes bad, right? We tend to sometimes, oh, it's only 1% of the times that the prediction is off. Okay, fair, but what happens then, right? And as I said, making sure that all the dots are connected. What kind of copy do we show them? What kind of design do we show them? What kind of feedback do we collect when these predictions are off, right? Making sure that all of this is in place and making sure also that all the different disciplines are aligned, right? Data scientists discuss it with the UX designers, with the developers, how the data is safe and so on. So these are things that you're going to start acting on tomorrow. That was the talk. That was all the things I had to say. So thanks for your time. I hope this was valuable. And as I said, if you have any questions or follow-ups, you can contact me through my website and yeah, we'd be happy to continue discussing. Thank you very much. Bye-bye.