 Hi, folks. Good afternoon. Hope you can all hear me. Welcome to ProductCon. It's great to be here. I am a former product manager at DeepMind here in the UK. And therefore, my talk's going to be about building AI products. And the first thing that I'm going to tell you all is that you shouldn't use machine learning in your products. OK, that's the talk over. I'm done. No, I'm joking, of course, but there is some truth to it. So I've been fortunate enough to be part of teams that have shipped products which could be considered to be AI products to hundreds of millions, if not billions of users. And what I'm going to talk about today, some of those experiences and some of the lessons learned. So let's see. Storytime. The first thing that I'm going to tell you is about the most common question I received when I was a PM at DeepMind. So DeepMind, one of the world's best machine learning companies, people would come to us and say, you've got machine learning. I want machine learning. This is what I heard. This was the number one most common question I heard at DeepMind. And it's true. People think about machine learning or AI as magic pixie dust, something that can solve all known problems in product. I mean, again, spoiler alert, it can. So let me tell you a little bit about my experience. So as I said, I work for DeepMind. And in particular, I work for the division called DeepMind Applied, which, as it sounds, is about applying machine learning research. DeepMind exists as this incredible research institution which is pursuing AGI, artificial general intelligence. That is the dream that a single AI can learn to do any task that a human can do. That we can have a single computer program which can learn to do anything to play any game, to complete any task. Some people refer to it as the singularity event. I'm personally not a big believer that that will happen. But what I am a big believer in is AI's potential to help us solve real world problems. And in particular, when you have an AI that is developed to solve a narrow problem, to optimize, to provide more efficiency, as a field is called narrow AI, that's what I believe in. That's what I think is going to revolutionize almost every field. It's already starting today and will continue to do so for decades to come. So I worked for a team in the Applied division called DeepMind for Google, whose job was to have impact at Google with DeepMind's research. Quite a big remit, right? So let me tell you about one of the most famous examples that this team was able to do with Google. So data centers. Data centers globally consume over 200 terawatt hours of electricity every year. To put that in perspective, that's about 1% of global energy consumption. That's equivalent to about the size of a country like South Africa, or every country in the world apart from 22. And it's only going to continue to increase as more of our digital lives move to the cloud, as more and more demand for services happens. Now, as a rough rule of thumb, the amount of energy required to power a data center can be split in two. One, to power the computers in the data center, and the second for cooling, that's the systems required to keep everything running at a safe temperature. Now, we decided that we were going to go and pursue reducing the amount of energy to cool the data center. And you might have seen already in the media that this was successful. They ran some great tests, and I think they saved up to 40% of the energy required by applying an AI system to optimize things like pump speed, water flow rates, all this other kind of really interesting things. But there's an interesting part of the story that not many people know, which is that when we're doing these tests, one of the tests came back with a much smaller efficiency improvement than the others. And so the team went to the data center operators and said, what happened? Where did our efficiency gains go? And the team said, oh, well, we learned from the other tests, and so we just changed their standard operating procedures. And so the AI was able to teach the human operators a better way to run a data center. I think that's incredible. To me, that's one of the most interesting things about artificial intelligence, is its ability to potentially teach us new things. So that was a really cool project. One of the projects that I personally led was helping Android improve battery life with DeepMind models. This was a really interesting challenge for many reasons. One was because DeepMind today, when it builds models to solve whatever game they're trying to solve today, or pursue whatever vision they're trying to pursue today, doesn't really think about compute powers. We've got data centers the size of the one I just showed you. And here, we were trying to shrink down the best of what we could learn from DeepMind and fit it onto a mobile device to save battery, which meant we couldn't burn the CPU and things like that. Also, a big consideration is simply the scale of Android. Android is massive. There are almost three billion active Android devices in the world. If you can save a better battery life, you can extend battery life, you improve a meaningful user experience for many, many people. It might mean that you get to the end of a busy day like today, and you still have power left to call an Uber to get home. That could have a material impact on someone's safety. The other thing that's amazing is that when you have three billion devices and you save a meaningful amount of energy required to power them every single day, that's a non-negligible impact on the amount of power needed in the world each day, which I think is pretty cool. So, we put together a team. We brought some of the best research scientists from DeepMind, some of the best Android engineers, data scientists, program managers, designers, and product managers like me. We worked really hard for a year at this scale where we're interacting between a mobile operating system that goes out to three billion devices, app developers, user experience, and we came up with a UX, which was one toggle each. And that one toggle each actually got removed in an expression of Android. And it just goes to show you that as a product manager, you don't need to necessarily have a fancy UX to have a co-product. In fact, if you Google adaptive battery on some forums, you will find that some people say, just turn it off for better battery life. I mean, we tried. Actually, it was the single biggest improvement in battery life in Android's history, according to the data that we had. So, those are some examples of building ML products. But let me tell you about the hardest problem in machine learning, because it isn't ML, it isn't AI. The biggest problem in AI is data. And this informed my own personal journey. I was a product manager at DeepMind, and when we were working with Android, the hardest thing was having reliable, high-quality data that we could trust to build machine learning models on. And so when I saw that opportunity, I decided to move and work for the Android team and help build out Android's data platform, which was to build a scalable data platform, one of the biggest in the world, to serve all of these devices, to monitor the health signals, to make sure that they were performing well, so that we could release safely new versions of Android in things like machine learning models. That was a really big challenge, because these devices are incredibly personal, and so we have to have the best privacy standards and take really good care when we're doing it to make sure we're being responsible stewards of our user's data. So that's a bit about my journey. Now let me tell you a little bit about building AI products, because I'm sure many of you are or want to be machine learning PMs. People come up to me all the time and be like, you're a machine learning PM. I want to be a machine learning PM. OK, cool. So here's some things that you might want to know. The first thing that I should say is that throughout this talk so far, I've been using the terms machine learning and artificial intelligence interchangeably. And for most cases, particularly business cases, that's OK. But you should be aware that AI is simply a field of study to pursue intelligent systems, a system that can be perceived to be intelligent through its capabilities. The classic test here is the Turing test. If you had a chat program and you were talking to it and you couldn't tell whether it was a human or a computer system, then it would be said to be artificially intelligent. However, one of the biggest subsets of AI is machine learning. And that's where a huge number of the advances in the last few years and decades have been. And so that's why the terms are pretty interchangeable. But you can build an AI system without machine learning. You can build a system with just a lot of complex rules. And that could be an AI system. Machine learning advances in the last few years have also been particularly powered by a subset of machine learning called deep learning. Deep learning is where you take a neural network with many layers, give it a huge amount of data, usually a vast amount of compute resources as well. And what you end up with is a learned model that connects the data to some outcome you're trying to achieve. And that is the type of advancements in the last few years, which enabled DeepMind to build a computer program that could beat the world's best player and goal, Lisa Doe. And that program and that feat was considered by experts to be 15 or 20 years out. So it's a pretty big advance. So let me talk more specifically about the product manager's role here. If you want to be a machine learning PM, what should you be thinking about? The thing is, you're not going to be training models. And you're probably not going to be even choosing the technologies that you use. Instead, what I really want you to focus on is creating clarity around the problem you're solving. That's very important. And also, set the success criteria. I'm going to go into these in some more detail. And I also believe that as a product manager, a large part of what you do is set culture for your organization, your team, and your product. And there's a lot here that you can do around privacy and user bias, which will help push things forward. And then, finally, communication. As PMs, we all love to talk. It's important to talk clearly, to communicate clearly the expectations about your product, what it can and what it can't do, and the impact it's going to have in the world. So first question that you might come across is when to use machine learning in products. So machine learning is great at some very specific problems. It is fantastic at creating personalized recommendations like my Netflix feed. It is also great at things like grouping like things together or predicting a future event. Where machine learning isn't great is where it would be doing something that you need to be repeatable, predictable, or understandable by your users. Let's say that you are creating an app that does bank transfers. If you're sending money to someone, do you think we should use machine learning in that process? You want it to be predictable. You want it to be the same every time. You want to take the uncertainty out of the situation. That is not a good place for machine learning. So to help you with this decision, I've created a small decision tree. I think this is the simplified set of questions you should ask yourself before you even consider using machine learning. So the first question is, do I have data to power an ML solution? Now, product manager's in the room. I can see you snoozing here when I'm talking about data. Data is the hardest problem in machine learning. If you're a product manager just now and you have a product no matter the size of it, no matter if it's a big data platform that I was talking about or a smaller web app, do you know where your data is coming from? Do you believe it is accurate? Do you believe it is reliable? Do you believe that it accurately represents the population you're trying to serve? The answer to any of those questions is no. You probably are not in a position to use machine learning. Next, this is people who used to come to me and say, can I have machine learning? I would say to them, have you tried 50 if statements? Serious point, 50 if statements would be simpler, more maintainable, more efficient, more understandable than a machine learning solution and might help you achieve the same goal. So just ask yourself, can I build 50 if statements? Next, does your product require personalization? What I mean by that is does every user of your product get a different version of your product, you know, some different content or some different experience? If so, machine learning is great at that because it can take the standard product flow and adjust its behavior based on data. Next, are my users comfortable not knowing exactly how my product functions? To go back to the banking transfer example, are users comfortable not knowing exactly what happens? Machine learning is very opaque. It's like a black box. You could right here be dragons on it, right? It has a bunch of stuff in it that was difficult to explain and it's very difficult to explain to users why things have happened. So if you have that requirement for your product, if you have maybe you're working with financial information or health information, probably not the best place to use machine learning. And then finally, will users tolerate errors in your products? Machine learning will go wrong. I can guarantee it and it will produce strange and unexpected things. And if your users do not have a tolerance for that, you're going to be in for a bad time. So let's say you answer yes to every one of these questions. You'll notice that the green box doesn't say use ML. It says maybe try ML. It's very important, buyer beware. Okay, so let's talk about the process of building an ML product. So let me talk about football for a second. One of the things that I find hilarious about football, the most popular sport in the world is that we cannot agree on the size of a football pitch. If you were to go to the Etihad, the home of Manchester City, the champions of England, you would find that the pitch is 105 metres by 68. If you were to go to Craven Cottage, the home of Fulham, who won the championship last year, you will find out that the pitch is 100 metres by 65. That means at the Etihad, you have 500 square metres more space to play in. As a product manager, this is relevant, what I believe is that your responsibility is to define the game your team's playing. And more importantly, how you keep score. Because in machine learning terms, how you keep score is the optimization function for your machine learning. It's what everything will be geared towards. Without a clear goal, without a clear objective, your machine learning product will never be able to meet what you hope it's going to be able to meet. This is very important, define the game and how you're gonna keep score. Next, and again, no falling asleep, data. If you're a product manager, you should be involved in all parts of your data lifecycle. You should know where your data is coming from. Those questions I asked, is it accurate? Is it reliable? Does it reflect my population? And then all the bits in between that end up producing a machine learning model. How is it stored? Is it safe? Is it secure? Is it private? Those are all incredibly important parts that you should be aware of. And then, you need to take a big step back. This is where other members of your team come to the forefront. Your data scientists, your research engineers, your machine learning engineers. Let them do what they do best. Take the technology, make the choices for what the best options are going to be, and they will build models that draw relations between the data you have and your product objectives. They will produce what we call candidate models. Candidate models is the first model that you're going to try in production. And that's where you can get back involved again. Now, you're all competent product managers, I hope, and so all of you are doing A-B experiments. You probably woke up this morning and looked at your A-B experiments, right? If you define the goal for your product, then you're going to run experiments. You run A-B tests on your experimentation with your candidate models. And you will look and see which one is performing best. And the one that's performing best is the one you're going to ship to production. But be careful. You need to look at all the other metrics as well, because machine learning can have unintended consequences. It can do unexpected things to your system. And so you need to have a holistic view of all the metrics that matter to your product. And now, once you have a candidate model, I guarantee you candidate model number one will never be the model you ship. You will move into a process of iteration where you continually refine until you get to something that you're happy with and happy to ship. So in this journey that you're going to go on to become machine learning product managers, there's a couple of things I want you to watch out for. In particular, you need to really pay attention to preventing poor outcomes. Machine learning is extremely complex. It will add complexity to your product. It will also do unexpected things. And unfortunately, machine learning is very good at reflecting, reinforcing, or even reducing bias that we already see in society. And so this is where you, as a product manager, can have a huge impact by anticipating what those poor outcomes can be. Think about how underrepresented groups could be impacted by your product. How maybe disadvantageous associations could be drawn between certain characteristics, things like that. And then build test cases, build negative test cases to test your product and protect against these things. Then if you do come across any problems in your product, your responsibility is culture. Your responsibility is to set standards. This isn't acceptable in my product. You can change your model. You can change your raters. You can change your policies. All of those things can help your users end up with a better experience with a product that works for more people. And I believe that as a product manager, a huge part of your role is setting culture and setting standards. And you should be continually pushing to improve the standards for your product. So TLDR, of everything I've just said, machine learning is not magic pixie dust. It is an incredibly powerful tool which can be used responsibly and is very good at solving certain types of problems. It is not for everyone. It is not applicable for everyone. Very often at these types of product meetups, I get people who come up to me and say, you're a product manager with machine learning. I want to be a product manager with machine learning. My response is something like, great, why do you care about machine learning? Do you care about relational databases? It's a tool which can help you achieve an outcome. If it's the right one for your product, great. I wish you good luck. And just beware of the pitfalls. This is some great resources if you want to learn more. I highly recommend taking a snap at this. And thank you very much for your time.