 Welcome to our webinar on AI product development. My name is Marisi Annabi and I'm a senior product manager at Amazon. Today I'm going to talk about the AI product management but before that let me start with the brief intro by myself. I'm a record scientist by training. I did my PhD in aeronautics and astronautics with focus on optimization and autonomous systems and a master's in pure mathematics from University of Washington. Right after graduation I joined Palo Alto Research Center known as Xerox PARC as a research scientist in AI. I was very interested in the business aspect of the technologies that we were developing in the labs and how we can bring those technologies into reality putting it into a context of the product. So I did business training at the Stanford Graduate School of Business and right after that I joined I joined the business unit at PARC basically and I was in charge of commercializing the advanced technologies that we have. After that I decided to move from R&D to more in the context of companies creating products and I joined as a senior product manager. I joined a startup, a unicorn startup basically in a valley it's called Automation Anywhere and I was in charge of bringing AI and more specifically natural language processing computer vision and speech to create a product that basically can automate enterprise processes. After that I joined eBay as a lead product manager and I was in charge of bringing AI and using AI to convert unstructured data provided by our sellers and buyers to and convert them into a structured format. And after eBay I joined Amazon as a senior product manager in the supply chain optimization technologies. Disclaimers, so the content I'm presenting here are just based on my personal experiences and are not reflecting the companies I worked for before or I'm working right now. And the other aspect is that the presentation is going to be forward thinking and some of the concepts and the way that I have been thinking about this AI product development may change and we'd love to get more feedback about how you are also thinking about this concept of AI product manager. I believe AI product management is a very important topic and there is a lot to cover here but the two specific concepts that I was hoping to cover here is the first aspect of it is that what it means to be a product manager for AI products. There are different perspectives to this. There are different things to consider but I'm going to talk about three specific pillars here. So about the mindset, the type of a skill set that an AI product manager would need and this concept of integration and collaboration between AI and human that I will elaborate more in a few minutes. The second topic I'm going to cover is just talking about some opportunities and more about the challenges that we would face as an AI product manager. Okay, let me get into that. So how to become and what it means to be an AI product manager. Before I dive into the concept, so I want to start with defining what I mean by AI and make sure that we are on the same page. So AI is a soft branch of computer science started in 1955. It went through a lot of ups and downs and we are at a stage that a lot more development has been happening at AI. So AI itself has different branches, some of them more popular, so some of them more have gone through more advancements recently and some maybe not much. So branches like machine learning and different aspects of machine learning like deep learning, reinforcement learning, supervised learning, unsupervised learning that we talked about. The second branch of AI that also has gone through a lot of recent development and interesting applications about the field is natural language processing. So others like vision, like a speech the same and so other branches of AI like expert systems on the planning and robotics are all into the category of AI. So when we say AI is very broad, it can include many different subfields basically. So become a PM in AI based on my personal experiences. So these are the three pillars that I was hoping to dig deeper today. So the first aspect of it is that what should be the mindset of a PM when it's working on AI products? So the second aspect of it, the set of skillsets that is needed and the third aspect of it that I'm going to talk about is human AI collaborations why we needed, why I think it can add a lot of value to the set of products that we are creating. And so I just, I want to start sort of bringing this and as a way for AI product managers to think about the products that are creating. So it may not be applied in all scenarios but I believe it has a lot of value and it is a good thing to think about it when we are working on AI products. So let me get into each one of these pillars and talk a little bit more about the details and what I mean by that. So the first aspect is the mindset. So AI product development in many sense is very similar to a traditional product development. So what we see in product lifecycle, what we see as an introduction, growth phase, maturity, the decline, it will be also valid when we talk about an AI based product basically. The same goes with product development process. So the way that we think about developing a traditional product, most of those set of steps and the tasks that needs to happen is the same for AI product development. So when we talk about the process in a product development, so we go through the conceive, we talk, we plan, we develop, we iterate and this iteration and it happens in all these phases. So the launch when we go through the study of state and when we go through the maintain and kill process, so all these tasks would be the same basically when we talk about AI product development. So this is, I think it is very important aspect even though all of these phases, when we talk about one product, it would be different from the other product, but the framework, the set of tasks, all those things that we need to go through, it will be more or less the same. So this is the first aspect. So we just need to think about AI as another set of tools at our disposal to solve a problem. So and that we start with the problem, that's a key point here. So the second pillar that I wanted to talk about was this key set. So a traditional PM needs to know the technology, needs to know the business aspect and needs to know the design aspect. So a PM in AI would also be the same. So we need to know the business aspect, we need to know the design aspect and when it comes to technology, I want to argue that we need to have a deep knowledge in AI. So we need to be able to interpret and criticize the results and not just understand and explain and communicate them. So what I mean by that, if we think about the depth of knowledge as four level, so the first level being able to recall a concept, the second aspect of being able to use it to solve a problem and the third and fourth aspect of it to be more of a strategic thinking and extended thinking, being able to criticize, being able to interpret and that's what I mean by that. So a PM in AI needs to have this fourth level of depth of knowledge when it comes to AI. And the reason that I argue that this is very important are just the following. So the same way that if a CEO cannot read the financial documents, cannot understand the data coming out of the product of her company, then she won't be able to make, make intelligent decisions. The same goes with a part, a PM who is working on an AI based product. So the PM needs to be able to analyze and criticize the solutions, the algorithms, the results that are provided to the PM. And so, and this is, so when the data, this volume of information coming, so and a PM who has a depth of knowledge in AI can make a better decisions. So the other aspect that a PM in AI needs to have this depth of knowledge in AI is that the PM have to understand and communicate the limitations of the technology. So we often come across people who know the AI as a buzzword, they think the AI is a magic and is going to solve all the problem in the universe. So we should be able to communicate the limitations. First of all, know the limitations exactly and then being able to communicate the limitation. So the third aspect of it is AI and the algorithms that we are developing are based on data. So data is integrated into any AI product development processes that we are going through. And understanding that every aspect of the data, the data acquisition, data handling, the privacy and security aspect of the data and basically being able to put a firm and valid data strategy in place can happen if the PM knows well enough and has this depth of knowledge in AI. The last part of it is that this depth of knowledge in AI, it would build trust with the scientists and developers as well. So that's why I think the difference between a traditional PM who knows who should know the technology well enough and is different from a PM in AI should know the AI technologies like in a deep level basically. In the fourth level of the depth of knowledge and will be able to criticize and interpret and make a decision about the next phases of where we should go with the results with this data. So now let's move to this human and AI collaboration and integration. So we cannot expect the AI algorithms and we cannot expect using historical data to teach machines to perform with high confidence and high accuracy in all scenarios. So there is this limitation to AI and that comes because of the volatility in the data that sometimes is hard to model. So because of the unseen events that are happening in the data. So there's these scenarios that the AI algorithms won't be confident and won't be accurate about the predictions that are making. So then we need some way to adjust for that. And I believe in most cases, not all for sure this integration and collaboration between the human and the machines and AI algorithms can provide a better product that will have higher efficiency, better adoption and more confidence basically in the product. So let me give you two specific examples of this and I wanna encourage all of us when we think about AI products, how we can basically embed this human aspect into the product that we are developing. And the two examples, one is in the context of predictive maintenance. And we come across this in advanced manufacturing for example, in context of more physical work probably. Think about this specific example that we have trains and we are trying to predict whether some pieces of this train that let's say doors, they need maintenance. So we wanna look at the historical data and based on that historical data predict if the door is going to fail. So if we make that prediction then we will do the maintenance ahead of time and we will stop the interaction to the train operation basically. But what happens if we just use the historical data? So we develop this predictive maintenance based on the results, even though if the algorithms are not high confident but we will send the crew to the train to the main line and they will try to see if there is any maintenance needed. What happens is that if the crew will go there a couple of times and they see that there was actually no need for any maintenance, over time they will lose the confidence in the algorithms. But if I integrate, if we integrate the human into this process and so they can add input to the predictive results that we had from the algorithms, then we will prevent this sending crew to the train while there was no need probably for any maintenance. This is one example. The other example is a customer service automation. Again, this is more in the context of enterprise and think about an enterprise like, for example, enterprise in educational services they receive about half a million at least emails per year. So without using AI, so a human needs to read the emails, understand the intention of the emails, send it to the right people and or perform the specific actions that the email was asking for. So we bring AI, we bring natural language processing and NLP will be able to understand the intentions, we'll be able to extract the right information and take some actions using some other services. But my argument is that what happens if the AI algorithm is not confident? So should I send it to the wrong person? So should I perform the wrong action? And depends on the context, some of these actions downstream could have bigger sequences. So let's think about context of banking or other scenarios that the consequence could be bad. But what if I bring a human into this whole AI solution, AI product and the cases that the algorithms are not confident it will be sent to human for review. And this is now the human can focus on the more complex cases, the workload would be much less. And we will be more confident in the results that are coming out of AI. So this integration of the human into the AI products it will be much more complex than a traditional product or even a traditional AI product. It will be heavy on the design because of this presence of the human and the integration that we are talking about. But at the same time, there's a lot of value that can be provided by this collaboration and by the way that we as a product manager from day one we think about this integration. And one aspect of this integration that can help is the human can provide a venue for us to collect more labeled data. And this labeled data would be a very high quality data that we can use it to improve the algorithms. So, yeah, as I said, I encourage you all to think about this human AI integration. And as I mentioned before, so I believe there is a lot of value but not always that is the right solution. So I stop here as a point that we talked about them how to become AI product managers. There are a lot more to it. I myself transitioned from research, from R&D to product but I will stop here on this subject and I move to the next subject. The next one talking about opportunities and challenges. So I don't think I need to talk much about the opportunities all these advancements in the technology advancements in dealing with the data, the computational advancements that happened in the last couple of years. It all made it AI more reasonable to use as a technology for solving more complex problems. And we see examples all over. So both for the consumer solutions as well as in the context of enterprise. I talked about customer service automation. So the technologies that we see the drivers cars, the advanced manufacturing, the better health and wellness type solutions that are in the market. And the list goes on and on and on. So in the context of e-commerce in the context of sharing economy. So all these things. So we see very interesting examples of AI. So there are opportunities there but I want to talk more about challenges. And the challenges basically, I divided into these four categories and I will talk about each category but I'm sure there are a lot more given the context of the product and the solutions that we are thinking about. So let me start with the challenges as in the context of stakeholders expectation management. And I believe this is a very, very big one. So who are the stakeholders? So the stakeholders, we can divide them into either external stakeholders and also internal stakeholders. And depends on the face of the product, depends on the bigger context that the product is living in, for example, the state of the company and so on. So the stakeholders might be different and the expectations might be different. But based on my experience, so given this buzz, word aspects of AI and some people think that AI is a magic. So this expectation management would be very important. So external stakeholders could be enterprise companies if you are working on the enterprise solution. So consumers, it could be analysts and so on. Or it could be the business owners, all those business owners in the context of an enterprise. And I have seen examples that people want AI solutions, but without any experience about AI needs data. So they don't know what is the label data, but they have this metrics that they need to follow like in their business units. And they think that AI will solve all of their problems. So the same, I'm sure, goes to the context of consumers as well. And then the same goes with the internal stakeholders and internal stakeholders like sales teams. So sometimes they believe, or if we add this feature that is AI based suddenly, so the sales will go skyrocketing. So the marketing team the same. So the all of this internal stakeholders also need to be educated about what is AI, what are the limitation, what it takes to create a product in AI. The second challenge is the data acquisition and governance. So AI basically is integrated into data. So we need data in order to develop models and to think about how to obtain the data, how to manage the data, how to what it takes to take care of them the privacy and security of the data. So that the IT requirements, all those aspect needs to be thought through. Some companies may not have those IT infrastructure. So and the AI product manager needs to think about what it takes and needs to educate the system and needs to team up at least with others to create that infrastructure. So the next aspect of the challenges are knowing the technology limitations and thinking about the return on the investments and return on the effort that it takes to create the product. So and what can be achieved, what cannot be achieved basically and sometimes some of the solutions may not need AI or may be able to solve those with a simple algorithms rather than going all the way to deep learning but a simple regression sometimes can solve some of these problems. So being able to focus on the problem and see how we can solve that problem rather than being excited by using deep learning to create a solution. The last aspect that I'm going to talk about today are the metrics. So if metric is on the mind of all of the PMs but as a pilot managing AI so new set of metrics come into the picture basically. So having a very good sense of what are the different metrics and why we should look and define the metrics that is another aspect of the AI given that new set of metrics are defined and are at the disposal of a PM in AI. So I'm not going to go through the details given the time but as I mentioned all these challenges we can dig deeper and talk more about each one. So the main takeaways I want you all to get from this webinar. The first aspect AI is not a hammer so it's just another set of tools to solve a problem and we need to think about what problems we are trying to solve. So the other aspect of it as a PM in AI I encourage you to have to develop that depth of knowledge in AI and understand different algorithms, understand so what are the limitations? Understand what can and cannot be done. And the third point is that AI plus human can help us to create products that are basically have higher efficiency and it will have a better adoption basically at the end of the day. And thank you so much for listening to this webinar.