 Hello everyone, welcome to this webinar on AI ML product management. I'm Deepak Mukuntu and I lead a product management team at DocuSign focused on building ML platform as well as enabling key ML scenarios for the company. Prior to joining DocuSign, I worked at Microsoft leading some of the key initiatives in Azure, Bing, and Windows. AI and machine learning is an area of passion for me. Today, I'm happy to share some perspectives based on my experience on AI product management. Let's get started. This is what we will cover today. We'll start with AI adoption and how a lot of real world scenarios are being pushed by AI these days. Overview of the ML process including then to an ML lifecycle. Then we will go over role of AI product manager and how to be successful in that role. We will end this session with giving an overview of how to get started for people who want to get into AI product management. As most of us know, AI has already been making a lot of influence on us on a day-to-day basis across multiple industries and sectors be it the social media, healthcare, entertainment, gaming, and many more. Now, this is possible today because of three key trends that I've immersed recently. Number one, growth in the data volume. Number two, powerful computing infrastructure availability for training large models for enabling these two cases. Number three is research advancements in algorithms that people use for training models. Those are the three key initiatives or key recent trends that have helped us get to the state where we are today. But again, this is just the beginning. There are a lot more research activities that are happening in the AI space where more and more of these advancements are going to come and we are all going to witness that. Let's take an example of online shopping experience that we go through as customers. We typically go through four broad stages. First stage is when we have a purchase intent where we don't know exactly which product to purchase, but we have a need and we start shopping. We start discovering for products. Then stage two is when we make our first purchase. Stage three is when we become repeat customer, meaning we purchase more and more from this online retailer as our need emerges. And then stage four is when we become loyal customers or fans for this online experience where we start promoting this to our friends and family. So with those four stages in mind, let's go through how AI and machine learning is helping each of those stages for the customer like us as well as for the online shopping company. The first stage, when we are still discovering when we are in the demand generation phase or purchase intent phase, things like search, like great search experience, powered by machine learning, smart content curation, all of those things help us discover the right product that we may like. Then in the next stage, when we make our first purchase, that's primarily driven by investments around app targeting or lead scoring. All of those investments pinpoint the online retailer to reach out to us as customers to enable us to buy a specific product. The next stage of getting into repeat customer, becoming a repeat customer, it's basically driven by investments like personalization, targeting or discounts offers. That way we as customers are motivated to buy more and more from this vendor. And the last stage is primarily driven by predictive customer service. For example, if our customer service experience with that company is great, we end up becoming a fan. And that's one of the investments from an MISML slash AI that helps us get to that loyal customer bucket. And another one possibly could be marketing notification, like proactive notifications that help us get more and more value from that vendor and we start purchasing more and more and become a fan. So those are some examples of how machine learning and AI helps us as customers as well as the business reap benefit through this process. So this is just one example. But as you look around, you will see that there are so many real world scenarios where ML and AI is helping us get our jobs done, starting from the virtual personal assistants like Alexa and Siri to search engines that we use on a daily basis like Google or Bing to getting traffic predictions as we go from one place to another using Maps apps. All of those use large volumes of data and machine learning in AI to give us the experience to help us get our jobs done. As I said, AI and machine learning, even though we have so many applications today that we are interacting with, this domain is still very early. There is a lot of research going on to extend and to give us more and more benefit out of AI and machine learning. For example, robotics and autonomous vehicles is an area where there is a lot of research happening and there are some real world implementation that are starting to come. How do we, as product managers, one of the questions that I would like us to ask ourselves is how do we remain relevant in this area of AI machine learning with everything moving towards intelligence, we need to start developing that muscle. So AI product management is a new PM specialization that started two years back and that's what we're going to talk about today. Now, before we understand the role of AI ML product management and how do we succeed, let's take a small detour and understand how machine learning works. And to do that, let's take a simple example of house price prediction. Now, if you are trying to sell a home or if you're trying to buy a home, one of the key questions you have in mind is how much is this home worth and how do we know that? If you look at online platforms like Red Spin or Zillow, they provide house price prediction or house price estimate for every house that is listed. How are they able to do that? How are they able to estimate the house price? What variables go into the determination? Let's see how ML actually helps with this area. Now, let's briefly talk about how machine learning is different from traditional programming. With traditional programming, we have a goal in mind and we need to write a functional program that can with certainty produce an output for a given input. Now, note the word certainty. With machine learning, you do not know the behavior of the program or function beforehand, but we have a lot of historical data that we can use to find that trend and to codify that. And that is exactly what how ML systems work. You have data as input and you know the output from historical data and your goal is to produce that program that basically can be used for any new data for you to find the output. So with that understanding, let's dig into the problem at hand. For house price prediction, what are the input variables and what data points would help us determine the price of a house? Here are some that I've listed, but there are many more. Size of the house in square foot would be one. Number of bedrooms, number of bathrooms. Neighborhood schools is an important factor that influences house price. Type of house, whether it's a condo or a town home or an independent house, that also plays a key role in the price of the house. So all of these are input variables, also called features in the machine learning world. Okay, let's see how we can model this as a machine learning problem. For simplicity, let's take one variable as an input variable. In this case, the square foot is the home and the output variable is the price, as you can see from this graph. In machine learning, there are two broad problem types. One is regression, the second one is classification. Regression problems are where your output variable is a continuous variable. In this case, because our price can range from anywhere from $0 to $1,000,000, it's a regression problem. Classification problem on the other hand is where you're trying to predict a fixed number of categories. An outcome is one of the categories from a fixed list. For example, if you're trying to predict customer churn, whether a customer is going to churn or not, it's a Boolean outcome. That's a classification problem. Now, in this case, if you look at the graph here, all the blue dots represent historical data points. Historical prices and the historical size of the forms. Now, if you look at the trend here, it looks like the dots basically form a straight line. And hence, we can try to model this as a linear regression problem. So regression, as I said, is predicting continuous variable and linear is that the dots fit into a straight line. So the task here is to figure out what straight line makes the most sense and can map all of these data, all of these blue dots, so that anytime we get a new x, we should be able to predict the corresponding y. For any home where we know the square footage, we can predict the price. So with that in mind, this equation here, the line here, seems to be the line of closest fit for the blue dots. This is where machine learning is helping us. Machine learning is helping us come up with that line. So we did not know the equation of the line to start with. But through modeling, through ML training, what the system does is it tries to plot the line that best fits these blue dots and that line is our model. So in this case, that equation that you see on the screen is our model. Now, that was a quick glimpse on how to arrive at an ML model for a given task. In our case, it was house price prediction and we came up with a linear regression model which was drawing that straight line. But building the model is just one out of the five steps in the overall ML life cycle. Let me quickly walk you through all those five steps. The first step is you frame the problem. In our case, for the house price prediction, we started with the business problem of predicting the house price and then we mapped it to an ML problem of linear regression. As part of framing the problem, you also define the success metric for that model. It's also called model evaluation metric. So for regression tasks, the typical evaluation metrics teams use are R square, root mean square error or mean absolute error. So those are the top three that I've seen people use. But depending on the problem type, you need to make the right decision on what model evaluation metric makes sense. That's the framing the problem step. Once you have framed the problem, the next step is to go find data, secure data that you can use for the training process. As you go through that, so that step is called data preparation. So that involves things like how do you secure data from different sources? How do you clean the data? How do you prepare the data in terms of treating missing values or performing transformations like normalization or scaling followed by feature engineering? So all of that falls into the data preparation phase. The next phase is when you actually build the model, which is train and evaluate. So as part of training, data scientists have multiple options to pick from. They have multiple algorithms to choose from and each of the algorithms have their own set of hyper parameters that you tune to get to the model to improve. And this is where data scientists spend a lot of time and it's a pretty time consuming and iterative process for figuring out which algorithm and which hyperparameter combinations give you the best model. And in many cases, this is an iterative process, meaning you go back to the previous step. You realize that as you start building the training model, you start realizing that more data could have helped or different type of data cleaning or data normalization could have helped. So that's when you go back to the previous step and redo that. And in some cases, you even go back and figure out like what new data can I secure for meeting through the model? So it's a pretty iterative process there. Once you have the model trained and evaluated, you go ahead and deploy the model. So deploying the model, there are multiple options depending on your need. If your need is to just do one call to the model and get an output for a given input, you can deploy the model as an S10 point and call that as an API. Or if your scenario requires a large volume or a batch of data coming in and you want to predict or consume the model for that batch, you can deploy it as a batch entrance and service and then use it that way. So it depends on your use case. And then once you have the model deployed, you may think that your job is done because the model is there and your scenario is calling the model and the scenario is up. But no, it's not the end of the world. That's when you start, meaning you have the first version of the model deployed in real world and it's serving real customer traffic, which is where you start now collecting customer data and feedback loop to improve the model. You continuously measure model performance and production and you start looking for things like data drift or model drift. These are concepts you will start getting into as you start deeper into this area of ML life cycle or ML ops, which is managing your model life cycle with real production. Now let's look at the role of an AI product manager. Based on my experience, I've seen four different AI product managers. First one is AI product focused. Here, you leverage your domain expertise as well as your ML understanding to drive opportunities to infuse intelligence to the product. So if you look at Siri or Alexa or search engines or personalization experience, all of those fall into this bucket. This is probably the most common AI VM role with scope of impact in any industry or product line that you can imagine. The second bucket is AI platform. This is where you as a product manager, you're focused on defining and building infrastructure and tooling to enable data scientists and ML engineers to be successful. This role is fairly limited because it is capped by number of platforms that exist and number of companies that are investing in building ML platforms. The third role that I've seen is called AI operation slash business focus product managers. So here, you are trying to identify opportunities to bring intelligence to internal business processes and analytics slash data science workloads. If you imagine like trying to understand customer churn for an area in the company or doing revenue forecasting could be scenarios and policies. Scenarios that fall into this bucket. This is called AI operations slash business intelligence area. The fourth bucket is what I call AI research. This is where you're actually working with AI researchers to bring breakthrough research from research to market, either through publications or through platforms that product integrations. This again is a very niche space because not a lot of organizations have deep involvement or deep investment in research. So this is very limited to large organizations which have a huge research budget. Now, with those four types of AI product manager role, let's look at what are some key traits of successful AI product managers or in other words, how to succeed in this role. This is a no-brainer for any product managers focusing on customer scenarios and business value. But I've seen that in the AI and ML space, it can be very tempting to lead with technology and ML and AI innovation. As product managers, if we remain focused on customer scenarios and business value and have the technology follow, it will be a lot more successful like with any other product managers. But I want to reinforce that in the AI and ML space, this becomes even more important for us to focus on. So instead of starting from ML as a solution and trying to find a problem, always start from who are the customers, what are the scenarios, what is the key business value that you need to add in this area and then work backwards from there to figure out what solutions help. I've also seen that for most of the problems at hand, you don't need to start with machine learning. You can start with a very simplistic heuristic-based approach, ship something simple, get an MVP out, get customer feedback, and through that process, collect more and more data. That way you can then improve the scenario by building more sophisticated models down the road. This strategy works a lot in many cases. So I would highly encourage that you follow this. Next one, which is the continuation of the previous one, is focus on the right metrics. Many ML projects fail because the team doesn't start with the right goals and business outcomes to measure. In absence of that, what happens is teams go in circles, trying to find approaches, different approaches and trying it out, and eventually coming back to the realization that they were not focusing on the right problem. So define what success looks like, model that into the right set of metrics to optimize, and then start coming up with solutions. So this is going to help you as you get into the journey of AI and ML product managers. This again, very typical of a good PM role, but it's even more important in the ML space, given ML is kind of non-deterministic, as we talked in the original definition of ML, non-deterministic and probabilistic nature of ML warrants this model. Next one, foster and experimentation culture. This is super critical because as we talked in the beginning, the ML lifecycle is a pretty iterative and time-consuming process. Also, user behavior trends and input data can change over time, which means that you, instead of spending a lot of time trying to get to the best solution with a big bang approach, if you can hypothesize small and incremental approaches and then have a close look with customer, where you can actually get customer validation on an ongoing basis to improve your model or your solution, the better off you are, which is where teams that have a good agile and experimentation culture have a higher likelihood of succeeding in the ML and AI space based on experience. Next one, AI can be very powerful, but with great power comes great responsibility. Now, think of a situation where you apply for a credit card or a loan and the bank rejects your application and they are not able to give you reasons for why they reject it. That can be very frustrating experience as a consumer. So as an AI product manager, if you have a great model which is super accurate but is a black box versus a slightly subpar model that's super transparent and can explain why the model is behaving certain way for a given input, choose the latter because you can always improve on the accuracy of the model but you need to be able to explain for you to gain customer trust. And this is especially important in regulated industries like healthcare or finance where transparency is super critical. Now, let's look at the other aspect. If your model is biased towards specific genders or race or nationality, that can be an extremely bad impact to your business or to a product and the consequences can be very bad. And this happens, this does happen because most of the model's behavior is based on the input data. If the input data has biases, the model will have biases and we all know that the data around us has inherent bias because the societal bias that comes in also comes into the data and can become part of the model. So it is very important for us to take a step back, make sure our data is good and is of bias and if there are biases or imbalances that you see, there are techniques to improve your data to remove those biases before training the model. Again, ethical and responsible AI is an emerging field. It is top of mind for many, many enterprises these days. So it is important for us to treat this as high priority for us to be successful. And last but not the least, while AI and ML is already pervasive, this is just the beginning like we talked. This is an area where there is not more research and innovation that's going on. There are also not more learnings happening around potential pitfalls of not using this technology in the right way or responsibly. So please keep an open eye towards these trends, opportunities and challenges so that you know how to leverage the power of AI effectively and responsibly. If you're new to AI and machine learning, you may be wondering how to get started. In this section, I will share some approaches that are very effective. First, online courses. There are a lot of great online courses available on platforms like Coursera and Udacity. Make use of them. My personal favorite is this machine learning course by Andrew Ng on Coursera. That is how I got started. After doing a few of these online courses, I wanted to get a bit more formal education. A lot of Udacity these days offer part-time certification courses specialized in machine learning and AI. I did one such course that University of Washington offered. The way these courses are structured is you spend a couple of hours in class learning, followed by a couple of hours of homework or assignment. And the unique thing about these courses is that you have a cohort of participants that are going through this course with you. So you get to interact and collaborate with them. The next thing you can try is AI and machine learning conferences. So there are a lot of conferences happening across the world where teams come and present how they're using machine learning for their real-world scenario. And there are also researchers sharing ML trends and research work that they're doing. Another great tool to explore is Kaggle. Kaggle hosts a lot of real-world competitions and challenges that involve machine learning and a lot of data scientists participate in these competitions. Even if you're not ready to compete with data scientists, you can actually look at competitions that are already closed and try your submissions. You can also look at other submissions and other kernels that people have shared for you to learn from. As you go through these learning experiences, make sure that you make sure to publish your work. Publishing your work in terms of simple blogs or videos or sample ML projects not only show that you're learning but also helps you build your brand, which will be extremely important as you get into applying for real AI product manager roles in the future. Last but not the least, internship and volunteering opportunities look for those because what these opportunities give you is a real-world experience of machine learning. While all the previous approaches I described help you learn, this is one approach that can help you solidify your learning by actually doing on the job. So those are the approaches I would recommend that you can try out to learn. With that, I think we have reached the end of this session. That is all I had to share today. Hope this was helpful. If you have questions or want to follow up, you can reach out to me on LinkedIn. Thank you all for this opportunity. Take care. Bye now.