 All right. Thank you everybody for joining today's webinar on launching your career as an AIML product manager. I'm super excited to be able to talk about the subject with you through product school. And thank you so much for having me on today as a speaker. Please feel free to use a chat along the way to ask any questions that you have. I'd be happy to get back to them at the end of the discussion. So without further ado, I'd like to give a brief intro about myself and then walk you through what we're going to be talking about today. So a little bit about me. My name is Chin Mai. I'm currently an AIML product manager at Microsoft within the Teams team on the engineering side. And I focus on making features intelligent and improving the quality of experience that we provide teams users with using AI and machine learning. Previously, I held roles at J&J, Shopify in the Ontario public sector and at Microsoft as well. So just a quick walkthrough of the agenda and the topics that we're going to be covering today. The first one is just a refresher on what the typical PM skills are. We're going to review the foundations of being a strong, proud person. Next is understanding what AIML is and where it fits into the bigger domain. The next is the skills required for an AI or ML product role and how it differs from maybe a traditional product role. Fourth is the responsibilities of an AIML product manager. And again, doing a little bit of a compare and contrast between typical product and what AIML product means or is. And then finally, the future of the AIML product domain. Really, what will this space look like in the coming time? What are the areas that people should be focusing on? You as professionals, I'm guessing are attending this webinar because you know how nascent the space is and how much growth there is still yet to come. So super excited to talk about this. So just going into the refresher. At the end of the day, the first thing that's important before thinking about being a good AI or machine learning product manager is learning how to be a good product manager first. The ML techniques are the add on the things that you develop over time and you can bring data science skills in but at the same time being customer obsessed and understanding the skill set of what makes a good product person. Even from a stated AI product manager or published by the product led alliance, everyone that was surveyed over index on being a good PM first. So what does being a good PM mean? Like what are the core skills that differentiate one PM from another, or maybe not so good from a great. So there's a couple of topics. One is translating those unmet needs into customer or crisp technical requirements. Writing both communication skills in both verbal and written are key to succeeding as a product manager. Because you're always talking with two stakeholders communicating things. Always explaining concepts that are inside your head to others and being able to do that as succinct and Chris play is extremely important. Second is understanding and empathizing with customer pain and customer obsession and these are terms that I'm sure you've heard before, but really what problem are we trying to solve. Why does it matter and what's the pain point that we're trying to address for the customer. And is this actually something that they need to have solved or is this something that it would be a nice to have, but it's not really solving the core problems. Always asking and having an internal dialogue with that. The third is balancing the use of data with product, customer and product, marketing, customer intuition to drive decisions. And where I say product, marketing, customer is so important because really it's at the nexus of all three where you have to influence others to be able to have them follow you in your vision to get what you want. Or what is in your mind done as a product manager. Fourth is running effective meetings and this is one of the things that people don't think about when they think of our product people. But running, making sure that whatever meeting you set up with whoever stakeholder is involved knows why they're there. And there's a very clear agenda provided at the onset of the meeting so everybody comes out with clear outcomes. There's so many meetings and in fact there's a project that I'll take you through that I'm working on today. Is how many there are so many meetings that we have in our day to day that just don't have a purpose for being there. Meeting shouldn't be there for meetings sake. Finally is influencing the authority. This one is a key item that you'll see over and over and over again, well, whether it's in public blog posts or whether it's in historical books about the art of product management. As a product owner, you do not have direct authority over any of the teams that you work with. They will follow you because it makes sense or your plan makes sense but not because you can tell them to do so. And that's a key skill to learn because it in fact, as an AML PM this only ratchets up and it becomes even more important because you're working with even more diverse stakeholders. So people are going to, I don't want to take away from this conversation to be go focus on data science skills and just focus on that alone. The customer centricity portion is extremely important. Again, from that same report that I mentioned earlier, 84% of all survey respondents said yes to customer centricity being the first thing to focus on. The other and the know are saying that yes, customer success and centricity is important, but it's not the only thing in that machine learning and understanding how the ML product lifecycle works is also key to success. But as I can, as you're seeing, there's still a huge margin of people that believe understanding your customer's pain points and building a solution from that versus just using the latest and greatest model isn't going to really cut it. So let's take a look at how understanding how AI fits into the broader world today. And I want to start us off with a quote, and I'll be using some of these really from AI ML product leaders along the way. And Peter Skomorowch, I'm not sure if I'm pronouncing his name right, but he really is an authority in the AI and product ML space. And one of his quotes that I really enjoyed was companies understand how to apply machine learning will be best positioned to scale and win their respective markets over the next decade. But the key word I want to highlight here is that understand how to apply machine learning and that too to solve customer problems in a new and novel way that haven't been done before. So where does AI sit in? Why even talk about AI right now? Why are we here? So this really goes back to the fourth industrial revolution model. And because we're past the steam electricity and now we're actually even past the computing in revolution where all the apps that needed to really get built, the core apps that we use today have been built. There's a ton of data flowing from them. And that data is now useful. It's extremely useful. It's what people refer to as the new oil. And I take a little bit of, I have a little bit of a qualms of that statement, but I can go into that a little bit later. But really, the next revolution is how do you make intelligent products and features for customers? The things that we expect from things like Google's autocomplete or the auto sentence generation that we see in Gmail. These are becoming de facto. It's not enough to just ship the next big iPhone because even that's just iterative. Really, it's about how do you make an iPhone or maybe how do you make the next hardware device? Or how do you make the next piece of software that you're going to ship more intelligent than the previous version? And by intelligence, I mean, how do you predict what the user is going to need and adjust the software to those requirements on the fly? That's really where the magic really happens. And so AI is cool. And it is a little buzz wordy right now, but I want to break that down here. And artificial intelligence really breaks down into, there's a lot of different areas, but the two areas that will be the focus of this discussion are machine learning and deep learning. So within the artificial intelligence broad spectrum, it enables machines to mimic human behavior. And that definition really is encompassing two areas. One is artificial narrow intelligence, which we are in today, and artificial general intelligence, which is what you see in movies like iRobot or what is described as a machine taking over and making autonomous decisions. And we're at this point in time, at this very time, we're not there yet, but we're getting there. But that's a topic for another day. Machine learning is the subset of an AI of an artificial narrow intelligence AI technique, which uses statistical methods to enable machines to improve with experience. And that is really the focus of a lot of the machine learning use case or the AI use cases that are being discussed in companies today. There's a lot of structured data and some of these terms may be new to you. So I'm happy to go into them and chat if you'd like. But structured data are all the things that you see like an Excel spreadsheet or a table or in databases, and it's organized with columns and rows. Unstructured data are things like audio, video, and text, and things of that nature where you don't have a specific row or column associated with that kind of data point. And finally, within machine learning, there's a subset called deep learning. And that's the thing that kind of astounds most people because that's where you're building neural networks. And neural networks, to most, are a representation of how the human brain works. And yes, there is some element of truth to that. But truly, it's really a series of layers and activators. And happy to go into that again in the chat, prefer some sense of links out for people to learn more. But I don't want to go too deeply into the theory of deep learning here. But just know that it's used to take unstructured data. Let's say even as a classifier, say you might have noticed that in Silicon Valley, there was a classifier that one of the characters built to say hot dog or no hot dog. And that's really how that's the most simplest use of deep learning is to classify images and text in real life. So what going straight into what is the ML product development lifecycle look like? So scoping is where the team comes together to understand the problems that they face with respect to the AI products vision and try to forecast the effort that will be involved. Experimentation is around like any other product development lifecycle, the team will need to build a prototype or an MVP. But instead of building a bare bones functional product, the team will need to select the right approach to utilize the data that they have. Or choose a mathematical model that seems to show the best outcomes. Around training, the bulk of the work in this phase is with shaping the data required to train a pre-trained model. This might be a surprise to many of you, but an AI ML team's job is not to reinvent the wheel with a brand new model. We're often taking pre-trained models, shaping the data that fits into them and then trying to get the best outcome possible. Finally, serving. So now is the moment of truth. The model will be tested in production with real data flowing into it. Many things could happen, but primarily they fall into two buckets. One is either the model provides the result it's supposed to, the confidence level similar to what the team saw during the training phase, or it doesn't fit because of the differences between the real world data that the model is getting and the training data that's used. If not, the team will need to go back to the drawing board and I'll go into some of these terms now. So what are the basics of building an ML ready data set? So data set is think of it as really 90 to 95% of the work required for any AI machine learning project. And this is true of any data science project as well. Classical machine learning has two types of data sets. One is has pre-categorized or numerical target values. And I'll go into this later and or the data is not labeled in any way. And that further breaks down into supervised or unsupervised learning. Supervised is when you have a label available for every row or record in your data set. So you can either predict the category to say, let's say I want, I have qualities about a sock and I want to divide them by color. Or I want to predict the number to say, hey, given all of these real estate indicators, I wanted to predict the pricing of this specific house or row or record of that house. Unsupervised is broken down into clustering, dimensional reduction or association. And I don't want to go again too deeply into some of these things, but this is a high level primer of what things that you should look for as you're beginning to learn more about the subject area. So there's a couple of terms that you might see along the way that I want to simplify for you. ML and statistics are really one and the same. The concepts and the fundamentals come from statistics for machine learning. And in statistics, some of the things that you might have heard are dependent response or output variable, and that's the label or target. That's the thing that you are trying to predict. Independent, explanatory or input variables are input data used to make the prediction. So those are, if you take a real estate housing example, all the factors that influence pricing would be the input data. So that is what machine learning engineers or people in the ML space refer to as a feature. Data transformation is cleaning and reshaping and preparing the data to make sure that it's usable. Data is dirty and that's why I took issue with data as in the oil because there's so much effort that goes into cleaning, understanding and basically cutting the data. And a lot of the data that we have, a vast majority of it isn't actually useful on day one for any machine learning tasks. So that is what we refer to as feature engineering. And finally, there's variable or subset selection. Not every variable matters equally. There's correlation of variable to targets, so of a feature to targets, and that's why using these terms. So you want to pick the data that's most useful given the computational constraints that you have or even the timing constraints that you have. So that's something that we refer to as feature selection. So I want to make sure that everybody understands what a good ML ready data set looks like and what in this case is supervised data set. So think of it as your typical CSV or Excel file. You have rows and observations from top to down. You have a target variable, so a target outcome and that's what we refer to as supervised learning. Where in this case, we are trying to predict that what grade the student will get based on all of these different features that are available to us like instructor, lab available, teaching assistant, overall GPA, gender and prerequisite grade. Now, the model will determine what features matter most in predicting that grade and that's ultimately what we want to do with this data set here. Now, you can't really go forward in AI ML without understanding the CRISP-DM framework and this is really the bread and butter of modern data analysts, data scientists and AI ML product managers. AI projects require a feedback loop in both the product development process and the AI products themselves because AI products are inherently research based. Experimentation and iterative development are necessary. Unlike traditional software development in which inputs and results are often deterministic, the AI development lifecycle is probabilistic. We don't actually know what will happen. This requires several important modifications to show how projects are set up and executed regardless of the project management framework. The data understanding takes a huge portion of any AI ML product lifecycle. But completing it and completing it well is one of the strongest indicators of future success. A product manager often needs to balance the investment of resources against the risks of moving forward without fully understanding the data that's available to them. Acquiring data is also extremely difficult, especially in regulated industries. Once we get the data and once it's obtained, we as AI product managers aren't the ones maybe doing the research ourselves. But our goal is to be able to guide data scientists, analysts and domain experts towards evaluating the data in a product-centric way and to inform meaningful experiment design. The goal is to make sure that you have a measurable signal for what data exists, the solid insights into that data's relevance and a clear vision of where to concentrate your efforts in designing features. Data wrangling in feature engineering is the next thing, is one of the most difficult and important phases of every AI project. It's generally accepted that during a typical product development lifecycle, 80% of a data scientist's time is spent in feature engineering. Trends in tools in auto ML and deep learning have really helped in reducing the time, skills and effort involved, but not the actual product itself. Building a superior feature pipeline or model architecture will always be worthwhile. AI product managers should make sure that project plans account for time, effort and people need it. Finally, the modeling phase is where people get most excited because that's where you put the data in and you get a result and it's magic. But it's actually very frustrating and sometimes very difficult to predict what you will actually get from the data that you're feeding it. It's iterative. You're often as an AI ML PM and often the team that you're working with will go in and out between the data understanding and modeling and that will have to churn a bit between the two to make sure that the outcome that we want, the prediction that we want, makes sense for the users that we want to ship it to. Finally, unlike traditional software engineering products, AI product managers have to be involved in the build process. Eng managers are usually responsible for making sure that all the components of a software product are properly compiled to binaries and organizing build scripts, but many DevOps processes make that part easier, but they were developed for traditional software products. These tools do not exist today for ML AI product managers and product teams. The skills needed and the tools needed need time to mature and they're usually high-touched and they're very custom to each project. Finally, monitoring is a huge part because the prediction that you train the model with may not be the prediction that you get with real-world data and there's often a huge disparity between what you think the world is and what really it is. It's a reality versus training data. Having alerts set up in the system to inform you when the model's results are drifting past a margin of error are extremely important for allowing site reliability teams or engineers to be able to take action on that model and make changes. That's where you as an AIML PM are really the subject matter and domain expert for your model that you've shipped and deployed. Just quickly recapping ML product development breaks down very much as a typical PM job with a couple of differences really in the design piece. So understanding what data you have, what data is available, what you need to get and then figuring out what the model's outputs, design and proposed UI for those predicted results will be. And ultimately it's not just you ship the model and that's done. You have to continuously build a pipeline to monitor the impact of what the model is doing in the real world and how you're going to adjust for changes on the fly. So what are the skills required then? How does this all translate into you as a professional learning, breaking into the world of AIML PMing? If you're already in it, this is an amazing slide. Like this is a great deck for you. But if you're not in it, what do you need to do? What are the gaps? So really it's understanding how ML integrates into products as a whole. And I think Clemens really summed this up well where he says, but while ML grows more important, few PMs know how to integrate it into their own products. So skills that matter most. So once you have the base level or fundamental understanding of how PMing happens, the next thing is really having a working knowledge of how machine learning development and deployment works and having hands-on experience with data and modeling technology. So having built a model from scratch from start to finish, having a predicted result, understanding what the steps are in the intermediate is going to give you an intuition for the things that your data scientists teams are struggling with. As a software engineer or as an AIML product manager, your job is not to build the models themselves, but be an authority or an expert on how they work at least. It's like asking a software engineer or asking a regular product manager to understand what their product is and does. Our product is the data that goes into it and the models that come out of it. So there are a couple of ML systems that I want to go through very briefly at a very high level. But often on an effort to time scale, improving existing features takes the least effort in time, even in an ML way. And of course, creating a new product is on the upper end then to the right. So there's a couple of different systems that are often you'll see that are integrated into the most popular products that you use today. One is the recommender ranking systems that provide a list, a rank list of documents from a corpus, something like these documents can be anything from apps in the app store to movies on Netflix. It can even be your favorite item prediction on Amazon or any e-commerce site of your choice. Event or action prediction models predict the likelihood of an event or user action. For example, at Google, machine learning is used to predict a click. So then showing you videos that you're actually likely to click. Classification models are classify arbitrary objects into known classes like categorizing emails as spam or not spam. Generative models generate output in a form similar to the input that they were trained with, like translation models that turn text in one language to text into another language. Finally, clustering is a form of unsupervised learning that you saw earlier in which similar objects are grouped together like segmenting users into different groups. So on the thing that people sometimes miss and don't realize until they're actually on the job is how important the integration path is for your model into the product itself. Where is the prediction going to be displayed? What is the actual outcome that you want for users? And how are they going to make it actionable? So in the improving an existing feature, really, if you're trying to build a better recommendation system, ultimately your goal is to make sure that the items that are recommended in your shopping cart, for example, in Amazon are more relevant to users. Enabling new features. Google implemented a reverse image search product or feature that allows you to, instead of searching with text, actually copy and paste an image into the Google search bar and then it'll show you results from that. Now that's something brand new because that means interpreting an image, turning it to text, querying it, and then displaying the results to users. And finally, an example of enabling a new product is providing insights, for example, in Microsoft Teams through Viva Insights to enterprise users about what trends or patterns they're seeing in their organization, and then making recommendations on things that they can change to improve the culture. And that can be everything from productivity to how meetings are done, to employee burnout. And so now you're enabling a new product line that never existed. So a small case study I want to take you through, and this is something a personal project that I've been working on at Microsoft during my time, is trying to understand how we can make meetings more effective and inclusive. Microsoft Teams, we use the telemetry that we collect from each call to predict whether a meeting was effective or ineffective based on what's happening in real time in the call. We're not listening to the conversations, but we're using all the surrounding data that we get to understand what changes can an organizer or an organization make to improve meeting culture in their respective domain. So we first identified product improvements that we can make. We then understand what signals we should use to train the model. These signals are the circles that you're seeing, and these are the signals that we're actually getting from each team's meeting. Then we're understanding how to define the experience itself and what will be surfaced. So for example, using Teams or an admin page to surface the insights that we're going to build on this. Next is helping to analyze the outcomes of different models. And then the fifth step is planning and managing a launch strategy. So asking questions like of myself and of the team should be launched internally or should we launch to a subset of users? What is our test group going to be? Is our data set generalizable? Are the insights that we have, are we confident that these insights are going to have the intended impact of moving meeting effectiveness from the left to the right and actually making an impact there? And how are we going to be measured? How are we going to measure that? And this is where the whole metrics aspect comes right back in. The core key and fundamental skills are still as valid in this project as there are in any other. And finally, you have to sanity check the recommendations. We have to ensure that the model actually yields good results on the test data that we're getting it, giving it. This feature should only trigger when the models perform and the confidence is quite high. We don't want to start showing insights or showing behaviors or showing actions that organizers can take without really having clarity or confidence that the insights will work as we expect them to. So how do you get ready for an AI or machine learning PM interview? And this is the question that most people have. It breaks down into three areas. One is product sense, which I'm pretty sure you're aware of. The second is statistics. And this is where it kind of differs because no traditional PM really focuses so heavily on statistics. But having an intuition helps a ton, especially with understanding P values, confidence intervals, hypothesis testing, and sampling techniques. And finally, setting success metrics. This becomes even more paramount. You have to understand what the model's output is in production and be able to say and conclude if leadership teams come to you and say, hey, is this model working as we intended? You have to think broadly. And MISI is a great framework to do that. A mutually exclusive, collectively exhausted. I'll post a link to that framework in the chat later today. And of course, mapping user actions to metrics and providing context on why the metrics matter. All of these are really the building blocks for any strong performance, strong performing product person interview. So I've gone a lot into what the background is and how, yeah, it's in and what the life is, but what are the actual role and responsibilities as an AI and MLPM? So I think to remember, always in the back of your mind, is that AI or machine learning is a hammer. It is not the hammer that should be used for every nail. It's only one of many tools in your toolkit. And it's not good or bad. It's not a magic solution. If and only if so, if it seems like the best tool to solve the problem, then it should be used. So the things that matter most on the job are really bridging the language of data science and product development and having the ability to solve and prioritize user problems. It's funny because the ability to deliver AI powered specifications to data science teams are important and they're one of the top three, but they're not the most important. A lot of, again, goes back to being a very strong written and verbal communicator and even more so in such a technical domain. Key responsibilities usually include deciding on what the problem you're trying to solve is and for whom it matters. Always understanding, evaluating the data pipelines and ensuring that they're maintained throughout the AI product lifecycle, orchestrating cross-functional teams from teams like data engineering, research science, data science, machine learning engineering, and software engineering. So it's a pretty exhaustive list. Next is deciding on the key interfaces and designs and user interfaces and experiences of feature engineering. Next is actually integrating the model and the server architecture of the existing software products. Another one is working with ML engineers and data scientists on tech, stack design, and decision-making. The seventh is shipping the AI product and managing after the release. And finally is coordinating with engineering, infrastructure, and site reliability teams to ensure that all shipped features can be supported at scale. This is extremely key because we need to ensure alignment at every step of the process. So common challenges that you're likely to face will almost always be poor data quality and lack of data availability, which is ironic because apparently we have so much data anyway, but why is that so problematic? Because a lot of the data that's been collected through legacy systems isn't usable for AI and machine learning systems. Garbage in, garbage out. You're going to hear this over and over again. But as a data as a data and AI and machine learning PM, you are now empowered to make sure that whatever features you build, the data that's being collected from them is cleaned at the time of collection or as clean as possible. You don't need to collect data that is just going to sit there and not actually provide any value for additional features that you want to build in the future. So what's the future of AI? And I'm going to touch on this very briefly. And I think Pete does a great job of telling that story is at the end of the day, we want to ensure that AI systems are simple. We don't want to increase complexity. We want to increase simplicity. And it takes something special to be able to say, I'm going to actually take something that seems very nuanced like AI and machine learning and dumb it down and build a feature that people love to use because it's so smart. So AI product managers, you might have seen the UX tech business then diagram that was floating around for many years, but really it's changing over to data and AI fluency, buy-in and alignment, and use case familiarity. There's a couple of themes that are going to come up, like bots using for communicating product launches or recording standard meetings or prompting PMs to assess the performance of changes weeks after launch. And a lot of this will be accomplished by the things that we're seeing and the models that we're deploying to production will have bots and services enabled to ensure that we're always getting alerts back on how well our bottle is doing over time and if it's being adjusted for data drops. Transparency is another key theme that's going to come up. Customers aren't happy with being told that there's some sort of black box model that's going to take this data and output this result. They want to know how it's being done and really the result that it's putting out should be trusted. And finally, consumer psychology. Product managers need to understand that their end users care differently about they have different changing habits and AI and machine learning can help capture those trends in a more generic way. So the coming time there's a couple of themes that I also want to discuss here. So today we're focusing on automation, communication and collaboration in the different areas of decision intelligence, everyday AI, AI engineering, autonomous systems and responsible AI. But what customers again care most about is are the results being shown to me relevant to me in context? Can I trust the results that are being shown? And are there experiences between who's building product, the teams that are working on making sure that it's running the way it should be and the actual output that any customers or any end user is seeing? End users are no longer happy with knowing that hey, my data is going somewhere, it's being stored in some server halfway across the world and it's now being used by this giant tech conglomerate to make decisions or to sell me things that I really didn't really need and I didn't really want or push ideas that can be disruptive. So transparency is going to be a key theme in every AI or ML feature or product that is going to be built in the coming time. Thank you so much for your time today. Please feel free to use the chat to continue the conversation. I'll be there and happy to answer any questions. Thank you so much.