 Hi everybody. Welcome to the product school webinar of how product managers use AI to 10x their productivity. I'm Sam Stevens. I'll tell you a little bit more about myself and then super excited to get started today. So today we're going to be going over three ways that product managers can use AI to help them in their day to day jobs between planning, creating a product strategy, executing and optimizing your day in a very tactical way, and then using AI as a measurement tool to help you analyze feedback and data. So before we get started a little bit about me, I am Sam Stevens. I'm the AI executive in residence of product school, building the AI product management certification. I have spent the past decade in product. I started my first real PM role at American Express, managing the Windows 10 app in partnership with Microsoft, if anyone remembers Windows 10 back in the day. I then went over to Tinder where I was the director of product leading multiple teams across my tenure there, including growth, revenue, new product development. And after four years of online dating, decided that it was time to shift gears and went over to Google where I spent a year on the Google assistant team working with early large language models, and then about a year at YouTube monetization leading a product called Superthanks, which we'll talk a little bit more about later. And then about a year ago, I left Google and co-founded a company called Catalyst, which is an AI project management assistant that automatically keeps projects on track and teams on the same page. And we are in closed beta right now. But as a special thank you for all of you here today, you can join the wait list and write webinar in the referral field and I'll reach out with a special invite to the program at CatalystAI.com. So I was first introduced to generative AI, leading a team at Google assistant in 2020. And I was blown away by the potential of this technology. I watched it mature over the past few years and firmly believe that we are in the middle of a foundational technological shift. And the capabilities of today's models are truly just going to fundamentally change how humans interact with technology. And the reason that I am here today is because I firmly believe that if you don't understand people, product managers don't understand how to leverage this technology and these models into products, then we're going to be falling behind the curve. So when product school reached out to ask if I would create the AI product management certification, I was so excited to say yes. I strongly encourage you to sign up and take it AI PC AI product certification as a course for product managers to learn how to build and manage products that are powered by AI models, specifically large language models. It's a great course, highly recommend. We'll have the link to sign up later. But today, we are going to be talking about the personal applications and personal productivity of AI in your day to day life. So in this webinar, we're going to go through my real world tips that I have used as a founder and CEO as a Google product manager and go through a lot of just hands on examples to really get into the details and the nitty gritty of how product managers can use AI to power their day and supercharge their productivity. And there's the link in the slides if anyone wants to sign up and learn more about the product certification. Okay, so let's jump in and talk a little bit about the methodology that I use. There are lots of specialized high quality optimized products that depending on your specific use case you might decide to use for working with working with models accessing models. I personally use GPT for via the chat GPT premium subscription. Sometimes I use it with chat in chat GPT sometimes I use it directly in the GPT for playground. I also highly recommend experimenting and playing around playing around with other types of models and products. Claude is really great for plexi super interesting. And you'll find that some tasks, some models are better suited for some tasks they just sort of get it better. So highly recommend taking an explorer's mind and playing around and trying a bunch of different things until you find what works for you. One tip I will recommend is that if you are using an AI model for work, be sure to check your company's policies, some of them have specific models you can and can't use. So just make sure that you're compliant with how you are using your workplace data. So with that, let's jump into the first phase, which is how to use AI to be a thought partner in product strategy. So the way that I like to use AI is sort of in three steps and I'm thinking about either, you know, kind of a big macro decision like planning my product strategy or, or making a big trade off. Usually I'm, I'm in the strategic mindset where I'm trying to think very big picture think through all of the different pros and cons, as we often do as pms to try to ultimately reach a conclusion. And so what I found is that going through sort of this three step funnel with using using an LLM as a thought partner is super helpful. And so my three steps that we'll walk through are first starting off with casting a wide net, then weighing pros and cons and finally identifying blind spots. So what does that mean? So we all know that AI is great at generating new ideas. And sometimes it needs specific guidance to generate the ideas that are actually high quality and useful. And so let's walk through how to generate the ideas and cast that wide net of options in a way that works for us. So these are the six steps that I like to go through. The first is brainstorm ideas. So use an LLM such as chat GPT to generate a range of ideas based off of whatever you're working on. So maybe that's a, like I mentioned a decision that you're trying to make or what should I do for my product strategy this year and have it come up and really generate a bunch of different ideas. You might also want to give it some specific examples, maybe related to your product strategy to help steer it in the right direction. So for example, maybe you clarify and let it know which options you've already considered, or even that you've already ruled out to start getting it on the right track and thinking towards the direction that you're looking for. And I like to think of this as a help me help you where as always the more information and context that you can provide it to steer it, the better, the better output you're going to get. Next we use our human judgment as we as these ideas start start flowing from the AI. Some of them might totally not resonate with you, but some of them might start to spark some some inspiration. And so this is where you use your expertise and understanding of your product and your market to identify which ideas are feasible and align with let's say your product strategy that you're working on and trying to brainstorm ideas around. So, a big piece that people sometimes don't recognize or remember to do is working with an LLM is really an iterative process. So it's probably not going to get it right the first time you might have to have like an elongated or prolonged conversation with it really treat your, your chat GPT interactions as an iterative process and refine your prompts based off of previous responses to get more targeted ideas. So sometimes I like to imagine it like a lively debate with a peer, and we're still in the creative process at this point in the phase so if there's a direction that it starts going down that really resonates with you, encourage it to keep exploring and generating ideas along that path. So you're kind of, you know, spearing it and similarly in step number five, just continuously guiding and redirecting by being clear and giving feedback about what constitutes a good or a bad output in the context of your product strategy. So that might include like focusing more on this target audience or focusing more on these business objectives. And the last piece of advice that I have is that you're going to get a lot of really bad ideas and that's okay. But what you're looking for is a spark, right, you're there's there's probably a needle in a haystack somewhere. And even if it's just one idea or a couple of, you know, sparks of inspiration that stand out for you. You can take those and run with them. So next, we now that we have a bunch of different ideas that we've generated, we can use AI to help us examine a decision from multiple angles, and really help weigh the pros and cons. So maybe we're trying to decide the pros and cons of which customer persona to prioritize or which North Star metrics to choose. So let's go through step by step how to work with the AI to weigh our pros and cons. Okay, so four steps here. We're going to start by feeding all of the pertinent data and the insights that you have about your decision at hand into the AI tool. And when I say all I mean feed all be as comprehensive as you can provide as much details as you have to help make the AI has enough context to generate meaningful options. So let's say we're, we're, let's say we're trying to weigh pros and cons between which customer persona to prioritize in our product strategy for the year. We might give our AI everything we know about these various personas like the size of the customer base how easy it is to reach them etc like just give all the information that you can to provide that thorough context. The next is we also want to kind of cast a bit of a wide net. So we can ask the the AI to list all of the possible courses of actions that it identifies based off of the information given. So maybe there's a potential choice that it uncovers that you might have overlooked or maybe you can compare what what it generates to ones that you were already considering for example maybe it identifies a third or fourth persona that you actually have that ends up being a really smart option. So we'll get into identifying blind spots later but this is a little bit of just making sure just using this AI system as like a mental safety net and catch all to always make sure that it's helping you have this this comprehensive worldview and that nothing's falling through the cracks. So at this point I want some all of my options are laid out and I made sure that the AI has generated more options maybe some of them work for me maybe some of them don't but now we're going to start comparing and contrasting all of the options so I like it to I like to ask it to create a table with really structured sections and asking it to weigh the pros and cons of each options to help me understand the implications and the potential outcomes of each choice that I might potentially make. And then last of course we are going to decide ourselves will review our list and combine you know this these thoughts that are AI generated with what of course we already know and our own knowledge and insights and judgment to make our best informed decision that ultimately suits our strategy and our goals. One thing I really like about this this approach is when I'm making a decision whether that's in like a product strategy or maybe in a decision memo I always include a section of what alternatives I considered and why I decided not to pursue those choices to lay out my reasoning. And so having gone through this process all of that is documented and makes it really easy for me to share my soft process with the rest of my team. So everybody is on that same page. And now the last part of the planning part of the of using AI is identifying blind spots. So it's a I always find it to be a super useful tool in finding potential gotcha moments that I might not have seen. And so we'll just do a walkthrough of that process and how I use it as well. So first similarly we want to detail the current situation so similarly to how we did it for weighing pros and cons. Give a comprehensive description of everything that's going on. Sometimes I'll have like if I have a draft product strategy a draft positioning document a draft decision memo I'll just upload that. So it has all of the relevant context and details and it knows what I'm optimizing for. Then I'll ask for a critique. I will tell it to basically tell me all the things that are wrong with this potential strategy or direction. I have these three questions that I generally ask so what are potential unintended consequences of the strategy right like what am I not thinking about that could go wrong. Where are my gotchas what are the biggest risks associated with the with this plan and then any suggestions for improving this strategy. And again sometimes like the suggestions will be really obvious or really not that value adding but sometimes you might find one or two where you're like oh yeah that's actually really helpful. So then next we evaluate we really apply our critical lens. Speaking of those like maybe potentially valuable insights that it draws this is where I err on the side of skepticism but try to remain open minded. So my evaluation bar is pretty high for anything that is generated but I'm certainly open to to new ideas and have truly frequently been pleasantly surprised by having an AI model just take a take a look with with fresh eyes and maybe see things from a different perspective and open up my the way that I'm looking at things to a different point of view. And then of course we you know we can incorporate anything that makes sense into our product strategy so I always find this sort of blind spot exercise super helpful in bolstering up a any sort of risk and mitigation section. So this is how like you can when the AI system kind of cast a wide net and maybe thinks of risks or unintended consequences that you hadn't thought of but like are valid. You can proactive me include those in your product strategy or your decision memo. And then you figure out how would you handle all these issues so you already have your mitigation plans in place. And finally, now that you you have all of this, you know, assembled together. You are even more equipped to prepare for critiques and counter arguments that you might receive, whether you're presenting your product strategy or advocating for a decision. And you'll have a much more clear thinking to back up your rationale which is a hugely important skills for product managers to have. Amazing. Okay, so let's go through to the next section, which is how to use AI to optimize your day. And these are two fun tips that I love to do some of my little favorite personal productivity. So I find that often as product managers, we are trying to balance long term initiatives or professional growth plans with projects like our fire drills. And AI is really a great tool to be able to bring the focus back on big picture goals and actually break those down so you can incorporate them into your day. So I live and die by my Google calendar. I use it rigorously to plan my day to time box activities, make sure I know exactly what I can and can't get done. So I'm able to communicate those expectations to my team, but not everybody is quite as your brain works the same way. And so luckily, we have this wonderful tool that is pretty good at taking a bunch of information and creating a plan. And so I have created a prompt template that everyone is welcome to use to break down, to first break down big goals into manageable tasks. And then next we'll talk about like the day to day planning. So let's say, for example, it is performance review season, and you have a very well intentioned manager that gives you some big career goal like work on your stakeholder management capabilities. And sometimes it's hard to know where to start, right, like you could spend hours researching and creating practical steps and meticulously planning. Or you could go over to a large language model and write a great prompt and work with an AI system to help break that down. So in this scenario where I'm trying to break down big goal into tasks, I first start with the preamble of setting the AI up for success. So priming it to tell it what it's good at, what to focus on, etc. So it knows sort of steering the ship in the right direction. Then I explain the task at hand and see if anything needs clarifying or if there's any additional information it needs before proceeding. I do this because I find that sometimes an app like chat GPT will jump into solution mode too quickly before it really has all the context and sometimes it makes it do a poor job. So I want to make sure it takes more time to think up front and have the LLM first come up with the clarifying questions and whatever supporting information it needs in order to effectively achieve the goal in that first part of the prompt. So that's why I first have it ask me if there's any additional questions that it has. And then I put that in we see what happens, you'll work back and forth with the with the AI, you'll put in your clarifying information. And then that will help you break your big goal down into tactical actionable tasks that now we can move to step two, which is about planning and optimizing your day. So, similarly, another fun prompt template that anyone is welcome to use. So I, people always ask me how I stay organized, I'm like weirdly organized. I somehow always remember to like follow up on everything at the right time and it's because everything is on my Google calendar, as I mentioned, every single task is planned and time boxed. And that doesn't, as I mentioned, that doesn't necessarily come naturally to every single person. So I created a prompt because a it's helpful to everybody and be like it does take a lot of time to think through your day and time box everything and plan everything. So one thing to remember with this sort of prompt, which basically entails like priming it with all the relevant information that's personalized to you, giving it a task of all of your, or a list of all of your tasks for the day, and then having it help you create that schedule. You have to remember that a system like chat GPT is garbage in garbage out right like it's not a mind reader. If it doesn't, if you haven't been explicit and what you've told it, it's just going to make a bunch of assumptions that may or may not be helpful to you. So I always suggest take, take the extra five minutes and write out more granular tedious information to provide it more context so that it makes much higher quality recommended actions for you. So as you can see here's my prompt, I give it a little preamble on the left. And then I fill in all of this information on the right about like my energy levels throughout the day, how long it takes me to send emails what goals I'm optimizing for right like what tasks we just generated from my big picture goals that we want to break down stuff I have to get done today all of this stuff. So I give it all that information, some amount of time estimates and let it create your optimized day. And then finally, the last way that I like to use AI as a product manager is to analyze user feedback. And oops, there are lots of like specialized apps for analyzing user feedback. If your team happens to use one of them, that's fantastic. But you can also use any LLM that allows you to add upload information and generate or analyze the insights off of it. So, right, there's a lot of like quantitative and qualitative products that serve to analyze data, there's like chat with your database, there's English to SQL products, their sentiment analysis products, all those are great. But if you don't want to necessarily have to pay for another like SAS tool for your team, you can just get away with using chat GPT. And so I will walk through how we can do this. And I'll walk you through how I so dearly wish I had this tool when I was a product lead at YouTube. So, as an example, I launched this product called Super Thanks when I was at YouTube. Super Thanks is a creator monetization feature that allows viewers to financially support their favorite creators. And when they purchase a Super Thanks, they get to write a special comment, and it shows up highlighted in the comments section. If you've seen like the heart button on the, on a YouTube video that says thanks and you click it, that's how you get into the feature. And so that's the feature that I launched. And so once we launched it, I had to analyze how users were using the feature before rolling it out more widely, right, we ran it as an experiment and we wanted to see like well what are people writing in their Super Thanks comments. So what did I do? Well, we didn't have a data science resource available to us at the time we didn't have a chat with your or analyze your, your database, LLM type of product. So I manually downloaded and reviewed 1000 YouTube comments, including cleaning the data, creating the categories of like what people were saying right like wishing like wishing a happy holiday, requesting a second specific type of video expressing gratitude right like creating all the categories. And then, having every single comment and so I could figure out like the distribution of the different types of categories. And this took me at least 20 maybe even 40 hours of work over the course of a couple of weeks it was so arduous. And it was super valuable to understand like we had to understand how these comments were being used. But I so dearly wish that I had a tool that could have expedited this process and so I actually recreated. I ran my own experiment after leaving Google and recreated this process of analyzing YouTube comments with with GPT for and was just blown away by the result and so we'll walk through how I did it. And you can apply it to your own practice. Okay, so first we just need to prepare our data. We want to start by just inputting all of your data into a spreadsheet. Maybe your data is in like raw YouTube comments, maybe it's App Store reviews, survey responses, Slack messages in your community, Slack, whatever that is. So, clean the data, right make sure that you're deleting any like spam comments filter out either either translate or filter out any comments that are in a different language so if you're doing it in English either translate or filter out any non English comments. There's a detect language formula in Google Sheets. That's super helpful that I that I've used. Delete and again like delete any role rows if they just contain like random characters or they're clearly like spam links or maybe just emojis that don't fit the scope of your analysis that's just going to throw off the data. Then finally just make sure your columns are clearly labeled so that the LLM can understand them and then download the spreadsheet. Easy enough, takes a little bit of time but you want to make sure that your data is high quality. Okay, so next we just, we, we upload our table or CSV into a tool like chat gpt. If you have your advanced subscription and you have your your data analysis your code interpreter function turn on turned on you can pretty easily do that. And then this is the prompt that I use. And so I will won't read this to you you're welcome to screenshot it copy and pasted etc. Adopt it to what you know your your specific use case is. But really the point is just that you are telling it what you want to do you're giving it a very long detailed prompt with clear expectations and running it and start to identify the themes. A couple of tips here that I found one is sometimes these systems will be a little lazy, and instead of reading all of the comments they'll try to just read like a sample of the comments and you don't want that you want to be comprehensive so sometimes you have to convince it to read all of the comments first before starting to like categorize create the categories of what the content says. So sometimes be sort of overly broad and categorize like a bunch of comments into some sort of like other or miscellaneous bucket like sometimes I've tried it and it's put like started with 50% of the comments and like miscellaneous and like iterated and gone back to those principles that we went over in our, our product strategy section where we talked about, you know, iterating and giving feedback and direction and example so bring back all of those best practices. Sometimes I tell it to create try to focus on creating a messy analysis so creating categories that are mutually exclusive and collectively exhausted. So it tries to encompass everything. And then of course using your own human judgment and knowledge to make sure that you're, you're getting a sense of the comments and making sure that these are the correct categories as a, you know, before moving on so make sure you're using your judgment and getting into a good place. And then finally, we say great now that we've identified all of the, the comments we want to sort each comment into the most relevant category, or maybe you want to tag comments with multiple categories or whatever you know it depends on your use case to ensure the accuracy or at least to improve the accuracy. I like to use what's called a few shot example, where you can sell many select select a few rows of your data and included in your prompts and then the category that you would apply it to, or you would apply to it maybe you do like one one or two examples of each, so that it gets a sense of like, what is a good, a good categorization and a successful output. So, you can also then have it so you can run run through this process have it generated spreadsheet, you can then check the, the accuracy of course and you should always review the, the output to have any LLM to check for accuracy these models make mistakes, people make mistakes it happens. In this sort of exercise I'd be okay with like a couple of miscategorized miscategorized comments out of 1000 I would probably make the same, you know, misjudgments as well. But you know, depending on your use case sometimes you need more accuracy as you might go through everything. And then you can have it even generate a graph, you can have it write a memo detailing your findings, you can have it put together, you know, a research readout with a graphs or a pie chart of the distributions to your slide and so for my personal use case what took me at least 20 hours ended up taking me about 30 to 45 minutes to go through this whole process so incredible application of AI as a product manager to supercharge my, my work time. And that is it. So, now that you have all of these tips, hopefully you will be able to spend more time working on your long term strategic initiatives, your bigger picture goals, spend more time deep thinking have more time to spend, you know, immersed in market research and developing relationships and all those things that we really should be doing as product managers but often get kind of pushed to the wayside because there's so many tactical things to execute on. And so that concludes our webinar thank you so much for tuning in. If you have any questions for me, drop them in the comment section of the LinkedIn posts and I'll go ahead and answer them there. I'd like to head to catalyst ai.com to join the waitlist. And of course, if you want to translate all of these cool AI productivity tips into actual product and learn how to build the products that run on these really powerful AI models. Sign up for the AI product management certification, you can scan the QR code and learn more information about all the work and the coursework that we're going to be going over there. So thank you everybody for tuning in and I'll see you on LinkedIn.