 Oh, so nice to be here. I took a lot of notes, Matt, from your talk. So now I'm going to wait for a time machine so I can go back and hand them to myself and be on stage here 10 years earlier. So it's great to be here. Awesome. It is kind of late. You guys are probably a little tired. Hopefully you can sit through one more talk. But it's about a topic that feels like it is just taking over the world so maybe we can all sit back. The reason I'm excited to be here is first, this is about learning. I love learning. Involved in LinkedIn learning and learning is one of my favorite things, like most of us. It's about product management, which I love. I love product management. But probably more importantly, it's in-person, at least partly in-person. It's awesome to have everybody in the room, right? Can we hear you? Yeah. It's just so different. I mean, it's unbelievable what COVID did to our heads. Fourth, it's in New York City, and which is really amazing because whenever I meet people and I tell them I worked at Google for so long and now I'm at LinkedIn, they're like, oh, you're visiting New York. I'm like, no, New York is a technical hub of products. So I'm really excited to be here. So let's talk about Gen.A.I. But first, I always had to introduce myself and it's kind of weird to introduce myself and talk about the products and things. So I thought we'd start first doing how many people have seen magic tricks from Gen.A.I., like crazy stuff. You just see it online, people trying everything. I figured let me try that here and ask some of the AI models, who am I? I did not do a good job trying to make my prompt engineering perfect here. I literally just said, who is Jonathan Rochelle? Just to see what it does. Now Bing is actually a product of my new parent company. So I'm at LinkedIn, it's a subsidiary of Microsoft. So I think of Bing as the product of my new parent company for the last only less than two years. And Jonathan Rochelle is an entrepreneur, technology product manager, yes. Currently VP of learning content, yes. Leads of development of learning products, yes. Co-founder of the Google Docs products, yes. And then they said, also I apparently have two jobs. I'm the director of product management for Google Apps, no. That was past. So we're getting our first taste of how to use those models. But let's see what Google says. Google barred. I don't know if this is a conspiracy theory because I went over to a Microsoft company 14 years at Google and they've forgotten about me. For those of you that don't see the screen, it says, I do not have enough information about that person. Apparently it was all hidden. I used to like to get comfortable on stage by explaining that I don't see myself the way you're seeing me right now on stage. I see myself as the person on the screen, which was the intern interviewee in front of my 1977 Celica GT, the most awesome car. And I didn't get the interview. I mean, I didn't get the job. I got the interview. But I thought, well, it would be fun actually to, you know, let's spice this up since we can. I thought, you know, why not? So this is the new way I'm going to introduce myself from now on for sure because I look so much better there. So let's talk about the biggest trend in tech. That's why we're here, right? NFTs, Web 3, and blockchain. No. It's Gen AI. Let's talk about Gen AI. What happened, by the way? That stuff just disappeared, it feels like. Gen AI is so hot. I mean, it's hard to actually get away from it. Now, it's AI as well, obviously. But Gen AI is what burst it into the scene where, you know, my family is talking about it. People, you know, at every age and every discipline. On LinkedIn, just some data, there's 33 times more posts with Gen AI keywords than a year ago. 33 times, 79% more job posts that mentioned GPT. A year ago, many people didn't know what GPT was. Many people did. It was there. But it's just a huge trend. And in the Microsoft Work Report, 86% of respondents say they are looking to AI to assist them with finding answers. AI is the new way to search. As product managers, as product leaders, as product people, everyone in this room really needs to keep up and watch what's going on in the industry. And it's really hard. But one of the ways I've been doing it is I watch YouTube videos. And I really do find it's a really fast way to get broad overviews almost every day. This guy tried 200 AI tools. You know how long it would take me to try 200 AI tools? It would probably take me 200 months because I never find time to do that. But I could literally be on my Peloton or something watching this video. And in 28 minutes, or however long that is, get 200. But then this robot tried 250 plus AI tools. And this guy had to come back and try more. So he did more. And these other people are just loading up the internet with reviews of these tools. This is actually an effective way to at least understand what's available and what's out there. I highly recommend doing it. What's the new trend now? What's better than chat GPT? You know, everybody's basically saying you can learn anything, you can create anything, you can earn anything. It's just everywhere. My head is literally exploding. It's just insane. But it's really important to stay up to date. Again, as product leaders, you are the agents of change. You are the ones that can make a difference not just in your products, but in the world. Because as you introduce new products with new capabilities, those are things that people could use, like we've heard before, by millions, by hundreds of millions of people. So let's just return to the basics first. Let's understand it's a little bit of terminology, but I find it's helpful, particularly for people that haven't done much with it. I hear everybody referring to chat GPT. That's the way they describe generative AI. They say, oh, chat GPT, chat GPT. Chat GPT is a product, right? It's the open AI's product. It's an incredible one, and as of last November, it is probably singularly responsible for the incredible boost in popularity. But it is just one example, one product in the world of generative AI, in the discipline of generative AI, which is within the big world of artificial intelligence. And when you hear people talk, you know, they see one of these magic tricks on the web or the 250-plus tools that are available and, you know, the people that have been involved with AI for a long time say, AI is not new, everybody. And of course, I just say, so what? It's not new, but everybody now knows about it. That's what they're talking about, artificial intelligence as a whole. And when you hear large language models, LLMs, that's, again, a technical discipline that's used in artificial intelligence in generative AI and even directly in chat GPT. So let's just talk about the elephant in the room. And yes, this is a generated picture, if you couldn't tell. Are we replaceable? Everybody's talking about how AI and generative AI now can suddenly touch creative roles and creative jobs. Are PMs replaceable? So I would say maybe, let's think about it. What do you do every day, right? Can generative AI identify a problem and a solution for that problem? Can it identify the right problem? Can it propose and pitch a solution? Maybe. I've seen people write pitches using chat GPT or one of the other tools. Can it build and motivate a team? I'm sure a lot of you build and motivate teams, right? You say, oh, that engineer we need, the designer, the product marketing manager, a BizOps person, you build the team and you motivate them. Can they be annoying? We're all annoying if we're good product managers. It's the only way to stay on track. And you do that in a way that keeps the team engaged and motivated. And can we be optimistic in the face of doom? Can chat GPT or one of the other generative AI tools actually move back from the facts of doom and say we're still gonna get this done? And I guess chat GPT could probably work on a Saturday, so. And can it prioritize? Can it narrow the scope of a problem in the right way in the way that is actually gonna be best for the business? Maybe with enough data. And can it make confident decisions without data? These are some of the things that I can just imagine. Can it hallucinate and convince everyone that they're right? That's what we're doing, right? We always do that. I know I've done it in the past. Although somehow those hallucinations become things like Google Sheets. There's two ways that I think of using AI. When I think about product management and AI, two general ways, really, at a very zoomed out level. The first is to use it in the process of creating products, of managing products in product management. So being a user of gen AI, basically. So generate ideas, research, and do it with care. You know that there's some constraints we'll talk about. You can use it to help you write a product that's generate slides, although I talked to a few people that tried that for today and they're like, it was a total failure. Develop demo scripts, edit videos. If you haven't tried some of the new editing tools for videos, I would highly recommend trying them. Analyzing data, finding that needle in a haystack that's actually really hard and sometimes takes teams of people. Generate prototype code. If you always wanted to show your idea as a product manager and you were looking for that one engineer to participate in the hackathon with you and you couldn't find them, maybe you can get a gen AI tool to participate in the hackathon with you by generating prototype code and marketing ideas. The key is, in that role, you're a user of the large language models. You're using them and to use them effectively, you really should understand prompt engineering. How do you prompt the machine? Like I tried to do for my intro, I said, who is Jonathan Rochelle? It would have been better if I said, write me an introduction for a presentation introducing Jonathan Rochelle as a product manager or as a product leader or as somebody who created products for N years. One of the ways I would recommend learning prompt engineering is watching this course on LinkedIn Learning. Yes, this is a paid sponsorship. No, I'm kidding. This is just a great course. I literally took it twice already because I really wanted to revisit it. But Chavi, who I work with actually, went through kind of the individual items that help you talk to the LLM and help you get the most effective response from it. Some people go to chat GPT, they try it and they're like, well, what's the big deal? And if they did that, then they weren't really thinking through how they were prompting the machine. The second general category, I said at the zoomed out level, is to put generative AI or AI features in your product. A lot of you, if you could actually, and I really want you to raise your hands, how many people here have put AI into their products? Have worked on putting AI in their products? Very few. Now that's probably also because, actually a decent amount, but not that many. I think that's mostly because of where a lot of us are in our careers or we haven't had the opportunity, we work on a lot of things. And it's actually, it's something that has been used a lot for things like recommendations and matching and scoring and things like that or identification of objects. But you can use it in a way that is presented to users depending on what your product does in so many areas. Improved customer support is one that I've seen a lot. In product, natural language search. If you have a search feature in your product, consider using AI or generative AI for natural language searching. Give people a better opportunity to get results. For recommendations, clearly, UGC is user-generated content. If you have in your product user-generated content, think about giving them help, the users help in generating that content. Creating dynamic templates so that the templates aren't just, well, there's the same list that everybody gets. Maybe you actually give them templates for that UGC, depending on who they are, what they've done in your product before. And then, of course, creative and generative features. But when you think about, is Gen AI right for my product? Just remember it's super early. Quality is really important to consider. Generative AI with large language models is generally a random response. You know, people that have gone to more school than we would call it stochastic. It's not factual, it's not absolute, you're not gonna get the same response again. So you have to know when that makes sense for your product or not. I really respect the way Bing has presented this because what they've done is combined that stochastic random response with actual facts using the search index as well. So that pointer I have there is showing you where that data came from. So attribution when you use generative AI in your products is really critical, especially now. This might change. If you watch this talk a year from now, it might be totally irrelevant. But I actually hope it does change. But right now, that's the state of the state. Can your users wait for how long it takes? That's the other consideration. The speed of generative AI is not yet fast. The cost is high, it's not free to scale. And the form factor, it might not work well for a mobile app. If you're not connected, if you're on a device that's not connected, maybe for auto or for the internet of things, you know, an actual device. And of course, there are significant risks and constraints. That's a dangerous road with a cliff leading off to the right, by the way, with no guardrail. I'm trying to remember my prompt. Scary road, no guardrail. Hallucination is a real thing. You have to get into the study of hallucinogens or something. You've got to figure out how you're gonna get around that. It's real. And it means basically that the LLM is making up facts. And you've probably seen the examples. If you say, show me a recipe for a cement cake. It will give you a recipe for a cement cake. That's pretty dangerous, actually, when you think about it. They retrieve good and bad knowledge. They retrieve all knowledge. Everybody thinks about the web as an amazing resource. But remember, besides collective intelligence, it's also got collective sarcasm and collective stupidity and collective opinion and comedy and radicalism. And that's all in the model, at least the public large language models that you see today. They have stale information. They're not up to date. They don't have basic data access. And they're really, really large. They're hard to run. And of course, they are a general purpose, which means they're not the best choice for everything. They're just general purpose. So use care if you're gonna use large language models that exist today. And prioritize safety. Again, that cement cake is a good example, perhaps. Don't give somebody an answer that's gonna lead them to danger. Don't walk them off that cliff. First, for your brand, make sure your brand is safe. One bad mistake can really ruin a brand. Now, at LinkedIn, our brand is trust, professional growth, professional identity, professional opportunity. Trust is at the core of our brand. We're being extremely careful as we introduce LLMs and generative AI features into our products. Remember, for your customers, their safety is critical, too. Their privacy, their safety, their security. Don't give them something that leads them to dangers. And for their customers, I've had instances in the past, products that are sold to enterprises, which they then use for their customers. And suddenly, I'm on the hook for their customer's safety. And that's really critical to consider. So LinkedIn actually uses a very significant set of principles of responsible AI. And some of them are very connected to our brand. Advancing economic opportunity, if we're gonna use generative AI, we're gonna focus on how we can advance economic opportunity, how we're gonna uphold trust, how we're gonna promote fairness and inclusion, provide transparency. And probably the most important is embrace accountability. We're responsible for the products we put out in the public. It's really important for us. And then, of course, if you're gonna use the technology, plan for constant improvement. Build in signal collection in a way that you can use it. Make sure you have a process for using it. Don't just collect those signals. So this is beyond activity signals. Today, it's very common, obviously. And there's a lot of products out there that do a great job at it, collecting analytics. Analytics are activity feedback. And that activity feedback are great signals. Somebody clicked this a lot. The click-through rate on that was higher than that. The A-B test worked out this way. It's all about those use signals. The signals I'm talking about are also human signals. It's really important to get human eyes sometimes on the result, or human ears. But human hands on the result and say, is this right or wrong? And that's like supervised learning. That's what reinforcement and supervised learning in AI works that way. So make sure you design those signals in and those loops. It's really important. Terminology-wise, remember this really hard to grok for some reason, RLHF, Reinforcement Learning from Human Feedback. That's a term now in the industry, which is just a generally accepted method within these products. AI is already in LinkedIn products. It has been for a long time, over a decade. And now with generative AI, I just think it upped the ante. First and nearest to my heart, LinkedIn Learning, we're trying really hard to help educate the professional world on AI and gen AI. We actually unlocked, I think in January, 100 plus courses on AI. LinkedIn Learning is a paid product, but we've unlocked 100 plus courses through June 15th. So you've got 15 more days to use these tools. But we've got a ton of courses and each of these courses on very focused or broad, basic to advanced topics related to AI and gen AI. There's some amazing ones in there, really worth watching. We also, across LinkedIn, moved from just matching and recommendations, all the AI we've been using for over a decade, to this co-pilot content creation. And I want to give some examples of this. So the one on the screen now is really the recommendations you see in LinkedIn Learning. You're getting personalized recommendations about what courses you should watch. When you go to the LinkedIn feed on the LinkedIn app, you see personalized recommendations. The posts you see in your feed are different than the person sitting next to you. But now with generative AI, we're moving in and have already implemented some very specific content creation capabilities. This example is AI assisted job descriptions. If you're a hiring manager or a recruiter, you can actually generate a job description based on the skills that you actually need and even based on a specific person in your network that you say, this is the kind of person, this is the skills of this person or the type of skills I'm looking for in this job. And it prompts you for more information and you can actually generate your first version of a job description. And then it presents for you to actually edit. And there's a theme here that you'll see across these examples. The next is personalized writing suggestions. So if you want to write your profile and stand out and you need some help, you can actually get help from the machine. The machine can actually help you and say based on your profile, your posts, your activity, and then some of the inputs you put, it can actually help you write something that feels eloquent, that actually feels maybe better, maybe just different than what you would have written yourself. It's really hard to write a self description or a description for your company. And again, then it's presented for you to actually edit. Another example is members messaging hiring companies. You want to send a message to a company that's got a job description, the machine will help you. The app linked in will help you write that message in a way that again, feels natural, but gives you the opportunity to edit it. But it speeds up the process and gives you the opportunity to make it your own. And the fourth example is a little different. This is almost prompt engineering for humans. This is generating collaborative article prompts and getting members to actually participate with their knowledge, their human input, their expertise. The key here is you've got to keep learning. This is an overwhelming, just mind blowing trend shift in the industry. And it's something that really will make a difference in our products. And if you don't believe it, definitely watch that robots 250 apps and see some of the examples. It's just incredible. Learn about AI overall. It'll help you talk to engineers. It'll help you talk to product people that are making these products. It'll help you talk in interviews. Understand what AI is overall and gen AI and how they fit together. It'll also help you understand, and this is the most important thing, what's available to you to solve a problem. As a product manager, I said your agents have changed. I really, truly believe that. A product manager that is basically managing only the operation of their product is only doing part of the role. The other part of the role is innovation and change and constant improvement and looking for better ways to do things. And if you don't have that technical insight and that understanding of what can be done, you won't have the best ideas. You won't have necessarily all the ideas you might have had. It's really part of the diversity, the importance of diversity in the creation of products. You're adding to that ability to know what tools you've got at your disposal. Understand language models and what they mean. Embeddings, it's a big area. It's not something we'll be able to get into here but look it up and get into it deeply if you're not already. And if you are already, teach the person next to you. Prompt engineering, another good example. And then, of course, all the available APIs and services that are out there. And like I said, watching even YouTube is great. So I'll just say, you are the agents of change. Learn everything there is to learn about Gen AI. Try it. Create prototypes. Create ideas. Create mocks. Pitch. Present. But use it to the advantage of your customers. Thank you very much. Appreciate it. Thank you.