 Hey, everyone. Thanks for joining us today. What an exciting year this has been for AI. As part of the discussion today, we are going to be talking about how do we craft a culture of AI in our teams. Let's kick it off with some intro. I'm Govind Kaushal, your host and moderator for this session. Our planned moderator for this session, unfortunately, could not join, so I'm stepping in to host the session with an exciting group of panelists, Stacey and John. I'm a good product manager at Google. I'm part of one of the lab's group at Google where mission is to identify long term opportunities for Google by focusing on advanced technologies. This year, my focus has been on AI, specifically focusing on AI applications in the enterprise domain, thinking about various ways in which AI is going to be increasing productivity across various organizational functions. With that, let's get to know our panelists. Stacey, do you want to go first? Sure. Thanks. I'm Stacey Cronin. I work at YouTube, also a part of Google. I'm the trust and safety team. So our team's responsibility is to identify and remove content that violates our policies and shouldn't be on the platform. And Gen AI has certainly been a big topic of conversation in our organization, both from a potential abuse factor. There's the potential for a lot of content to be created very quickly. And then the creative aspects. How can Gen AI be a really powerful tool for our creators to create things that are more original or faster and easier to do? So there's a lot of exciting opportunities that would be very beneficial. And then also in our tooling, in our ability to detect things and even to simplify our tooling for us to be able to identify potential violations faster. So there's a lot of different uses on all the different good and bad categories within YouTube. That's one that was thank you so much for the introduction, Stacey. John, do you want to go next? Sure. Hi, everybody. I'm John Mark, Product Director at Waymo, also an alphabet company. I'm from the family. And I lead product for what we call commercialization. Which is basically how do we take the self-dealing car that works pretty well at this point and scale it to 200,000 of cars and build a rideshare and delivery service with paying customers and all that could stop. And before this, I would go to after a bit and then Uber for a long while before that. So I've been kind of working in the logistics and rideshare delivery space for 10 years now. And yeah, it's been very fascinating to see AI's evolution over that time. Obviously a big part of algorithms for matching and pricing at Uber that I worked on and both at DoorDash. And then obviously with COVID-19 cars kind of the impact on the technology behind how self-driving actually operates. And then obviously on the commercial side as well, it's lots of applications in terms of understanding customers better understanding feedback, hoping to design more intuitive products that can better understand individual customers and who they are and where they want to go and all that good stuff. So yeah, happy to be here. Thanks. Great. Thank you, John for joining us. All right. So let's get started with an easy question. We are all PM here. So let's talk about how has AI influenced our life? And maybe we can start off with John if you wanna go first. If you can share an example on how AI has helped improve your personal productivity as a product manager? Yeah, for sure. I mean, probably a couple. I mean, one big one is actually understanding rider feedback, like back at Uber when we used to look at all the ratings and the comments that people put in and what was good about the ride, bad about the ride. One of that originally was very manly triaged by large numbers of people and sort of kept classified and categorized. And we've had a lot of success starting to use AI to understand the window side now the rider feedback and understand and try to classify things and find patterns, actually things that humans didn't even have thought before to get into the nuances and pretty fascinating to see. We've definitely done some experimentation with helping to write documents, PRDs to use content with content together and that's been pretty cool to see. I don't work as much on the economy side but my colleagues on the economy side have obviously been using more advanced AI developments to rethink how perception and planning and Ubering and all the kind of piece of cake on the staff work. That's very fascinating, John. You mentioned something around like, helping getting the assistance and writing the PRDs. You wanna elaborate on that a little bit more, like what does the process look like for you and what do you still look like and what is it looking like now? Yeah, I would say I've been super happy with all this so far, but maybe you all have something to point to, it would be great. But I think there's a lot of potential there. I mean, when I think about PRDs, they all have this different template out there but a lot of the structure is very consistent, right? It's like, what is the problem statement? What was the opportunity? What are the things you're solving for? What are the constraints? And how do you think about different solutions and trade them off and decide if they're good or if they're not? And so what I've noticed over the years is there's a lot of road work that happens there from PMs, right? Where it's like, you know, if you build that or PM might get template, but it's just a lot of road work sometimes to pull things out. And then also to reference things and pull them into different places like presentation or some program with you or other kind of forums. And so for me, I think it's early stages, but I see a lot of potential in making the FPM a lot more productive by helping to crack knock into them quickly to summarize them, help you understand. It's a reading a 20 page document. You know, help me understand this, write the TLBR for me, right? Writing a great TLBR, you know, that hopefully the machine can do better than I can some more. I cannot agree with you more. Stacey, over to you. What are your thoughts? Yeah, I think, you know, the summarization that John mentioned is certainly something that I've made use of. Often it's hard, it's easy to write a long document of all of your thoughts and it's hard to distill it into something a little bit more crisp that might be more appropriate for an executive communication and our short email update. And so, you know, doing your just thinking and writing and then invoking some AI to help you summarize has certainly been helpful. I think also on some of the creative fronts, right? So we've used it for naming, like project naming or, you know, team naming and saying, and you can just have fun with it, right? Like give me a name, that's something like this. And then I'll make it funnier or make it, you know, more sci-fi or whatever you might want to do. And similarly, images, images and decks. So decks can get pretty, you know, boring. And so putting some interesting images has always been, I think, a really nice way to add some color. And I've had some fun recently. I had a big update recently and I searched on the last page where I'm like asking for everybody's buy-in. I searched for an image that said, you know, leadership supports my plan and it gave me this wild picture, just sort of abstract art. And I just put it in the slide and I was like, maybe it'll help. At least it looked interesting. So, and the review went well. So maybe it was the magic bullet. And to build upon what you mentioned, Stacey, one of the things that I personally use, I've seen myself using it quite a bit for is to be able to provide a critique. So if I'm writing up a document, just making sure that, you know, it has been written in the format that audience, the targeted audience who might not have understanding of all the complexities, all the devices in the document are able to clearly understand. And this has been like really helpful. You know, there were days like when we used to get like PM PR reviews on the PRDs that we used to write. But I think now AI can actually be a really good co-pilot and fingers unlock that. So now to extend that line of thinking and maybe we can start off with you, Stacey. What is your favorite resource to stay on up to date? It's just information overdose right now, right? Like every day there's like a new breakthrough that's happening. So share some tips that you have for staying up to date on what's going on. You know, I wish I had some like favorite podcast or great article or something to cite here. I think what I mostly do is talk to people and particularly some of the engineers and hey, let's go have coffee, let's go have lunch and you know, what's new and what's going on in your world. Things are changing really fast. And some of it is, you know, I will read articles in the paper and that's kind of the big picture and you know, the drama of CEOs getting fired and you know, all of that's kind of fun. But the stuff that really impacts my work I mostly learn from talking to people on my team. And sometimes even before it's been officially like, yeah, we're sort of noticing that maybe it's, you know, we don't need a huge volume of labels but we need some really high quality labels particularly in this area. And I'm like, hmm, okay, we can go think about that and get kind of the early insight on how we might wanna change our strategy from the product side to better serve the engineering side. But maybe I'll get some good recommendations from you guys too. Yeah, it's not too hot. John, do you have any tips that you wanna share with us? Yeah, sure. I would say two things for me that are super helpful. One is a curated list of people I follow on Twitter which I refuse to call X. And you know, there's a bunch of, yeah, researchers, sort of founders, engineers, I would say that's where I often first heard about like, oh my God, I can't wait, they can do this now. You know, kind of chat PPP stuff or whatever. So I find Twitter is a great resource. It's like, obviously people all over the world do really cool stuff and if you call the right people I think it's helpful. The second thing that's been really helpful is here in San Francisco is obviously a lot of hackathons and on the side I do some advising and investing. And so I'll just go to like a hackathon in San Francisco and talk to a bunch of engineers that are like, hack it on something or people that are struggling to company it on something and kind of like do that ethos. I'm sure quite a bit of like, really just thinking what's going on. Now I think you guys covered a pretty good list. The only addition that I would make is I sign up for a bunch of these TLDR type of newsletters that arrive in the morning. And that kind of gives you a good overview of what has happened in the last day before. So that's kind of like making sure like, you know, nothing important relevant to your work is getting this stuff. So that's a good starting point as well. So now let's get into some of the product design and the product management related topics. So we are likely going to see two parallel waves of AI influencing product design. The first one is likely gonna be around AI getting incorporated into existing products. And then the second one is gonna be a new class of products that are designed with the AI first principles. I think in this last year, we have definitely seen a lot of both of them actually. We have seen companies like Microsoft, Google trying to incorporate AI into the existing products. We have seen bunch of like new startups coming out which are really putting together the products which are AI first. So as a PM, you may actually end up working on a project over the next year that does one or the other. So what are the essential skills and knowledge you need as a PM to be prepared? And maybe we can go start with you first, John. Yeah, so the question was like, what skills are relevant to stay on the edge of that stuff? Yeah, I think as I think about it is there's like tactical stuff and strategic stuff and there's like different time horizons that we touched on a little bit of things like, yeah, I hope there's some right to document or some of that and I think some of those things are more short term. For me, the long term question is very interesting as a PM or how does this sort of change like user interfaces, right? Like I think it's a big reason why it is essential for like Microsoft and Google because there is, obviously it's been things like Alexa and Siri and they've been helpful for some specific things but they certainly haven't replaced the Google search box for example, right? Or like the kind of typical ways that people interface with their phone or the computer. So a big one for me is like over the next, yeah, several years, decades, like how much ADC, eventually interfaces between the computers changing, moving to voice or moving to other augmentative ways of interfacing and so like, how do you think about potential disruption to your product or right that would change radically or allow competitors to more efficiently kind of like interface with the customer. Stacey, do you want to go next? Sure. Yeah, I think, I mean, it relies on some of the same core skills we use for everything. I would say in particular, I would think about understanding customer needs and really thinking about product market fit, right? There's probably gonna be a lot of really cool ideas that don't necessarily solve a customer problem or deliver a business benefit that you're looking for. And so I think having your spidey senses up for asking those hard questions like, yeah, this is pretty cool, but what is it doing for our customers or will people pay for this or how might this actually support what we're trying to do? Will be an important question to be constantly asking yourself and building that culture within your team to be willing to question things and experiment, right? I think there's probably gonna be a lot of things that you try and you might try 10, 12 things and one of them might be huge and the others might be kind of either duds or just not as much adoption as you had hoped. So be willing to experiment and learn and see where the market goes. Now, the fantastic responses from both of you, the only thing that I would add, which I'm kind of like seeing working on a little bit of an experimental stuff is really around like the iterative cycle of the product development, which is typically in the software engineering, you kind of like go through a very traditional path of like, hey, here are the requirements, here are the test cases, here's like the PRD, here's like, you know, how the engineering can actually plan the work. And maybe this is a reflection of where the technology stands right now. Right now, if you're trying to work on an AI project, the things are gonna be very iterative, where you really have to first start off from an idea, think about how you can quickly prototype and see what you are getting out of the box and then be able to go through these cycles of quality improvement in certain hypotheses where you would may not find any luck, any success and in certain cases you may. So it's gonna be a little bit more of an iterative product development cycle, at least for the next year, that is what I would expect till the time the technology kind of like gets established and becomes more and more mainstream. So that's probably one of the observations that I've had in the impact of AI with respect to the product development. Now, we actually do have a interesting question coming in from the audience. Let's take that one. So the question is, how do you recommend driving change within engineering? And more context on this question is, AI tools are going to drastically change how teams are structured, but might result in needing somewhat different skill sets and even deductions. I have noticed engineering leaders adopting JAN AI with different levels of commitment. So with that context, any insights Stacey that you wanna share that on how you would recommend driving change within the engineering organization? Yeah, I do think there's change coming, right? In terms of what are the skills that are needed? You know, I think it's also, I don't know just thinking about my own experience and things that we've been talking about necessarily that we need smaller engineering teams or that some of the tools that are going to eliminate the need for good engineers. But I think obviously folks who are good at building and measuring and iterating on models are more and more important. So that skill I think is gonna continue to be a really critical one. And then building on some of the things that we were just talking about, I mean, the still the role of the application engineer and the engineer who's thinking about what products do I build on top of this is essential, right? But building faster, more iteratively, understanding what the market could need, I think will be the direction. The question asked about engineering, I almost wonder, in my experience, it might be some of the other teams that are more impacted places where we are more likely to use a lot of more human labor to do things like we have human reviews, human reviewers looking at content and the more we automate the fewer humans we may need or the different jobs that they may need to do, right? So the skill again may change in some of those operations, support types of roles for sure. That's fantastic. John, do you wanna add anything? Yeah, I think I just see that trend of engineers moving up the stack and it's sort of going away, right? But it's like, I'll beat myself, but I graduated from college in 2005 and like when I studied computer science, a lot of it was still about data structures and like pointers and memory management and like all that good C++ stuff, which of course, and then before that, it was like people were writing assembly and even then people were like, okay, you're not thinking about assembly, right? There's a lower level thing happening service and now you're seeing that move up more and more, right? But it's like a lot of the basic blocking and tackling and that people used to have to like to find out, right? As engineers is like being done by the activity or the tools. So I think that actually frees the engineers up to think more about system design, architecture, kind of the higher level conceptual thing that I think humans are often still better at and just get more leverage, get more done. But I don't think yet it's gonna be a smaller team. I think, I've always put an engineer constraint, right? Like you get the price anyway, every year you have a roadmap, you'd love to do 50 things and you're like, well, I'm gonna have an engineer to do half of it, right? So maybe we can get more of it done, right? If we have like more leveraged engineers. Makes a lot of sense. Thank you both. Let's continue over to the next question, which kind of like fits in into the bucket that you were talking about Stacy, which is like, it's not just the engineering, but some of the other functions which are also seeing the change in the way that they do their day-to-day job. And as a PM, we actually get a chance to work with the cross-functional teams, whether it is design, legal, marketing, business development. So for the purpose of this discussion, let's talk about the non-engineering team members. Let's see if someone from marketing or business development comes to you and they're like, hey, I've heard this AI thing is going to change how we work. How much do I need to know and what do I need to know to stay up-to-date or stay relevant? What Stacy, do you wanna share thoughts on this? It's a good question. I mean, I think a lot of the conversations are fairly technical. So it might be more of kind of demystifying it to someone who is in another function or even someone who you know socially and I would maybe just start with, what do you know about it? And people use it today and don't realize it, right? It's like, I start typing in the search and it autocompletes, right? Like that's been around a long time and people are pretty comfortable with it. And so how do you make it feel familiar and not scary and think about the ways that it might make your life easier, right, when you're writing emails or design? I think design has a good opportunity to really use these tools to expedite and expand the creativity of the work that they can do. So I think grounding it in the actual experiences that a non-engineering person would have with AI would help kind of demystify and make it not seem like, oh, there's this weird crazy thing that I need to totally understand and just understanding. And I think the more people understand it from the consumer side, the more we can be thinking about what types of things do we wanna bring into the world? Not just the how and the models and the tuning, but like what is this delivered, you know, and what can we imagine in our hands, so. Can I, John, do you wanna add anything to it? Sure, yeah, I think some of what we were saying about engineers being higher leverage, I mean, I've seen some of that, like we're, yeah, with marketing, like copyright and generate copy tests for ads or take copy from a successful ad campaign and rip on that and like further, like find possible permutations, right? Marketing breeze, all that kind of stuff. Again, higher leverage, same design, if you're the same safety, I think, you know, helping to generate concepts or building blocks and kind of speed up the design process. Definitely a lot of interest in that as well. Great, so let's switch gears. Let's talk about, you know, at the PM, we are always thinking about the pricing of the products as well, which is, you know, do we, what kind of revenue model do, how do you wanna make money out of what we're building? And every product goes through like a different journey to be able to come up with the right revenue model as well as what the pricing for the end consumer should be. Now, and this is a little bit of a futuristic, so none of us have seen the magic band, things may shift next year, but knowing what you know right now, how do you foresee AI influencing the product pricing for the end users? And to give some context on this, you know, there is a definitely a common consensus at this point that AI is gonna reduce the operational cost. And if you think about the product development process, operational costs include like, you know, the amount of number of engineers that it takes to actually build something. The cost functional teams that it takes to actually put together the product. And if you are seeing productivity gains over there, that would translate into the cost of building a product to go lower. How do you see this playing out? Is it fair to say that the end products to the end consumers will become cheaper? And let's start with Stethi. Sure. Well, I mean, we don't always use cost-based pricing. So this could translate into more profit or more reach. You know, I think that is something that you could potentially, you know, your product could just sort of expand more quickly to more people, more highly leveraged. And so you have these reduced costs to build. And then, you know, you can get it out to more people. You know, there's also, there are some increased costs for compute power and that can get quite expensive. And so depending on the demands of your model, there, you know, even though you're getting some efficiencies in one places, you could have some increased costs elsewhere. You know, what one thought I have on this too is that it reduces the cost to serve certain types of tasks, but potentially increases the value for others, right? So I could see potentially more kind of freemium type models or something like I'm reminded of the, some of the mental health apps and services, right? You might start with a chatbot and you're talking to the chatbot and that's gonna get better and better. So you could get more and more people into a free or low-priced product, but that might create more demand for your higher-priced products, right? For your more specialized support from a, you know, from a one-on-one type of provider. So I could see both extending that your funnel by having a really, you know, cheap and easy way to bring people in and at the same time driving value toward where do you need a higher-touch experience for whatever it is or a more experienced product or, you know, person. So, yeah, I don't think everything will get cheaper, but there'll be some opportunities. What I'm hearing from you, Stacey, is it's more nuanced and it's not one model fits on. John, anything that you wanna add? Yeah, and I agree with all that. I think, yeah, it will vary a lot by vertical. I think, yeah, for SaaS products, right? Like the portion of the cost that this can cut out and be pretty significant in certain professional services, yeah, I do think there'll be dramatically cheaper access to tutors, to lawyers, to doctors, right? So we have to end the level kind of stuff and I think that's super positive. You know, where I've worked, like at Uber and Waymo, it's probably not gonna change the price too much, right? With Uber, you shut the pay to the driver, right? So even AI makes product a little cheaper or support cheaper. It's a small portion of the full cost of Uber to ride, right, Katie? So we drive the cost of the driver. So like, and then with Waymo, it's pretty asset-heavy, right? But the expensive cars and sensors and we still have infrastructure and charging and people that prepare the cars and all that. So again, I think it'll, it'll lower the R&D cost and some of the product development costs and make some of the product run in better but a big part of the cost probably won't come down in the near term. Got it, got it. Awesome. So I hope you guys enjoyed the discussion today. I know we are right on time. Anything that you, anyone wants to add before we close the session today? Stacey? Yeah, I think this has been, it was really fun to think about these questions and I hope that it was interesting for people to hear some of the conversation. Tom? Yeah, I enjoyed the chat and I think it's an exciting time to be alive. So enjoy it and make the most of it. All right, with that, thank you both for the few for joining us today and we can end the session now.