 Good morning product people How are you? I'm super excited to talk to you today. I'm still looking at this. This is a different slide I'm super excited to come talk to you today about one of my favorite topics It's around the gen AI and what I want to share with you the next 20 25 minutes is basically how it's booking We're thinking about this and sure a little bit about the product that we launched a little while ago and lessons and hopefully those are some of the Information that I share with you. You can take back home today and actually act and incorporate it into your product plans Do you guys like the title of the session? Actually, I didn't write it that was chat GPT when I gave it the contents and I asked for a catchy title That's what it came out with so it kind of stuck So before I get into the session just want to share a little bit more about me You can tell by my accent. I'm probably not from around here It was born in the US, but I spent most of my childhood in Europe. I've lived in six countries Worked my way back to North America doing my education in Canada and then my most of my professional life and in the US I Live in Netherlands now. I've been in Europe for four years and loving it Prior to joining booking which I did about 14 months ago I worked 15 years at Microsoft and it's where I transitioned from being a product developer to Focusing on product around 20 years ago I've dedicated my life to using technology to solving problems for our customers I've gone through a lot of in my years I've had the luck or I guess To be part of many many disruptions when I first started coding I actually we were taking character-based systems and moving more over to graphical user interfaces I when I started at Microsoft we started moving apps or to the internet I was very first in working on mobile search Mobile payments as the mobile revolution came around. This was a few years before the iPhone even and Then probably the most thing them I'm most excited about and lucky to have in my career is disruption that we're going through with January by I Joined booking because I wanted to help using technology to make Make the make it easier for everyone to experience the world It's super exciting Looking at booking with some of the numbers. We have actually a fairly large scale I think we're the largest OTA in the world right now started out of Amsterdam in 1996 So we're one of those mature e-commerce companies The thing with scale, it's it's pretty impressive While I'm going to be talking to you We're plugging a book anywhere from fifty thousand to a hundred thousand trips for our travelers Which is super exciting because most people are right now are thinking about what they're going to do this summer Dreaming and planning and then executing on on booking and that With large scale also comes large responsibility as a PM What that means is every time you make a slight change to your product You can easily positively impact the end users and your business or negatively impact your your business and your end users So we've kind of been doing a be testing on every little thing that we've done for for as long as bookings been around Now let's start thinking about Jenny I so this was about what 13 14 months ago when I'm sure you've all seen chat EPT launch back in November 2022 and If you were like me you were in awe with what the technology brought together So we quickly this was very very early in my in my booking career. I think that was day 10 While I was there thinking about what does this mean to us? What is this disruption feel like I talked to some of my friends back in the US? That I worked for a very long time and they also shared the same excitement as I have so I thought there was something there We need to do something about it So we looked at three main buckets One is kind of like our product traveler and partner facing were two-sided marketplace So we serve both our partners and our travelers We thought about demand generation Not sure if you've seen if you go on to Google you've probably seen an ad by booking We rely very heavily on demand generation For our for our business and we were worried that if the demand generation business like the likes of Google or other search engines gets Disrupted we need to figure out what's going to happen and how we actually play into that ecosystem and then lastly we thought about internal productivity Especially for for developers for engineers. I've I've never in my career ever seen engineers just sitting around with nothing to do So the idea of making engineers 10 to 15 percent more productive is super exciting for us to build build the products really want to build Much faster than we are capable of doing that today Then we looked at what is it actually that we need to address and while all of those things are super important data and infrastructure Legal risk ethical AI talent and people today's session. I just want to really focus on something very specific Which is the product opportunity? So with many disruptions in the past It's super important to really understand where things are going to and I believe that the art of possible is going to change Just like it changed and all the other disruptions that I've gone through in my career So it's super important for product product people for designers user research and so on to kind of get on the same page And really understand and try to not predict the future because that's super hard But to really see where things are going to be going Before I get into the examples, I just want to ground everybody on what Jenny I Is this is an example? I think you can go and probably tell your parents about This is something that one of our product product person and designers came up with an order for us to tell the story internally When the first came out just to make sure it makes sense for people So the example here is let's say a chef with the best recipe and I'll use the analogy So if if you want to become a chef You most likely will go to some sort of culinary school You'll start learning about some very basic recipes and you'll get a little bit more advanced recipes. You will create the dishes Based on those recipes, you'll get feedback. You'll start understanding. You'll start fine-tuning So basically as time grows and you become an expert, you will create your own recipes in a way that no one's ever done that before Now imagine with this Jenny I Right now what we can do is take every recipe that has ever been recorded in human history And feed it in there Imagine every cooking show ever recorded every cooking book every blog vlog whatever have you you give it to this special brain It's going to be very much like that chef. I talked about that goes to color in a school It will start learning analyzing look at at the outcome and then over time It'll actually start creating its own recipes something unique More at a scale that more than any other chef in the world could ever do The magic will then start learning the the patterns and we'll start creating these recipes the interesting thing that it will also start doing if you are not careful is it will start even creating Ingredients that don't exist and some people call that hallucination I just call it like interesting content creation But it's really up to you to figure out when when when there's an answer that is hallucinated or incorrect to ground it and make sure that the the content that it's creating is correct there was a in a travel scenario, I'm not sure if other seen it Air Canada was just sued last week and lost where a chatbot was talking to a traveler and the traveler was asking for a refund and They shouldn't have got a refund because based on Air Canada's website. It told them exactly What the policies were and refund wasn't due but the chatbot? Looks like it hallucinated the answer and offered the refund to this traveler Then it took him to court and the court said that the chatbot speaks of speaks for you. So Super important to to make sure that content's grounded Kind of touched us already. So what does this mean to you? How do you approach it? We're all not gonna go home and start building chef type of software, although it like it would be kind of fun I've seen a video a couple weeks ago by by Hendrik Nieberg who referred to this as kind of like Einstein in your basement I really like this Analogy so imagine chef is like one kind of Einstein and this Einstein knows all of the knowledge has ever been recorded in in human history I get my my things scrolled here But imagine for you what what can these Einstein's be so imagine that you could have Let's say architects doctors lawyers Designers and then for our case in booking it would be like imagine if you can get the best travel agent That that ever lived that could scale to millions and millions and help our customers So not just the travel agent, but let's say even a concierge a concierge that would know all the ins and outs For where you want to travel that could share that information with you Built into your app something that you cannot do today So what's also very interesting like I mentioned a what we've learned is that Imagination is great start figuring out what what's possible But then start playing around with doing prompt engineering because that's really where you start teaching this Einstein of what is that information that you want to convey to your customer? What is allowed? What isn't allowed? What is on brand? What are the things that you absolutely do not want it to answer because It could be very very chatty and it would love to answer all kinds of things even give away all the secrets Early on there was an example Or somebody launched a chatbot type of feature Based on open AI and if you went and asked it for all the secret prompts actually told you so imagine It would share all your secret code I Want to also spend a little bit of time looking back in the past. I know I've I've already mentioned a little bit about the Evolution software evolution, but it's interesting to look at history about 50 years ago Bill Gates and Paul Allen had this dream of having a computer on every desk and at that time Even experts from IBM or Hila Packard at that time, which were the leading edge software companies thought that maybe we can have thousands But we'll never have a computer on every desk little did we know that 50 years later I bet most of you have what at that time would be thought it was a super computer in your back pocket always connected Another interesting one to me was Mark Andreessen. That's that's his photo in Around 2011 talked about software eating the world and at that time many people didn't really understand what that meant But what his context was he believes that all the products and services in the future Will be powered by software will be made much better by software and to me a great example would be a vehicle I just bought an electric mini just a little while ago and One of the reasons I bought it is because the app that it comes with it I'm able to remote-recontrol the vehicle I can actually my favorite feature is Preheating it because it's very cold in Amsterdam in the in the winter And I love to go to the gym wearing my shorts early in the morning So that's a feature that I guess maybe 15 years ago People wouldn't have thought of but it's something that people expect and maybe not even buy a vehicle without that And I feel the same with Jenny I so Jenny I is although there's quite a lot of hype and We really haven't seen Very much consumer adoption yet other than let's say chat GPT or Gemini Bard Or perhaps Microsoft's co-pilot, but I really I believe that it's something that's going to transform the way we do things the way we look at the world a little revolution lies creativity help you automate tasks With personalized content generation and help you with decision-making as well All right, so what does this mean to product managers? How should you approach it? Another one of my favorite quotes from my time in Canada who knows who this person is a Few of you. It's the great one Wayne Gretzky Actually, I didn't realize the name was on there. He had this quote that I use I Used with my my product managers or the team quite a lot especially in times like this And I think it stands the time a good hockey player plays with a puck is and a great hockey player plays with a puck's going to be If there's anything that you should remember from today I think that's one of the one of the better quotes as think about what will happen in 18 months in 24 months in 36 months What will happen to what your customers will be expecting just like I mentioned? I will not buy a car that doesn't have a remote app People not be looking to use your product the same way that they use it today So the idea is well, it's super hard to predict the future I'll share some things that that we did in order for us to make sure that we're moving just as fast as possible And the thing do you guys how many of you have some sort of a customer journey? Great, I think it seems like most of you Probably been around for a while if you're a mature company, right? You probably know exactly what your customers are doing one of the things that I urged our team is to really go back and Understand is the customer journey? something that we believe our customers are doing with respect to our business or Is the customer journey really reflect the unmet needs of our customers? That can potentially be met now with this new technology, and I believe the case is more like the later As great, I think it is right now to use our apps to Buy and manage your travel. There's so much more we can do Just imagine back if the if we had the Einstein version of the travel agent all the things some of you may be old enough To have actually use a travel agent before They could really give you that tailored customers experience Which is really not available yet today, especially in a single app So we went and did that We thought about what are the what is this disruption? What is it that it could superpower and really understanding customer intent now for us? It's I think it's a little bit harder. I spent many years prior to booking and I thought I understood how to do Understand customer intent. I worked in Microsoft advertising group. I started up Well, one of the groups and understanding really user intent and and for us, it's it's a little bit more difficult because You take on different personas Every time you travel Just an example, maybe some of you flew here or some of you fly to let's say, Manchester for to your head office And you stay at some budget in if that's all the travel you do and We look at try to create that persona We think you like to go to Manchester and you like to stay at this budget in so now when you come and you try To plan an anniversary trip. Maybe it's your 10th or 15th anniversary trip It wouldn't work really well if we try to personalize you based on the knowledge we have So for us thinking, oh, how can we get a better intent that we can get today? Could be super powerful and then lastly the thing that I mentioned about hallucination or creating content Imagine if we know what you're looking for and right now we have millions of property descriptions Which are all the same have been fine-tuned for conversion customer lifetime value all of those things But imagine if we know that you like pools or a great breakfast or a great gem We could change that As one-to-one personalization based on what you tell us much easier than we could have ever done that before Little bit of a recap This was actually a very interesting for us when we thought about our lifetime. I'm sorry our customer journey and Our understanding of trip intent really trip intent isn't just that you want to go look for a specific hotel That's just part of it trip intent would be something like cheap places to go for a week The summer that is beachy with a direct flight Probably most of you could relate to something like that If you want to go and book that travel planet travel today, it's it's super hard You probably do it in multiple places. Maybe you get your inspiration on Instagram Then you start researching on a search engine like Google and then when you kind of have some information that you want to double Drill into or get some specifics you go to an OTA like booking or another one So it's clearly that we have an opportunity for unmet needs that nobody is serving our customers yet The second part is around understanding user context and I mentioned I mentioned a little bit about this if all we know Is a little bit about you Then the way we used to do personalization probably doesn't make sense Imagine if we can ask you this information in a much easier way that you would share with us how we can personalize that and It's interesting. I'm not sure if you guys seen the opening. I launched a couple things last week One was memory Which they'll be sharing in plain English all the things that you ever shared with them that they use as input and Also the things that they kind of came up with based on your interactions So this is another opportunity for us to to look at so I'm gonna share a little bit of a video I think it's about a minute. This is something that we launched last June We first launched in the US. We've actually launched it in in the UK as well Last December and it's fully ramped. So maybe during break or later You can download the app and try it out and give me some feedback Could you guys see yourself using a tool like that in your next? planning event All right Thank you Don't give it a try I Want to introduce another topic which I've been working through with my team just to see if I'm a little crazy or not But I think there's gonna be a paradigm shift in how we design apps So if you look at like the example that that I just showed you there That's a combination of like a chatbot natural user Natural language user interface and some other traditional pieces as well like the carousel with the recommendations So my theory is that today We've been teaching our users How to use our system based on how they use other systems like ours and We've been fine-tuning With a B tests on how we measure success and how we iterate on that. Let me give you an example Today if you go to an OTA you you give us your destination and a couple of dates We taught you to do that as soon as you show up. You just know how to do that 30 years ago You probably wouldn't guess why that's the way it is but all the other systems teach you that so then you put the value and Most even if you've never come to our site, you expect to see a search result Which then you could start manipulating. So this is kind of like learned behavior, right? this wasn't out there before when computing started and We and then we start getting better at it by moving around some of the filters We create some more filters if we're personalized if we remember use the filter before we go do that So that's kind of today's world But imagine like the video that I showed you and then tomorrow's world where The user doesn't have to learn this new paradigm or the paradigm that we fit in today The user could just speak in their own natural language So then the paradigm flip is that instead of us teaching our users how to best use our system Which gives us the highest conversion Customer lifetime value and all that and make our customers happy We need to build a system that understands what the users actually want to do And then complete the task that way this does that resonate with some people or is that crazy? Crazy So when we when we did this we came up with some some principles five's quite a lot I wish would have had three, but we needed to have five and For us we thought if we nail these things as we go through this transition The user should feel magical just like the video I showed you it would feel pretty magical in order to use software like that The one thing that I do want to call out that I worry the most about and it's really Retaining or increasing gaining trust with our users as we take him on this journey from our current user experiences To what I view as a user experience of the future whether that future is 10 12 18 months from now I believe that's coming. So trust to me is a super key and let me give you an example I was just looking for a new pair of sunglasses last week I'm going on a trip and I want to have a cool pair and I think I clicked on an ad in one of the social networks and I started to get bombarded with a bunch of different sunglasses It feels like everywhere I go Some people think that's creepy. I actually don't mind at all that much It could be annoying once I already bought my pair of sunglasses the point I'm trying to make is Imagine with the personalization ability and content generation of today's tech that's using there How creep it feel? Imagine if we take this and supercharge all those capabilities in the future You could quickly come up with a system that users will just reject because it's just too too freaky for them So that's my my personal opinion. So make sure you you keep and earn more trust as you build this out So our learnings have been Revisit the unmanned needs Maybe you have some maybe you don't I think if you really go back to basics and understand what your users are trying to accomplish You could probably come up with some if it's possible Validate the extended scope if you put up a natural language interface. It's very easy I personally read thousands of anonymized ones People are wanting to do things with us tomorrow That we never gave them a chance to do because we've only asked you to put in Destination and dates now. They're telling us all kinds of information This was something the second point we learned kind of halfway through We're a mature business a high-scale business. We've got our own metrics when you introduce something new And you really don't know how to measure it or how to extrapolate it to the overall measurements that the company worries about It may seem like what you're doing with the business results are lagging What we really need to do is really understand how do you actually measure a funnel? That's conversational versus their total normal e-commerce funnel that people are used to and like I mentioned Earn trust with users So to take away for you, this is your homework three bullets not five Figure out who's your Einstein and figure out what they can do for your customer The next thing is build as much as you can as fast as you can we had Ran a hackathon bass back last February in order for us to try to figure out. What are all the different options? What are the possibilities and This product that we launched was actually based on the hackathon only 10 weeks later The last point is we're all just learning Embrace this even when you look at things like open AI's last week's Sora release You're just in awe of what some others are doing but the reality is Most of us are just still learning and there's so much to learn and the more you do the more you will learn In order to make your product better I'll leave you with a disruption of another kind and this happened Probably a little bit over a hundred years ago But there's so many similarities of what happened there to what's happening with gender AI or the other tech revolutions that we've had This is one of my other favorite quotes about Overestimating the effect of technology today the reality is nothing's really changed as of today even though it's been out for a year and then we underestimate the effect in the long run and This image is actually very interesting because if you do a little bit of research on this At the time when cars are starting to enter the roads The biggest concern for everybody was that the horses are getting scooped spooked by the cars They weren't worried about other things. So there was actually you can you can Google this now Somebody had a patent somebody actually built a car with a wooden horse in the front Because they thought that if we do that all the problems are solved and then horses and cars Will be sharing the road little did they think at that time is while these cars are gonna get much faster, right? They're gonna start delivering a lot more people will have buses will have trucks There'll be accidents. We'll probably need lights and intersections Very similar to how Jenny I is being treated right now. I was a little bit spooked last summer when Italy actually banned Generally, I I don't know if you guys remember that for a few weeks But we were really scared of what's happening. How do we build a global product? Do we need to build in all these features in order to turn some things off or behave in a different way? So With that, I think the call to action for you guys is to kind of go back Go back to your work Listen about some of the things. Hopefully I provoked some things for you to go take back and incorporate in your product Thank you