 Well, thank you guys so much for being here. This is the first of our AI Dev Summit at the Linux Foundation. I'm the AI program chair, so this is exciting to see everyone. Special thanks to all the speakers who've come near and far. Karina is here, and she'll be great at being in great talk tomorrow about the state of AI as visiting us from the UK. So anyways, I'm really excited to talk about gen AI. I've had the great pleasure of building two products with LLMs this last year. And so I kind of wanted to give you a little bit of my human-centered AI journey. I was a lowly investment analyst at Stanford back in 2014. I spent a lot of time thinking about data quality. And the big rock stars on campus were Andrewing and Faith Bailey and this thing called ImageNet was really exciting. And people were talking about business intelligence and product analytics. And we could see across all of our asset classes this wave of AI and sort of what big cloud had done for big data and then sort of how that was being applied into enterprise applications. So I got really excited about AI. And I went to UCL in London to study Management Science Engineering. I then got it kind of waked out about Brexit and came back to the Bay Area. Very nervous. But when I was there, I helped launch the deep with a bunch of folks from DeepMind, the AI Seed Fund, which was Investor in Wave. And yeah, got really excited about this thing called AI. I thought it might be a big deal. So I came back to Mountain View not far from here. I started working with Peter Norvig, who's one of the godfathers of AI. Got to see all of his Zany shirts, which was really fun. And learned a bit about how do you build a data first product, service, and company. At the time, Alphabet had made a number of strategic investments out in Southeast Asia who were doing a lot to bring the informal economy into app and technology. So you could start to see these experts in the loop, whether it was the merchant or the OJEC or the teacher. Suddenly you were able to have this sort of expert who was both providing data for the model as well as helping it train it to be better. So I got very inspired to study this thing called Human-Centered AI, which no one really knew about. I think I was the first in my lab, and that was back in 2020. Another thing happened in 2020, which is a pandemic. So suddenly it became very hard to study human interaction. So I spent all my research about midwives in the loop and brought it to Holodoc. For those who don't know, Holodoc is the largest telemedicine platform in Southeast Asia. It's kind of like the NHS on steroids. There's 35 million monthly active users. At the height of COVID, we had 60 million monthly active users. I led product in AI there. There's probably no shortage of AI use cases with that Holodoc, whether it's automating or back office admin or doing insurance tagging or sending out patient reminders or doing quick triaging. All with the idea that an expert in a loop is actually the best way to deploy AI because it means that you reduce hallucinations, you reduce any sort of confusion, as well as making sure that you have the best data going into your models. So my midwives project suddenly became a platform. Oh, sorry, okay. So I was talking about midwives, which many of you might not think midwives or that may not be connected to AI, but they're actually kind of fascinating. So they're responsible for 20,000 data points per pregnancy. They basically run data lakes and they are responsible for 80% of child birds in Indonesia. They are the backbone of community health. They are basically the entire healthcare system, but kind of done in this community effort. And they're responsible for so much data, even though that's not why you go to a midwife. So we built this as midwife assistant with the Bill and Linda Gates Foundation and it went viral in a positive way. And they were able to take care of more patients and increase patient care access. We were able to send out patient reminders and tailored patient education through WhatsApp, aka the world's largest OS for most people. And yeah, midwives loved it, patients loved it. They came back more frequently and healthier and it was a win-win. So the Ministry of Health then launched it as the standard of care and now over 200 million patients receive care through this midwives app. So all to say, human-centered AI and AI assistance is something that I've been thinking about for a while now. And when OpenAI was starting to build an assistant, I joined and then that became ChatGBT and that did well. So now I'm at Kendo where I'm thinking also about how do you empower workers? So at OpenAI, we're thinking about how can we build an assistant for Morgan Stanley? And at Kendo, we're focused on how do we build the product and security layer for the open source community? And so from the last couple of years of thinking about how do you build with LLMs, I've kind of distilled down like here are the five things that I would do if I was you. So first off, who is interested in building a product with LLMs? Okay, great, the whole room, go ahead. So today I'm just gonna walk you through like here's how I would do it if I was you and this is how I've sort of navigated my way through what is not a normal product design. This isn't a static product. If you think about YouTube, it's a dynamic product experience. When I open YouTube, it's a bunch of cooking videos. When my grandmother opens it, it's a Christian monitor reviews. We all have different tastes and preferences and the more that the model or the product knows about you, the more useful it is. And that's something that was very true for traditional AI or what I would call TRAT AI and is even more true when we think about generative AI. So the first step is choose a collaboration style. So you have either something that is very, I think if everyone was here for Dev's talk about smaller tasks, which was a great talk, you can either have it be something very specific and available as a feature or you can have it be far more horizontal and act as an assistant. And so what you'll see is that the more that you start to think about is this something that I want to unlock a new skill set for my user and therefore be more focused on providing convenience and also providing expertise that may not be there. So for example, having a French tutor versus something that is, oh, I need a French translation. And then the other sort of thing that we're seeing a lot of is like, how can you take on a persona or AI being a buddy? But that's not something that I'm super keen about to be totally honest and President Obama just recently talked about it too. So I think there's a growing disinterest in having AI be your buddy. But for those who want that, that's also the category that I think consumer is gonna be very excited about. So the next thing is like, now that you figured out how you want to do the human AI collaboration, do you wanna be small tasks? Do you wanna be some board? Now you start to think about what type of persona do I want my LM to take on? So it can either be extremely well-defined. So today we're seeing a lot of code interpreter applications around being an analyst or being a data scientist. We're seeing roles like, for example, the features that you might see with the notion of this is your new editor. Can it be more casual? Can it be more concise? We're also seeing sort of the other end of that of open-ended applications, whether it's taking on the role of a celebrity. I don't know if anyone's tried to chat with one of Meta's personas. But it's a real fun time, I'm not gonna lie. It's like you have Snoop Dogg who's like, there as your masters in Dungeons and Dragons and just like totally goofy and fun to think of, how can we take the sort of celebrity or artist or completely new interface and bring it into the enterprise as a result? So then just taking the same framework, what do we see today? We see a number of companies that are trying to tow this line between having a general purpose assistant, so the chatbots of the world, that both are playing a consumer and prosumer world. And then we're seeing much more co-pilot examples, whether that's typeface with marketing or Harvey with legal. We're going to see an emergence of these as well. And so what does that mean? So now that I kind of give you the context of like human plus AI, what type of collaboration model do you want? What type of persona are you gonna lean into? You can start now thinking about the product design of like how can I get the type of interaction data that's required to start training my model to become better? So if you think about similar to like the midwives example of midwives are providing data to our model that then is helping us become better at predictions around patient health, what are ways that we can within your own product start collecting data to make sure that the model is actually better at recommendations, for example. So this is an example from our product at Kindle. This is the chat interface. And here you have the LLM providing three recommendations as to what to do next. First, this is an example of when you upload a type of model output, so in this case video, how can you surface with reasoning like things that you think would be best to use next? And when the user clicks on one or the other, you can start feeding this information back to the model to then make better predictions later on. This is just like classic YouTube, Google like recommender systems, but LLMs are starting to become a lot more horizontal where it's not just recommending next steps, but also starting to, you can start placing in other aspects of your work. So for example, from this meeting, I honestly, I think on meeting transcripts there's like golden goose within enterprise. You can do meeting summaries, data entry summarization. You can do sort of at scale. You can do performance reviews. It's becoming this like gold nugget onto all these other ways that the organization is collecting information and analyzing it against different rubrics. And the fourth step, so once you sort of figure out like where are the places that we want to collect interaction data within the product, what are the ways that we want our model to learn? What are the ways that we're going to drive convenience and utility for the user? The next step when you're thinking about LLMs is that you really need to abstract the complexities. So I'm not sure if you guys saw this tweet, but it actually is really helpful when thinking through what parts of your product are gonna be zero shot versus few shot versus rag versus fine tuning. And definitely fine tuning is gonna drive the most performance once you figure out how your user is going to use the LLM. But for a lot of cases, actually few shot, so making it possible for the user to pull in different examples and do something for the first time. It's actually far more useful in the product experience so that you can, especially in the early days while you figure out how we're gonna use LLMs and how our user is gonna use LLMs. So the way that we've done it is that we have style references. So as you guys know, individuality at work matters. How I write about LLMs is very different or unique to me versus if I use ChatGBT which kind of has a one size fits all model. And so by enabling you to use different style references or examples, you can start training the model on how you write, what is your brand style guide and what types of examples do you normally use within your own writing. This can also obviously be really useful for any marketing or all the homework assignments inside the enterprise like writing meeting notes. But yeah, so this has been really useful. So the other sort of like thing that I would leave you guys with is that you should start with some quick wins. So what are ways that you can make it super easy to work with LLMs? In our case, we use actions which are basically back in prompt engineering which you feed the model of like this very long text that we've been working on for a while now. And the user only sees compare. And that means that A, the user doesn't have to be an expert in prompt engineering. B, you can get the strings of LLMs quite faster and you can also start building workflows which I don't know about you but Chat is like very active design. It's a very active process of having to engage with an LLM and this is a way for you to control and steer the model towards results that you want. And that's also a big part of this conversation. It's just how do you enable model steerability through product design? So yeah, so anyways, so I would highly recommend playing around with a product. We have lots of different ways that we're sort of taking from the research community and apply into product design to drive better convenience, utility, reliability. And if you guys are interested in building your own, there's a number of projects that are available through the open source community that I would highly recommend playing around with. We're super lucky today to have Dev as well as Jerry and Mestral just recently launched their model over the weekend. But it's becoming increasingly easier to build these applications, which is super exciting and which means that I think LLMs are gonna be a part of our life sort of whether we want to or not. And so what that means is, thinking through how can we, how can we get LLMs to be collaborators? How can we get LLMs to take on the repetitive work? How can we get LLMs to be a source of inspiration? LLMs are able, they have far more data than we have, but as a result, it comes down to the user being able to steer it and guide it and shape it into something that you actually wanna use. My co-founder Ron likes to think of it as a piece of marble that you're chiseling away at. And with that, I have actually an ask for the open source community. I personally would really like to work on everyday evals. So it's, I don't know if you guys have seen, but whenever they come out with a new LLM model, they talk about how it passed the GMAT, it passed the LSAT. It's now smart at the same level of a paralegal. That's great, but it's gonna get a lot weirder when they become agents. So it's one thing to pass the GMAT, it's another to be giving me financial advice, to have access to my brokerage and to also have access to my bank account. And that is the world that we're heading towards of not only is the LLM doing reasoning, but it also has the ability to go into my tools and also has the ability to remember things. And so I think we need a way to be able to evaluate these models in ways that are just practical use cases or tasks that we imagine models to help us. I definitely believe in the power of human plus agents as a way to unlock new potential, to spend time on the things that matter most, to reduce the time spent on repetitive tasks. But I think in order to get there, we need to have a better way of evaluating these models. And so I'm calling them everyday evals. But if anyone's interested in working with that together, that'd be awesome. So the other sort of thing about this is that as government starts thinking about red teaming and thinking about how do we test the strengths as well as the potential weaknesses and faults of these models at a government's level, I think enterprise also needs to come together and figure out what's best for us. And so that's also part of this effort. But yeah, thank you so much. So here's my step stack where I write about how do you work with LLMs? And then if you guys are keen, here's a Kendo app you can play around with all the latest and greatest open source models. It's a user interface that you can securely deploy within your enterprise. But yeah, happy to answer questions about product. And yeah, it's like being up here. Thank you. Yeah. I might not know what that is. Is that like the LLMs expert or is they keen in the expert that's in the loop? Yeah, totally. So yeah, it's the person in the loop. So today, when you think about Google or how like AI is traditionally made today, there is what I would call a knowledge factory floor in Southeast Asia predominantly, where you have different people annotating work, some call it ghost work. But that is typically the human in the loop when you think about, is this a picture of a cat, yes or no? Or is this a picture of a bicycle, yes or no? All that data tagging is done by a human in the loop or a mechanical turks. And a lot of that now is being shifted onto, depending on your collaboration style is actually the user. So if you're a doctor and you're using a medical diagnosis tool that an AI is providing a recommendation on, as a doctor and you're saying, I agree with this or I don't agree with this, that is being used as critical data to then train the next model to be better. And so how you sort of position the product as well as how you set user expectations can set the standard for what type of data you're gonna collect. And I think what was described earlier at the keynote was super interesting of, there's a new data requirements with LMS. If you think about the human feedback that's required for post training, that's really where you can lean into your user to help you collect that data. So as much as it's great to do open data, I think at the end of the day, you still are gonna need specific task data and that's where you're gonna leverage this expert in the loop. So part of also my, I think my call to action is to think of your users as experts, not just humans, and to think of your experts as a critical resource and making both the better product more convenient, but also your model better. Those who are the two sides of the same coin. Thanks. Yeah. You kind of alluded at the end to AIs being an agent, sort of acting as a fiduciary, as they would have to in California. It strikes me that large investment firms, hedge funds are probably already using this kind of technology today as their own fiduciary to make decisions in the market to cause big swings in the market. We've heard about stuff like that. Do you think we're gonna get to the point where individuals are gonna be able to use a fiduciary? Are you think we're gonna get to the point where the regulatory stuff is gonna descend on individuals? What do you see for the future of that? Yeah, so I think the consumer path is that they're building out assistance for all parts of your daily life and part of that is your finances. To me, it gets very wobbly very quickly. I think it's gonna require really expert product design to ensure that, you know, ChatGPD is a research preview, right? And they've only recently took that off after hitting 100 million users. So I think part of this is also the role of product of ensuring that the user is well educated as to what is actually going on behind the scenes. And that's really what open science is about, right? The ability to have transparency, accountability, and in that case, in the end, trust. And so I mean, I haven't recently spoken to a financial advisor, but most of it goes over my head, to be totally honest. And so I would love to have an assistant that translates it to something that I can understand, right? When people start talking about derivatives, I'm like, who? So I definitely see the role for that. But it's really the product builders responsibility to ensure that the user is fully aware of the trade-offs and the risks and what's being done on their behalf. Yeah. Let me get the mic for you. And this might be the last question. Okay. So I was wondering if there is any best practice that you would like to share in adapting the product, either the user that is logged or what is doing. So practically to give the context of the user that is using the product or what the action that is currently doing to the LLMs, to then have a better prediction and help the LLM in the product. Yeah, so something that I think a lot of people are doing, is first starting with private APIs that have a lot of reasoning, figuring out what your users are doing, what the subtasks are, and then swapping out that private API for a fine-tuned, smaller model that's just far cheaper to use. And so figuring out where your users are spending their time, figuring out what types of, and users are gonna surprise you. Like we launched enterprise search, no one wanted to use enterprise search, it turned out. But they did really like the chat interface when it comes to professional comms or writing up proposals or being able to feed it data. And so they were embedding different documents, but they weren't embedding it through the library, they're embedding it through chat. And so being able to follow your users as they go along while at the same time holding true to your product roadmap of what new capabilities do we wanna unlock? What, how do we build upon our strengths versus sort of spreading thin and hoping on a prayer? If they think a lot of product builders probably feel. Yeah, that would be my advice is definitely let your users guide you and also build upon the fact that as the infrastructure and the scaffolding continue to grow on the open source community, you'll be able to pull a lot of this into your product very soon. So today you guys can use RAG, today you can start playing around with agents as the interface before RAG. And I just think this is gonna become semi-autonomous pretty soon. Thanks. I think that was the last question I could answer. Thank you. Thank you.