 Welcome to SuperCloud 6. I'm Hawaii Xu, panel host here, and I'm a serial entrepreneur and the AI executive in Silicon Valley for a very long time. Today, I'm inviting my friend Harrison Chase, founder CEO of Lanchand to my panel. You just came from the GTC conference and are giving a talk at a panel there, and welcome. Thanks for having me. I'm excited to be here. We've got a lot of chats, so fun to have one on there. Yeah. Lanchand is a well-known company these days. Just a year into it, amazing, got amazing traction. But still, give us a little bit about, tell us a little bit about how you get here. Graduate from Harvard, worked in the place, and how did you get into this space? Yeah. So my background is in ML and ML ops. I studied stats in computer science and college and worked as an ML engineer at a fintech company doing some NLP and time series things, and then joined startup in the ML ops space, where we did testing and validation of machine learning models. That was about four years ago you joined. Yeah. I joined about four years ago, stayed for about three years, knew I was going to leave, didn't know what I was going to do, and started doing a bunch of hackathons and meet-ups in the Bay area. This was right after stable diffusion but before chat GPT. So people were exploring with generative AI, but it hadn't quite taken off. So I was talking to a bunch of folks who were doing early-stage experimentation, basically noticed a few common trends and abstractions, and thought it would be really fun to just put them in a Python package called Lanchand, and I launched that and just as a side project, not intended to start to come in. Again, this was a before chat GPT moment. Yeah. This was like November of 2022. So about a month before chat GPT, and then chat GPT came out, the whole space just became a roller coaster, very much right time, right place, and Lanchand, the open-source package, took off, and so ended up starting a company around it about a year ago. And then Lanchand became a household naming, Silicon Valley, at least. Yeah. Not quite there, but I think the whole space is absolutely crazy, and what we really do is we try to, we're a framework for building these apps. So we're an orchestration, well, we do a bunch of things now, but the thing that we're probably best known for is being an orchestration layer that helps create kind of like these applications that use a language model, but connect to external sources of data and computation. And so as part of that, we have tons of integrations with all these different companies, all these different language model providers, all these different vector database companies, all these different tools. Different embedding models. Different embedding models, and there's this massive community, because there's all these integrations, there's this massive community that's popped up around Lanchand to add all these new, all support for these new technologies. And so I feel very lucky to get to meet a lot of awesome people in the Bay Area, like yourself, and have these conversations, and yeah, just where we sit in the ecosystem is kind of like a central spot that's like the glue that connects everything. And so it's a very fun spot to be in. Yeah, I was joking with people even yesterday when Jensen Huang released this much bigger chip, GPUs, and I was telling people that, well, the hardware's there, we're going to see big and a bigger model, but who's going to deliver the applications? That's going to deliver the value, either you need a systems like Lanchand and a few other, whether open source or closed source software platform to help people to build JNAI applications, right? Yeah, I would say like the capabilities of the models are here and the apps are still catching up, they're down here. So even if the models don't get in, the models will get better for sure, but even if they don't, there's still a lot of really amazing things that we can build and we want to help people do that. Yeah, the model is there, the application is there, that means there are a lot more things, right? People need to catch up in the app space. So let's actually just dive a little bit deeper into this, right? Why building applications is hard? Is that because not enough use cases? Or, well, people know the use cases, but it's hard, right? They need to do all sorts of integrations, that's why they need a launch, like why it's so hard? Yeah, I mean, I would say part of it's hard because it's just so early in this space. Like it's been a little over a year, right? Like I think a little over a year after the iPhone came out, I don't think the major iPhone apps had yet come out, I think it just needs time to experiment what this technology is good for, and this technology is constantly changing as well. So things that were just like a dream a year ago are realistic today, and things that are a dream today will be realistic in a year. And so I think that's a big aspect of it for sure. There's just a lot of experimentation and tinkering that needs to happen to figure out what the right use cases are. I think a lot of the use cases so far have been more on the creative side. So we've seen like character AI, mid-journey, Suno, all of these awesome technologies, but very much on the creative side of things. And I think a lot of the things that enterprises in particular are excited about are a bit more on the less creative. They want to automate business processes. And there's massive, like if you're in a large organization, there's massive value to that. They want to automate customer support. Again, absolutely massive value there. But you have a lot more constraints than you do if you're building creative applications, right? In creative applications, it's almost a feature if the LLM gets a little crazy. Yeah, exactly. Hallucinations are a feature, not a bug, but customer support that's absolutely not true at all. So you need a lot more control, a lot more sort of the quality, guarantee out of it, right? Exactly, exactly, yeah. So what do people need to do? Yeah, that's a good question. So I think it depends on, I think where we see a lot of people spending time is on prompt engineering and on flow engineering. And flow engineering, prompt engineering, I think is probably more commonly understood. You got to prompt, you got to tell the language model what to do. You maybe provide some examples. And there's still a lot of research and experimentation to be done around how many examples do you want to show? Is it better to do zero shot or five shot or 32 shot or whatnot? Or fine tune a model as well. So I think there's that side of things. But increasingly what we see is a lot of these applications, they're not just a single call to a language model. They're perhaps multiple calls to a language model. And so, Codium, which is a coding startup, released a paper or a white paper called Alpha Codium that would complete coding challenges. And a big part of that was what they called flow engineering. So they basically have this flow of data. And in this case, it was like the coding problem you wanted to solve and initial solution. Then it broke up, it had it write tests and then it would run those tests against the initial solution. And so they spent a lot of time on this flow engineering, the architecture of the system of which multiple pieces were language models. So this is about planning how to utilize the language model primitives, right? Exactly. Not just the throw, hey, here's my question. But also carefully craft maybe into multiple stages or multiple phases. And hopefully when I piecemeal things together, it's a wonderful solution. That's exactly right, that's exactly right. But it takes a lot of effort and experimentation and testing to figure out what that right orchestration bit is. And so we see a lot of people spending time there. So, and then let's talk about the luncheon. Do you provide a value for prompting engineering and then flow engineering? Like what do you do to help people? Yeah, I'd say with LangChain, the open source, there's maybe like three main layers at which we kind of like provide value. So one is the base layer. And this is kind of like a very low level kind of like runtime and way to connect components together. So it's not super opinionated. It provides a lot of flexibility. Part of it you could think of as similar to kind of like a DAG orchestration framework. We actually introduced a new runtime LangGraph, which is explicitly not DAGs, because a lot of these agents, which maybe we'll talk about later, go in cycles. And so you need a framework where you can create cyclical things. But that's a generic runtime. Then there's kind of like all the integrations we have. So we have about like 700 different integrations, 70 different LLMs, 70 different vector stores. And the main value we provide there is a common standard interface for all of these. So you can interact with, you can easily switch out between a profit. Swapy, swap power, the different vector database, different model. Exactly. And one of the main things we see, which is why this is so important, is that the ecosystem is still incredibly fragmented. And so there's just so many different providers. It needs to be able to work with any. And so we have this massive collection of integrations. And then the third bit on the top is more use case specific chains and agents. So this would be kind of like an off-the-shelf way to do a RAG, an off-the-shelf way to do question-answer over a SQL database, an off-the-shelf way to do extraction or something like that. And this lets people get started really, really quickly. And then when they want to customize it, and more like, pretty much all the time when we see enterprises go into production, they're customizing aspects of this, then they can switch to like the lower level primitives. So just on this one for a second, RAG. In the RAG, so RAG is one of those things that it's pretty easy to get it going and to get some accuracy, but it's pretty hard to get to super high or very high accuracy. What's your thought and what does a land chain do for developers? Yeah, I'd say probably the main reason why RAG doesn't always work is, or where RAG fails more often than not, I would say is probably in the retrieval step. Yeah, high level overview of RAG, you retrieve some information, you then pass that into a prompt, to a language model and ask it to answer. And so if I ask a question and the answer I retrieve is not relevant information, then obviously I'm gonna get a bad response. And I think more often than not, that's what's going on. You retrieve the wrong information. Yeah, exactly, yeah. Or I mean, I think even trickier is retrieve partial information. Because I think a component of this, and this gets into like, I think there's a few ways that land chain helps. One of the ways that we help is with this flow engineering. And so that really simple kind of flow of retrieve and then go to prompt is one way. But you can start to do more complex things where you can have the LLM think about whether the retrieved information is all it needs or whether it needs more information. And so now you maybe get into a loop or you get into a branch where if it has incomplete information, then it asks the question or asks the human like another follow-up question or something like that. Anyways, so we provide a lot of like that flow engineering bit around there. So flow engineering you mentioned earlier is a sort of way to help to improve the RAG quality. Exactly, and specifically I would say probably on the generation side of things. But another huge area where RAG can often be improved enormously is on the indexing and ingestion side of things. And so having reliable methods to load the data and load metadata about documents because oftentimes that's really important for providing context about where these documents come from which is often needed to answer questions. And then there's a bunch of questions around how to best split that document. Chunking. Chunking, indexing, just a vector store, some layer on top of the vector store. There's traditional document search methods as well. And so I think on that side we have a, it hasn't been a focus of ours but we probably have the best open source collection of like text splitters just cause it's kind of like a necessity. And then we have a bunch of different document loaders. We obviously have all the integrations with the vector stores. But then one thing that I'm excited about as well is integration with like more managed retrieval services. We have a retriever abstraction in lane chain. It's incredibly general. It's super like we integrate with like Amazon Kendra, Azure Cognitive Services, like all these things that are way more than just a vector store. And so, yeah, whether it be through kind of like our implementation of text loaders or this kind of like very easy off the shelf abstractions and then integrations, there's hopefully a lot of dials for people to play with and tune their retrieval strategies. So, you know, land chain, at least when you started a company it's 100% open source company. And then now you probably have more like less, you know, open source solutions, right? You know, what is the, like what kind of things do you open source? What kind of things do you not open source? What's the thought process and how should the people think about it? Yeah. So we have two main product lines. One's basically the open source. So Python and TypeScript versions. And that's a lot of like, you know, it's an orchestration framework. And so it's, it's everything I just described. The other thing that we're really focused on is Lang Smith, which is a platform that does a bunch of things. Observability, monitoring, testing. There's a prompt hub, human annotation queue. And so a lot of this I would kind of describe as like LLM systems ops. So basically, if you think of land chain and to be clear, Lang Smith works without land chain or with land chain, it's agnostic to land chain. But a lot of what it's grounded in is this idea that the applications that people are gonna be building are these LLM systems, where you maybe have multiple LLM calls or complicated sequences of flow engineering type things. And so there's a lot of pain points when you're building those types of systems. You wanna know what are the prompts that I'm calling? What is the sequence of those prompts? And then what are the, what are the exact inputs and the exact outputs? Because the main new thing here, and I think it's always really important to not get caught up in all of this and sort of like stay grounded and like what's the actual new thing? And it's the LLMs, right? Like that's the main new technology we're talking about here. And so when you put that in the center of your application, it introduces a lot of new like issues and things. And like they're unreliable, they're stochastic. And so like you kind of like need better observability there, you need a new way to think about testing. So we have like a testing and evaluation framework. And honestly, the main value that that provides is I think kind of like a mental model for how to think about testing these things, just like you run software tests, you run LLM system tests, things like that. And so right now it's a lot on kind of like, yeah, that observability and testing. One of the things that we're really excited about though is getting closer to the runtime. So right now I think a way to think about this is you've got LangChain. You've got LangSmith. And basically LangChain is kind of sending data to LangSmith and we're debugging it and we're analyzing it. But it doesn't really work its way back into the application yet. And so a lot of what we're thinking about is okay, how do we get that feedback loop back into the application? Because I think that's where we can really start kind of like super charging some of this application development. And to the question around how do we decide kind of like what's open source and what's closed source, I think a thing like LangChain always has to be open source. Like it's an application development framework, again aimed at developers, not really sure how it would work too well if it wasn't open source. Something like LangSmith, I think there's a much like it's not a perfect analogy, but you could think of like Datadog or something like that. And there are open source Datadog alternatives, but at the end of the day, if Datadog's the best product, it doesn't matter too much whether it's open source or not for a majority of users. Right, right, right. And then Datadog, they have some open sourcing stuff, connectors or whatnot. You sort of treat the LangChain more like that in some ways, right? Yeah, exactly. Yeah, there's just great synergy between the two. We also have integrations with other frameworks as well. So LangSmith has an integration with like the raw open AI SDK. And Instructor is a Python library that builds on top of that SDK. So we integrate with that. And in TypeScript land, we integrate with like Vercel AI SDK. So I think there's a few other frameworks that we think are interesting and we integrate those with LangSmith as well. And then by the way, LangSmith is cloud only or it's kind of a on-prem and then cloud version both? Yeah, so we do both. So the main offerings cloud, we do have an enterprise version which can be self-hosted but that's more targeted at enterprises. Right, right, right. Cool. So let's talk about agent. How does the agent fit into everything we just talk about and then or it's beyond that? Yeah, I think it really nicely fits in with everything I'd say. So like I think an agent, high level, you've got one simplistic way to think about it is you've got an LLM and you're kind of running it in a for loop and you're asking the LLM to decide what to do and then you go back and then you go do that and then you go back and the loop and ask it again to decide what to do. And that's a simplistic view but I think like that's the core idea. And so that... Do you view the flow management stuff you were talking about basically is the sort of the aging or aging? Yeah, yeah, exactly. I think the flow engineering bit, I think where it starts, where that starts to come more into play is like that really simple loop that I just described often isn't good enough to get things in production. You have to be a bit more opinionated about I don't really want this to do anything under the sun. Like first I wanna write unit tests, then I wanna write my code, then I wanna run the code, then I take that error and I plummet back in and ask it to fix it or something like that. And so that's where it's still these cycles but it's a bit more, the developers are kind of like imparting their knowledge of how they develop things into this agent. And so you start to get these more complex flows that are agents. And the core idea is basically using an LLM to interact with the outside world. And again, the core idea of laying Smith is to provide kind of like observability and monitoring and reliability for these kind of like systems. I think the state of the world where we're in now and I was just talking about this at the panel at NVIDIA, which is why it's top of mind. They ask kind of like, what's stopping agents from being reliable? Like what's kind of like next? And I think the state of the world right now is about a year ago their auto GPT came out and I think the whole space, the whole agent space really became popularized. Well, everyone was excited but then when people look into it, it didn't work. Exactly, and I think it's important to note a few things. One like the, and I think this is the main thing, like the auto GPT flow was actually like relatively simple at the end of the day. It really was kind of like a for loop with an LLM in it. But I actually like, I don't say that, I don't say simple in a bad way at all. I actually think like if the flow could be that simple, like that's like, that's fantastic, right? Like you don't have to do all this flow engineering. You can just run it in a loop and that's awesome. And I think, but the fact of the matter is the LLMs aren't good enough at this point or there's a few reasons. One of them being the LLMs aren't good enough at this point in time. And so I think there are a few things that we see people doing to kind of like overcome that and it all comes back to this like flow engineering. And specifically that a lot of the flow engineering is done because the language models are not great at planning and reasoning, at least not to the ability that they need to be to just run them in a loop. It's not quite there. And so a lot of the research papers that have come out in the past year kind of introduce, I would say two styles of things. One is basically a planning module or planning task. And then the other is like a reflection thing. So planning is like at the start you think about what you wanna do as a long-term plan. And the reason that that helps is because the language models, they often maybe, they realize, okay, first I need to do this one step and they can do that pretty well. And then the second step, they maybe struggle with a bit more. And then the third step way more. And so they kind of just like, they almost like forget and that's because they're the context windows like filling up. And also errors compound. Errors compound, yeah. And that gets into the reflection bit which is on the other side which is after the agent run, did it do a good job? And like asking it all to like check that. And so both of those are like examples of flow engineering, the planning step and the reflection step. But these are all just to overcome the limitation that the language models by themselves are not good enough at planning and reasoning. Well, precise planning. Precise planning and long-term reasoning to really run in a loop unassisted. At some point, who knows how soon, they probably will be. And that's when things like auto-GPT in its simplicity become really powerful and really exciting. I think there are a few other issues, but like that ability or that lack of ability to kind of like reason and plan, I think is probably the main limitation with agents. And that's why people are spending so much time with this flow engineering type thing to overcome. Well, so one is a flow engineering can hopefully compensate and make life a little bit better or make the precision a little bit better, right? But another thing is the model improvement, hopefully will compensate that, right? Yeah, yeah, I think flow engineering is a short-term solution for the lack of planning that the models can do intrinsically. So let's talk about the use cases, because sitting where you are, you see so many developers from many, many different sectors. Tell us about the use cases. You are excited, right, in the recent past. Wow, I've started seeing this kind of use cases that are thriving, can you share with the audience about what you are excited in terms of the use cases you are seeing? Yeah, so I think some specific ones that we're seeing starting, so I think there's maybe two big categories of specific ones that we're starting to see become kind of like reliable. One is kind of, or not reliable, but like a lot of interest there. One's kind of like coding style agents, and this is top of mind with Devon, you know. Cognitions, Devon, yeah. Exactly, exactly. And so that came out last week, and I think that's top of mind. I think there's actually a lot of really interesting learnings from that that I'd love to get into later, but coding agents in general, I think, are good. And I think the reason they're good is a few reasons. One, you can execute code, so you have a really tight feedback loop of whether the code you wrote actually compiles or something like that. And then two, the people building things are developers, and we know coding well, and so it's easy to kind of like do that from abroad. The feedback loop is shorter, right? Exactly, yeah. Okay, that's interesting. And then the other style of things is customer support agents. I think with enterprises, we just see this being a massive kind of like focus area of ways to kind of like reduce costs. So let's actually dive deep into both of the areas, right? Let's just talk about the customer support for a second. Yeah. You know, we all saw that people started rolling out some customer support, external customer support based on the journey. Yep. And they're leading A line, all those things. A few jokes or a few things happened, right? People sort of are taken back, right? Hey, does that really work, right? Because there are certain customer support that you don't want to be wrong, not even one out of a hundred times, you don't even want to be wrong one out of the million times, right? So, but clearly, GenAI is not quite there, right, in terms of the accuracy, precision. So how do you think about it? Like a customer support, are we really sort of closing the gate? Like a, what's your thought? Yeah, I think this also speaks to a larger point, which is like, what's the right UX for all these generative UI, or as generative AI applications? Because as you mentioned, it's not going to be perfect. And so how do we design a UX to overcome that? And I think an instructive case is actually looking at Klarna. I think they released some press a few weeks ago about how they've saved millions and automated a lot of customer support. And I think if you read more closely what they did, it wasn't kind of like fully automated everything. There is basically an escape hatch where it could go to a human. And I think the issue is when you're running in a customer support operation at that scale, there's just so many questions that are actually pretty simple and you can't answer. And so even if like the AI by itself can only answer like 20%, that's still a good amount as long as it knows when to route to the human. And so I'm sure they did a lot of work around prompt engineering and flow engineering and working with language model providers to really nail like, hey, I shouldn't give an answer here. I should revert to a human. So I think what you're seeing is that, look, customer support that there are more complex cases and there are trivial cases. It's not about how to get a super, super high precision on the complex cases, right? It's about, there are a lot of low-handing fruit to get them out of the way. So yeah, when Connors, I think they released some data in terms of how much money they saved, some of my friends were skeptical about it because their own experience with customer support, use cases that didn't go as far as well. But you are saying that, well, they might have done some low-handing fruit. They do that well, right? That might still save them a lot of money. Yeah, absolutely. I think when you're running things at scale, there's a lot of low-hanging process automation type things that you can build out and get running. And I'm sure they aim to tackle more and more of that as the models get better and as they build up their own data flywheel. But to start, there's absolutely a lot of... And then where we are as an industry, in terms of the technology maturity, from that point of view, you're fair like you would recommend people, start with low-hanging fruit cases instead of tackling the complex cases. It depends on who they are, right? Like if you're running a customer support operation at that scale, or if you're an enterprise where the scale of internal questions is just so large, then yeah, there's probably a lot of low-hanging fruit that you can do relatively easily now and get out of the way. On the other hand, I think there are, and I actually don't know too much about how they do things, but Sierra, Brett Taylor's new company is in the customer support space. And I'm sure they are aiming for the very hard problems, right? Because they're a startup tackling this space, they want to prove value that way. Again, I don't know actually like what, like I would guess they probably have some aspect of like routing to humans in particular cases. I don't know that for sure. I don't know kind of like what level it is, but I would guess they're way more focused on that. And they should be more focused on that than an enterprise that can kind of like, yeah, there's a lot of low-hanging fruit we can automate. Right, right, right. I mean, for me personally, if anything kind of, I would say after a year and a half looking at a GNAI applications, including customer support use cases very closely, the biggest learning for me personally is, I think I have a better understanding of the boundary of the technology, right? Where it would shine versus, it would be less sort of a perfect technology. And then you maneuver that way, right? So that, hey, for whatever you want it to do, make sure that you don't have the wrong expectation. As long as you have the alignment between the technology maturity and the use case, then that's good, right? If you have a misalignment, then that's awful. And out of curiosity, like how did you build up that intuition for what, you know, where these boundaries lay? I think, first by felling, right? By seeing issues. And then there's also part of that is applying first principle because, you know, certain questions, right? Even human, it wouldn't be hard. I mean, where the technology is, you know, it's not going to work, right? So you kind of start thinking about planning, be thoughtful about what use case is to attack, right? Unlike like a year ago, right? You know, the aging, the child GPT, everyone was excited, right? You know, from whether you are executive, you are engineer, everyone is excited, right? So a year later, I wouldn't say, I think people are still very excited, right? You know, you probably can tell. But I think that the excitement, you know, is accompanied with, you know, some failure, some experience, some, you know, trying the error, sort of a lot of the trial and error experiences, right? So, you know, when you ask me this question, I would say it's harder not to know this because you have all those trial and errors, you see all those things, right? That's sort of my answer to that. Yeah, no, I agree with that. And I think the thing, like, I think the thing to also remember is, like, you've probably interacted with this and seen more things than 99.9% of the people in the world or something like that. And so as, like, I think, you know, we're in Silicon Valley, we're in the Bay Area, people are talking about this every day. That's not often the case for a lot of enterprises or people outside this region. And also, I think the other interesting thing is the model capabilities keep on changing as well. So I think that- Sometimes it's for good and sometimes the other way around. But yeah, I think this idea of, like, probing out, and I think we see this with, it's slightly different because I think you're coming at it from the angle of, like, how to build these things. Although, but I think it very much applies to how to interact with these things as well. And I think, like, the everyday consumer is still learning how to interact with the chatbot and what is this good for and what is it not good for. And so, you know, I think there's UX things you can do even as simple as just, like, showing a list of, like, questions that you could ask the chatbot to, like, prompt the user and figure it out. But I think people in general are still just figuring out how to interact with this technology. So let's talk about the other categories. The use case, you are excited about the, you know, for developers, right? Cognition, and one question is, you said that there was a lot of interesting learning I would love to hear from you. And then the other thing is, you know, everyone's talking about AI replacing developers. What do you see, right? You know, can you share with the audience what's your perspective on this? Yeah. So I think there are a bunch of interesting learnings from the Cognition Releasers. I think actually apply to agents more generally, not just coding agents, maybe I'll tackle the latter question first. Like, I think there's a lot more to kind of like software engineering than kind of like just writing code. There's a lot of scoping things out. And I mean, there's like the low level implementation of writing code. There's the high level like architectural decisions which often involve kind of like needing context from the appropriate kind of like stakeholders and synthesizing that. You know, I use chat GPT a bunch to write initial drafts of like a boilerplate backend or a boilerplate kind of like function in JavaScript or something. And so I'll use it to get started super quickly. But then I'll iterate on it from there and there's still like a lot of like, you know, I still spend a bunch of time thinking about the architectural decisions and things like that. So that's a long way of saying, I don't think it's gonna completely automate a way kind of like engineering. It's possible engineering changes or engineering. Developers doing things way more than just writing code, right? Yeah. So thinking about architectural requirements, you know, there are a lot of the nuances beyond the writing a few lines of code. Yeah, and so I think like, you know, and so how like, what exactly does that manifest in? I'm not actually sure. Like, is it just everyone, you know, is 20% more efficient because they can use AI to write 20% more lines. I think there's some, I think that's probably like, you know, the most easiest to grok case. And I think another really interesting case that just maybe a bit further out on the spectrum is basically like, you know, maybe there's, maybe basically what we turn into is like code reviewers, basically. Like, you know, when I was managing a team at Ken Show, a lot of what I did was code review. And, you know, maybe these AI are more junior coders and it's my job as, you know, an engineer now to be like a tech lead almost. And so maybe that's kind of like some of the dynamic that it takes on and engineers turn into more like tech leads and with managers. Or managers, yeah. I still think they need to be technical though, because right, like, I think like, you know, like a lot of what you're doing, and I can really suspect Devin as well, but a lot of what you're doing is like relating whether they're doing things correctly from like a technical perspective as well. And fortunately at the moment right now, you don't need to have like a talk with your AI assistant about whether they're getting a raise or something like that. So there's probably some managerial aspects that, you know, aren't as relevant, but tech lead, absolutely. And so maybe getting to like what I think was really interesting about the cognition kind of like release. And I think speaks to a really interesting UX for agents. It's basically they, you know, it would do a bunch of things. It seemed like it was running for like five or 10 minutes, right? So it wasn't just like a single alum call. It wasn't just like a really quick response. While it was doing that, you can kind of see exactly what it was doing. So it would log all its steps and then you could actually rewind and you could edit it. You could edit it at a point in time. You could go in and you could change the code files. You could say, no, you're doing this wrong and correct it. And so I think that like rewind and edit is really, really interesting because the whole issue with some of these applications is there's kind of this weird tension where you need a human in the loop in some form. Because if you just let it run wild, then it's not going to produce anything reliably. But if you have a human in the loop too much, then it doesn't actually provide any time savings for whoever's using it. And so I think this was a really interesting way where you didn't have a human in the loop, but you had a human like on the loop, kind of like watching the agent, but then they could go back. Optionally in the loop. Optionally in the loop, yeah. And so I think this idea of like being able to rewind and then edit in agent state is really interesting. And so at a more technical level, we've actually added a lot of, it like coincides amazingly well with some of the ideas that we've been working on in LangGraph, which is just like agent runtime built on top of LangChain. And so that functionality already exists. And then we're working on a few, like, it was great because I think the Devon demo is kind of like a perfect, you know, whenever we build things in LangChain, we always build demos so that people can see, okay, oh, this is what I'm supposed to do with it or this is why it's useful. I think this really helps motivate it. So I didn't get it. What's the relationship between LangGraph and then Devon? So Devon had this like rewind and edit functionality. And basically, we've made LangGraph kind of like stateful and support basically this type of rewind and edit functionality. And so I think this type of rewind and edit functionality is really useful, whether you're doing coding, whether you're like writing an essay, whether you're browsing the web, like this ability to go back and correct is, I think. So you are saying that people can use a LangGraph and then Devon together? Not even that. I'm saying you can use LangGraph to build applications that have like, you know, like a Devon sort of thing. Yeah, exactly. Got it. So, so Devon give you the inspiration data. This could be the future of how, you know, humans or developers engaging with the AI. That's how you think about it. Exactly right. I think one, I think maybe the best thing they did was this UX. And because I do think a lot of JNAI applications, the main issue is the UX at this point and finding the right UX. Yeah, I think you call it a UX. I think the way I think about it is, you know, maybe the same thing, how you engage with the, you know, JNAI, this JNAI is a wonderful machine, right? In some ways, but how do you engage with it? When do you engage it? I remember when JNAI first came out, you know, I wrote up a blog, I said something along the line that suddenly everyone has a five-interns, almost for free, right? Maybe $20 a month. But guess what? Most of us do not know how to use those five-interns. And you and me and then pretty much no one knows how to use those five or maybe 50, right? It depends. And, you know, we need to figure out how do we engage with the model so that we can get those five-interns for free, really utilized. I think what you mentioned is, this is the A inspiration, A way to think about how to, very, very cool. So, going back to my original question, right? You know, is AI going to, you know, replace developers? You know, you are saying that, look, you know, certain part it will be, but you know, a bunch of what you do day to day, what your developer do day to day, you don't see that being replaced by JNAI or even GPD5, GPD6. You don't think that's the case? I don't think so. I think like it'll definitely work its way more and more into the application and maybe, yeah, as mentioned, you know, maybe we take on more of a role of like a tech lead or, yeah, a manager, a supervisor of these agents. So, just a little bit further, your advice to kids, you know, like applying for college or, you know, or should they still learn coding? I think coding is a great way to build your problem-solving abilities and I feel like problem-solving abilities will never go out of style. So, probably. So, you have a different view from Jason Huang. I'm just kidding. You know, I don't think he's saying that don't learn. And he actually clarified very much, you know, hey, if you're passionate about doing that, but if you think that coding is the only technique, you know, do you think you know and you can get a job from whether Google, Facebook of the world, that they may be gone, right? So, I mean, yeah, more seriously, I'd say like one, like, you know, like I actually didn't start coding until sophomore year of college. So, I'm very much a believer that you do whatever you want as long as your passion as a kid. And like, if it's your passion, then you'll do it intensely. But maybe a little bit more practically. I think there's so much alpha and just playing around with these models and understanding how these models work because they're gonna change how things work. And so, if you understand how they work, you're gonna have a leg up over people. And so, I think like, yeah, I would highly encourage everyone to just interact with these models, probably directly to start, but then, you know, you'll start noticing it in your Gmail. And if you know how to interact with it directly, you'll know how to interact with it when it shows up in your Gmail or your code editor or something like that. So, lastly, I wanted to talk about something a little bit future-ish, right? Maybe not so future-ish, you know. When are we going to see AGI happen? I don't know if I have a great answer for that. I think very much depends on the definition of AGI. I mean, I'm probably, you know, 10 plus years out, depending on the definition. So, I'm 15 plus, I'm not, I don't think it's here now. I think it will look like hooking LLMs up to other systems. I think it will look like language models today, but hooked up much more intelligently and with just like better innate kind of like planning and reasoning ability. So, that's AGI, you know, a little bit further out in your opinion, but something more concrete, right? CNBC had an article in December, you know, the title is along the line that, hey, 2023, great year for GNI. We'll know about that. Lots of profits for NVIDIA, lofty experiments for the rest. That was literally the title, right? When do you think we would get out of this, you know, lofty experiments for the rest? Instead, we would have lots of profits for the rest. When do you think it would happen? I think we're starting to see some of that. I think like maybe not, not as much profits as NVIDIA, but like, you know, I think Klarna's saving a bunch of money with their customer support, but Sierra is a very legitimate kind of like business that's going to make a lot of money and save companies a lot of money. We did a case study with Rakuten, you know, they're building an internal platform on top of OpenGPTs, which is an open source project that we're working on that's going to let, you know, a lot of their internal users access this technology and create their own kind of like knowledge bots. We did a case study with Elastic. They shipped an assistant, an agent in production on their thing. So we're seeing things starting to be shipped in production. And so I think we are beyond just the experimentation phase. I think we're into the phase where we have, where we have initial versions of these in production. I think the phase where we're not and where maybe we will see more of these things, I think like the really transformative things will probably be these like agentic type systems. And I don't think we're quite there yet. I'd say, but you know, Devin's a great example. Devin's also not in production, right? Like I think they're just in some beta or some alpha. End of this year, I think we'll start to see some of these become widely available. There are some. Lindy is a great example of an agent-based system that's in production. I don't know if they're, I don't know how they're doing commercially, but their tech is extremely impressive. So, end of this year. Any other advice is to start up founders, entrepreneurs, or people in Silicon Valley, outside of Silicon Valley, they want it to be part of the wave. Any other advice is? It's still so early on. I think sometimes people say, oh, like I'm getting into it so late. No, you're not. Like it's so early on. Like there's so much that's changing. No one should worry about it being too late. If anything, maybe a little bit too early, that's possible, but not the other way around. Exactly. So just get started building and, yeah. And you didn't mention earlier at the beginning of this conversation that it took us a few years for, iPhone got out and then people started seeing Instagram-ish kind of the interesting applications, right? Yeah. On the consumer side, right? Have you seen, are you seeing any interesting, the sign of consumer applications emerging? Everything that I've seen, I'd say, is more on the creative side and is not building on top of models, but controlling the models themselves. So, some of them are using, by the way, some of them are using luncheon. So yeah, well, so I think the standouts in this space are training their own generative models. And this would be like character AI, mid-journey, Suno, in text, image, audio. Then I think there are, and so I think those are the only clear, kind of like breakout ones. I don't think we've seen, like there are a bunch of other things that have attempted to be, like the GPT stores, one example, apps built on lane trains, another example. None of those are at the scale of character AI, mid-journey. So I don't think we've seen the breakout success there. Any prediction when we will see some of those killer applications emerging? I think one of the missing bits that we're really excited about as well is memory for these applications. So I think just like, if you think about all the, you know, personal assistance of the futures in movies or whatnot, like- You have to have some memory. Exactly. And I think we're still figuring out what exactly- What is the solution for that? You know, we've got some ideas that we're prototyping, I think, but I'm not claiming that we know. I think it's very, I think it's very open-ended. I don't think many people are doing, there's been a few interesting academic papers, but not much more in this space. Vector search and similarities are probably part of the solution, but not the full solution. Possible this is solved at the model level. Possible this is solved at the system level. I think different teams are taking different bets, so. So my last question, just beyond that, you wonder where you and I are able to influence in the near term, but just the large language model, right, opening up the world. What's your sort of the prediction of their trajectory? Are they going to be just like Jensen Huang, right? You know, every year he came out with a far bigger, far better GPU. The model is going to be far bigger, far better, you know, not just bigger, right? Bigger is the means to an end. Far better model every year. Is that what we are going to see? Well, what do you think? Yeah, I absolutely believe in that. I think better, cheaper, faster. I think anytime we're building or thinking of an application and objections raised like, oh, this will take too long. This will be too expensive. You know, I think, yes, right now that's true, but in the year I think all those objections kind of- So you would think, you know, the marginal cost for the gen AI AI would approach zero over the next few years. So that would- I mean, like how much has open AI cut their costs in the past year? Like, I don't know, but it's going to be like order 80% or something like that. So my last question is, what would the world resemble in that state, right? You know, the cost of AI, cost of a gen AI is several auto-magnitude that lower in a few years than today. What would the world look like? That's my last question to you. I think things like agents become way more viable. Like, you can run all these- You can run it in a loop and it's just far more fast. Like, right? Sophisticated applications. Sophisticated applications with multiple LLM calls. Highly personalized per user, because now you can start to think about fine-tuning per user or at the very least, using a bunch of few-shot examples per user. Yeah, like sophisticated multiple LLM calls, perhaps multiple different LLMs as well, mixing and matching as we figure out which ones are better for which and maybe really small, cheap fast ones are good at classification and then personalized. So my question, you know, there are two-fold, right? One is the technical side, but the other part is just we, you know, humans. Like, what would the world look like? Like, any imagination? Yeah, everything. I mean, okay, you wanna think, if you wanna think weird, like, everything's an agent. Like, you know, right now we interact with like a website. Rather than interact with a website, you'll interact with an agent. Rather than interact with an app on your phone, you'll interact with an agent. And I think the reason for that is it just does more for you. Aging, meaning your assistant. Aging, meaning your assistant, yeah. So you get your assistant, personal assistant to go shop, go do shopping, go do things, collect, do research for you, that's the world that we are going to live in. Yeah, and I think one big question here is like, how much of it will be dominated by like one personal assistant? Like, maybe you're Siri or something like that, versus like, will you have a GitHub agent and an Expedia agent? And, you know, who knows what, well, like smaller specialized. In your take case. And my take is more on the smaller personalized agents. Like, I think we'll have, yeah, very focused on a particular domain with access to a particular data source and particular prompts and particular learnings about the user. Yeah. So Aging is the new app, basically. Yeah. And then, but it's new app, but it's far more intelligent, right? It can accomplish a lot more. Yeah. I think that's exactly right. Thank you, Harrison. This is a wonderful conversation. And then we talked so much about, you know, the AI, gen AI, open source, luncheon, and of course the world should look like, you know, once that AI infrastructure, GPU costs that goes, you know, several other magnitude lower than today. So thank you for coming over here. Thank you everyone for watching SuperCloud Six.