 OK, welcome, welcome, everybody. Hi, I'm Daza Greenwood from MIT Media Lab and also executive director of law.mit.edu, which is the convener of today's workshop, the eighth annual MIT Computational Law Workshop. I just want to start by saying, having done these things since actually the late 90s at MIT on this topic of law and technology, I honestly believe this is the best program yet. And that's owed largely because of a breakthrough with widely accessible, generative artificial intelligence and its applications for law and its impact on law and legal processes. So with that, let's get right into it, shall we? Jesse, are you with us and can you come off video mute? I am. Thank you for handing over the stage and for setting this up, Daza. Very excited to share with you guys what I've been building. I think you'll find it very helpful for some of the legal use cases that you've been considering. If you will allow me to share my screen, I'll jump right into a demo. Absolutely. Go ahead and hit screen share and let us know if you have a problem. Great. And Jesse, by way of introduction, am I correct in saying that you previously worked at OpenAI and were involved with ChatGBT? Yes. So I'm the co-founder and CEO of Multi, which is the startup that is responsible for this tool, which I'm about to demo to you. This was started back in May. Prior to that, I was working at OpenAI. I worked on large language model infrastructure, did some early work on ChatGBT dialogue systems and grounded question answering. I also achieved a new state of the art in machine translation using large language models and was a major contributor to OpenAI's theorem proving release, where they applied large language models to mathematical theorem proving inside of proof assistance. So I wanted to echo a sentiment that I heard in the last presentation, which is that we need systems that are going to be able to deal with the unprecedented amount of scale that these large language models are going to enable. And that's not scale in terms of parameters, but scale in terms of the volume of text processing that has to be done. So I think that that same consideration applies equally well to all knowledge work in general. The kind of intelligence that we see in large language models right now is kind of like an alien sort of intelligence, which is a pretty good approximation to a somewhat unreliable teenager. And as these large language models become more and more commonplace, we will begin to see large amounts of knowledge work, not just knowledge work inside of the legal field, but knowledge work elsewhere, which involve processing large amounts of text, which involve operating software tools, begin to be more and more automated. And so the problem becomes, how do you orchestrate that kind of knowledge work automation at that kind of scale? And that is the problem that Multi is trying to solve. We are building a software platform so which anticipates this future where there is this massive abundance of near human level intelligence and automation. And our first product, MultiFlow, which we released back in November, provides a visual, intuitive, and low-code interface for people to assemble AI-first workflows. And let me give you an idea of what I mean. So I'll give you guys a bit more background on myself. And this kind of showcases some of the features that we brought online in MultiFlow recently that might be especially relevant for people working in the legal field. So we recently added a way for our users to upload PDFs. So this contains an uploaded version of an unupdated resume of mine. So this is a pretty long document. And with resumes, it's kind of hit or miss whether or not they'll actually fit inside the context window of a large language model. But we've gone to the trouble of doing the kind of plumbing that Daza mentioned during his talk. And we can actually handle question answering over documents of arbitrary length. So we recently added a document Q&A block which takes in a document and a query and provides a bullet-pointed list of answers which were extracted from this document. So we can ask, what schools did I go to? It kind of says here. But let's pretend it's further down in the document. We can also ask something like, well, what was Jesse's SAT score? And we can also ask, what are Jesse's notable publications? So what's happening here is that I am building a program in a visual programming language for which multi-flow provides both a user-friendly front end and also a runtime that keeps track of all the state. And we can step through the execution of this program by clicking Run. And this is all powered by large language models underneath. And the amount of abstraction which is inserted between the user and the actual large language model API calls is entirely up to the user. I'm operating a fairly high level here. But later on, we'll see an example where we actually get a bit closer to the metal in terms of how much we're actually micromanaging those large language model calls. So let's take a look here. So it correctly extracted the fact that I went to UCLA. I got a bachelor's in mathematics in 2015. I got a master's in math in 2018. This resume is not up to date. I actually got my PhD a month ago. So it also correctly sees that my SAT score is 2280. And it correctly extracts the fact that some of my major work has been in formal theorem proving and the applications of machine learning to automated theorem proving in mathematics. So I'll pause here for any questions. Thank you. Yeah, let's go. I actually use multi-flow to write the PhD. That's awesome. So if we wanted to, say, generate some copy. Right. So here's another example where a visual front end for prompt chaining and for being able to recursively assemble prompts and then feed them into language models and use them to produce a complex result might be useful. Suppose that you need to write a short blog post about some arbitrary topic. So in this case, I've asked it to create a blog post about the fact that clowns across the world have decided to go on strike. So what we've done here is we've combined few shot prompting where we show the language model a bunch of examples with the kind of structure of saying, well, now we're going to few shot prompt an introduction. We're going to few shot prompts some body paragraphs. We're going to few shot prompt a conclusion. And we use the topics which are extracted from the language model call as prompts for actually producing the actual body paragraphs, which will be inserted inside the blog post. And so you can see that once this entire thing is run, in fact, we can just re-execute the entire thing. So we see that it regenerates this blog post and it also creates a bunch of images. And these images can be prompted so that they arise in different styles. And this is something which is possible through the string formatting that we have through our text boxes. So as you can see here, the way that we manipulate text inside of this interface is that we use template variables, which are denoted by these double brackets. And so when you surround a piece of text with these double brackets, they become variables which are then exposed as ports. And you can pipe input into these. And so all we're doing here is we're concatenating the copy about clowns with some styling prompts. And then this becomes the input to a call to a stable diffusion model. So now let's maybe jump over to a use case that might be particularly interesting for people working in the legal field. So here's an example of a flow which performs dialectic reasoning on some topic. So suppose that you wanted to analyze the pros and cons of some controversial topic and then synthesize them. And you wanted a highly interpretable trace or audit trail of what the language model was thinking throughout that entire time. So what you can do in this case is you can just ask it to create bullet-pointed lists of pros and cons and then ask it to argue against itself. So I'm going to try something different here. So I'm going to try building more housing in San Francisco, my favorite topic. So you can see here that what's happening here is that we've created the instruction to create a bullet-pointed list of pros or potential benefits of the following topic. And this is input to a language model API call. This is our text generation block. There are some settings here where you can control various parts of the OpenAI API. And you can see that here it generates a list of pros. We can then ask it to expand each of these short bullet points into a full-fledged paragraph. So you can see that this is piped into this text generation block. And this provides a paragraph which expands on each of these points. Finally, you can ask it to produce a point-by-point rebuttal for each of the claims, which are made inside of this essay. And then you feed that into another language model API call, and that creates a rebuttal. So this kind of structure-chained reasoning is especially important for high-touch applications or particularly high-stakes applications, which involve lots of language, like legal, because this provides a clear audit trail of the model's reasoning, what the ingredients were that went into each of its choices. And it also gives you the ability, or at least the interface that we have here, gives you the ability to quickly rewire the way that the model is prompted in order to achieve the outcome that you want. And what's happening at the bottom of this flow here is that we've just gone and reversed the chirality of this. So we've asked it to come up with a list of cons instead to expand that into an essay and then produce a rebuttal. And then from these, we can actually So one thing that we could do here is we could ask the model to take these two things and synthesize them. So this is actually something that we can do right now. So OK, so let's pipe in the text above and let's format it so that the model knows that these are two different pieces of text. So this is rebuttal 1, and this is rebuttal 2. And let's separate these with some kind of demarcation. So above, you have been given. So this kind of prompting works with the instruct models on the Open API. Above, you have been given a rebuttal of the pros and cons of topic, even a rebuttal, the pros and cons of topic, and a rebuttal of the pros and benefits, cons and drawbacks of topic. Synthesize these into an essay with multiple paragraphs. And then let's do a little bit of prompt engineering and tell it to write an eloquent essay, an eloquent, well-articulated essay. OK, so with this in place, we see that due to the formatting, which I mentioned earlier, there are three inputs. So we need to pipe in the first rebuttal, and then we need to pipe in the second rebuttal, and then we have to pipe in the topic, which is all the way back here. And finally, we can run this through the text generator to get a final output. And now let's rerun this thing again. So this gives you guys an example of the kinds of rapidly and increasingly sophisticated use cases that you can achieve with complex prompting and a tool like this. And what our front end gives you is the ability to inspect the intermediate outputs to rapidly change the prompting style on the fly and also to deploy these to an API. So if you're a developer who wants to integrate this kind of technology into your own application, we are actually constructing a function in a visual DSL for programming large language models. That's why you see these input and output text. And so what we have here is a function that takes in a single input, which is a topic, and produces an output, which is this well-argued, articulated essay, considering very carefully all the pros and cons of this topic. And this can be deployed to an API that you can call from inside of your own application or also to a web app that you can just create and share and which hides all of the intermediate outputs. We're standing. And so I hope you can all see why it is. I wanted Jesse to share this. So a lot of us have got at best a concept of a very flat interface. I mean, the great thing about ChatGPT is that it provided very wide, almost population scale, immediate access to the technology. But it's hardly the ceiling of how this technology can be configured and composed and integrated into other systems. And so thank you so much for showing that, Jesse. I forgot to mention that I'm bending our rule a little bit here on product demos. Obviously, MIT doesn't endorse this product. This is for educational purposes to see what's possible. And let me just ask, let me start with, we're going to have to move to the next session pretty soon. But Dan or Michael, if you have any reactions, questions, or comments, I invite you to jump right in. Go ahead. I would just say this is what, when we talk about why you want to use an LLM and what the results that we've showed mean, this is what we're envisioning their actual use case, right? Like not a human directly asking GPT for answers, but something like this, let's say the Illinois Legal Aid Online, which Dan works with in his capacity at Chicago Kent, instead of building a rules-based triage system, which is what the ILIO does today to answer questions about, let's say, landlord-tenant disputes for tenants in the state of Illinois, would replace the rules-based system or parts of that rules-based triage system for people who are trying to deal with their legal problems with something like this. It would be inside of this larger ecosystem or, if we're being honest, a real product. And this is just a component in a product, not a product itself. Yeah, absolutely. I think that's the future that we're heading towards. And eventually, systems like this are going to become more and more widespread. And we're going to be living in a world that is going to be orchestrated by language model programs like this. No, I just thought this is great. Congratulations on this. This, I think, is a great follow-up from what we had shown, because this kind of shows you where you can take all of this, particularly I like this idea. I mean, just for an education perspective, teaching my students about, when we do legal composition, we think about the relative weights or merits of arguments in law. This is a way to enable them to see those kind of, I don't know, the battle royale between these arguments or what have you. I can imagine just in teaching legal writing. You asked this question earlier about what's the future of the bar. I mean, how about we have a measure of people's performance, but you can use the best tools available to then solve some theoretic client problem and that's your demonstration. And so if this is the tool you have, then you get to use it, and you get to use any tool. And that's the world I'd like to head to, which is you use the best in class tool to solve the people's problems. Anyway, I don't want to turn this into a revival, but it'll happen very quickly, otherwise. Thank you. Yeah, and I didn't get a chance to show you, but the document Q&A feature, which I started the presentation with is actually very useful for analyzing contracts. It nails most questions like for a purchase agreement, what are the obligations to the buyer and seller and so on. So there are lots of use cases. I'm going to drop a link to the website that I was using in the chat. And I encourage you guys to sign up for the wait list and come check it out. Thank you so much for giving everybody an introduction to what this technology is and what's possible. And not only are tools evolving, but the technology itself, just put your seatbelt on for Claude from Anthropic and for GPT-4. So this truly is the beginning.