 Good afternoon and welcome to another episode of MIT's Computational Law Idea Flow series. And today we're going to actually pull the curtain back, in a sense, on some of the research happening at home in our lab at MIT Media Lab from one of our very own PhD students, who also, I think, very interestingly, is doing a concurrent degree, getting his law degree at Harvard. So we've got a really good example here of law and technology kind of hybrid of the warrior king type that we always advocate. And so, or maybe philosopher king would be a better example of the blend. And his name is Robert Mahari. You've heard him before in this series in the past as part of a demo of a different research project on AI-generated music and all the interesting copyright implications of that. But now we're going to delve in even more on point for the topic of computational law and take a look at two very interesting research projects that Robert's been at the center of in the Media Lab that exemplify how to surface actionable insights from legal data into very interesting different ways that help us sort of scope out the answer to the fundamental question, what is or what could computational law be in the first place? And so with that in mind, I wanted to say also that Robert is a member of our advisory board of the computational law report and just a cherished member of the community. And I'm so glad that you're willing to and able to share some of your kind of behind-the-scenes research with us today, Robert. And so please go ahead and to the extent that you'd like to fill out your kind of bio a little bit from my rather single point of view introduction of you, I invite you to introduce yourself more fully and to share your projects with us now. Thank you. Thank you so much, Dazza. As always, the introduction was very flattering, so I will add nothing and leave it at that. But it's always a pleasure to be with this group and I'm glad to be back. And so like Dazza said, so I'll share two projects that were two research projects that we're working on. I'll follow maybe slightly unusual format in that I'll kind of give you a high level overview of both projects and then dive into kind of some of the details and some of the technical details. But I would really appreciate, you know, stop me along the way if anything is unclear or also if you just have questions or marks, these are both ongoing. So we're still, you know, very much like looking for feedback and trying to integrate new feedback. So with that, let me see if I can share my screen. That seems to be working. And then let's see if I can do this. Cool. So now you guys can see that, right? Yeah, so the first of the two projects relates to so both projects actually focus on legal citation. So in common law jurisdictions like the United States, like England, judges and lawyers rely on citations to precedent to build their arguments. And so the first project says, well, you know, that's really similar to how scientists operate, right? We have this knowledge system that builds on prior knowledge through citation where the citations serve to compress and store knowledge for the future. And there are patterns that we've observed in scientific citations. So there's this discipline known as science of science that is sought to understand how science is done, why certain scientific ideas become popular, why certain scientists become popular. But we don't know if these patterns that have been identified are unique to science and all the kind of weird historical accidents that made science operate the way it does today, or whether there's something fundamental about the human pursuit of knowledge. And law, which also relies on citation, appears to be a really different system, right? So a couple of the differences between law and science are the strictness of the hierarchy, right? You, if the Supreme Court says something, that goes. If someone at MIT says something, you can disagree with them. There's no kind of strict hierarchy. Judges don't get to pick what disputes they work on. There's a little bit of specialization. We have bankruptcy courts and tax courts, but beyond that, you don't get to pick what kind of disputes arise in the report room. Scientists go out and they find problems. They specialize and then they're restricted, but for the most part, they're free. Judges also serve for life, and scientists very much come and go. The judges we're focusing on are federal judges. So judges that are appointed by the US president and they confirm by the Senate. Of course, there are all sorts of judges in the United States too, but those are the ones we're focusing on. We identify three, so we actually checked all sorts of different citation patterns and I'll tell you about more than just these three, but kind of everything that we compared between law and science, we find similarity. Three kind of interesting patterns are this law of preferential attachment. So if you've been cited highly in the past, that is indicative in science that you'll be cited a lot in the future. You might imagine that this makes it difficult for kind of new ideas to compete sometimes, right? Because if there's one thing that gets cited all the time for a certain proposition and then that's kind of set in stone, we find the same thing in the law. The second, maybe more surprising, and I think for scientists more upsetting is this rule of random impact. Your most impactful publication can come at any point in your life. If you have a really impactful publication early, that could be it. It can come really late in your career. There doesn't seem to be, for example, a tremendous value in experience in terms of just being able to attract citations. And finally, we have these citations that go unpublished, yeah, unsighted for a long time, referred to as diamonds in the rough or sleeping beauties. And we find them in both law and in science, and I'll give you kind of a fun example of legal sleeping beauty a little bit down the line. So now switching gears a little bit. Whoops, let's see if I can. Yes, so switching gears a little bit. We have these legal citations, right? And so we can do something completely different with them. Namely, we can think about, I told you they're critical to making arguments, so we can think about, well, can you predict what legal citation to use in order to make your argument? And to formalize this a little bit, so what do these citations look like? They're not just judicial opinions as a whole, right? You're usually citing something specific in a prior judicial opinion, like a certain passage that might represent a legal text. And so the question that we looked at for the second project is given a legal argument that you'd like to make. Can we predict using natural language processing, and specifically this model known as a BERT model, can we predict the missing precedent? And the way we approach it, and you'll hear more about this in a moment, is we say, well, in a way, we can kind of construct many arguments from judicial opinions. By taking the introduction, the conclusion as global context that tell us what's going on in the case in general, and then text before and after a thing that is cited by somebody else, right? So for example, text before and after a legal text, a test as like local context that tell us about what the judge is writing about in the instant case. And so we can mine these kind of many arguments or many opinions from lots of different judicial opinions. We use 1.7 million federal judicial opinions to do that, and train a model. And then we can take that model and give it kind of more reasonable, more real legal arguments. And we find that it does tremendously well at predicting missing legal precedent. And this has interesting implications on access to justice on the one hand, but just like judicial efficacy and the efficacy of lawyers. So two different projects both related to citations. I will start digging into the first one and some of the technology and kind of research we've done. But I want to deposit this stage if anyone has questions or remarks, feel free. Yeah, including just clarifying questions, like to make sure you understood what Robert was saying or anything else at this point. But he's not kidding when we say we're about to dig in. So like this is a comfortable platform to get off and stretch your legs for a minute if you want to. I've got something real quick. I mean, Robert, you might address this as we're about to jump in if you do, obviously. If you are about to do that, obviously, just ignore me. But yes, I have a background as a scientist like by degree in chemistry and physics and published loads of papers and then got tired and wandered off. And one of the other things I see is the difference between hard science at least in law is the desire for or the kind of presence of the notion of objectivity in the scientific domain, at least in an epistemic sense. The idea that there seem to be some facts that are knowable in the epistemic domain of science, whereas everything in law seems to be very much relative and subjective and more on sort of circumstantial. I wonder if that's also something that's informed this similarity in different seeking that you've been doing across the fields. So we haven't thought about this explicitly. And I would wonder how many judges, if you poll judges and you ask them to what degree do you think the law is something subjective? I would think that a surprising majority would say, it's actually very objective. Like we have these statutes and we have case law and we're highly constrained and you can have like there's a separate like moral question, like are these laws good or just or something like that? But like it is what it is and seems reasonably kind of like it binds you. At the same time, right? Like it binds you in a sense that certainly feels more arbitrary than like laws of nature, right? Things fall because they fall and there's gravity and there's nothing you can do about that. But there are things you can do about the tax code. And so it's interesting in that sense. But I would add that to the list of differences between the two systems that make the similarities all the more surprising perhaps. Nice framing, good looking forward to the rest of the talk. Thanks. I think I'll dive into it and then people can interrupt me as we go. So I mentioned already, we have legal citations kind of core to making arguments here's a nice example of the chaotic convention, this convention, I think it happened in Minnesota, goes completely off the rails, it's in a hotel, someone brings in a mule, someone else brings in an alligator, people are firing guns. And so the Minnesota Supreme Court is tasked with resolving this case and saying, well is the hotel liable? And so then they cite this case in New Jersey that says, well, yes, if a hotel knows of the danger that its guests are causing then when they're firing guns, that seems like pretty clear then it's liable. So done the hotel is liable. This is like a pretty typical example of how citations are used and we'll use these kinds of examples throughout. So I've already talked about kind of the differences between law and science, although they both use citations, they have lots of differences. We just talked about an additional one, but we have this hierarchy, the fact that scientists are free judges are confined. Another big difference is just the number. We have relatively few judges. There's like 6,000 ish district court judges, appeals court judges and Supreme Court judges in US history. And there's millions of scientists. And then there's this lifetime appointment. And so then like diving into the similarities and I'm sorry that the graphs don't look so nice. It's weird, but the first thing, hopefully it's still clear enough. The first thing is the number of opinions and the number of scientific publications have both been growing exponentially. So you can kind of make it out so the axes are on the log scale. But, you know, exponential growth is always a little bit surprising because it can't continue forever. Like kind of by definition, it can't, you know, it consumes everything, right? So like why these things have been growing exponentially for the time being kind of remains unclear, but both systems are growing exponentially in the time that we've been observing them. Second, there's a mostly constant number of citations received by an individual paper. So over time, over the hundred or so years that we've been looking at it, the number, the average number of citations is constant. You see the drop with scientific citations towards the end. That's simply because they haven't existed long enough to get cited, right? So it takes some time for you to accumulate citations, but you can see kind of the number of citations has remained constant despite the exponential growth, right? And so what does that mean? That means that there's a growing number of references contained in each publication. So that means that in order to make your argument, you have to cite more things. And so that suggests that kind of the complexity of arguments and the number of citations, number of maybe sub-arguments you're making has grown in both science and law. So this is kind of relatively basic. So we can kind of move on to more sophisticated comparisons. And one such comparison is the impact that individual authors have, right? So clearly, an individual judge has tremendous impact on anyone that comes before her, but there's this kind of idea of impact over time, right? Like how much are you cited? How much are you remembered? There are judges that come up in law school classrooms hundreds of years later, even though others that other contemporary judges are forgotten, the same as very much to a scientist. And so we can measure this individual impact and it's known as a Q factor based on work done by some other folks. Reason, yeah, there. And so we can measure this Q factor and we can kind of stratify judges and scientists into groups of high, low and middle impact authors. And we see that in both cases, if you're a high impact author, you simply publish more. And so there's a question about correlation and causation, but the fact is that in both systems you publish more, this is especially surprising in law and I don't actually have a good answer for it yet because nominally you're kind of assigned cases at random. And so the question how it's even possible for one judge to publish significantly more than other judges is kind of not super clear. I mean, the obvious answer is like, some judges are judges for longer, but that doesn't quite answer it. That doesn't fully explain it, but in any case, so you publish more if you're high impact. Second, the average impact of authors is relatively constant over their careers. Again, surprising in law because you can get promoted from the district court to the court of appeals or to the supreme court, we would expect your impact to rise quite significantly, but it seems like for whatever reason, if you look at the whole system, either you're kind of high impact from the start or not, and that is true in both cases. And then finally, I mentioned this random impact rule. So essentially your most impactful publication come at any point of your career. The charts are a little confusing. So the graphics, essentially what we do is we plot the actual kind of timing of the highest impact publication and then we compare it to a random timing. So we shuffle the timing for each author's sequence of publications and the fact that the blue line and the orange line coincide mostly suggests essentially that there's not a lot of difference between the random shuffling and the actual sequence of publications for authors. And that suggests that there's this kind of rule of random impact. So the next thing that we looked at are the sleeping beauties. And like I mentioned, there are papers in science or opinions in law that kind of are invisible for a long time and then suddenly experience an uptick. And so here you see a really dramatic example on the right where you have goes unnoticed from the early 80s to the mid 2000s and then suddenly this opinion receives a ton of citations and then drops off dramatically. And so we can calculate this sleeping beauty coefficient that measures this kind of surprise factor. That's based on the publication year, the number of citations received in a given year and the year when the maximum is achieved. And there's some details about how we calculate it using this reference line, that's the dotted line here but that's not super important. But I'd like to give you an example that's the case that we just saw which is United States of Yehudin. So this was published in the Ninth Circuit, I believe, in 1982 and it goes unscited until 2002 and then it's cited over a thousand times. And the interesting thing is that the case stood for the proposition that the question raised in an appeal is so insubstantial as to require no further argument. And so each of the thousand times that it's cited by the Ninth Circuit Court of Appeals, it's cited as part of a super short opinion that essentially says, there's no substantial question, see Hutan, goodbye. And it's like one sentence opinions. And presumably, but we haven't actually found where this happens, presumably someone said at some point, guys, you can't do this anymore. Or a different opinion arose and the practice continues but a different opinion is cited for the same proposition. But it's a very convenient thing. And so we can see this kind of like rapid rise and then drop. And so comparing, again, science and law, we see that the, you have to believe me because the figures don't really show it as clearly as I'd like, but the distribution of these sleeping beauty coefficients is really similar. As is kind of the distribution of awakening times, how long it takes for a case to be noticed. And so we're kind of figuring out how to explain this and also what is the significance of these sleeping beauties. And you would think that kind of with improved technology in both systems, you would expect these sleeping beauties to become rare, but that doesn't really seem to be the case. I mean, we just talked about a pretty recent example. This continues to happen. So these are a couple of surprising to us similarities between the two systems, despite their tremendous differences. And so the main takeaway for us is, well, we can probably transfer approaches from one system to the other. And if we think, for example, about the Q-factor model, that helps us assess the quality of an offer. And so you could imagine that the American Bar Association, when they make kind of their competency assessment of judges could use a metric like that to assess, how good is a judge? So this is kind of this work, relatively academic. Any questions on that? Because otherwise I'll dive into the NLP side of things. See, there's also chat questions. Maybe if you could just say one thing before you get to auto-law, I think there's going to be a lot of interest in auto-law and discussion. So could you just talk a little bit more about how... One of the things that I think that's interesting just to back up in terms of showing this sort of quote behind the curtains, end quote, look at what's happening on this research lab at MIT with respect to computational law is how did it arise? And so could you just say a couple of things about YY from our team and what that professor is doing with science of science that you mentioned, but didn't describe and then how you had the glimmer of an idea to apply that technique within the legal space and then that cascaded to a bunch of breakthroughs, but you just described that, like where did this arise from and what is science of science? Yeah, totally. So YY is one of our collaborators. He was a visiting professor from Indiana in our group and studies, you know, science of science and networks more broadly. And so what happened was, I think about a year ago now, I had extracted the citation network from a data provider called the Case Law Access Project, which is out of Harvard. And they conveniently scanned all US case law, like almost 8 million opinions and converted them to plain text using OCR. And so that's kind of where it arose, but you know, kind of to take a step back and kind of think more philosophically about it, right? YY and that whole community of science of science researchers, think about, you know, how can we evaluate the process of doing science? Kind of like how can we assign quantitative metrics to it? How can we understand if it's working well, if it's working poorly? And it seems like, you know, we have a lot of the same questions about the law, but we don't really have the tools to talk about it, right? I mean, I think there's often the kind of pervasive view that like the law is working not so good sometimes, but like better than long time ago and better in some places than in other places. But that's all very qualitative. And you know, this paper doesn't do this, but I think it scratches at this idea. Well, you know, can we assess the quality of the law and the quality of judges and can we do it in an empirical way at scale? So I think that's what you were getting at, Daza. So that's kind of the background. I hope that's helpful. Yep, it is helpful. And so just in the spirit of idea flow, just please recognize everybody that ideas from one field and endeavor can sometimes be adopted and adapted in other fields and that there's some, you know, maybe superficial layers in which they're all but identical like citation networks and scientific papers and, but at a deeper level, I guess I will say one more thing substantively that you may or may not be aware of, Robert, but part of what I love about this project is that when you look at the scientific method itself, you can see that to a material extent it arrived on the shoulders of the law and the processes of, among other things, I'm thinking in common law, England, of what is the process for finding truth in the legal system? You know, you have an adversarial process. You know, you have the opportunity to cross-examine, you assume nothing, you have to prove everything. How do you prove something beyond a certain, you know, kind of burden? You know, what is evidence to start with? And there's other approaches as well that were adopted at the, you know, during the beginning of the age of reason as part of how we established facts. In fact, about the natural universe, you mentioned gravity very poignantly as something that was different. Well, same, different, different, but same in certain respects. And there's some great books that detail, you know, what the lineage of legal systems to the scientific method was. And so now that finally, just a few hundred years later, the law is starting to catch up to science and become more computational and empirical in some ways. I think it's all too poignant and fitting that we should borrow back from our cousins, you know, more advanced techniques. But at the end of the day, aren't we all seeking truth by some definition? So anyway, with all that, those philosophical kind of muttering society, let's dig into auto law, Robert, because I think that one is like right on point for what we're doing. Can I ask a stupid question first? It's a stupid question because I don't know. Yeah, go ahead. Yeah, I don't know anything about the law. So like it may be stupid. But so we're using, we're comparing citation dynamics as on the slide now. And one of the things that's interesting that has come up in scientific discourse around publishing recently is the notion of negative citation. So like you're citing a thing, not because you think it's a good work, the opposite in fact. And I don't know if a corresponding phenomenon exists in the citation dynamics of law. I imagine you're citing things because you want to build arguments based on them. So you're implying that they're good by citing them. I don't know if that would factor into your analysis. So it doesn't factor into our analysis in neither. So we don't worry about negative citations in either the scientific or the legal data set, but such a thing exists in the law too, right? And specifically, higher courts can overturn lower courts and will almost inevitably cite the relevant opinions in some form. So it definitely happens. And in fact, if you go on to the kind of for-profit, big legal search engines, they will tell you whether an opinion is still good law. And that involves kind of whether or not it's been overturned. So very much kind of important to the practice of law and a real thing, yeah. Cool. All right, I'll move on to the next project then, to auto law. And so we're still dealing with citations, but a very different question, which is given a legal argument, can we predict what you should cite? Can we predict relevant, not just opinions, but passages of opinions that you ought to cite? And that's kind of for the purposes of most of this, I will focus on a subtask, namely given the context around a quoted passage of precedent in a judicial opinion, can we predict the passage? And then I'll go back to kind of the original task towards the end. And so this is a convenient way of handling the problem, right? So we have one opinion that quotes another opinion. So I know first of all, that whatever's written around the quote relates to the argument that's being made. And second of all, I know what is being quoted because I can map that back to the original opinion. So how do we treat this kind of from a technical point of view? We treat it as a multi-class classification problem. This is generally surprising to people because they say, well, you must have a ton of these passages and we do. So how can you treat it as a fixed number of classes? And the way we can do it is because these citations obey a really long tail distribution. So although there are like 1.5 million unique cited passages of precedent in US case law, again, this is all federal law, the top 5,000 alone account for 20% of all references. And so for the purposes of this all law project, we focus on these 5,000. And as you'll see towards the end, these 5,000 don't seem to relate to any like one area of law in particular too much. So you can use them kind of different legal settings and still get really interesting results and predict really useful precedent. And I'll compare two different types of models. The first is it maybe doesn't look that way, but it's a really simple model kind of as far as the NLP literature is concerned. So it's kind of like a simple neural network that uses a simple way of taking words and embedding them, representing them as vectors that doesn't take into account, the words surrounding. So whatever, each word in any context will always get the same vector assigned to it. And we train those vector embeddings using lots and lots of legal text. The second is a fine tuned legal BERT model. So BERT is a much more sophisticated approach to doing these natural language models that was kind of has become the state of the art. And some folks have trained a legal BERT that has been fine tuned on legal data. And so we take that and we then further fine tune it on this multi-class classification model. And so to remind you, what we do is we identify these kind of missing quotes and we generate many opinions by taking texts from the introduction of the conclusion of opinions as well as text from right before and right after the passage. We generate 7.4 million, many opinions in total and half a million of those correspond to the 5,000 most cited passages. So there's a lot of training data that we don't use and you could definitely kind of build this out into a full product by using much more of this data. But this is technical, but we do some preprocessing, kind of cleaning things up. We, honestly, most of this is probably not actually helpful but one of the challenges that you have when you do these neural networks, maybe it's interesting, is you have a class imbalance problem. So if you have this power law distribution that I told you about that, some categories are vastly overrepresented then your neural network can just predict those really common categories and it won't be wrong that often. And so for the simple neural networks you need to find ways to overcome that. And you can do this thing called balancing and where you kind of synthetically generate training examples. So I told you, these training examples are in some vector space. And if this one is category one and this one is category one then we can assume that anything that is between them is also in category one. And so this is kind of the synthetic over sampling technique just to help us with this class imbalance. Whereas BERT allegedly can handle class imbalance kind of there's some research on that. So we don't do this balancing and we kind of follow the industry standard in terms of how we train the models. And the results are really quite exciting. So the macro and the weighted averages here corresponds to how well the model does kind of on the data as a whole and how well the model does if we remove the imbalance issue. And so you can see that the BERT model has a really high weighted performance. So it kind of gets it right 72% of the time. It does a little bit worse on the macro level. But you'll see. So these numbers don't seem that great, right? Like 37% doesn't seem that great. But in fact, this is kind of surprising, right? So like 30% of the time it picks the right passage out of 5,000. Yeah. And so... It's a little hard to put in, but don't seem that great compared to maybe your dream, but they seem incredible compared to everything I've ever tried. I mean, like, I think the big news is that this is working at all. Just sorry to interrupt you, but like, come on. Well, so it gets better. So a lawyer wouldn't actually use it this way, right? A lawyer wouldn't just say, give me one passage and I'm gonna cite that and move on with my life. What a lawyer would do is say, give me 10 passages or 20 passages and I'll use those. And here we really get like kind of fantastic performance. So the BERT model kind of gets it right, gets the right passage among the top 10 results 96% of the time. And even the feedforward neural network, even the kind of simple approach gets it right 80% of the time. So this is really encouraging, right? Like this suggests that you could actually use this, with the caveat that we're focusing on common citations, but this begs the question, like is that really a problem? Like is focusing on the common citations a problem? And the way I tried to get around this is let's construct some like real examples and see like does this make sense? Like does what the model predicts makes sense. And so we did this by kind of testing it in the wild. And what we do is we take two legal briefs that were written by kind of like preeminent lawyers. And we summarize one argument in the brief and then we see what are the results. So the first brief that I looked at written by Solicitor General and now U.S. Supreme Court Justice Elena Kagan, the whole brief is 86 pages long, but the argument that I focus on is that the petitioner's right to a fair trial was not violated because the pre-trial, because during pre-trial there was a lot of publicity, right? So you might imagine someone could say, well, everybody is saying that I'm guilty. It's in all the newspapers. Like how could I possibly have a fair trial? That was kind of the argument. And of the 10 passages that were predicted by the model, three appeared kind of completely irrelevant. Three were relevant and actually appeared in the brief. Two belonged to an opinion from which a different passage is cited. And two appeared at least to me relevant, but we're not cited. And this to me is like really fantastic, right? It's a pretty specific legal question. And yet the model is able to predict relevant precedent. So we try this again on a different legal brief, which was kind of one awards is the best ever legal brief. The argument here is that an alien that was stopped, so kind of an immigrant that was stopped at the US border has a constitutional right to be free from false imprisonment and the use of excessive force. So here the question is, if someone who is potentially coming into the United States illegally, do they get the same constitutional protections as a US citizen, that kind of question? Note, super different legal contexts from before. And yet the results are really quite good as well. So three irrelevant results. One of the results actually appears in the brief. Five belonged to an opinion from which a different passage is actually cited and one appears relevant, but isn't cited. So this seems incredibly encouraging at least to me and suggests that these kind of like NLP approaches could really do a lot of good. And they too can do good in at least two ways, right? Like first by just improving the quality of legal arguments, right? Kind of giving people tools to make better arguments. But second and perhaps more importantly, by lowering the barriers for access to justice in various ways, right? They reduce the costs associated with judging. So judges can get through more cases, there's less of a backlog, but they also make it cheaper for someone to work with a lawyer and kind of be able to make arguments in court. But even before you even get to court, right? You might be in a settlement negotiation and you wanna know, well, if we did take this to court, like what does it look like? And these kind of approaches could help you. There's lots of room for more work so we could evaluate it on more real legal briefs. We could go much further than these 5,000 most common cited passages. We could think about a slightly different problem namely assessing the quality of legal argument by maybe thinking out loud a little bit. What does the model think you should cite versus what did you actually cite? And kind of assigning scores to that. Or maybe given what you actually cited, what might you be missing? All sorts of things like that. So with that, I've reached the end of my kind of double presentation and I'm excited for your feedback and questions. And I hope that was interesting. Yay, thank you so much. That was so great. Okay, well, the floor is open. We do wanna reserve a few minutes at the end to bring you all into the fold on a cool initiative that Brian is mostly leading on composable governance but and what seems gonna share some thoughts that I think are good examples of the direction we wanna go. But right now the floor is open. And I guess the question is how much do you love that? What Robert just said, huh? No, I'm sorry, the question is if you have any questions or comments or ideas or objections. So one thing was, yeah, I was talking with Deza earlier in the week and one thing that we were discussing related to the auto law project would be that it would be really interesting just to see what the kind of like buckets of law are. I'm sorry, Brian, but some of that is not yet announced, but I'm working on a follow-on project that could be cool, but we gotta choose the right time to talk about that stuff, especially after I have any idea that it might work would be the best time. I'm sorry to interrupt you. Yeah, just let me rewind them. Okay, I'm sorry, man. I'll pass it back to the floor. Okay, I'm sorry, Brian, but look, if you guys make it into our lab, then you too can see crazy ideas that may or may not work before if we choose to announce them or not. And somebody else was starting to speak, I think, at the same time, who was that? Might have been me. I shared the link to the auto law kind of preprint on archive, so if people wanna dig into it, they are welcome to. There were also some questions in the chat. I can start with those while people think. Chris asked how widely distributed are the NLP-based tools. I think it's hard to say, right? Because a lot of it is proprietary. Certainly kind of the for-profit legal research companies are saying that they're doing NLP. There's more and more interest from the NLP community. What doesn't seem to exist is kind of widely accessible free or nearly free tools that people can use. And there are maybe reasons to that, and you mentioned the barriers. This is kind of a whole conversation in and of itself, but I think there's a resistance to relying on technology, and I think that that is reasonable. Lawyers don't want to get sued for malpractice, for example, because they relied on some tool. So there are certainly barriers there, but it seems like there's a growing desire to use tools like this. Robert, are there any other jurisdictions that you know about that are a bit more forward thinking and applying this kind of technology? Maybe you'd like smaller countries with more less regulated legal systems? The short answer is no, I've looked a little bit. We once explored doing a collaboration with some Pacific islands because they are very small, and they also have common law systems. But, you know, and maybe what put it over the edge is a trip to Kiribati wouldn't be the worst thing in the world, but it never materialized. So if folks are aware of jurisdictions that are interested in kind of experimenting, I'd be super interested. One of the issues is data, right? So if you want to do any of this, you need to have access to data. We once initiated kind of conversations with Rwanda, also a common law jurisdiction, and there it turned out that especially post-genocide, a lot of the case law wasn't even like typed. It was because they have to have these emergency courts. And so a lot of that was written down by hand. And then that was compounded by the challenge that the language Kenya Rwanda is spoken by so few people that a lot of the NLP tools we have available to us, like BERT wouldn't work. So that's a real issue, but we're exploring it. Hand inside? Yeah, someone you should reach out to is Steve Tendon. He's working with the Marshall Islands on some crypto-related projects, but I just checked in their common law jurisdiction. So that might be a good signal that they're interested in doing something. Yeah, one step beyond that is look at anywhere that's implementing a CDPC, because it's back digital currency now, because they may also be in a similar legal situation to be forward-thinking with that ecosystem. And we started doing more work on CDPCs. It strikes us as a good example of a lot of this. And we happen to have a sitting judge with us. Renata, did you have something to contribute? Yeah, well, I was just wondering if you have ever tried this too with any civil law jurisdiction. And if you would happen to find a source of open data that maybe it would make this, all this resource and research a lot easier. For instance, in Brazil, we've got 18,000 judges and 78 million lawsuits going on. And that would be just amazing just to have all this data and this citation collected and built upon. I mean, it just got me wondering here, you know? Yeah, I haven't tried this. We haven't tried this on civil, in the civil law context. And we ought to. I think we're just less familiar with it. And we're looking for kind of like someone who can show us the way for, because presumably the, okay, excellent. Yeah, no, I'd love to explore this more and figure out, because I don't really have a good understanding of how we would have to think about citations in a civil law context. I can totally help you with that. I mean, just, if you wish, I mean, I don't know. I'd love to kind of engage in a conversation. Absolutely, yeah. And I know that there's like an specific institution that's actually working with AI. And maybe we could just try to gather things up and it's part of the structure of the justice. I would have just to check how things are going because it changes every couple of years, something like that. But I mean, I believe that this could be a real game changer for a place like Brazil. And it's like, it's crazy here. Yeah, no, I think it could be a game changer for a lot of places, unfortunately, but absolutely no, and then there's basic questions like availability of data and also somebody who needs to speak Portuguese, right? Like, it's surprising how much you can, yeah. There's a surprising how much you can do with NLP if you don't speak the like relevant languages, but only so much, especially in the legal context. I think it's really good. This is just amazing because of access to justice and that's just something that gets me every single time. So congratulations on your project. I just wanted to just quote us about that. That's beautiful. Thank you. No, hopefully you can put us in touch and we can explore. Yeah, for sure. Okay, I see a nod and we don't require much encouragement so I will put you in touch. And just one other little thought there is, and I know we have talked in very broad ways about, could we repeat this with statutes and regulations in a different way? Obviously it's a citation network, but there are, but statutes and regs are a critical part of the law and they're frequently what's cited to, right? And you can cite from them, especially when you look at the paid databases and they do provide like a real ontology and like proper rules of law. And obviously that's gonna like take a lot more, thinking through that that may be among the last things we do in common law jurisdictions. In a civil law jurisdiction, maybe there'd be far fewer steps needed because of the nature of the framework being so comprehensive. So maybe it might be worth reaching out even more closely to our friends and colleagues in Brazil, for example, to see if there's a team up there that in the future or even just to help articulate the questions and possible approaches, even if we didn't take it forward in the development phase, it has its own advantages. So I really do encourage you all to keep pulling on that string. It's I think very promising. And fundamentally it's like adding a second access to the corpus of law. Okay, more. Okay, and so, and I notice that boy, that was the fastest 52 minutes of like my recent life. So thank you. We do wanna kind of pivot a little bit now. We're not good. We wanna save some time for composable governance. So I just wanna thank you again on behalf of, you know, the whole community, Robert, for taking the time to explain this so well. And as Renita said, kudos on really digging in in I think like the MIT spirit into computational law and building something that we can react to. So thank you and thank you so much. And thank you for giving me the opportunity. Thank you so much. I appreciate it. You're here. Cool. And so now giving the mic to the person I so rudely interrupted, whose instincts were right on target in terms of what is the coolest thing to talk about? Just maybe slightly premature. It's not premature anymore for Brian Wilson, our editor in chief of the MIT computational law report who had yet another awesome idea about what we should be doing. And it's, I'll just say the name and then hand the baton composable governance. What is it? How are we gonna do it, man? So I think that's really what we're trying to figure out. We during the IAP course just announced a new call for submissions to an exciting project. The collected works on composable governance. And here what we're really trying to get to is to cut through some of the, cut through some of the myth and understand more of the cool aspects of what's happening with some of these web three innovations that are taking place right now. Because there is a lot of stuff that needs to be made sense of. And I think here, some of what we're doing is sense making. There isn't a really great definition for what composable governance is. So in the coming weeks, we'll be releasing something akin to an FAQ where we start to answer some of these questions. We'll start speaking with people getting some recorded videos out there of what does composable governance mean to you? Because I think in different contexts, it'll mean different things. For regulators, it'll mean something completely different than it does to people who are building things. And that will mean something completely different than it does to lawyers. And that'll mean something completely different than it does to business people. And so I would encourage anybody who's interested to engage in the Telegram channel that we have for computational law. You can find that on the contact page, I believe. It's either in the contact page or the about page. There's a page and I'll find it and I'll post the link. But before we run out of time, I wanna kick it over to Waseem because his column is about a lot of these governance mishaps and what happens when they go wrong and exploring and disentangling the bad from the good. And so we'll see him, it's over to you. Yeah, thanks. So we don't have so much time. So maybe we'll just give a little taster. So I suppose the column that I wrote, I'll just post the link in the Zoom chat, is the first in a series on missing mis-brackets, mis-adventures in crypto governance. I'm trying to go beyond good and bad. It's all experiments, it's all very early stage technology. It's very speculative and we don't know the outcomes of these experiments as we embark on them in good faith or otherwise. And I would say that I started off with the simplest case which is Bitcoin. I would say that this is kind of a case of where modularity of governance and composability of governance is absent. So we don't have, there's no governance in Bitcoin. Everything is off chain. It relies on human coordination outside the network and we just rely on the incentives that Bitcoin mining provides to the various stakeholders in the network that the kind of partial alignment of the different stakeholders in the group, it's kind of creates a Mexican standoff. And so like, the devs have guns pointed at the miners, the miners have guns pointed at the exchanges and so on. And the thing just keeps spitting out coins to keep everyone happy enough to stop the thing from descending into literally anarchy. And there's this really helpful concept which I'm sure many of you have come across before, the tyranny of structuralistness. And this is kind of a way of describing this kind of governance minimized space like in Bitcoin where there is no set of, you know, there's no way of, there's nobody that's gonna answer your proposal. There's nobody that's gonna assign you funds or give you reputation or give you credibility for doing X, Y or Z inside the network. So you're relying on everything happening outside it. So you're kind of navigating this structuralist space which is not empty. It's just that everything is poorly defined. You know, so you've got like norms and you've got kind of hidden agendas and technical limitations and you're just trying to find your way around this. And so there's one concept which came up in the Bitcoin space, Nick Szabo, one of the progenitors of Bitcoin who created the DigiCash came up with this idea of, so we talk about attack surfaces with crypto networks and other kinds of computational systems. Well, he was talking about argument surface. So think about governance in terms of complexity and he was trying to use it frame in a good way saying less governance the better because the more complicated system is at the more vulnerabilities there are whether they're technical or they're social or they're somewhere in between. But having this idea of this minimized argument surface is all well and good. But unless you have some kind of way of helping stakeholders whether they're human or otherwise, reach agreements and make decisions, you just end up in this kind of perpetual, a best Mexican standoff worst civil war. And the civil wars in blockchain systems are forks and you've probably heard about hard forks before like Ethereum had one when the Dow collapsed in 2016. Bitcoin had one in late 2017, I believe it was early 2018 over the disagreement over the scaling pathways for the network in terms of the block size of each Bitcoin block. And unless you've got some kind of way of like a du jour system or something like that within these networks, then you're always going to end up in these kind of tyranny of structuralistness human off chain coordination moments. And I'll give you an extreme example of how badly things can go wrong when there's no governance inside the system which is a paper from, I believe it was 2017, 18 by Patrick Macquarie and colleagues. It's called smart contracts for bribing miners. And the idea in this paper is that you can use incentives on Ethereum with smart contracts, automate the algorithmic incentives to corrupt the game theory balance of incentives on Bitcoin. Because we know that Bitcoin inside the system the incentives work, that people start like stuffing other people's pockets with cash outside the system. You may not be acting in this kind of economically rational way anymore. And so I think that's another really interesting kind of pause for thought that these networks also don't exist in a vacuum. They also exist among other networks and they also exist among jurisdictions and legal systems and central bank currencies and all the rest of it. And so yeah, I don't have any solutions. Frade, I set my store out by foregrounding the problems. So maybe I'm useless here. Maybe I'm just a critic and I'm not really helpful but I will promise to continue to unpack more governance maladies as we go through the column series. And we'll be getting next to Ethereum, the DAO and then all the DAOs that we're seeing now which are flourishing or expiring depending on fate on pretty much a daily basis as I'm sure you've seen in the news. I'll wrap there, thanks. Here, here. Thank you so much, Brian and Waseem for sharing with the community through IdeaFlow. What's been going on in the MIT Computational Law Report with respect to governance on Waseem's Beat and his column, which is off to a terrific start and I might just say one more point. Ordinarily, you know, like our vibe is more like let's build not sit back in our armchairs and critique everything all day, but there's a balance there. And I think our friend Elizabeth Reneres has been a good example of how, you know, critique is a critical input to design as well. In fact, it can be a constructive source of requirements and constraints and describing, you know, how we would test a system that operated in a way that addresses or maybe even transcends what the dead ends of the past have been. So I think it's incredibly useful. It's also a really entertaining column and Brian most especially for having the leadership and creativity to articulate this very cool direction of inquiry. So everybody, everyone I encourage you to participate in this, how do you participate? Number one, write something about this and then submit it at the law.mit.edu forward slash Composable Governance link that Brian has shared but better yet, Brian is and the rest of the team are gonna be emailing you and talking in future idea flows about more sort of like incremental things we'll do like little symposia, kind of discussion events, other kind of sort, but maybe ask me any things where we can just have open discussions on the topic and get more iterative input and start to flesh out these things. There's a lot more to say about that and so much more in computational law but we're out of time today. So thank you again everybody for participating and we so look forward to seeing you at next month's last Friday, 12 p.m. Eastern time idea flow. Until then, I'm gonna wish you well. Thanks everybody. Bye bye. Thanks. See ya.