 I'm going to pop over into the frame for the kickoff. OK. And we're live here from MIT Media Lab. This I'm Dazza Greenwood. I'm Brian Wilson. And we are the co-instructors for the 2020 MIT Computational Law Course, an IAP offering that's now in its fifth season. And today is final presentation day for all the participants in the class. And so why don't we kind of get right into that with the looks like a, oh, Andre, you are with us. So Andre, are you ready to present the message? Great, why don't you? Yeah, yeah, I'm ready. Why don't you? I'm listening. Do screen share. And then you have five minutes to introduce yourself and focus through your presentation on the use of blockchain and the transparency and ensure accountability for government actions. Just let me see if I can put the slides, because I'm not using my computer. I was, as I told you, I am in vacation. And I'm improvising in bed. Then I would try to put the presentation. I can pull the mic if you need us to. Just by way of background for everybody who's watching this later on after the fact, we are having all the participants in the course who decided to submit a final project, walk through and say five minutes about what their presentation is, and show us any slides or other materials that they've prepared for this. Indeed. And Andre, if it's helpful, we can go to Neha first and then come back to you. Yeah, yeah, I would prepare all the, I think I will be able to present. I found how to do this by day iPad. But Neha could present first, and then I go later. OK, that's perfectly fine. So Neha, if you are ready, we would love to hear your presentation on tax as a computational system, and especially your experience trying to basically specify something that could be buildable and what you learn from that. So Neha, welcome, and you're on. I'm going to try sharing my screen. So what I was working on was trying to create a class. And the idea was, what would it look like if instead of laws, the IRS put out just code, and then you could feed in your own numbers into the IRS's code, and that would be the single source of truth for what your taxes are, what your tax liability is, what tax deductions you get. It was just a thought experiment. So yes, my goal was to turn laws, which are on the left side, as an example, law, which is, I think, 26 USC 1014, which describes what happens when people inherit property, and turning it into code, so something like what's on the right side, which is just pseudocode. And in the process of thinking through what this would look like, I was trying to measure twice, cut once, to be really careful about what the system will look like, and then code it only once. But the problem was so dense that I spent all of my time measuring, and I didn't actually get to write much code at all. But I just wanted to talk about here what problems I ran into when trying to go from left to right, trying to go from English to code. So the problem was just how to simplify code. And I took inspiration from what one of my professors did for corporate law class. So what he did was we were looking at Delaware corporate law statutes, and an example of one is on the left. And he has a website, simplifiedcodes.com, where he's taken all the Delaware statutes that are important in corporate law. And he's just written them in simpler English. It's still not entirely simple, but it's a lot easier to understand if you're, A, just coming in as a student, or B, if you just want to know the outlines of what a law does without knowing all the little finicky details. So this is sort of similar to what I'm trying to do. I'm trying to take English laws and convert them into a very simplified version. So let's take, again, 10, 14 as an example, which is the law that describes what happens when property is inherited. And the problems I ran into mainly fell into three buckets that I've highlighted in different colors here. So first is purple. It's safe to accept as otherwise provided in this section. So that seems simple, but when people are handling the system, because when you want to decide what the exceptions are, you can look at everything else in this section. Or you can take it to court, and the court will tell you what tax laws take priority over others. But when you're doing it in a computer, you need to know what the exact priority order will be to be able to code it correctly. And that requires making explicit things that are just implicit right now, because you won't have the court to help you. The second problem I ran into was duplicates. So this is something what it's love doing where it's all say something, and they'll insert an or, and then they'll say something again in the same meaning, but in different words. And there are edge cases where a situation might be covered by a, and it won't be covered by b. But if you're trying to simplify things ideally, you wouldn't have duplicates at all. If you're trying to write a system from scratch, you would only say things once and make sure that saying it once encompasses all the cases you need to encompass all those. Then problem three is sometimes the tax code asks for things that are really hard to find. Like here it wants you to find the fair market property of inherited value at the date that the original owner died. But this is information that's really hard for people to know. On the day that a loved one dies, you're not going to be trying to evaluate the fair market value of their property. And in real life, I think we fudge it. We assume that we have the information when we don't. So a person who's relying on this code, they'll just put in the fair market value on the date that they evaluate the property. They won't look at the date that the descendant died. But if you send a computer, you don't want to use fudge factors. You want to have the code describe exactly what it wants the computer to do. So you would either rewrite the law or you would try to pull in information that you might not have access to easily. So those are the three problems I ran into. But I think if we address these, the reach goal would be to create something like what my bank currently does for me. It tells me my credit score, but not only that, it tells me what I can do to improve it and where I'm failing. And I think this would be really cool to have for tax where not only do you know how much you owe the tax IRS, but you also know what can you change to decrease your tax liability? What can you do differently with your assets to make them simpler to understand, to reduce your paperwork, to reduce the time you spend filing taxes, whatever metric you want to measure it on. So that's what I did the last two weeks. Outstanding. Congratulations, nice work. So maybe I could get the discussion started with one question. If you were attempting to codify, say for example, that rule on death tax, would you look to, do you think it would be a better strategy, or what are the pros and cons of a strategy of having the legislation or the reg express the fair market value of the decedent's property on average during like the quarter of the year of their death, said something that you could maybe get some market markers for similar properties that we're selling, or if it was experiencing like a precipitous decline or increase in value based on some neighborhood, or sort of, or on the other hand, go the other direction more of a typical legal mealymouth direction and say like within a reasonable time of the decedent's death, for example, or what other strategies do you think might make that less obscure and more objective to be able to measure? I think there's some, let's go back to the discussion that we always have in law school about rules versus standards. The rules are really specific, but they create a lot of weird edge cases that don't necessarily have the best outcomes and standards are like the lawyers love saying in a reasonable time. But then they're so vague that you end up having to go to court a lot more to mitigate exactly what reasonable means. And I think if you're using a computer to codify tax law, you should take advantage of the fact that it, A, can do computations really cheaply and B, it loves specific information. So I think I would go with the first alternative we just did, which is something like within the quarter of the death that gives the inheritors enough time to get their affairs in order. But it also gives the computer specific information. That's not too honor is to get, but the computer likes having numbers instead of reasonable standards. And then you can reduce the amount of time people spend going to court. And then actually, I'm just gonna, a quick follow up. Have you heard of programming languages like prologue? No, I don't think I've worked with that one. Okay. That's only maybe we can, well, why don't you just Google it? And it's a logic-based language. It's a thing called declarative. And one of the things that it does is it requires being very explicit about things like the, basically like the priors and the, you have to sort of declare upfront, what the assumptions are and what the axioms are going to be. And then every rule is stated in a simplified version of pure logic. So looking at things like the duplicates, but like basically each of those rules would be able to be like atomized into its own expression that may reference other rules in part or distinguish itself from other rules and have different kind of criteria. But each thing is declared more or less and they're all parsable logically so that it's very easy to do something like formal verification or certain inferences that you kind of draw from them in natural language. But they would be different from the inferences that you would express in some language like prologue. That are very specific about what's included and what's assumed not included and what you can deduce from that. Because anyway, there'd be a lot of relearning for people if there ever were like a formal syntax and kind of grammar for computational expression of law and the way that you're talking about to make it completely clear and ambiguous that may be somewhat beyond what's realistic for lawmakers and lawyers to be able to master. Yeah, and I think too there's probably there's some lesson to be learned about because of some vulnerable error or some certain circumstance. So I think a lot of people have started incorporating by reference some of like the underlying principle. And while you know something kind of like contest it and go back and see does this actually fit within the principles that we have for this program for this algorithm? And then you can reevaluate it under the right set of principles essentially. And so we come back to the common law of like having like people with the ability and a reason at some point setting parameters and making judgment calls about what's in and what's out and the computational CCL. Any other feedback or questions or comments on Neha's presentation? Well, I think we should just thank Neha for a great presentation and a thoughtful, very thoughtful project. Thank you, Neha. I think that... Thank you. And Andre, I think you're, if you're ready, we're ready for you to go next. Okay, I will, yeah, I will charge you Oh, if you want, I can share your slides on my side and you can just tell me when to advance. Yeah, yes, if it's possible, it's Brian speaking, no? Yes, yep. If it's possible, it's better. I was trying to connect with Google Drive, but I'm on a very bad connection here and I'm sometime and we, okay, thank you. Okay, so just tell me when... Well, my... Okay, I will tell when we can pass to the next slide. First of all, I want to present myself. I am Andre Guskov-Carduz. I'm here. I'm here. I work with regulation and public law in general, administrative law. And for a while, I've been studying the use, especially the use of technology in public law. And one specific field of this is the use of DLT. In general, the Cybertron Ledger technologies and blockchain are very, very rich fields in which we can evolve. And I think that it's possible to enhance the... The... The... The... The... The... The... The role of the state in the... If you can go to the next slide, please. Yes. Okay. Okay, is that it? Sorry. We can start by some initial comments. We know that despite all efforts, despite all legal provisions in polls and transparency, governments in general, governments in the US and Brazil and Europe, are still difficult to citizens to comprehend in times of very often confusion, contradictory in various sense, and antiquate. We can perceive the lack of use of current and available technology. We... Everyone has the experience, probably has had the experience to be in a public department and think, what I'm doing here? Probably all this situation, all this formalities could be done by the web. We're still in the stone age when we deal with governments. But this is a... In a sense, it's a contrast. Because if we still see the experiences in the stone age, there are some fields of government that are very, very, very advanced and use advanced technologies. And usually we see this with the tax authorities. They're eager and very, very good at using new technologies to detect frauds, detect and so on. And this is a problem that DLT is in and blockchain could deal with, could tackle with. All this considerations, all this situation. And just a quick time check, you've got about a minute and a half remaining. Okay. I'll go very, very, very rapid, very fast to the lack of trust in government, corruption in efficiency. And in the presentation, I made some citations, a well-known or new Francis Fukuiya and even a scheme of... But we need to know how to deal with this situation. And to do this, governments need to act more and truly transparent and accessible to achieve both goals by using DLT and blockchain technology. And there are pilots, program pilots using this kind of technology in Brazil, in US. I mentioned in the slide 12 an initial program at US Treasury that we use blockchain-based grant payment system in order to control. And in Europe, the ecosystem of blockchain is being discussed a lot. It's possibilities and it's probabilities to use. We even discussed the use of smart contracts to reduce bureaucracy and to limit the discretion of government action. And then this goes exactly to the idea of computational law. Blockchain platforms and DLTs could be a solid platform to use... to govern an action, enhance transparency and accountability. We have a lot of constraints and these are the difficulties that we can see all over the world. A lack of proper infrastructure in maturity of the technology inside government itself needs to develop the skills inside government, needs to convince governments of the benefits of the use of this technology and the problem that we have to convince the public bodies that they... probably they will have to renounce some power to use this kind of tools that will enhance government actions. And we have some other problems too as the risks we have, personal data and privacy. We have a necessary evolution of the legal reasoning and parts of understanding as Nihá has already mentioned. It's difficult to convert, it's difficult to translate legal code in digital code. This is very difficult. It's very, very difficult to translate to make a smart contract based on blockchain. But we can say that this is a long way forward but they are very promising technologies and very promising ways to do this with the use of DLT and blockchain technologies. And this is the general idea. We can go to discussion and we can see if we... Thank you. Thank you for managing the slides. Excellent. Thank you. Great presentation, I want to say. And since I sucked up so much time on the last discussion, let me open the floor to anybody that has any feedback or questions or comments on Andre's presentation. And if you're online, just come off mute and start talking. I think the fraud use cases is a really interesting one because it lines up with a lot of the traceability and immutability that blockchain has. And I think a lot of people can get involved with different certification schemes where you have to prove the origins of maybe where certain funds came from for political donations or things like that to actually verify some sort of... Formally verify that you have funds that are from legitimate sources instead of funds from shadowy car wash sort of things. Yeah, so this is Bob Taylor from Liberty Mutual and I would just add on to the fraud use case. I really like your presentation. I think there's tremendous upside to what you're talking about, especially when it comes to professional licensing. So if you're hiring somebody that's supposed to have a valid professional license, there'd be a very easy one source of truth to be able to check and if that licensing has to come from a government entity or even if you need like a certificate of insurance, right? But anything that's issued from a government entity that I want to know that the contractors licensed in the state of Massachusetts to do work this type, I don't have to rely on the contractor or word of mouth. There's an easy way for me to verify that. I think this transparency certainly is... Yeah, I know the town of Zug in Switzerland did their entire land registry on Ethereum so that I think is another case for something like that. Yeah, indeed. Any other feedback from more participants? Yeah? Okay. Hearing none. I'll just end by saying there's a technology that you should run your radar. I don't actually even think this is the right... This is not final or ready for, you know, wide-scale deployment, but it may be pointed direction. And that would be basically so-called verifiable claims, which is a worldwide web consortium cryptographic protocol that involves basically three parties, the issuer of a claim. In Bob Taylor's example, it might be the insurance commissioner of a state that can say whether someone is licensed to some insurance in that state, or the board of bar overseers in Massachusetts is a licensed attorney or is not, or is retired status or whatever. So that's an authoritative source. The person about whom the claim is made and then a third party that wants to see and be able to verify the claim. And basically it's just like... It's always a rudimentary application of public key cryptography with a digital signature. It just sort of takes a hash of the claim, digitally signs the claim, and then has the standard syntax to associate that with an envelope of the claim and pointers to where you can verify it. Some other bells and whistles. And it has the property of allowing the person who the claim is about, like you mentioned a certificate of insurance, actually have a client for a consulting who has informed me that to go forward, you're on out, they need to be listed as a beneficiary and they need an additional assurance. I'm sorry. He takes an insurance guy, which actually asks you about this after class. And the certificate of insurance it shows it's kind of paid up for all of the workers coming up and things. And if I actually had that from my insurance company, I could just send it and they could verify with the insurance company, but it wouldn't have to be another workflow to get another one every time from the insurance company, because they could verify it with the public key and you could have a verification database or whatever. But these are some of the types of protocols and implementations that could maybe be architected and orchestrated together to form something like a legal infrastructure that would support the transformation you're talking about at a larger level on trade. So everybody Google verifiable claims and see what you think of it. Okay, so next up we have, we promised you Christina and we're going to deliver Christina. You're up. Okay. Okay. Can you manage my slides? Let me get them pulled. Just a moment. Yes, we can. And we're just, Brian's going to be the, the maestro. Okay. And this is on personal data for sale and EU and perspective. So what could be more comparative? Okay. First of all, I would like to thank you for this opportunity. This course is really bringing to me many ideas and that's fantastic for our researcher. Just introduce myself. I'm a lawyer and specifically I'm a comparative lawyer. So I'm focusing for example of comparative law and block chains or comparative law and computational legal systems. So quite an interesting perspective of this hybridization between law and technology. My presentation is about personal data as a counter performance. I'm looking to the comparative perspective on the problem of trading personal data. That's the main point. We had a meeting in Milan where we were discussing within the European Law Institute about the problem of transferring or trading personal data in the European Union and where I'm based. The problem is that the EU is in, I mean in between the philosophy of economics and the philosophy of identity to quote the philosopher Floridi in considering personal data in the sense that we have two main legal provisions. I'm just telling this for American colleagues who are not aware of the state of the art in the EU. One is the famous GDPR and the other one is the consumer contracts. We have a directive consumer contract directive and then we have another directive on the supply of digital content. So the EU is quite complex legal landscape basically the result of the interpretation of the legal framework brings us to say in the EU that the picture is unclear in the sense that it seems that what we can do is to barter what I define as bartering personal data for having as an exchange previous services. In the next slide I question the grounds of such a system of exchange we have in the EU in the sense that we base this on the informational model in the the consent of the people by saying we provide you information we get your consent and we can exchange your personal data for very trigger services. Consent is based on the notion we use in contract law in competition or in privacy. This is quite a paradox in the sense that some economists have pointed out the sense of people in the EU is not perceiving the personal data as a great value. They think it has but at the end they are ready to exchange personal data for very trigger services. So what's the problem? We could not openly discuss of ownership in the sense of trading because of the fact that part of the legal scholarship is very keen to protect the idea that the personal data are part of your life. So this is a matter of fundamental rights and you cannot trade yourself to quote the sociologist the consumer is now the product in the sense that you are trading part of your life. So the problem is to find a legal category for something new because of technology and there are many different perspectives. One is data protection the other one is market, the other one is what people are doing in reality because we are already exchanging personal data for services with a number of US companies indeed. So the problem is to find a way to categorize or to create a new understanding. The problem is also a problem of frameworks. When you decide you are not so happy of this boundary what you can obtain as a person as a consumer you can ask termination or restitution of data you can have the right to know how companies are using your data but what you can obtain at the end is a sort of data set that the majority of common people could not understand indeed. So all these rights that are provided by the GDPR are not very effective if you decide to terminate the trading the exchange of data for services. What's the conclusion? I'm looking by taking part to this course to get new understanding of a wider picture but I'm very interested to know about the understanding of colleagues from other countries especially from the US. I'm going to write a paper on this so I would like to understand solutions you have or you are dealing with. I'm aware about the state of the art in the US I'm aware of the new act in California from right but I would like to understand have you managed the same problem just to me. One option is to think in terms of leasing a data set to companies for a limited time or to consider data not as goods but as IP rights. That's an option. I question the point someone has raised during the course that the GDPR is based on civil law. I'm very interested to know why our colleague is thinking so because this is quite an interesting comment and so I'm trying to find a solution. Individually the data burglaring is not so bad I mean the problem is the collective dimension of this in the sense that by giving data to so many companies you have some sort of liquid surveillance in the sense that data sets are giving companies the possibility to know your life indeed. So thank you very much for the time and for your attention. I hope to get your comments. Thank you so much. Great, thank you. Great presentation. Again we're on a roll. So one quick thing she's in Davos right now but a friend of our of the MIT computational law program and frequent collaborator is Elizabeth Reneres so we should get you in touch with her. She's done deep work on exactly these issues largely from a European perspective but she's United States attorney and it's very deep in the comparative law approach for personal data protection and from an individual's perspective. One of the things that she would really emphasize now I think if she were here is understanding the type of legal thing that the personal data is and I think there's aspects of it once it's reduced to data that I think clearly can be classified as property, a type of personal property that happens to be intangible like under IFRS or GAP or whatever but there's another dimension of it as well and it gets to these legislative and deeper jurisprudential legal frameworks that sounds more like in human rights especially from a European perspective and some of those from those perspective aspects of it may not be alienable there's an alienable universal definition that makes it harder to kind of encapsulate and sell like property or even lease or license potentially much less barter the other thing that comes to mind is this new vantage point on surveillance capitalism so called I think it's made a lot of very useful distinctions but the chief one is to get deeper from the slogan that if you're not paying then you're the product or the person is the product and they have more of a real politic vantage point which is we're not even actually the product if you look at what the companies and agencies that actually create value from the personal data are selling or the value moment it's not literally what we gave them we didn't give them the product we gave them a closer analogy would be the raw materials that they refine into a product more like an oil field or like a granite ditch or something which in some ways doesn't change the idea that we're providing a valuable resource but it does change like where we are in the life cycle and it also suggests there's certainly been people that have formed collectives that own oil fields or that own for telepharm land or have other property rights and know how to get together and negotiate a better deal with companies that want to extract those more raw materials in order to refine them into products and so that they can be a fair value exchange at the right point but understanding getting more definition on what is the legal thing that we want to define personal data as I think will help us identify maybe more useful legal frameworks going forward yeah and building on that a little bit like looking at environmental regulations as kind of the way that you measure or the way that you manage you know the extraction of raw materials so to say those would be more in line like the artificial legal personality rights of you know property and in that sense would be more like you know something like a human right rather than just literally a property right there these unalienable considerations that you can't like decouple from like the data itself so true so I just wanted to add that good any other feedback questions comments musings on Christina's presentation so this is Bob I think your work is really important company in here in the U.S. that has had to react to CCPA California Poverty Act we anticipate that California is only the tip of the sword and that many other states will enact laws similar to this we've taken a holistic approach across the entire country and set up an entire team so there are entire corporate legal privacy departments that are dying for solutions around this data ownership problem and how we might be able to mitigate that. So you're in fertile ground and quite frankly might have great opportunities to consult with a number of companies that are looking for solutions around this. There are empires to be built. Thank you, that's correct. Any other feedback? Okay, well then let's all thank Christina for a very thoughtful and timely presentation. Thank you, Christina. Nice work. So next up we have Samuel. So I'm not sure if you're in a position yet to present, but if so, come on off mute and this is your time. And share your screen. Yeah, and share your screen. So just as a fail safe, the next two we have are Bob Taylor and Megan Ma. So Bob, why don't we go to you and if Samuel is able to join us, what we'll slot him in after you if you're, are you ready? Just gonna share my screen. Okay, perfect though. Sir Robert Taylor of Liberty Mutual. Thank you. So let me just go ahead and put this at presentation mode. So what I'm gonna share with you today is a truncated version of something that we did at an actual use case at Liberty Mutual. So just on a very high level, what we wanted to do is train our in-house lawyers to be better consumers of data and to apply a data-driven approach to their litigation. And what we're finding and we're having this conversation before class is that many new lawyers, some of them are getting exposure to how to consume data and coming out data literate from their law schools, but not enough. But we're finding that mid-career professionals are having this crisis of not feeling prepared on how to practice into the future. And so we're looking to not so much look into the future about what might be possible. What we're looking at is what's available today in terms of data-driven law and how can we exploit it to make better decisions faster in the life cycle of a litigated matter that leads to better outcomes. So that's what we're thinking about. So I'm gonna kind of cruise through a couple of these slides. You don't need a data litics overview, but some of our internal folks need it. Then we explain this to them. The one thing, the reason I'm showing this slide that I wanted to point out is that the second bullet from the bottom around the fact that data-driven insights do not replace legal research or reasoning, but they are a supplement. I just wanna enhance that thought a little bit. We had another slide where we talked about, and many of you remember Gary Kasparov when he lost his chess match to Deep Blue, right? That was long ago, right? I mean, I'm talking about ancient history, but he had a very interesting comment after that. You would have thought he would have railed against it. He went the other way and he embraced it. And he came up with a new game called Advanced Chess, where the human would partner with the computer against another human with the assistance of the computer, and they were playing a much higher level game. So what we're proposing is that our practitioners practice advanced law, meaning that they have computer-assisted data at the ready combined with their own experience to make something greater or better. And I think that that's really what we were trying to do from a cultural perspective is take away the fear that we're supplementing the judgment of these professionals with the data. That's not it at all. It's really enhancing it. So we're looking to create a competitive advantage there. So in our pilot, this is kind of the overview. Really what we were looking to do is to enhance the effectiveness of our department, identify and understand what the current data usage and needs are of the individual folks. So these are litigation attorneys that oversee litigation outs that's being handled by lawyers outside the company. It could be employment litigation. It could be bad faith against Liberty Mutual. I know that's a shock that we might get sued for bad faith, but it happens occasionally. Or it could be a large construction suit against one of our insurance. So they're overseeing and strategizing around litigation. We wanted to be able to produce analytics that were being able to be utilized and were actionable. Otherwise, what's the use of consuming the data unless it gives you something that's actionable? We looked to measure the impact of the tools. We were really using these tools, and I'll get to them in a minute, on new cases. So we're still seeing the results of those cases come in. We would periodically survey the lawyers throughout to say, are you sharing this with outside counsel? What has been their reaction? Has it been useful to you in setting strategy or resetting strategy? Has it given you asymmetry of information, meaning that you have more information than the other party so that you can make strategic decisions sooner that might give you an advantage to litigation? So we're doing these periodic check-ins. And then lastly, and this is something we're still working on admittedly, we want to combine our proprietary data with publicly available data to do something really even more meaningful. So that's overlaying a public dataset with our private and really giving us much greater insights. So we were calling that analytics 2.0 in our organization. So I talked a little bit about assessing the impact, what we committed to deliver to these folks was customize engagements with certain analytic providers. And I'll get to who those were in a minute. Give them individual case consultations as they requested. So a lawyer might come to me and my team and say, I've got a new piece of litigation. I really could use some data on this. We did consolidate their feedback and give that back to them. We gave them evaluations. We looked for evaluations on the platform and we did online demand training. And this was a big part of helping our lawyers advance and being able to use the tools. So at first we would fish for them, but ultimately we wanted to teach them to fish for themselves or at least their paralegals and their support staff to fish for themselves. And that was really the idea. And then what we were looking to understand of what we got deep insight was on litigation strategy impact on case outcomes. Did they really truly understand how to leverage analytics? Many of them didn't even understand how to incorporate analytics into their practice. And so this was a deep learning for many and kind of an aha moment. It was really nice to be part of that. And then we were looking to see what kind of tools outside council were using. And quite frankly, we found very little. We did look across the evolving space of analytics tools and legal litigation analytics tools. This is not designed to be a comprehensive list by any means, but these are ones that we did take a deep look at and that we have experience and understand. Some of these tools we have in-house, some we don't, but some we partnered with to kind of evaluate their data. But this is just to give you a little sense that there's a lot of players in this space and a lot of people trying to enter into this space and do interesting things. So here's the pilot platforms that we currently have been using. No surprise in the upper left hand quadrant is Westlaw Edge. They've come a very long way in their ability to do state level analytics. This is one of the reasons we really like them. Although state analytics is where it's at for us. I mean, for our company and for many others, I think it's 94 or 5% plus of our litigation is all on state courts rather than federal courts. On the right hand side, Lex Machina, some of you might be familiar, they are wholly owned by LexisNexis. They do an excellent job on federal cases and leveraging PACER. And they are a little bit more intentional about the case types that they go into and domains. The lower left hand side, we went with a startup out of California, Gamalytics. Some of you might have heard of that, Rick Merrill is their CEO. Very interesting because they are going after behavioral analytics around them, judges and state level court judges. And they started in California, but are expanding out California as a big jurisdiction for us. And they've been expanding the court of New York and beyond. On the right hand side is something called Judicata. Very interesting there, the interesting story there is, while there's a lot of startups in this space, it's very expensive to do kind of this AI. And so this company has pulled back and so they no longer are offering what we were looking at, which was a brief analyzer tool to kind of predict what the viability of your particular motion might be like a summary judgment motion and find the gaps in it and also kind of give you a prediction of how the strength of your brief was relative to your opposition. Amazing interesting concept, but they had to pull back a little bit because it's just an expensive endeavor. So those are the platforms that we had been offering. We've developed a series of checklists. This is just one example of when a case comes in, what are the things that people were interested in understanding and knowing about? So a paralegal or even a legal librarian or a technologist might go in and use this checklist of surface materials before a lawyer even gets the case itself so that they're getting both the case dossier along with the analytics. We do that intake at a midpoint in discovery and then at the decision point of prior to trial or settlement. So there's three separate checklists that we use. We are a big devotee of the checklist manifesto. And that's a book that we've really incorporated into our practice. So our learnings here is that there was no real winner among these. Each one has its use for, depending on the use case, interacting directly with the platforms is better than giving static reports because the platforms themselves are dynamic with the filters. So it's much better to teach your lawyers to use this than to surface static PDF or being a report documents. Combining the platforms gives optimal results. We often found that you get different results from different platforms. This is one of the issues that they're having right now and something called the PASER problem. The data is full of typos. The code does not look like the true focus of the matter. The names of parties are all in mess. It's prohibitive, the expensive, the user interface and that go on and on. And then we just help allow us to get a better sense of the strength and weaknesses of the various products themselves. But we've had tremendous adoption as a result of this pilot over the last year. So we did a scorecard on the current state. We found that the quality reliability, I think, I thought this was a generous grade as a B minus I might have given the C plus for where we are in the evolution, but we're getting there. The state level data is clearly evolving. We've got that PASER problem. The platform training materials really varied. We had to develop some of those ourselves that were more customized. Integration, there's a little bit of a barrier in adoption and culturally in some organizations. Outside council acceptance, we found to be relatively low. They wanted to rely more on their own experience and intuition, but I think we all understand one individual's experience is really, ultimately when statisticians would call a hunch rather than data-driven decisions. So we did find that it did work well, although you kind of gave them a personal experience. These tools are very easy to use. You can get up and using these really just by giving them a login and letting somebody play around. They're very intuitive. There was expert guidance available and they're very well suited for collaboration. So we felt that that was good. So our rate of adoption has been very high, but we had to have a very intentional program to push this out. It's still every day kind of a cultural challenge for us to push this into our organization. It's becoming more of a mandate and we were able to get our head of litigation to be quoted in an article talking about how this is the future. So fully committed to this going forward. So that's really kind of what we've been doing. I think that there's some amazing tools that are available today that you can start using and you just need to start collaborating with them. For me, I am highly interested and if anyone has any ideas about how to upscale or increase the technical competence of mid-career lawyers so that they can be relevant over the next five, 10 or 15 years in the way the law is going to be practiced. And I'm keenly interested in the way that the law is going to be trained today to fill our pipeline for the demand and our needs to have people that are technically confident in our legal department and they can practice data during the law. Here, here. Excellent. Great presentation, Bob. I have five or six days worth of feedback for you, but my tongue here is he is anybody online have any reactions or follow-ups for Bob based on all of that? I'd be interested, and this ties into a conversation Des and I were having this morning, actually, about the process by which you came up with the checklist and how you decided to focus on the various areas which were being utilized. Yeah, I love that. So we really, it was through kind of ethnography, right? So we went and watched what the lawyers were doing when they took in a case and asked them questions as they were doing their intake and their strategy and how they were filling out their case assessment and we interviewed them. What is the first thing you need to know? What is the information about the judge matter? Yes, why? What information are you trying to claim? So we did a series of interviews and just watching the way that people did their work and that was incredibly valuable to us. And we would develop a series of checklists, go back, get feedback, iterate, go back, feedback, iterate. So these checklists are not designed to be permanent, like they're designed to be updated as we get better information or more feedback. But there are pretty good sense of what you should be thinking about at various stages in the cases. So we really went to our expert litigators and said, what are the things that you think about that help you get to a successful outcome and really extract that knowledge from you and put it into a checklist so that the newbie that comes along can have the same level of thought process at least or try to gain the same level of thought process. So a lot of, you know, kind of in the trenches, design thinking, ethnography and interviewing. Nice. Beautiful. Wow. It really is this very, this is very relevant to what Brian and I have been doing. She said we talked a little bit before class, but we're slowly starting to structure something through the human dynamics lab by way of sort of mid-career executive education course. And, you know, some of that has to do with getting explicit about tacit knowledge from experts with certain types of tools like analytics, but also like legal reasoning and like litigation, which is a process tactical and being able to express it in ways that are, you know, that are, you could diagram and that you could role play and that you could simulate and basically ultimately extract knowledge from that, you know, that you could absorb and digest and then be able to practice and then ultimately can become unconscious knowledge basically that sort of guides how you work. I totally agree. I mean, we have a saying that I keep pushing out to my team, which is provide insights over information. And so, you know, I think people get overwhelmed with information, but they appreciate insights. So anytime you provide them data in a way that can direct them to the insight they should be extracting from that, it's far more valuable than just shoving a data set at them or a set of graphs and expecting them to magically understand what the insight is they're supposed to draw from that. Yeah. So a lot of it's been made to sort of classify at a moment in time like intake or mid-point or toward the end. What matters and why and what sorts of tools could get me there? That's exactly right. And I want to emphasize our program and I think many other programs like this and within companies or law firms should all be designed with the focus of how can I make better decisions faster, right? They give me a better chance of getting a better outcome at the end of the case. That's really ultimately what we're here. You should be looking at not the least side of that. Yeah. I think the notion of checklist goes to an idea that not many in the law think about very often, but that would be the idea of scaling law or scaling legal processes. And that's exactly what happens when you move to an approach that's rooted in checklist that identify the key components of a legal process. It collects data about them and then you're able to use that data to subsequently automate or compute legal outcomes. Yeah. I just, and I'll probably know what this is. I totally agree. And I think the checklist are cross functional in that. If you want to create an expert system. Yeah. Right. One of the places you may want to start, right? Is by developing a checklist and extracting the knowledge from the subject max expert through that and use the checklist almost as your first level draft of code as to how you're going to arrange, you know, your low code or no code, you know, expert system. So, so here's a parting thought. When you're doing like a big, a big software system, one way to one approach to understanding it is, is modeling and one approach for modeling that's, that has the, well, the benefit of being complete is, it's codified and something called unified modeling language UML. And so if you, if you break down UML, there's seven types of diagrams, you know, like, like state change. There's entity relationship. And then one of the core ones is sequence diagrams, which is, is a way to encode checklists among other things and other types of sequences and you're like logic trees or whatever, you know, forks and so forth is use cases. There's a few others, but now that you've gotten really good at checklist, do a Wikipedia search or like a web search for just one of the seven types of diagrams and get like, you know, 10 examples of each one and maybe think about what, what it might look like to start to fill out your elicitation of the different dimensions of what's going on. Because if you are able to diagram out all seven dimensions of a UML diagram of a system because of the nomenclature of how, you know, the arrows and the circles and the squares mean you can actually hit a button in some software packages like rational rows or others and it will encode it as like C plus plus or Java like it will make a working system and people that are good at these languages swear to me you wouldn't want to necessarily use that code, but the code compiles and it works and it, it says that there's a one-to-one equivalence between like the complete expression of, of the software package in pictures like diagrams, like hieroglyphics that anybody should be able to understand and working code. So, but the sequence, you know, sequence Uber on this in some ways like in checklists, you know, deserve a manifesto. So thanks, Bob. Great work. Thank you. Yeah. I think we need to come and do a lecture next year. So now we've got one and a half more. I think. Oh, there's a question for Bob. It looks like in another system. So before we go to that question, let me just ask, is there any, can we have unanimous consent to go to 345 basically 15 minutes beyond what we thought we're going to do, 330 now? Any objections to that? Yeah. Okay. Here are no objections. If anyone, if anyone has to jump off, we'll send the video so you can, you can catch up, but, but the session will remain 15 minutes longer. Let's see if we can nail this question very quickly. And then we'll go to Meagan and then show Samuel's video. A question for Bob. According to behavioral science judges may be irrational. How can a checklist manage that? No, I would agree the judges to be irrational. I think if you have a large enough data set of any individuals results over time, you can draw interesting conclusions from that or deduce, you know, a way to kind of confront. So if you know you've got somebody that's irrational, that is enough to at least guide your behavior in some way. And that's one of the reasons why we like to do that. However, I would suggest, yes, you can get people to be irrational, but most of the time people act fairly consistently. You know, and so it's that, that we're relying on. Yeah. So the real, so one of the great things about doing, you know, kind of like judge behavioral analytics is that you don't get in the trap of what would a law professor say they should have ruled on that motion or that objection. You get to what, you know, rational analysis. We have a predictable analysis. And that's a trial. That's what you want to know. They would look predict what's likely to happen. Well, so one of the things that we actually train is don't tell the judge what you think the law is. Taylor, you're right. You're right. You're right. You're right. You're right. You're right. You're right. You're right. You're right. You're right. So you think the law is. Taylor, your argument to what the judge thinks the law is. So, so much wisdom in that. And so many people want to, you know, argue and prove themselves right. And it's very easy using these tools to drill down to the actual dockets of the individual cases, extract the actual rulings that a judge made on the summary judge in motion, Find out what the president was that they relied on, whether you agree, whether that's the president, they should have relied on or not. that are cited in your case going back to that judge. So I guess that's the best way I can answer that question. Here, here. Thanks, Bob. So next up, we've got Megan Ma. Megan, if you have slides. Hi, Megan. Hi, everyone. Hi. Thanks for letting me present. Of course, basically what I want to talk about today is actually an ongoing collaboration that I've had with myself and a computer scientist and a mathematician, actually. And the project is fundamentally on translation or testing sort of translation from legal text to numerical form. So I'll just pull up some of the slides. I'll go as quickly as I can just so that I don't waste any more time or in case anyone has to leave. How do I? There should be a green box in the bottom of the Zoom window that says share. Anyone see it? Oh, yeah. I don't see my words. OK, so I'll just turn off my camera. So essentially, of course, when we look at other people's projects for Google Neha's project, it was fundamentally about sort of this movement from legal text and descriptive natural language to numeric form. A lot of what we see in existing law, it appears in this kind of formally logically reducible if then statements. So of course, the question that we brought before us is could translation be possible? A lot of what people have seen is how do we simplify it? How do we turn this into something more structured? So one of the interesting articles that came across was one written by Douglas Hofstadter, who is a translator. And he talked about the shallowness of Google Translate. And he's largely spoke of the Chinese Room argument, which for those who don't know, is its thought experiment first published by John Searle in the 1980s. And it's about how syntactic rule following is not really equivalent to understanding. So of course, he probed at the sort of important question, does translation require understanding? His experiments obviously tried to say no. The purpose of the language is beyond this processing of text. It requires imagining and remembering. It's a lifetime of experience and of using words in a meaningful way. So I tried to reflect on what he had written. And I looked into actually a historian called Julia Fumer. And she's a historian of sciences. So science is obviously a field where the language that is used has a sort of reputation of universality. So what she had considered was a historical example of these 18th century Japanese scientific texts. And what they were from were actually Dutch translations. And so between Dutch and Japanese, they don't actually share similar intellectual languages. Sorry. I am not using my PowerPoint very well. And what she saw was that they required actually conceptual transfer. So it's a sort of situation of ideas in different conceptual worlds, and that you needed a migration between these conceptual worlds in order for translation to actually take place. So does this mean that with new experiences, we will get sort of these new interpretations? Marie Hildebrand, who is a legal scholar, actually questioned or teased this premise on addressing the challenge of translation. And what she saw was that actually in order for translation to occur, we need to first understand what exactly is the sort of language of statistics. We need to learn the language of statistics in order for us to properly reason and understand between what exists in text-driven law, so what's in descriptive natural language, to a sort of numerical form. Of course, when we think of translation, there is the risk of being lost in translation, things that don't quite move in the exact same way. So the larger question is, of course, could law behave like mathematics? And what was brought to the attention before was the World Justice Project, the Rule of Law Index, and it's fundamentally kind of taking a qualitative assessment, the rule of law, and transforming it into a quantitative tool. What's interesting, actually, is not really how what this rule of law index is sort of created from, but actually how they measured adherence to the rule of law, which is that it kind of uses their own World Justice Projects, their own principles, in order to make this evaluation. So then the question becomes, what are the sorts of risks involved with kind of identifying the rule of law in this numeric context? And a numeric context that was created based on the evaluation was somewhat infernally created. So I'm going to skip a part of what I've tried. What is also in sort of our ongoing paper was this historical perspective of how governing numbers is not novel. It's existed since Aristotle and two Leibniz and Boulle. But I want to kind of center it on what exactly is our project given this background. So ours kind of tackles this semantic conundrum of what is the significance of meaning in legal language. And what we saw is from a statistic standpoint, meaning is approximated. They could apply word analogies as the logical basis, and meaning is gauged by the statistical probability of the response. So in recognizing the context and relationship between words, meaning sort of hinges on the frequency of its appearance in a particular setup. The project test translation, then, by identifying the rate of convergence in the meaning of legal terms. So in determining the sort of rate of convergence, the project is tackling the existence of legal concepts. The crux of the project is twofold. It analyzes the processes involved with legal interpretation and reasoning, and critically assesses them against the function of law. So how we'll try to do this is that we're going to be using US Supreme Court cases from the 1700 to 2019. And that's sort of our working data. Then we'll be applying NLP technology, so specifically word-to-vec, to parse legal documents. Our hypothesis is that by analyzing the components of legal language with these techniques, we can actually begin to translate law to numerical form. Moreover, it's interesting to understand what's the importance of contextual understanding in order to sort of recognize the significance of meaning in legal language. So what I mean by all of this is that, actually, this isn't necessarily new. What it's existed before, and there's a paper that came out about roughly a year ago, called the Deep Learning in Law, Early Adaptation and Legal Word Embeddings Trained on Large Corporate. Essentially, what it is, is that they've created something called the Law-to-vec, and they've taken legislation from English-based countries, including some sort of legislation from translations, and they took this and tried to sort of pull out word associations. So they have a table where, for example, they pulled out words like article, and what they noticed were sort of like the top similarities or top meanings were convention section clause provisions. We are trying to take it one step further because a lot of the cases that we do find in the US Supreme Court opinions is actually that meaning or interpretation takes place in words that aren't what we would theoretically consider as legal. So the words that this article that existed the Law-to-vec article was pulling out things like crime, pulling out things like felony, security, fraud, things that they might see as legal words. What we're trying to do by parsing these legal documents is looking at cases, for example, like the 1993 case, Smith v. United States, Scalia has a famous dissenting opinion where he questions the word use and the use of, for example, arms. And in this case, for those who don't know, is that what happened is there was an illegal exchange where someone had exchanged firearms for cocaine. And it was brought before the court, is this considered a use of arms? So this sorts of stuff is what makes interpretation interesting. So we're trying to see, can this sort of translation exist from this descriptive natural language to numerical form in things that are like very normal words like use? And we'll obviously be looking at the context in which this word is seated. So yeah, so thank you very much. Outstanding. We love it. Anytime word-to-vec is part of your presentation, we get excited. Nice job. Any reactions to that? I have to notice that there's some resonance with the first presentation from Neha. Anybody have any feedback from Negan? So Negan, this is Bob. I'm just curious about, you might have mentioned, but are you publishing the results of the analysis from the Supreme Court cases? Like how might we be able to consume this? So this will be, so right now, so Dmitri, so I think I've listed their names. So myself, I'm sort of, I guess, the jurist behind it. Avalon is the mathematician and Dmitri is the computer scientist. So he's actually working with the corpora right now. It's going to be, it's taking a bit of time, but we're hoping to publish this by the end, we're hoping by summer. And I think we'll try to get a first like running draft with the MIT Computational Court, if that's okay. Yes, I think you're saying that's okay. Yeah, that's one of the reasons why we want to sort of get, put out feelers with my presentation. I couldn't act, a lot of my part is the, where I'm attempting more of like the philosophy of logic and language and legal theory. That's sort of my portion. And I'm going to take sort of this knowledge and try to communicate in the best way I could to liaise with Dmitri, who will kind of assess what are the words we want to pull out? Because right now, what exists with Law to Beck is that obviously they're taking what is perceivably just legal word. We're looking at focusing on meaning interpretation and what approximating meaning and how that differs from actually the act of interpreting. Beautiful. Yeah, I especially love the reference to the translation efforts that Douglas Hofstadter's doing. He's got a book called, I am a Strange Luke, that I would recommend everybody to read. And he also has a lecture, the Stanford presidential lecture, which he gave some years ago, on the idea of analogy as the core of cognition. And in that lecture and in his book, he does a lot of describing ways to formally represent words, numbers, and concepts like this. And so I think that is definitely like one of the big meta themes that we have is like, okay, well, how do you distill law, which is kind of this kind of nebulous thing that's not as precise as it could be. How do you distill that into these kind of concrete, checklistable sort of functions that then you can start evaluating the relationships on them. And so I found this especially, like I think this is very needed. And the, I had a question, which is, are you guys using the Harvard case law access project data or where are you getting your data from? Yes, actually we, so there was a, I can't remember the school for now, but it was called legal linguistics and they've put sort of these cases online through their database. How cool. So I need to go back to which school that was, but basically they've put it all online and we just took that and we ran with it. If that's public, I think it would be really great to share that in the telegram channel. I know we'd like to look at it. Yeah, and just to follow up on your previous beautiful statement about publishing the MIT computational law report, we had a quick editorial meeting after you said that here. You announced that we would accept your data. And so Florida a little bit and then ultimately where maybe there's some aspect or aspects of your, or parts of your article where people could try to reproduce your results. And that's really the gold standard that we're going for. And that's something we could bring to the table unlike starchy old paper law reviews can't help you that way. I do have to mention that in terms of the data that we've taken, it wasn't in a plain text format. What happened is, is that they simply given a list of sort of all of the US Supreme Court cases from the 1700s until 2019. And what Dmitri has helpfully done and why it's taking a while is because he sort of had to pull it up and he's created sort of this, he's created this algorithm to actually pull out the plain text. I don't understand. So it was effectively had a list of all of the cases. Beautiful. One other quick observation, as you go to the, you know, kind of like next levels of analysis, some things to, I'm just sort of synthesizing best of some of the things we've tried to do here at law.mit.edu or computational law research program, but also pulling threads from, you know, groups like International Conference of AI and Law, which has been around for, you know, decades. You can, if you can, you know, like the simple way to say, to talk about it for modern data scientists entity extraction, but I think the deeper, what's most relevant there in a legal context is you can start to identify the roles. So yeah, it's Exxon, great, or it's, you know, Acme, LLC, that's critical to identify the parties, but what role did they play here that was relevant in the case? So, you know, this most superficial one is plaintiff, plaintiff, defendant, defendant, plaintiff, plaintiff, but there's a substantive role. So they were like, you know, they were, you know, like landowner, you know, pursuing a trespass claim. They were tort-feasor. They were, you know, whatever. They had a certain role and other parties had certain roles. Identifying what those were is like jet fuel for analytics. And then from there you can start to get to relationships, or well, it's actors and actions, but beyond the action, like, you know, there's some event like, you know, the tanker exploded and leaked oil or whatever, something happened. But then there's also, the action can be understood once you've identified the legally relevant roles as basically relationships. So, you know, you had like someone that was, like their relationship was they were in a butter, like sure their landowner was a role, but they were in a butter and that really matters because that's a whole different legal framework. So roles and relationships of the actors and actions is something that you might be able to make progress on once you've got the data in a structured way. You may be able to start to assign those, or even if you can assign it to, you know, like a 10th or a quarter of the corpus, well, then you can make potentially real progress on that subset of the capora that you've extracted with the algorithm. Thanks. It's almost like the head notes that, you know, Thompson was would pull out, right, to kind of draw out keynote, you know, items, right? Bullseye. Yeah, that's a really smart idea. Thank you. So this is still an idea, you know, whose time is in the future, but we think that this will be one of the purposes of computational law is to have, you know, almost like automagically self-deriving head notes, including for things that are emerging right now or coming across your desk that are being published by Westlaw. So let's see, any other questions or comments for Megan? Brian, you listening with your pride of how many times the word corpus was used? Yes, yeah, we love your use of the word capora, the plural of corpus. So yeah, thank you very much. This is an outstanding presentation. Thank you so much for sharing it. Great, yay. And I think we've got one last, one which we'll have to do quite, actually I don't think technically we're in a good position, I don't think Zoom will let us play a video. I guess we can share it. You could, but the audio doesn't. Oh, the audio wouldn't come through. Yeah, we have to pipe the audio. We don't have the software for that. Yeah. So let's just give Samuel one more chance to see if he is able to. Yeah, I noticed connections back. Present. So, Samuel, I'm sorry, I don't think we're gonna be able to do the presentation for you, basically. And so part of it is Q and A, okay. We will post a blog going, and tell it going for it usually, I think. Yeah, I'll do it. Actually, we never kill them really, but. I'll throw the video in right now so it can view it. Great, so that is our kind of final, any final feedback, I'm offline. But so, I wanna thank everybody for making this a really great class and the quality of the final presentations I thought was really, really illuminating. Almost starting to think next year we should do the presentations like on the second day and then spend the third day discussing them and going deeper. Yeah, we can space it out. Yeah, maybe space it out differently. But really, really impressed and delighted by the presentations. And we hope that you will stick with us through the year by going to law.mit.edu, get on our mailing list if you're not, and we're going to start a monthly community building call where we'll come together and have some people will do quick updates on breakthroughs in computational law or something like that. But then mostly have an opportunity for people to talk, ask questions, introduce themselves and have catalyze more idea flow is what we call it. So it's among the things we'll be doing coming up next. And we also have a big event brewing for the end of April or the beginning of May here at MIT. It'll be a computational law summit. So if you've ever wanted to come to MIT in the springtime where it's actually nice here and on the frozen tundra that would be a great opportunity for you to do it. And yes, of course you can join us remotely. So with that, let's close up the class, shall we? I think... Thanks very much, guys. Yep, so the iPhone gavel app has been invoked and this session is hereby adjourned. Thank you very much everybody and we hope to see you next year and before. Bye bye.