 So, like, jump in with questions, if anything's clear, let me know. So essentially, I'll have three parts to this. I'll teach you enough law to be dangerous, at least with respect to class action lawsuits. And then I'll tell you about what we've done, what kind of results we've gotten, and where we see things from here. So, let's imagine the following real scenario. It's 2010, 210 million gallons of oil spilled into the Gulf of Mexico. 22,000 businesses have been affected. And all these people are seeking legal relief. And there are two issues for the legal system. So the first is all of these people have been damaged, but not necessarily in a big way, right? So if you have a bed and breakfast and the tourism industry is down a little bit, you're probably not looking at multi-million dollar settlements. And you're probably not going to hire a lawyer to help with this. On the other hand, all of these people have the same claim against the same company for the petroleum in this case. And so the law is pretty similar. And if you had everybody go out and try and bring their own lawsuit, even if they could afford to hire a lawyer, it would completely slow down the legal system. So we need to aggregate legal claims. That's essentially what class action lawsuits do. They aggregate legal claims. But they're interesting for several kind of special reasons that I'll discuss shortly. And if we kind of generalize the DP case, in general, a class action lawsuit has lots of people who experience a small harm. In the end, there were 100,000 legal claims and the economic impact of the BP oil spill was 8.7 billion dollars. And this is pretty typical. Really large harm spread across lots of people. A lot of lawyers are going to find out about this harm. And they're going to file class actions. In the BP case, there were 80 different class actions. The lawyers are incentivized because they're going to be paid a portion of the settlement. And so they have an economic incentive to pay these class actions. But then something interesting happens in the class action workflow, which is that a judge actually decides who the class is. A judge decides what the definition is that is going to be used to certify the class. And everybody who matches that definition without having to opt in is automatically part of the class. So in this case, it included things about their geography, businesses, and people located around the Gulf. And what they did, people in tourism, people in agriculture. The other important thing that the judge does is the judge selects one lawyer to be the class counsel, to represent the class as that class's lawyer, and to kind of go forward with the lawsuit. The second important thing that happens in class action lawsuits is that the judge has to approve the settlement, which means that in contrast to the other types of civil litigation, which settle and 98% of cases they do settle, in private, and we never know what the settlement is, in the class action lawsuit case, there's a public opinion, a public document that's written by the judge that actually tells us what the settlement is. Not only that, it tells us where the money is going. So in the BP case, we know that the settlement ended up being $13 billion. The lawyers got $600 million, which worked out $1,000 an hour. And we know this data, and we can work with this data. So I would put this rigorous understanding of class action. How many pounds per hour do you say? A thousand. But probably more for the partners because that included paralegal funds, and paralegals are paid. So the incentives are clear. So I would put this understanding. Sandra and I formulated a couple of initial research questions. Sorry, quick additional question. Who designed and decided this process? So it's kind of designed naturally through the common law system. But the modern process was a reaction to various kind of social issues in the 60s. So this process was kind of born in 1966 and invented in the United States, partially as a result of school segregation issues. But that was the issue, right? We have lots of harm distributed. And so that actually meshes nicely with the research questions. It's unclear whether this is a good way of deterring this kind of distributed harm. We think it might be, but no one's really measured it. It's unclear whether the lawyers incentives are aligned particularly well. And it's unclear whether another structure for funding these things, for organizing these lawsuits might not be measured. And so we started thinking about how we might start to answer these questions. And as of last year, there's a data set that might help us do so. So for the first time ever in 360 years of U.S. legal history, someone's actually put together the data and made it available to researchers. This isn't just limited to class actions. This is all U.S. law. 6.7 million cases, 40 million pages all scanned in a relatively convenient JSON format that we can work with. And so what Centro and I did is we wanted to find all the class actions buried in this data set. And so as an initial approach, we did combination of word counting and random porous would work surprisingly well. It showed us that there were about 13,000 class actions, which seems to work out if you compare it to other research. We're working on more sophisticated word of vettings and LSTMs, but essentially we want to first find the class actions. And then we want to extract information. Some of the information is obvious, like what the attorney is and how long the cases, things like that. Other things are harder to figure out, right? What kind of opinion it is, whether it's a settlement or a certification opinion, and whether it was granted or denied. Those things require us to build their random porous model or something else to try and tease out those features. We're working with data augmentation, replacing words with their synonyms. That's a lot of fun, if you're into that. And then when we find the information, we can try and gain some sort of statistical insight. And I'll present the initial insight that we got now. The first question is how long did the things last? And they tend to settle quickly. And this matches our legal intuition. There's a rule of thumb among lawyers that if you make it through certification, you dispute those from $20 million to $200 million. The reason is that suddenly, instead of having one plaintiff and one defendant, you have a class of plaintiffs and one defendant, and you have a lot of leverage. And so there's a huge incentive to settle quickly, and we see that reflected in the data. We also see that sometimes these things last for 20 years, creating tremendous legal costs and probably leaving very little for the class at the end because you have to pay the lawyers for all that time. So that's interesting to know about the struggle. Another interesting thing might be to try and figure out how these lawyers do. And this is a striking graph. So what we did is for each lawyer, we figured out what percentage of the successful certifications. We've heard that this is kind of a big step. What percentage of those are they personally responsible for? And you find that 27 or so percent of lawyers are responsible for all successful certifications. The other way to look at that is that 75% of lawyers never win anything. And given that this is kind of a specialized case and people spend their whole careers doing it, it's shocking that so many lawyers don't seem to be particularly successful. What I thought even more, I figured this would be a mistake. We've asked people who are either practicing attorneys in the space or who are academics who engage with the space. It seems that this is actually unsprising. It seems kind of in line with legal intuition. Most lawyers don't do that well. The concerning thing is that the court is appointing these lawyers, because the courts seem to be picking lawyers who aren't doing particularly well to represent potentially very large groups of people. Aren't there just, because there are not so many class action lawsuits, right? 17,000. Aren't there just a lot of lawyers that do one? And isn't that just most class action lawsuits are not one? And therefore, there are just a lot of lawyers that do one or two, which are both not one, and they're there for 0%. Next slide, I would say. But to answer the question, it's like a specialized process. And you have some attorneys who bring thousands of these things, or hundreds of these things. So there are definitely attorneys who are two, kind of got lucky, because they found a class and they were appointed as class counsel. But for the most part, these are kind of specialized. But if we look at the next slide, we can look at, for each attorney, how surprising is the result that we're looking at? So essentially, we figured out for each attorney and for each judge, how many successful certifications where they involved in. And using like a binomial test, we can figure out how surprising is that. So the right side of the distribution are those people who are unsurprising, right? That means that they either perform the average or below average, they might have been involved in one class action. They lost it or one class action. They wanted it. It's not surprising. The left side of the distribution is a surprising side. Those are people who might have been involved in 50 class actions and 49 of them have been one. Or 50 class actions and 49 of them have been lost. The same is true for lawyers and judges, right? You can think of them in similar ways. The judge sees these class actions of lawyer, brings these class actions. We want to see like, what are the outliers and they're on the left side? That's another question. Isn't this confounded by sort of industry or specialization that there are just some lawyers that do smoking things that are always one or other things that are always lost? Isn't this more of a topic-related thing than a person-related thing? Because now the claim is quite strong, right? Lawyer A always loses. Lawyer B always wins. As if the other variables are the same for both lawyers and judges once. So if we found that, that would be a good explanation for it. And if we found that, then that would worry us about all those. So securities class actions are exactly an example of this. Every time there's a big change in stock prices, some will bring a class action on behalf of those people who fought right before the stock price change. Or those people who sold right before the stock price change. And those people tend to not win because there's nothing there. But then the question is like, should our legal system allow for those people to bring very extensive lawsuits? These are thousands of hours of judge time that flow into it. And is this the right structure? Yeah. Yeah, one question. Do you know how this graph in the previous one compared for like class action lawsuits versus just like other... No, we don't. So this is like the first time that everyone's tried to do this for any kind of lawsuit that we're aware of. At least with this scale. So people do like 100 or 200 kind of high level. But it would be really interesting to compare. Absolutely. Really this related with the differentiation of the skills because when you deal with a septal, you are not inclined to legal knowledge but a merchant or dating knowledge. So it's another skill. And maybe lawyers that are good at law are not good at the target. That could be true. But again, then we would wonder why are we... Shouldn't we have some sort of... We have all these, you know, hurdles to becoming a lawyer. So maybe then we need to go back to legal education and ask like, are we teaching people the right things? Or are there too many lawyers? So this people seem to be tough. And one kind of obvious explanation might also be well, all those kind of successful lawyers are with all the judges that tend to grant certification. And so we can kind of get... Exactly. So whether the certification was granted, did the judge approve a class action to go forward or did the judge say there's nothing here to go in? And like we said earlier, right, that changes the dispute from like a relatively minor settlement to something really big. And so it's a big deal. You can define it differently. But that's an intuitive way of defining. And so we tried to match, right? So maybe the judges were on the bottom. Sorry. Yeah, the judges were on the bottom. And the lawyers were on the top. Are matched in such a way that the successful lawyers are always working with the friendly judges. And that explains the discrepancy. And so what we did here is we linked lawyers who work with judges above average. So if somebody, if there was a lawyer that had five cases and four of them were in front of the same judge, there would be a link here. And these are the 20 most prolific lawyers and 20 most prolific judges. Although you get the same picture if you look at the top two. So we see no systematic connection between the successful lawyers and the friendly judges and something else is going on. This is overclass action. This is overclass action. So you could have somebody who's like Mr. Andy Smoky and he does a lot of things. He's very friendly to things that are smoky and he doesn't handle them in class. The kind of smoking lawyer I know he's very friendly to this type of issue. Yes. And you could try and play that out. I'll take your data and sort of look at all the things that the judge handles and try to get a grade for them on the topic. Absolutely. Even within class actions or beyond class actions. That's right. Yeah. Because this doesn't work. There's nothing going on yet. And so that kind of brings us to the next steps. And I'd be curious kind of to see what the room things where this might go. But one obvious place to go is to say well these things are meant these class actions are meant to rent these as sort of social issues. And so let's measure whether they do so. Right. There are class actions that are brought for smoking. There are class actions that are brought for school segregation for workplace discrimination for securities fraud. Let's see if a class action actually results in the social impact that we expect by trying to kind of cross correlate those different data sets. Another approach would be to try and understand the financial dynamics of the C. What is the expected value of a class action on day one? And what does it depend on? Can we build a factor model for that and extract financial data from court documents? This is particularly interesting if you think about alternative ways that we constructed these things. A lot of people propose options. So have the lawyer buy out the class on day one put up their own money and then they own the legal claim and they get to go fight it and they get the whole settlement but they have to pay everybody on day one. And if they do that we would expect less frivolous lawsuits less lawyers who are willing to go down this whole path having spent potentially millions and billions of dollars. It may be slightly more realistic variation of this if you have a third party financing for lawsuit and that would have an interesting effect on the way incentives shift. And we can also think about this kind of more globally we're only looking at the U.S. So in Australia this third party financing is the way class actions are done. Virtually every major class action is actually funded by some other third party investment firm and there's actually publicly traded litigation financing firm in Australia called Bentham that is like specialized in this. That's what they do. In India class actions are a new invention. They were brought in in 09 as a result of a lawsuit that's been dubbed the Indian Enron. And it's interesting to see in a jurisdiction where you didn't have class actions and now you do how that might have changed the way people approach handling this kind of social impact type of litigation. In Rwanda there are no class actions at all but people propose that they ought to exist and looking at Rwanda's history. Maybe this would be a really interesting area for class actions to be introduced and so we can see that. What is the impact of not having class actions on a legal system? What they did in Rwanda in the end is actually instead of aggregating claims they just created more judges. So they appointed people in different villages who were village elders to act as judges. They would meet every Sunday and they would just sentence a bunch of people. So that's how they approached it there. But maybe a class action would have been a better different way to handle this. The Pacific Islands are yet another example. Some of the Pacific Islands propose that there should be a lawsuit brought against the people responsible for rising sea levels because those pose and makes a central threat on somebody living in a remote Pacific Island close to sea level. So the question is could we have a global class action on how that might look? They don't presumably play in in the sense that you need to figure out who this class would be and how large it would be. It would also play into trying to figure out who would have jurisdiction and how you would divide that up. So that's kind of exciting and interesting areas. I have like a couple more minutes if you guys want to ask questions. That's all the content I have. But thanks for your time. I have one thing that's pretty important. The second one. So one was there's probably lots more from natural waste than there are anything else. And so the question is about impact. Do you want to be able to do something that's sort of narrow? Do you have any question on which is that one? Because there's more class rock for you. So there's one data set at Stanford that's kind of kept under close guard. But that would help us because they have a securities class actually clearing us. So there's a lot of data there that we try to use that as training data for what we're doing. We might be able to start answering that either with time series data of share prices or various other things. So that would be kind of an obvious way to go. Because I mean the big questions people have is this helping? Or is it just a scam or money in the legal profession or whatever, right? And the other thing is of course we have a really interesting relationship with Australia. So we came up with something that was interesting in the sense that or in particular that it would promote jobs and the economy and blah blah blah that they would be interested in contributing data. Okay, we'll keep it in mind. I just don't know exactly what the question was. Yeah, you have to phrase it in a way because it's with the government there, right? It's with the government but it's a new government so they don't like screwing the old government. Um, they probably love it. But their big priority is that they face uncertain economic times and they want to have more people in the long term. So if you can point out something that is a day to day innovation that would help them change that role it certainly would help. Maybe the Rwanda model, like local charges and all the various, huh? Yeah. Yeah, well, maybe. Yeah. Um, did you have you done a kind of category breakdown for financial and product liability environmental? But that would be the next thing I would do and then look at basically run the analysis again in my category and see what you see. Yeah, I think that makes a lot of sense because you're going to have these sort of networks of people everybody's ego or tobacco or whatever it's sort of It may also spark ideas about other datasets or other dimensions you could add to it to see things like impact. I mean, another thing actually might be over category and over time just talking about asbestos. So asbestos. Well, you know, the sense that I have on no evidence at all is that, you know, it was a good thing to turn into an attorney to do another one type of and it's not actually having an effect anymore because all of these cases are paper. They're about And the hope with this is that it closes the book, right? Because you'd say the class would be everybody who's been affected by asbestos. Yeah. And then once they receive their settlement, once a class action is done they're barbed for future suit. So really after the first class action, there ought to be nothing. Now we know empirically that's not true. And so it would be interesting to see how people can construct new classes around existing classes. The way you do it happens in place, right? Work that way around. That would be interesting because people don't have a great idea of that one in particular. I mean, the original thing. Sure, right? Maybe the original two or three. Sure. Yeah. Yeah. Yeah. Yeah. And it's also from a restorative justice perspective. Like what different, like deterred bad behavior, but then also there's like the ability to restore the people who have been harmed. And if these different systems have a differential effect on people who have been harmed, my kind of like cartoon version of the class action is like you get answered from like a, you know, site and then you get like a hundred bucks at the end of a differential in our system. Whether it is differential. You can kind of adjust it. You can follow up if you can have a good lawyer, I guess, but like never inequities in that distribution system. There's this huge like legal debate about whether this is about deterrence or whether this is about restorative justice. And most people seem to be on the deterrence side. And you actually see like empirically if you could opt out of the class action, like if you've ever received one of those letters it says you can opt out and people who opt out are much better than everybody else. Which suggests that everybody else isn't really getting the best deal. And so that's interesting as well for most restorative justice. Yeah, so kind of like following up on what a bunch of people have been getting at with the breaking it down into different types of class like, you know, environmental products liability, whatever it is. You know, if you could start creating almost like a dashboard, that can be really interesting to like look at, you know, cost, duration, likelihood of success, like success of those who opt out even. Because then you could kind of like have like a transparent way of understanding what your rights are. Not just as a lawyer, but also as like a consumer who could potentially be affected by what we see. So, when it was started justice and one of the things that immediately occurs to me is usually people being sued are not long enough, big enough to do restoration. The only person that can do restoration is the government. Because they're the only one that can like put up a trillion dollars a year for whatever and so, which can't suit the government. Generally, yeah. Yeah. It might have to be some different sort of mechanism. Although often these are deep pocketed defense, BP, so Morris, kind of. It's people not. Well, it's pretty serious. A lot of people know it. It's much more than the company. Or like school segregation, like the school district probably doesn't have money to compensate everybody for the amount that they've set. And of course, you know, a large chunk of the whole population for the whole life and what is the value that certainly not we were covered. I guess the time aspect, because I know at least I don't know a lot that the approach a lot of people are taking to I guess the opioid crisis is very different because there was some issues with settling tobacco in the 90s. So a lot of the strategies kind of change for class actions over time. So I'm wondering if that comes to the point at all. So there was a big revision in like 05. And we should segment it by type and then see where the big revisions were and see if you could use that. Sorry, just to follow up on that. One of the things that irritates me about opioids is that there are doctors, specific doctors that just prescribe all sorts of things. Like you can't go to a dentist without coming home with a prescription for opioids, which is ludicrous. I mean it's just crazy. But you can't sue all dentists. They probably wouldn't be at the use of your legal resources. Well, I mean that's a lot more than some particular company. And also the deterrent that we have would be huge. I think that's what you're saying is normally it's a bunch of people suing one person. But what about the other way where one person can sue other people? It's called the cross-claim. With class actions it's not uncommon in the cross-cares for them to cross-claim everybody in town for suppliers or partners to sue people to pass by one day. That's awesome. But for instance, like the opioid thing in particular, I mean, the company is not, they did all sorts of things, but they were not the ones that prescribed it in the end. I remember going to a dentist's office and the dentist had put up an ad or you should ask for opioids. That's what you really look like. What do you have? They don't need to say that. That's someone who just had a role. It has got to help you. I wonder if you're smart enough. So, I wonder, I don't know much about class actions, but I wonder if there could be a class of like tort fees or a class of defendants. So you're going after a class of like male fees and dentists. You're going after a class based on prescriptions to distance yourself. They probably violate due process, because you have to be served notice and if it's a class of dentists and a determinate class of dentists so it's an indeterminate class of dentists, they wouldn't all have notice. You would have to serve each of them individually and have some proof of service and things like that. So that would be the challenge with trying to sue all of this. It could be really interesting, because every one of those dentists has a certification and pays taxes in a particular way and there's other sort of government things. So you know exactly who they are and in another completely different database there's food, wealth, prescription for what? You would also, I mean, you would swamp the legal system. It would be fascinating. If you found a way to just like bring all of these, like sue all dentists who have, you know, encouraged people to do some very quick together the case, do with notice and so on and so forth. And find like 10,000 cases, right? And that would be interesting. I don't know how the legal system would react. But a human doesn't have to actually look at all of that. A judge? Right, under the current system. So you create and file a law, you could do that computationally. And then a human judge would sit down and say, okay. Well don't the judge, I mean doesn't the judge sort of like look through them and say, I'm sure they'd be hearing for each of them. I mean, it would be really, really interesting. Or they would say, you know what, let's have some of this application of law stuff to deal with it. Yeah, that's the hope. Yeah. Well, I think, I mean, from the group's point of view, this sort of thing, where you can really have very different dynamics to it, is really interesting. And then, I better get on my podium for a little bit. So, you know, if you're going to hang out with the ABA guys and talk about access to justice, but in fact, people like us, you don't have access to justice, because you can't afford it. You have to be part of a class action or somebody else is paying more or go paying somebody to re-promote. So that's one of the things that you're going to see, is that you have the ability to change the dynamics of the hurdles in the losses of the way that they be very different. But we have to be careful because if you're new to all dentists and then nobody can get their rollers pulled, dentists are all passed. In law, it's my impression that a lot of the decisions come down to this kind of human feel, like reasonability. What are reasonable dentists have given hope to this person? And that's why people appeal to this like a human person must look at this. It can't be computational. I'm just curious what you think about those ideas and what I see in the law, which seem to be very, very much about you can't put numbers on this. It's about what are reasonable. The party line is that the reasonable person standard is an objective standard. You can follow up. You can follow up. But that would be an objective standard. And that, you know, we look at custom and we look at what would most dentists do. We ask dentists, what would you have done? And it also will pay you like an expert would this be. And things like that. So the interesting thing would be that trying to do it computationally. Representing the reasonable person computationally would probably be more in line with what the legal fiction is than what is actually happening. What I'm told is that increasingly what you're seeing is you're seeing computational methods looking at cases and saying this one is going to go this way and that one is going to go this way. So in the sort of automated discovery process you get to a point where you can say 92% of the time you're going to be granted this or you're going to be denied. So it's not going to be legal in France. So to run analytics on the way judges are going to act, there's been outlaws. Really? It probably means it works. Yeah, I think it works. There's important Germany and I think that outlaw automated use of anything in cases which is kind of... Really? Yeah. So I can't see Google. Probably it would be difficult. I think there's a lot of competition. Yeah. So I guess one point that's interesting to me is that I read maybe one or two years ago there's a lot of companies that you can have class action through them anymore. So does that mean class action is kind of going away or is you're signing off to do that like you can't even sue companies anymore? Like Jay sent me a letter saying they're setting up like an arbitration system. I can sue them but they choose who the arbitration is. Which means like they're going to choose somebody besides for them. Yeah, I mean arbitration is like this big hot topic for any of it everywhere including like all employment contracts. I don't know if it's been litigated like they can say whatever they want in the contract. But you can still try and see where it goes. And I think that like with the BP case the people who were suing were clients of BP right they didn't sign anything. And so I don't see I mean maybe some people couldn't sue BP because they like phoned like you know some asset that was from BP but for the most part like these are third parties and so it would be hard to ban it. The only thing about the pros and cons is that often it's good in other contexts you don't drag on for years it doesn't cost a lot of money they tend to be, they can be fair I don't know all the statistics about it but it's not you know it's not necessarily bad it reduces stress on the people right so if they're choosing who the arbiter is that's my proposal but you know if there's some sort of fair operations then my view would be is that if it's more than one standard deviation out from the arbitration point of view you don't get the brutal lawsuit because it's only in the gray area yeah I mean it kind of gets into the concept of online dispute resolution generally there are places that have implemented like schemes for like civil claims for online disputes and you can just go online you fill out all the information in it delivers the judgment like automatically and there's that group from Australia who came into the legal hackers here and I think they did some report where over some period of time they went through like 600,000 cases with this system whereas in the past with like the same amount of time devoted to judges they would have gotten through like 4,000 cases and so it seems like to be this like incredible efficiency gain and I think you know just like taking the discussion from something that's static and like reframing it as something that's dynamic and computational will get people to see more of the utility of it in fact I was an online dispute resolution arbitrator for domain names versus trademarks for a few years one of the interesting things about arbitration is there's a lot more capability in a voluntary way to set the process and so it doesn't need to be very a neutral third party arbitrator but you can design the civil procedure for filing it and the workflow and everything so that it goes through like a platform so to test computational law in a legal context like this it actually might be better to start with arbitration where you could very quickly design and deploy a system that can be computational that could maybe over time be adopted in public courts that don't move so fast so you're starting using arbitration as sort of a test for stuff because it's a much lighter system it's less constraining by tradition you can just have a few parties agree to do it with different methods and then you can do it that way I think about that a lot because like the idea of like data and property law one of the issues when I talk to people about the idea of data arbitration laws is the idea that all sort of litigation or insert action is going to do with data property is going to have to go through these kinds of claims courts that are already checked where you're not really actually going to get into people and talking about how do you set up forced arbitration agreements between consumers and providers where they're like holding data or when they like violate an agreement that you have about how your data is going to be used by the client and all of that could be completely automated it's a perfect fit for personal data yeah so that's a great place to start I mean generally financial area is a great place it's all digital we sort of know what outcome the number okay this is a really rich discussion I appreciate it so let's see well you're not the company is a lot that is not open right