 Okay, so welcome to this fourth and yet in this HMI Data AI and Society series of seminars. We're delighted to have with us today, I'm Clinton Castro, I'm from Florida International University. Let's start by acknowledging the traditional custodians of country throughout Australia, and the continuing connection to culture, community, land, sea and sky, and paying my respects to Elders past, present and future. I'm recording this in Michalago, which is Narago lands. Sorry, my respects to the Narago people in particular. So this seminar by Clinton Clinton's an assistant professor of philosophy at Florida International University. We went after a PhD at the University of Wisconsin, Madison with where we're supervised by Mike Titlebaum, a close friend of the ANU. Clinton is going to be talking about just machines. And as usual, we're going to keep the introduction short and sharp. So Clinton, if you'd like to share your screen with your slides, we'll get started. Sure. Thank you very much for the introduction and for the invitation. I'm very excited to be with y'all today. And this will be my first, this will be my first go at a zoom presentation. So hopefully it all goes smoothly. Fingers are crossed. Okay, so I guess I could just jump into my next slide. I may just as a heads up, I may talk a little bit quickly because I do want to get through everything on time and I have a little bit to get quite a bit to get through. So I hope I don't go too quickly. I'll try to be articulate, but also I will be trying to stay on schedule. So it might go a little quickly. Hopefully not too quickly. Okay. So my broad topic, my really broad topic today, most broadly speaking is fairness as it applies to predictive scores, which I understand as mathematical expressions used to describe or predict a group or individuals attributes or future behavior. Just some quick examples. I think a lot of us might be familiar with this, but here's some examples of how they've been used. Famously criminal justice, for example, whether to grant bail, predictive policing and figuring out where to send officers or who to target. Education is an example we'll talk about a little bit more today. Namely, one example being determining effectiveness of teachers. And then finally, here's a really big set of examples. This isn't the only other context, but it's a big one, the market. So predictive scores have been used to determine consumers age, ethnicity, gender, frequency of purchasing general apparel, television usage, job security, allegiance to buying main brand or generic drugs, likelihood of moving to another merchant, likelihood of smoking, and maybe one of the most famous examples, likelihood of being pregnant. But there are hundreds of hundreds of more scores, maybe even thousands in the context of the market. Now, so I'll be talking about those today and fairness as it applies to them. More narrowly, this talk actually has two goals. I usually try to stick to one, but I'm shooting for two today. One is I really, one motivation behind this paper is I have a lot, and I'm sure we all do have some really smart friends, maybe friends that we think are smarter than ourselves that have not jumped in on this topic yet. And I take it as not only do I have friends that haven't jumped in on this yet, but I think very highly of, but also lots of philosophers haven't jumped into this topic yet, or these issues. And so one motivation of this paper is actually to provide something of overview of machine learning and fairness in machine learning, so as to invite other philosophers and other academics into this, this topic. I'm hoping that what I have to say isn't so basic to this redundant for a lot of the people here today. I am hoping I have something for everyone. But if some of this is redundant, especially given our interdisciplinary crowd, I would love to hear feedback on whether at all this is a good way of trying to summarize sort of a high level overview of the, I think, kind of key concepts of machine learning and fairness in machine learning. And I should mention I've switched over to use the word machine learning. The reason that's appearing here is because that's a common tool for developing predictive scores. And then my second goal is to make a diagnosis. And this actually is served very well by giving an overview of things first. And this will kind of come up later in the talk. But what I'm going to do is I'll look at a bunch of different proposals of sort of what fairness in machine learning amounts to and a bunch of counter examples of those proposals. And then what I want to say is why have we been able to find counter examples to all these proposals. And I want to kind of go up beyond the local counter examples and kind of give up general diagnosis like what's what seems to be like the larger problem here. And so I'll be offering kind of a diagnosis of why we have not landed on a measure yet of fairness in predictive scoring for machine learning. So that's kind of like the basic, those are my goals. Here's the plan. I'll work through some machine learning and bias machine learning and how bias can kind of work its way into machine learning. I'll work through as I said some fairness measures and their discontents. I'll offer a diagnosis of kind of why these haven't been working. And then some really broad general lessons maybe even so broad that they're like more like invitations for kind of further comment and discussion. So some lessons from what I think is the correct diagnosis of the situation. Okay, so here's an example I want to talk to you really quickly. We'll talk through sort of three examples that'll be touched stones throughout the paper. Here's the only I think non-controversial one. This is sort of my model for machine learning. So I call the example Frenchies and Boston Terriers to anyone who doesn't know Frenchies and Boston Terriers are breeds of dogs that looks at least somewhat similar. They're both short have short hair have some noses and they're the kind of the reason I could imagine people confusing. And I can and this is somewhat contrived example of course but we can imagine that you have trouble telling Frenchies and Boston Terriers apart and you want to get better at this and so you develop you want to develop a system for telling the difference. One way you might do this is you might go to the local dog park with an expert make some notes on the dogs they classify as Boston Terriers and Frenchies look for some patterns in your notes and use those patterns as a guide for future sorting. Like I said a little contrived but hopefully like it's somewhat intuitive model of how you might solve this problem that you have. Now, that isn't all that different from what a computer does when it uses machine learning to mind data sets to find patterns. So we can, if you can kind of get how that that that process works as nice to describe, you can imagine doing that in a computer assisted way, filling out some spreadsheets maybe looking for some patterns and and you're off to the races. And this isn't how all machine learning works obviously but it's like a touchstone nice little simple model. Okay, the method I just talked through is something that we can call classification. And the next thing that I want to talk about is sort of, I'll do two things quickly in two steps. I want to break down the process I just described into some steps, three steps, and then I want to talk about how each step bias can kind of slip in to the process inadvertently. And I'll raise the question, well, how do we detect bias, there's ethical concerns, I gave all these examples machine is learning is used in all sorts of really important aspects of people's lives. And that motivates the kind of discussion of these fairness measures. So I'll look at classification really quickly in a bit more detail. First, so I'm slowing things down just a tad bit. The first step or one of the first steps in the process I walked through is choosing what to take notes on some very quick terminology, when you choose what to take notes on your engagement what will be what's called feature selection. And when you choose which labels to use for the thing that you care about Boston terrier Frenchy, you're defining your target variables the thing that you care to learn about. The second step, you go out and take some notes with your friend. When you do this you're engaged in data collection, and you'll use this data to train yourself so this will be your training data. And here's a quick example, I think people can see my cursor, people see my cursor. Good. This is like maybe an example of what your notes might look like. Moving on. Then you look for some patterns and I think if we kind of put this on a graph we could see it maybe doesn't jump off the page here but here you can see that there's kind of a pattern going right where I'm using dots for Boston terriers and X's for Frenchies. And if we wanted to be a little bit more circumspect we might say, we'll draw two lines. One vertical line to separate as many X's from dots as we can and one horizontal line to do the same. And now we might have a rule. Supposing you have access to this data, you might say if if a dog's attributes put it in what I've labeled as quadrant D, you can go ahead and call it a Frenchy. And if that rule works for you, you're done. Kind of depends on how accurate you want to be, but maybe if you were no better than guessing and you just wanted to improve a little bit, maybe for the sake of say targeted advertising. Then maybe maybe you're done here. I guess one thing I'll say really quickly while this slide is close is obviously we could do a lot better. We could have sort of gone gone a little more fancy and drawn like maybe a line here kind of or maybe we could have tracked more more data points. That might have been more work than we cared to do at the moment. Like I said, maybe this is good enough. But we could get better by making things a little bit more complicated and as simple as my example is that's when you kind of see the need for computers, right. So we could spot those those those patterns that I just put up there. But if we had more dimensions, more data points, more complex relationships, maybe not. So the basic thought there is that classification can get difficult and complicated very quickly. This is why machine learning has made its way into many aspects of our lives. Especially that that that's part of why another reason is that computers are cheap data is cheap story data is cheap now. And so like this is why we see like I said in my example with the market that we see kind of consumer scores everywhere. Okay, so now hopefully we have kind of a grip on how that process works. Now let me just kind of talk about three ways in which bias can kind of slip in that each of the three steps identified. So at step one, by a choice is what to take notes on. So here's another example we talked to Boston terriers and the French ease. Here's another example will return to a couple times I'm calling it hiring teachers. And I'll just read the case really quickly. You're hiring teachers and you want to know which ones will be effective. You decide to look at your past hires to see which teachers have added the most value to their students education measured using test scores. You find that teachers that went to certain colleges added more value so you give more teachers that went to those schools interviews. And that might look fine, but we can suppose, and this is going to be true in a lot of cases that the following correlations hold minority and non minority teachers tend to go to different colleges. Minority teachers gravitate toward schools with higher portions of minority students, the higher the portion of minority students the poor the funding for the school poor funding causes lower test scores. And now just by kind of from the outset before we even really started we might have geared things up so that we're going to end up with what's going to be an unfair scoring system. I'm going to look at step two and I'll use that to introduce a second case. So at step two we can introduce bias by how we're collecting our data. So here's another example. This one we're going to really talk about in some detail. And I'll call it pre trial release. You're deciding who to release while awaiting trial. You only want to release people who will not re offend while awaiting trial. You have data about past defendants that suggests that the number of past arrests a defendant has is positively correlated with future arrests. You use this to construct a rule that recommends only releasing those with you previous arrests. And again, somewhat predictively, we can end up with a problem. If we suppose certain correlations hold, which they do in a lot of cases. For example, if we suppose that there are many crimes that are committed by white and black people at similar rates, but that the black people get arrested for disproportionately often. So this is like an example would be drug drug crimes in the United States, I think is a I think pretty uncontested example of what I'm talking about there. If, if, if we're using the pre trial release system, the way I describe, and we have this background fact, we will again have a system that just by tracking the data is not going to look fair and that's just because the data, the data collection has not been a good process. Okay. So lastly, this will be my last example will actually turn to an old return to an old example, but this is like the last way in which might have biases seep into the process. And that's by just choices of how to balance false positive and false negatives, or basically like what our standards are for accuracy. And so I have this familiar slide about the Frenchies and the Boston terriers. And we can see here if we use that rule, anything that's in quadrant D just call it a Frenchie otherwise maybe call it a Boston terrier is going to have the effect of falsely accusing. At least some Frenchies of being Boston terriers. It will also never accuse a Boston terrier of being a Frenchie. And that might not be a big deal. In this case, it depends on maybe what we're using the score for. But we can imagine that it would be a huge deal if if this is being used for hiring or detention or whatever else. One thing I didn't say previously but now that we have this again that is going to be important and maybe a good transition to the next thing I'll talk about is we could imagine that if we had two graphs. And we had tracked male and females. Maybe there would be a clean line separating all Frenchies and Boston terriers and by introducing that that third data point, maybe we could have no inaccuracy and kind of not have this problem. Which this is going to this is like maybe now that we have the slide in front of us it might be helpful to see that that maybe by doing something as simple as just having two graphs one for males and females, maybe the kind of ambiguous cases in the middle would just go away, and we wouldn't have any inaccuracies. So hopefully that's all clear. Hopefully it's all not too simple or familiar to everybody here. I had to worry about that. But I guess we'll have to just kind of soldier on. Okay, so now I'm going to talk about so hopefully now you at least if no one was worried about this or maybe you're now worried. Oh, these things are being used in for all sorts of purposes we should care that they they're fair. In non simple examples, how might we tell. Well, I'm going to work through three proposals. The first one is less quantitative. And it, and it goes like this. So one, one claim is what here's what here's something important about fairness and machine learning. So the principle anti classification, don't take inputs about as to whether someone belongs to a protected class. So don't ask for people's race ethnicity gender don't use those in your scores. Now obviously, and now we're on to the counter examples, and I'll do this for each one that we'll talk about I'll say why I think it's not sufficient and why it's not necessary. So it doesn't seem to me that I think, well, maybe everyone will agree that satisfying anti classification is sufficient for being fair. Why, well, we can already see this if you bought either the hiring teacher or pre trial release case if either of those aren't fair. And the way that I set them up was that they don't ask for for, we can imagine at least that they don't ask for race or gender or anything else. Then we can see that satisfying anti classification isn't going to be enough for being fair. But maybe we can say okay well maybe it's maybe it's necessary maybe it's not sufficient but it is necessary. And here's why I don't think it'll be necessary. But at least some contexts women tend to reoffend less often than men. Taking gender as an input then can prevent a score from problematically over estimating root of is recidivism among female defendants so you kind of can kind of recall the the the the graph example, and how we could maybe improve our accuracy by by tracking gender for the dogs. Here, we could say, we can kind of apply the intuition here and see that at least in some cases doing the same thing with people might allow us to avoid being an inaccurate ways that might seem unfair. Because if we blind ourselves to that data, we might over estimate recidivism among women which would feel unfair to them, presumably. Before I move on to the next slide I should say really quickly, I haven't marbled in my references into these slides, I have a reference site at the end. And a lot of this discussion comes from several computer science papers with with similar authors there's a couple. There's like a group of people who read a bunch of papers together. I'll just kind of introduce those at the end and call back to them. I think where these definitions are coming from. Okay, so moving on to classification parody, or moving on to the next two measures. And I'll be presenting the next to the last two in tandem. Let me backtrack a second, I'm going to present the next to in tandem, because they seem very similar. And also these two have been more, I think at the heart of more controversies the last one. The first of the two is something called classification parody. Classification parody tells us to make classification errors equal across groups defined by protective attributes. And basically what that means is it shouldn't turn out that in my scoring. I tend to be less accurate about men and women, or women and men, or black defendants, as opposed to white defendants, I should have kind of error rates that are equal across those groups should sound intuitive enough. I think we can hopefully see why that sounds like attractive. Calibration in contrast, which can sound very similar to parody, but we'll tease them apart. That's kind of why I'm starting with them side by side as I'll tease them apart first. Calibration in contrast tells us to make outcomes, for example, re arrest among people we identified as high risk of reoffending, probabilistically independent of protected protected attributes, given one's predictive score. In simpler terms, it's telling us that, let's suppose I judge somebody as 50% accurate. If I'm good at what I'm doing, then there's about a 50% chance, sorry, if I judge someone is about 50% likely to commit a crime, and I'm doing my job well, there should be about a 50% chance that they're going to go on to commit a crime. Knowing how I've kind of performed in the past, you shouldn't upon learning someone's score and their gender deviate from that 50%. What calibration asks us is basically to, it asks us to how our scores mean the same things for each person across protected attributes. Now, I think that classification parody and calibration can sound very similar, but they're not and I have a graphic that I think should really help see why this is the case. But first I have a note on my slide that another thing that people might know who follow this is, it's been discovered that actually you cannot satisfy both of them in many, in many cases at the same time so you have to choose between them. So if you like one, you might have to choose against the other is the kind of a result that has kind of come out recently. Okay, so here's a graphic. This is inspired by some of those computer science papers I talked about. I've created this image for my own purposes. They have numbers about real systems so their numbers are a little ragged. I thought to kind of communicate the intuition it would be easy to just kind of make some hypothetical numbers. So these are made up numbers that do track a real phenomenon, but I've kind of smoothed them out so that it's easier to kind of follow the important point. So if we want to see, if anybody wants to see why calibration and classification parody say different things, this graph can help us see it. So what I have here is basically the some statistics on the performance of a hypothetical pretrial risk assessment tool. Each of these bars represents a different category. So here on the left side. The defendants that were the black defendants that identified as low risk, and we can see that it judged 100 defendants as low risk 100 black defendants as low risk. And that 80% of those defendants did not go on to commit a crime or to be arrested. So it's pretty accurate. And that's the same thing for the recipes we have high risk and black defendants low risk, white defendants high risk, white defendants. And one thing that we can see immediately is that for the low risk black defendants and the low risk white defendants the they they went on only 20% of the time to commit to be re arrested. So low risk means low risk whether you're black or white it means exactly the same thing. The same thing is also true of the high risk categories even though it's a little harder to see, because this system judged more black defendants as high risk. Only 40%, only 40% of them did not go on to, sorry, only 20% of them did not go on to commit to be re arrested. And that's, that's also true of the white defendants 10 out of 50 versus 40 out of 200. So we can see that low risk means low risk, high risk means high risk, in exactly the same way, whether the defendant is black or white, so it's calibrated. But it doesn't satisfy parity. And we can see this if we pay attention to again these 40 black defendants that were identified as high risk but did not go on to commit a crime. So all those misclassified it sort of identified them as people who would go on to be arrested that these people were not re arrested. And we could see that 40 out of 300 total defendants were identified were misclassified as high risk. That reduces to two out of 15. Whereas in the case of the white defendants, even though it's calibrated and all of the statistics for each defender the same. Out of 150 total were misclassified, which reduces to one out of 15. So even though the system is calibrated. It's twice as likely to identify black defendants as hybrid misclassified black defendants as high risk. And that's just a way of showing how classification parity and calibration can come apart. Sorry, I don't know if you guys can hear that we're having some thunder. Hopefully the internet doesn't go out. Okay, so now we'll go on to my counter examples and hopefully I haven't taken too much time with this. I have a little bit to go through, but I think we can end in close to on time. So, again, classification parity tells us to make classification error equal across groups defined by protective attributes. Quickly, let's see why this isn't necessary for fairness. Recall the fact about men and women and reoffense a calibrated score will behave a lot like our hypothetical pretrial risk system. So back like this one. Except now, it will be twice as likely to identify men as misidentified men as as as reoffend as high risk. But if we kind of massage the context enough, this might not seem this might not seem unfair. Maybe for the men it first of all wasn't through uneven data collection. Maybe this was actually a true feature of the men in the kind of system that we're talking about. And further, if we can massage the case even more by saying like, it could also be the case that this is like a nuance system that we're working with. And being identified as high risk doesn't mean you're going to be treated harshly. Maybe it means that you'll get social services or something like this, right? So there are ways of maybe deviating from parity in ways that are that don't strike this as unfair, right? We could do things to compensate for this and maybe it's the fairest thing we can do. So alternative being if we calibrated the system, we would be overestimating recidivism among women and maybe detaining them without need. Okay, so here's why let's move on to the sufficiency question. Here's why it's not sufficient for fairness. Well, I just said that forcing parity in this case would mean systematically overestimating reoffense among women, which I think would seem unfair. So it seems that classification parity, satisfied classification parity shouldn't be necessary or sufficient for being fair. If you guys, if you share my intuitions. Okay, so moving on to calibration really quickly, which again asks us to make the same score mean the same thing for everybody for each person. Now we can just kind of go through this quickly. Here's why I don't think that's sufficient for fairness. We can imagine a case like pretrial release where any inequality in the scores are driven by arrest rates that don't track reoffense. So basically, if the case I walked you through where black defendants were twice as likely to be misidentified as high risk and maybe therefore detain. If that struck you as unfair, we could see calibration at odds with fairness as at least as a, as we could see attention in the claim that satisfying calibration is sufficient for fairness. But again, we can ask whether it's necessary for fairness. And I don't think that it is either because now this one imagines requires a little bit of imagination, but we can imagine a situation where we try to adjust for the problem in that graph by forcing parity to at least some extent and maybe that's the best we can do. So maybe being identified high risk for black defendant means something slightly different from being identified as high risk for white defendant, because we're sort of over compensating and forcing fairness, forcing parity on the system in name of fairness. That might not be feasible in the real world for all sorts of reasons, but at least maybe we can kind of see how we might deviate from calibration in pursuit of fairness to compensate, for example, uneven data collection or some other sort of issue. Okay, so that's kind of, that's kind of like maybe the meat of the kind of the talk is just kind of working through a bunch of those fairness measures and some counter examples. Now I want to make like a little bit of a diagnosis beyond the local counter examples, why have we ran into some issues. The terminology here isn't perfect to offer a diagnosis I want to introduce a distinction. I'm going to deviate from the terminology here but I couldn't resist because this is a, I think a great story maybe people I think I assume a lot of philosophers have heard it before. But if not, let me introduce a distinction by way of telling a little story from the history of philosophy of possibly a possible history, a story from the history of philosophy. Well, Sidney Morgan Besser, at least allegedly, I think some people have said this is maybe not true but Sidney Morgan Besser once attended a protest where everyone was hit beat up by the police. When asked if he was treated unjustly or unfairly he said unjustly yes unfairly no, he elaborated an officer hit me on the head with his night stick, which was unjust, but they were doing it to everybody so it wasn't unfair. He describes a nice distinction he's using the words justice and fairness I'm going to use slightly different terminology, but hopefully that gives you an intuitive grip on on the on these two concepts and introduce this terminology and these definitions by the way come from that rat hooker. So what Morgan Besser or what hunt in the Morgan Besser story is identifying as fairness all called formal fairness, the equal and impartial application of rules. So the officers treatment of the protesters in the story was formally fair. There was a rule hit each protester on the head of the night stick and they did that. But of course, that doesn't actually seem fair to us. While the officers may have been formally fair. They are not what I will say. They were not, as I will say, substantively fair or as hunt says, and Morgan Besser unjust where substantive fairness is the proportional satisfaction of a certain subset of applicable moral reasons such as dessert agreements needs inside constraints. So in the story. Again, if we want to call that unfair it would be substantively unfair not formally unfair. Okay, what does that have to do with anything that I had to talk about today. Well, let me say a few more things. So first distinguish a classification rule. For example, if a suspected if a suspected Frenchies height and weight put it in quadrant D predict Frenchie separate that classification rule from what I'll call I don't know if there's a word for this. But I'll call the ultimate rule or you maybe even could use the word maximum. That the classification rule attempts to satisfy. So the rule in our case was label Frenchies as Frenchies or something like that. The measures of fairness that we considered. Tell us whether a classification rule, which attempts to satisfy some ultimate rule by giving instructions about what to do results in the unequal or part or partial application of its ultimate rule. Right, so I think one way here's my claim of understanding where all of those rules we were looking at were up to is they were trying to give us conditions for formal fairness. But I think many of our complaints about fairness and machine learning have to do with whether classification or ultimate rules are substantively fair. So I think a lot of our examples were grading on our intuitions. Not because there was an issue with formal fairness but because there was an issue with substantive fairness. And so that's my diagnosis is why are these why are these rules not working out. It's because we're kind of chasing up with the wrong concept. Or some white claim. So, like I said, I promised some really big somewhat vague lessons, but here's sort of sort of three follow up thoughts. And then I guess we can kind of open it up for comments and hopefully I don't think I went too badly over time to my. I'm glad about that I was worried. So first, one quick lesson is maybe we need to think less about formal fairness and isolation from substantive fairness. I think we can see that maybe there are a plurality of possible candidates for measures of formal fairness. And I think it would be helpful to think more about what substantive fairness requires, and also to think that maybe it requires different kinds of formal fairness in different situations maybe classification parody. So we've seen that classification parody seems like a good objective in some contexts and parody and others, and maybe there's room to think about sort of when which kind of formal fairness is desirable. Another lesson is I think we can use formal fairness as heuristics for substantive fairness. What we saw in the example where we try to what we saw when there were when we saw that there was conflict between classification parody and calibration is that when those both can't be satisfied. There's some issue with an inequality between protected classes protected groups. Now that might not settle the score as to whether or not there is in fact anything unfair going on, but it's definitely a red flag. And so the formal measures here could be used as heuristics not arbiters but for heuristics for detecting unfairness right. When you can't satisfy both you are in that situation. And finally, I think you need a better understand better our understanding of substantive substantive fairness as it relates to predictive scoring, and that's like a really really really big lesson maybe if you call a lesson. But that's maybe, I mean I think it's an important point. I think sometimes, at least in practice maybe sometimes in theory as well. You get this vision of the kind of like the technological context or the machine learning context as like this context, where you can kind of think about things in a certain kind of way and maybe come up with some principles of fairness or whatever. But there is no such context. As we started in the beginning of the talk. These scores are not are using all sorts of different contexts criminal justice, hiring the market. And I think it's it behooves us to think really hard about what substantive fairness calls for as it relates to predictive scoring, with an understanding that that's a huge project that's just more or less the project of asking the question of how people should be treated, because it's not like machine, I don't know this point is coming across, but it's not like machine learning is like this one little context where we can ask narrow ethical questions. It's being used to do all sorts of things in all sorts of different ways to all sorts of different people. Which means that it's a big ethical project to figure out how to do this correctly and that we should resist thinking that there's going to be one circumspect mathematical rule that's going to tell us the right answer here. If that was the case, we would have finished ethics a while ago, I think. So thank you very much. Thank you very much for your time. I'm sorry that I went over a little bit and as promised really quickly. Here are some papers that I relied on pretty heavily for this presentation. And these three in the middle, the Corbett Davies papers are the ones where I got a lot of my examples and my definitions. And this one in the middle is a very helpful one where I kind of that inspired that graph that I use it for me has been very helpful. I think it's like one of the most helpful images I've come across. So with that said, thanks. And I guess we'll turn it over to questions or discussion. Thank you very much. Thank you. That was fantastic. So the way we're going to do Q&A is panelists will be able to raise their hands. I'll sort of call on people trying to get a sort of distribution of questions across our various disciplines. If you're in the audience, you can put a question up by writing into the Q&A section, and I will relay that to Clinton and integrate into the discussion. Okay, so we'll start with a question from Atisa. Okay, thanks a lot, Clinton. So I was wondering if like I wanted to push you basically to a little bit connect your discussion to the discussions of fairness and justice in philosophy. So for example, when you give this counter examples about necessary and sufficient conditions to fairness, I was wondering what do you mean by fairness there? Are you just using an intuitive conception of fairness or are you using like a theory of justice or theory of fairness? And the reason is that I became a little bit like a susceptible that maybe some of these problems are not really problems with formal fairness in the context of machine learning. It's just because we don't have a theory of fairness that is going to be applicable in all kinds of contexts. And when you look into the history of theories of justice, there are like 10 different kinds of theories and each of them has their own benefits. Some of them are very ideal. Some of them are not ideal. So I was wondering whether you see some nice analogy between these problems that arise for fairness in machine learning to problems that arise for choosing which theory of justice or fairness actually is the good one in philosophy and law. Right, thank you. Thanks for the question. Let me see. Yeah, so as you see, I sort of tried to resist a little bit the justice word for partially for this reason in the way that I was setting this up. And I guess that what the short answer to the question of whether I have a theory of justice or fairness in the back of my mind, I don't. Because I wanted to make kind of a general, a really general point that might look attractive from kind of any point of view on this. So I don't I'm not coming from a place where I have like a subtle deal on this, and I think that there is, I think it's fair to ask and I think a reasonable next step to think hard about different trade off between different conceptions of fairness and justice. So, I haven't, I haven't thought through. I haven't thought thought through that yes, I don't have a view on it. And I could, I, I guess one other thing I should say is, is that I beyond just using I guess fairness in an intuitive way I meant to use it in a somewhat broad way, obviously broader than formal fairness. But again, I think I'm just going to repeat myself here. I was hoping to make a point for myself and maybe this is just for me. The thing that I kind of brought in at the end of my diagnosis was illuminating to me to figure out why things have maybe gone awry. And I wanted to make that point again in the general in a general way that didn't kind of have any commitments with what kind of fairness or justice might amount to but I could see how that methodologically might be a problem as well. And if you have any, if you have any suggestions, I would be very curious to hear them the specific ways in which you were skeptical or any connections that you see. So I don't know if I don't know if you if you do or if we're not doing follow ups. Maybe we'll do follow ups on the slack just because I've got a bit of a bit of a cue to go through. Yes, but please do do that. So next question is from Damian Clifford. One of our lawyers. Thanks very much for that. I suppose the separation between formal and substantive fairness, I was wondering, to what extent that can actually also be classified as a distinction between equality and equity. And, you know, is substantive because, like, you know, substantive fairness is going to end up being so context dependent. Is it actually ever achievable. I, you know, I mean, you were kind of pointing to that and that, you know, it's the big question, but I mean, I'm wondering to a certain extent, whether it's actually ever going to be achievable in your view. And then to what extent that actually results in just declaring certain applications and machine learning and predictive tools. Unfair de facto from the outside because they're always going to be. I don't know, not equitable. Yeah, so I think I mean, to give a really satisfying answer to this question, I'd have to think through the last one a little bit more carefully and maybe have an answer so maybe having a theory would be would be nice here. But I actually don't. I mean, I'm, I'm open to being wrong about this I'm open to be wrong about everything I said. But I actually don't think that I'm sort of setting the bar too high, or kind of setting out an unachievable goal. And I would say more other. I mean we have to look at this, the cases kind of case by case and side by side, I guess. But it seems that there are, I guess, reasonable demands that people could have for how they'll be treated by sort of automated systems. And those demands can be sensitive to what's reasonable and what's kind of a possible. So, I guess like in the example of the pretrial risk stuff. For example, I sort of in my examples had some references to how people are going to be treated for example. And so I think that when we're using things like a pretrial risk tool. We do need to be careful about where the biases are lying and stuff like that but also like what the system that they're part of looks like and how people are being treated. And on balance the whole the thing taken as a whole can be in my in my opinion substantively fair. In ways that wouldn't be super impossible to achieve. So maybe to put a little bit more meat on that. So for example, if we know that we're going to have to trade off between classification parody and calibration. Okay, maybe that's a fact and we have to figure out what to do about that. But also like if we look at the whole system, we can say, Oh, we're going to be inaccurate no matter what we do here in some ways. And maybe that'll inform whether we send everybody home. And maybe give some of them ankle monitors or ask them to check back in versus detaining all the ones that we think are high risk for example. I don't know if this is like coming out as articulately as I would like it to. But I guess, um, like I said at the end, when we treat someone through one of these kind of automated systems. What we're doing is we're just treating someone another we're just giving someone some treatment. And presumably there are ethical ways of treating people. And that's kind of not much more than what I'm talking about here and time out like maybe a subset of ethical concerns. But just meeting those demands, and it's going to be kind of harder to give more than one liner there but I think it's I think it's doable. I don't think it's like at the outset saying it's going to be a super high standard. In fact, it might be a very reasonable standard and in part because if you look at the context, I think that the what people can demand in any situation is going to be sensitive to kind of as feasible as possible. I hope that doesn't sound like empty and trivial. I hope that kind of gets to the heart of the question. Next question is going to come from Michael, PhD studying computer science. Great. Thanks. So thanks for that talk, Clinton. I think this is still in line with what you've been talking about already, but when, when do you think there are cases, like what, how would you figure out if there are cases when just the use of predictive scoring itself violates like substantive fairness. I think people have said already about the use of criminal recidivism scores. And I noticed that I saw in your field people profile that you have some work about consumer scoring data so that's sort of a kind of something I'd be interested in hearing more about. Yeah, so let me see. So, well I guess is the question just sort of, are there, what are the cases where it just can't be used is it sort of kind of. Yeah, yeah, I was like, I think I love to hear what you think those cases are or there may be some criteria. Yeah. Yeah, so I mean I guess I guess I have some thoughts about that. The overarching thought would be, I don't know if I would want to say, I wouldn't want to commit that there's like one context where it couldn't shouldn't be used. But I would want to say that it kind of depends more on how it's being used. So I guess. I mean this is going to be, I wish I could say something more interesting but like, I think that you know you should be certain that you're reliable. And that the extent to which you're being reliable and accurate that that's sensitive to the stakes involved right so just for an example right. I think there's a good way to use these for hiring and recidivism maybe not, but it's not whether you use them but it's how and if you're going to use them for those kinds of situations you should have a high standards for being accurate. And also I think that. So I think should be should be accurate. Sorry, I don't know if you guys can hear this there's a big thunderstorm right now. My lights just went out so hopefully again, I don't lose everybody. It's just kind of distracting. I also think that you, you should be sensitive to other things that people might might reasonably demand, like some level of transparency and other things like that but I think personally I don't know if I have any kind of hard and fast rules. Because more the more I think about this the more I think it's like, what should judges be doing when they do this what should hires be doing when they do this and those those kind of local and we do have to do these things and maybe as the computers can improve accuracy I know that with a lot of the, with a criminal justice they can improve consistency a lot, which could seem like very important. And then there's kind of a question is, at what cost of that come and how can we achieve that consistency without trading it against something else that's more important. And how can we kind of. Yeah, so I guess hopefully it gives you kind of an insight at least how I've been thinking about it. Yeah, that's great. Thank you. The next question is from Will Bateman in our project but then I'll go to one from the Q&A after that. Thanks, and thanks Clinton that was a really fabulous presentation. My question and I haven't sort of worked out all of its implications. The question is really about the kind of broader impact of being able to quantify and really identify reasoning processes that involve some what we would understand as discriminatory patterns of thought that involve protected attributes, race, sex, age, disability, etc. And I come at this from the perspective of working on a paper with machine learner and a philosopher where we're thinking about the dilemma that designers of ethical systems are placed in because if they work on historically biased data sets then the automatic response will often be a type of disparate impact discrimination to use US legal terminology. But if they try and avoid that by tweaking the decision criteria by reference to protected attributes they embroil themselves in liability for disparate treatment. And the thing that's really come out strongly from thinking through this dilemma and how you'd fix it from a law reform perspective is that the broader implication of being able to really clearly quantify and identify exactly how exactly the weight that you're according and each step of your decision making process to a protected attribute. It has these broader implications for the way that we design and think about discrimination law full stop. Because the way that we've gotten around the black box of the human brain in terms of discrimination or liability for discriminatory behavior is by saying, look, we can't work out exactly how prejudiced you were. And we can't work out whether you were discriminating on the basis of protected attributes in a type of good faith way or if you were doing it because you're just racist. And so we will just implement a blanket rule that says any reasoning process whatsoever which includes any of these protected attributes is ex facie without anything more unlawful and discriminatory. And it just it seems to me that there's this and this is my I'm asking you I suppose to reflect on my now very long setup. It seems to me that there's this broader implication if you can actually start to really clearly quantify and identify exactly the way that you're reasoning in respect of these attributes and in respect of non protected attributes kind of legitimate attributes that aren't protected that might impact the way that we design our laws more generally. Because we might be able to or at least we might be able to say look for automated systems we have a different rule where it's not just any involve any engagement with a protected attribute in your reasoning process which involves liability. It's whether or not you're doing you're including that particular attribute for a kind of legitimate purpose or you're doing your best. Not to be prejudiced and not to be discriminatory which has a whole set of other really complicated evaluative conclusions but so yeah that that's my sort of before 10 am coffee ramble. I'd love to hear your thoughts on it because I think you thought really deeply about it. Yeah, it's interesting question I. So I guess like I take it that the kind of the core of the question is, should there be maybe different rules for the automated decision systems with respect to maybe blanket bands on using protected attributes as part of your reasoning or something like that. Yes that's thanks that that's one one core that is one core the other core is do we have to rethink discrimination law more generally if you start to get to a point with with different types of ML systems where you can actually really clearly identify the weightings given to different types of attributes in a very broad set of applications and so you're starting to use these systems very broadly. And displacing a lot of independent human that you're displacing a lot of black boxes with a lot of really clearly identified algorithms clearly specified algorithm. Yeah, so I think it's really interesting that somebody who's like not a lawyer I get a little bit uneasy with some of this stuff because I don't understand the law perfectly well I but let me think through the first the first question. And I think it bears on the second of whether there should be a different rule for this for the algorithms I think I think you want to touch on this. So like, given the caveat I just made I'm not sure how to think about this, but I guess one benefit of these systems is you could give you could show. I mean I think that there's going to be a really hard question of even if it's going to improve fairness or whatever it is going to improve one goal in one way, if including race and gender. Presume that including race or gender or whatever will will have some improvement. There's like this further question of whether of how to make the kind of more holistic decision of whether to start rigging things up that way you might you might worry about all sorts of things. Like reifying some ideas, for example, but one benefit I guess it's I think it's going to be really messy business but I think you already said this one benefit of this could be that you could be completely clear about what the reasoning is and show that it's being used in a certain kind of way, right and kind of lay out explicitly that these are the rules this is what's going to happen. Not maybe perfectly deterministically, but you can at least think about that in a way where you're not going to have to worry about anything implicit in the reasoning that's going on there. And you can also, I guess you could also run some tests on it to make sure that it's not going to have sort of implicit problems. But beyond that I don't I don't really have a good theory about how to think about this and I get really, I get really kind of queasy talking about the law and and kind of policy solutions because I know that that is a that's sort of outside of what I know how to do as a philosopher, and it's kind of a it's true I just I just recognized as a really tricky business. Let me just cut in there. So, so first lesson is you guys should keep talking on on the slack and you can sort of avail yourself of the of the folks in our project and to know who have experienced Damien also would be a great person to talk to as well as well. But let me just try and get to a question from our audience. So Jessica court has asked, and I'll just pass this straight through to you. Could we treat machine learning bias in the same way as cognitive bias in real world implementation on human systems. So, for example, you might say, look, it says there's a 60% chance of recidivism, but we can also see that the person is in low socio economic status and black. So be aware of the bias and making a final decision. So do you think we can sort of compensate for the problems with respect to formal fairness by sort of judiciously using these systems in order to make a decision that also is a human decision. Um, I think that there might be some hope for that and I and I'm pretty sure I can't remember exactly which one but one of the computer science papers identified by Corbett Davies and company. I think gets into some of this and one of their recommendations actually mean I don't this is quite a recommendation. I'd have to look at the paper again. And, and I might invite Jessica do that too. But I think that one of the recommendations is I gave this contrived example of how we might force parody to try to achieve fairness in one of the cases where that we have a calibrated system that looks unfair for various reasons. And one of their recommendations is you could probably do it that way. I mean you might be able to kind of break things up that way but it would be better to let the system run calibrated and then make a judgment outside of it. I think I think some more some further context. And so I do think that that that there's at least some one good reason to have a human on the loop is maybe that they could we could say like what there is this system that tends to be better than people but it does systematically make some kinds of errors. And so we can consult its outputs. And then we can look at the broader context and make maybe some further judgment calls and hopefully improve improve accuracy that way. And, and, and I mean I guess on that note to there are other reasons good reasons to have humans on the loop with these kind of automated cases because we've seen them kind of. They can be really spectacular certain things but they can also kind of make mistakes that only make mistake only seem like reasonable mistakes from their bizarre point of view but humans can kind of spot certain kinds of mistakes as kind of obviously incorrect. So I think that I think oftentimes having a human kind of on the loop in the loop is a good idea for various reasons but then I guess you get into the tricky business of like whether people are going to over correct or not and I think that's also going to be very difficult. It depends on what that person's kind of training and skills are I guess this is a good point to move us over to the slack. So I'm going to track down there's a great paper that showed the judges responding to predictive scores were more likely to override them when they went against their prior prejudice. So you got the bias algorithm and then you got the racist judges, and you just get the worst of both worlds. So I'll put that one up on the slack and Matthew Phillips had a great point at the end there as well about the consequences for individual judges of going against a predicted score where if the score proves to be sort of confirmed that that might end up being something that goes against their record so they might want to protect. But we can continue that kind of chat over on the slack for now let's all thank Clinton.