 Hi, everyone. My name is K.J. Boggi and I am Senior Policy Counsel at the Open Technology Institute. I wanted to welcome everyone to this conversation today that centers on the role that algorithms play in our daily lives and in our society at large. I'll be trying to incorporate audience questions throughout the conversation today. So just as Angela had mentioned, and as a reminder to our listeners, please submit your questions via the Q&A and chat feature on Zoom. The comment feature via YouTube or by tweeting at OTI. Now to get us quickly on the same page, I wanted to make sure we're using a similar vocabulary. Algorithms that we'll be talking about today are a step-by-step set of directions coded to accomplish a specific task. There are many definitions of artificial intelligence and I'm sure our panelists have their own thoughts on this topic, but today we're using the term in a broad sense as a scientific field focused on designing systems using algorithmic techniques. Now one specific subset of approaches to achieve artificial intelligence is machine learning, which uses algorithmic building blocks to achieve an outcome. Now you may hear these three terms used interchangeably today and it's only because they're in the same ecosystem that we're exploring. Today's discussion attempts to bridge the connection between two issues that are usually spoken about separately. The issue of how algorithms are used in the era of big data and how they have a privacy intrusive manner and how the output of these algorithmic systems often have disparate impacts on communities of color and other marginalized groups. Now I would be remiss if I hadn't acknowledged the backdrop of our conversation today. Across the country in the past few weeks, civil unrest has spread across the nation as thousands protest against inequity and abuse power. Today's conversation hits at the heart of the systems that are meant to be uplifting and restorative and yet the technological designs that are created with the idea of establishing objectivity are reflecting the same inequities that we have in our society. One of the questions around these issues that we're exploring today is who should be accountable and what mechanisms can we employ to identify privacy intrusive algorithms or algorithmic bias. We are very fortunate today to have a keynote speaker who has worked on these issues in her role as vice chair of the House Energy and Commerce Committee while she represents New York's ninth congressional district. She's also introduced the House version of the Algorithmic Accountability Act of 2019 that proposes an oversight framework of automated systems to ensure that large companies and other beneficiaries of algorithmic design do not turn a blind eye towards unintended impact. I'll hand it over to her now to provide her remarks. Congresswoman. Let me first of all thank you KJ and thank a TI for inviting me to participate in today's critical discussion. Thank you for the kind introduction and I think the new America for hosting today's panel on such a critical topic. Increasingly, algorithms instead of people help determine whether Americans are hired for a dream job approved for home mortgage or sent to prison or jail. However, algorithms can be dangerously biased and results in discriminatory decisions. While AI systems come to conclusions based on calculations, the outputs can reflect the biases of the programmers or the data sets used to train the system. That's why last year I introduced the Algorithmic Accountability Act, the first ever bill to directly address this challenge. For my legislation, the FTC would be required will require companies with data on over one million users or revenue greater than $50 million to conduct bias and security assessments of highly sensitive automated decision systems and then fix any issues they identify. Right now we have an incentive structure for that that rewards willful blindness. If companies don't audit their algorithms for bias, they can act. I'm surprised when a researcher discovers that their software is perpetuating discrimination. But under my bill, the incentives are flipped. If you aren't undertaking due diligence to make sure your systems of fear you, you're not just at risk of receiving a bad headline, you are liable under law. As our country grapples with two simultaneous pandemics, the health pandemic of COVID-19 and the societal pandemic of police brutality. Algorithmic accountability may not seem like it's a priority. But it's that's mistaken, because AI bias is inherently linked with these two crises. Last month, I led a letter with Senator Wyden, advocating that the next coronavirus relief package require recipients of federal stimulus funding to audit their automated decision making systems for bias. This is critical because AI is particularly disturbing in the context of our healthcare system. AI will play a key role in monitoring the spread of COVID-19 among individuals predicting future outbreaks and perhaps even allocating scarce healthcare resources. Already, algorithms are being used to identify high risk patients. If the pandemic intensifies, the hospitals and hospitals experience shortages of ventilators by other supplies, these algorithms could be used to prioritize care. That sounds great in theory, but in practice, there are existing examples of biased AI resulting in patients of color, black patients in particular, receiving less care than their white patient counterparts. Life and death decisions should not be informed by algorithms unless we have confidence that they are putting people of, that they are not putting people of color at risk. Confronting AI bias is also critical in the context of our economic response to the coronavirus. For example, with companies receiving so many job applications right now, many are turning to automated systems to sort through resumes. It may sound fair to program the algorithm to look for resumes that share similarities with the, with the resumes of people who have previously been hired by the company, but in reality, because those previous hiring decisions made themselves be biased, favoring privileged applicants over poor applicants and stacked against women who may have taken time away from the workforce, the new algorithm is going to perpetuate those biases. With our country facing the highest unemployment rate since the Great Depression and spoiled businesses desperately seeking capital, preventing automated discrimination and employment and lending is absolutely essential. Finally, I'd like to say a few words about how AI bias is increasingly perpetuating discrimination in the criminal justice system. Right now, people across America are standing up and speaking out about the systemic racism and police brutality that has led to the death of countless African Americans and the beating abuse and incarceration of so many more. George Floyd was murdered while a camera was rolling. But we know so many other people have suffered the same injustice when a camera wasn't there. So what would you say if I told you that historical data from the criminal justice system the same system that jails stops searches and convicts African Americans at much higher rate than any other group is now being used to create algorithms to decide who will post bail, how much that bail cost and how many people will amend the length of prison sentences to judges. It's happening in states around the country. Using raw data from a broken criminal justice system to assign risk force is certain to lead to unjust outcomes. Machine learning algorithms and use statistics. Excuse me. Machine learning algorithms use statistics to find patterns and data. If you feed the system historical crime data that that data, excuse me, it will pick out the patterns associated with convictions. We all say defendants from a particular neighborhood are more likely to return to prison. And what the raw data doesn't show is the way is is that the reason why more people return to prison from that neighborhood is not because of more crimes, that more crimes are occurring, but because it's overpolice and even minor infractions in that neighborhood are likely to be caught and results in parole violations. Maybe it's because people from that neighborhood are more likely to be convicted by bias juries. These there are endless other examples of how bias AI impacts real people's lives and harms people of color and women in particular. The bad news is that unless we act soon, given the rapid deployment of AI, many of these biases will become even more deeply entrenched in our society, baked in to these formulations, if you will. However, the good news is that this problem has a solution with better data sets, continuous auditing, both before an algorithm is released and after its operational and increased diversity throughout the technology ecosystem. We can successfully mitigate algorithmic bias. So I want to thank you for having me for this very stimulating conversation today. And thank you for the opportunity to participate. And KJ, I yield to you. I appreciate that Congresswoman. Thank you so much for those really important remarks. I want to now welcome our panelists to join the webinar. I'll let each panelist go a little more into the work they do, but I do want to introduce them by name and by organization. I'm excited to welcome our panelists, including Daniel Khan Gilmore, who is a senior staff technologist at the ACLU speech privacy and technology project. Iris Palmer, who is a senior advisor for higher education and workforce with the Education Policy Program at New America. And Antoine Prince Albert III, who is the technology and telecommunications fellow at the leadership conference on civil and human rights. I should note that Prince is subbing in for Shakira Cook, who was unable to participate today, given her need to focus on supporting communities and community leaders dealing with the civil unrest across the country. I want to start off our discussion today by talking about a recruiting tool that was designed by Amazon, but shut down before they could widely use it. The company created a program in 2014 that would review a pool of resumes and identify the top candidates. Now by 2015 the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender neutral way. And that's because Amazon's computer models were trained to vet applicants by observing patterns and resumes submitted to the company over a 10 year period. So it's a lot of data. Most came from men, a reflection of male dominance across the tech industry. Now keep in mind the training data had all direct representations of gender removed. However, some found a way to make inferences and reflect the bias preferences of the tech industry as a whole. Now I want to bring in Daniel now to lay some foundation for us on some of the issues highlighted by the story that was reported a year or so ago. Daniel, what does Amazon story tell us about the demand for creating automated systems in the tech sector, and what are the implications, if any, on the privacy of users engaging with tech products. And also, what are the implications for systems still creating bias output, even if the training data use is trying to be quote unquote neutral. Those are a pack of good questions and I'm happy to be in the conversation here today. So the story with Amazon with that particular training set there's a bunch of other examples as well. But the story there is really a reminder that first off I think it's a reminder that the systems are designed primarily as cost shifting measures. Those are the systems are designed to make it possible to do things at a scale that your traditional mechanisms wouldn't be able to do. And the result is that they tend to cut corners. And the data that comes that comes in here. I mean as you described Amazon had removed explicit markers for gender. But we know that there are lots of data features that are sort of proxy measures for categories that we would care about. And so the system is able to pick up on those proxy measures. And if the way that the system is applied is to try to say well, let's see who does well at our company. And it could very well be that men do better at Amazon, then women, right and the reason for that may be there may be a number of reasons right there may be a ingrained culture of sexism within the Amazon workforce, which drives women out of the workforce. And so if you simply if you blindly say well we want to make sure we hire people who do well at our company. And the reason some people aren't doing well at your company is an internally discriminatory regime, the systems will pick that up. And they will apply that data and they'll tell you hey you shouldn't hire this woman she's not going to be a good fit here because your sexist coworkers are going to drive it out, except of course the systems won't say that last part, they'll simply say, here are the people who are going to do well under the regime that you have. So I think it points to a couple of things one is it points to this question of, where are the costs being shifted from, and why do we need these cost shiftings to happen. To who bears the brunt of the decisions that these things make you could use the same system to look internally at the policies of a group like Amazon, and say, Hey, it looks to me like we're losing a large percentage of potentially good workers. Why don't we shift the corporate culture so that we can make sure that we retain, you know, these kind of employees. But the systems are being designed not to do that they're being designed to say well let's just let's answer the easy question that's going to do as well. Internally, that's Amazon's credit they killed that system before it became live but there are many other systems that are out there that suffer from the same thing. With the ACLU, we did a test of Amazon's facial recognition system of matching people against a mugshot database and we used it, we used colleagues of mine used the headshots of the members of Congress, and said, Hey, like let's match these against this mugshot database how many of these folks have prior convictions. And of course the mugshot database doesn't generally include the sorts of convictions you might expect members of Congress to have. Rather, it represents as representative Clark mentioned the decades of or centuries of racist policing. And as a result, the people of color in Congress were misidentified as felons from this mugshot database at a much higher percentage than the than the white folks in Congress. So I think, you know, again, if you really use this to say like let's learn something about the underlying system and think about how we can use it to apply systemic reform, these tools might be useful. But the discriminatory impact comes when you apply the systems as with individualized consequences. I appreciate the context there. So, I mean, I want to bring in iris to talk about some of the issues you're seeing in the higher education field around this rush to create automated systems with troves of data. Yeah, higher education is using these in a in a couple ways, and they're very similar actually some of the ways that Daniel identified with slightly less horrible consequences, perhaps we could argue. So colleges use these things to choose who to enroll and admit in their schools, they use algorithms to some of them use algorithms to identify people who are at risk of dropping out of college and try to intervene early, keep them on track. And they also can use them to help people choose their like course of study and the type of major they want to take. And while I think obviously the risks in the criminal justice system and housing and a lot of other areas might seem a little bit more extreme, I will say that there are some issues with this if they're not used carefully, similar to the way Daniel was talking about. So, you can perpetuate inequality racist inequality that is the access to higher education just in perpetuity so similar to hiring colleges look at different applicants for fit and who's going to fit and succeeded our institution and if you embed that into the algorithm itself you can perpetuate the inequality and higher education and the access and success in higher education. There is also a risk of tracking students so we talked to someone in our research who had spoken to a Latina student who was interested in going into a STEM major. She was told by her professor that the algorithm said that she wouldn't be successful in that major and she should not go into it and so using these algorithms to systematically bring people who traditionally were not successful in certain high status and lucrative majors and putting them into different majors or getting to take different kinds of courses. There's also an idea that this can facilitate profiling similar to the way Daniel was talking about but also just in general, we've heard about systems where all they tell you is who is a person of color on your campus and who has a lower socioeconomic status when they enter college. If that's all this algorithm is telling you that it's not a useful algorithm it just perpetuates bias. And then obviously there's security and privacy violation issues so if you're not transparent about how the data, how student data is being used in these systems and what are being used for then they can be a real issue. That's helpful context in this conversation and then to sort of round out our sort of illustrating the issues that we're talking about. Prince, you know, we're living in a moment right now of nationwide unrest as individuals protest the treatment of black lives because of a deep rooted belief that our institutions are abusing their positions of power and influence. Many individuals as a result will engage with the criminal justice system, a system that uses automated systems to track individuals or produce recommendations around sentencing and parole. What are some of the concerns that are coming to your mind when we think about the role of algorithms and the criminal justice system. Sure, thank you kj and as you kind of highlighted before I go on to talk about the concerns. I really do want to acknowledge this watershed moment in black life in the United States and abroad and I want to put the technologies we're talking about here to dialogue with our national frustration about systemic racism. So just quickly in memory for those who have been killed or irreparably altered by racism, as Representative Clark mentioned I just asked for nine seconds of silence. Thank you. Now I want to lay out the technologies most prevalent in the criminal legal system, those that the leadership conference has lobbied around and educated people about. So I'll mention for but for the remainder of this discussion I really want to drill down on to that really highlight the inequities and our concerns. So the four are risk assessment tools, facial recognition technology, predictive policing systems, and body worn cameras. And I'll primarily be talking about risk assessment tools and facial recognition today. Risk assessment tools or risk assessment instruments RITs our eyes they are actual actuarial instruments that estimate the likelihood of a defendant's rate of recidivism, or failure to appear before the tribunal, or their likelihood of being re arrested on a similar or more violent crime in the future. And the score is a function of frequency of interactions among such of infractions rather among similarly situated defendants. These systems rely on a stock of defendant samples, if you will, that has trained the tools to emit certain outcomes for certain kinds of people. The factors they consider our substance abuse criminal history, failure to appear before court employment stability, education level housing residential stability, family peer relationships community ties and others those are some of the common ones around the number of about 36 different systems we see being used throughout the United States. These can be used at really any juncture of the criminal legal process where critical decisions are made of freedom, right, we can see that the pre trial system, sometimes replacing bail or augmenting bail decisions, these algorithms can be used. This can be done in consideration for early release from a sentence or post conviction stages while formerly incarcerated persons on probation or parole. And in this respect, the risk assessment tools are not really predictive at all because they do not use any artificial intelligence machine learning or any progressive self learning technologies that you mentioned earlier risk assessment tools are merely like calculators using smart technologies. And some of the concerns we see around risk assessment tools are that they use historically discriminatory data sets. Scores are not individualized to a person rather they project the generalized assumptions about recidivism from a group of similarly situated demographics onto this one person. And we believe they do run afoul of constitutional protections tools are rarely used to identify needs though they say they're used to identify needs they're really mostly used to make liberty decisions. And we fear that sometimes they supplant or replace a judge's independent decision making. We don't know how, in many instances and can't know how these tools work, both companies and governments block disclosure under respective theories of trade secrets or security classification. And both individual and the public are denied due process right a tool that is making a liberty decision about people should be available to the defendants they should know how that tool works they should know the factors that are going in how frequently that tool is is audited how reliable it is. And so they can challenge those decisions. And lastly, before I turn it over just facial recognition right this technology records measurements between structures and markings of the face to save them as a unique profile of that face for either one to one verification like your cell phone or your cell phone is using a stored image of your face to verify yes or no, the person holding up the phone is you, or one to many identification right. And this is the particular law enforcement use taking a photo that they find from either a mugshot or an image used on a recording camera and running it through a system of names and this is what DKG was talking about with a CO use study of members of Congress and trying to find if that one person is found in the system. And this is really important because right now right protesters are being intimidated from exercising their First Amendment rights to assemble and to speak freely with the deployment of these technologies. Law enforcement and white supremacist hate groups are using even access to social media and facial recognition technologies to identify protesters to docs them or otherwise target them. And our biggest fear here is that there are inaccuracies and facial recognition facial recognition is known to inaccurately identify people who are not white, not male and not gender static in particular, and this could bring real harms from misidentifying the wrong persons. No, that's helpful and I wanted to drill down a little bit on on facial recognition and the the use of algorithms and an AI in. I think we've sort of acknowledged a little bit and I talked about in the beginning that story about Amazon sort of rushing to create this system. I believe the term that Daniel talked about was sort of its cost shifting right it's cheaper to use these systems instead of, you know, essentially getting human beings to review this massive amount of data. So, but is there a concern with, and I'm thinking about the story, the news story that came out where a company was essentially to try to perfect their facial recognition technology and they essentially took pictures of homeless folks. And they had contractors right go out and take pictures without really any real consent. And so this was definitely a very privacy invasive sort of attribute for, you know, for Prince or Daniel or I guess you know iris in the education field I mean is there a concern with just establishing the training data to make these algorithms and sort of being privacy invasive there is that a concern. I think it's a concern. I mean, we see, you know, the systems are designed particularly the ones that are based on large statistical models. They only work if you have access to large amounts of data so just the presence of these systems drives a hunger for large data sets. So there's some ways you can assemble large data sets that are not as bad for people who are in the data sets but sometimes when you have a large data set if you don't manage it well, it becomes an attractive nuisance and that could be like a problem for law enforcement or immigration services or for foreign hackers or for identity thieves or whatever right if the data is not well managed we've seen instances of that with just for example not even talking about machine learning here but OPM the Office of Personnel Management had a system that collected all the information they had on doing background checks over decades and it was compromised. They hadn't been doing adequate data destruction. And so if you build a large data set in order to feed these systems because you want them to be better and you don't think consciously about how you maintain that data. That's a potential privacy risk for the people whose information is stored there. I want to also point out that many of these, these systems are developed for commercial purposes. Right. Like they want to predict who's going to buy a certain pair of shoes or who's going to want to buy a car or who's a good, a good match for this particular restaurant. And again these systems are being sold to the advertisers as saying we're going to find you the people who are going to be your customers. There's an entire economy around building these data sets about people that are that that's marketed towards salesmanship. That also has discriminatory impact. Imagine if you have an algorithm that's trying to decide whether to show an ad for new homes. If it decides that only wants to show new homes to people who have historically been known to buy homes, and the models have the exact same sorts of racial markers that were that were talking about proxies for racial characteristics whether that's zip code, or education level or family members who've had contact with the, with the carceral system or anything like that. Again, those algorithms may well filter out and do a sort of digital redlining. And this is happening entirely in the private sector right, and still having these impacts. And so you have this, this pressure from the algorithms for large data sets on the one side that's a privacy risk because the data sets are a risk. And on the other side, you have this more ways that your information can be misused because these same systems have an impact on people's lives as the information gets pulled in. So, you know, in addition to losing, you know, people losing control of their data, people are also impacted more closely by the fact that their data is out there because the systems are making decisions on their behalf. This is by people who have no malicious intent, but they simply are creating systems that have this impact, which is one reason why ensuring that we have clear visibility into what these tools actually do is important right that's step zero for us knowing that the systems are discriminatory is insufficient for us to protect the population and, you know, ourselves from their, from their impact. It's very difficult for us to figure out how we can address the inequalities and how we can find the privacy violations and ways that are being run by. It's helpful. And so I want to incorporate an audience question here. The question is as it comes from someone who works in information security, and they have concerns about the integrity of the systems that we're talking about. And, you know, and I'll pose this to Daniel, then I'll open it up to others but could you speak to the dangers of a hacker being able to tweak certain data points to force a less than pleasant outcome. I mean there's a bunch of different ways we call this adversarial modeling. So, if you're so so Prince described some of these risk assessment tools as just being calculators not smart tech. And I think I see the distinction there although I actually think that smart tech is basically some fancy calculators as well. The difference I would characterize it is a calculator is something where you can actually see everything that's connected and what's going on it right so you can say hey look. This is just a spreadsheet and it says do you have parents who've been arrested before. And if you say yes then that makes you twice as likely for us to just hold you in jail anyway. That seems like kind of a messed up outcome, you can inspect it you can see the path for the messed up outcome. The smart tech is really obscure tech right it systems that where the calculators have become so complicated that it's difficult for anyone, even the designers of the system to pull it apart and say here's why this particular decision was made. And so if once you get into this opaque tech system. It becomes very challenging to figure out whether something has gone wrong. And furthermore, you can get folks who push that opaque tech system into a really problematic space right we. I don't know if you've seen these these researchers who figured out stickers that if you slap the sticker on anything than the standard machine learning algorithms and think it's a banana. Doesn't matter what it is. This is a banana sticker right you put it on and the machine learning algorithm thinks it's a banana. Why do they think it's a banana. We don't really know right the system is a complicated one. Nobody can pull it apart well enough to understand it, but hey this is a banana sticker and if any obviously put it on the machine will think it's a banana. Imagine somebody tweaking a system either by solving the training data with bad stuff, or perhaps by getting into the model and actually fiddling with parameters you know training their own model getting parameters that they like, and then going and substituting that in an attack that people don't somehow notice and causing all kinds of mayhem on that right. So, so the more that we can understand these systems, the better we can defend ourselves against them. And I want to point out that the, you know, having that kind of review. This is not a theoretical concern right this is a, these systems are. They're basically like let's throw a bunch of data at this and turn a crank and then machine will give us an answer and we don't have to think about what's going on inside that. That attitude might be useful for like saying let's generate some hypotheses that we then want to test in the world. And if you say well let's just, let's just apply this to, you know, say a security system to figure out whether someone's access to the bank is a legitimate access to the bank. And I can manipulate that I could lock people out of their bank accounts, right, I could lock people out of their ability to access civic technology what if I made it so you simply couldn't talk to your representative anymore because their, you know, machine is a hybrid model of what is a, you know, what is a hacker trying to get into the, trying to get into the web forum that you talk with your representative on just excluded my neighborhood, right. So these are the sorts of things where these systems have real impacts and the more that we treat them as black boxes, the more that we say hey, we trained it up on a bunch of data and it's probably fine. The more we are opening ourselves to manipulation of those systems. I want to double down on this conversation about the, you know, like data security and iris and just want to get your sort of thoughts. When we're thinking about education systems, I mean they're collecting a lot of information on individuals as they enter these institutions. And a lot of, again, as Daniel talked about a lot of these automated systems require that those troves of data to be able to make decisions quickly so what are the concerns from the education perspective on that sort of data security and how can violate privacy. So there is an actual federal law called for that governs educational data privacy. That being said, I don't believe it's ever or is very rarely enforced in any way so while it's used a lot in conversation it's actual impact is mixed I would say. And most of the systems we're talking about existed in an exception for although there are some blurry lines here because the, it's basically institution data institutional data which is extensive on students, but it also includes things like learning system data, which these students interact with as they're taking classes, it can include courseware data it can include CRM data so the tracking how the institution has been communicating with the student it can include data on the kinds of clubs or things that the student is involved in on campus, it can include location data. It can get quite extensive and generally speaking colleges partner with different types of vendors who help them create the algorithms with this data. So the security and privacy concerns are pretty extensive and the security concerns in particular I think that there's definitely been a lot of panic might be strong but there's been a lot of concern and focus on GDPR and the recent law that went into affecting California and some of these I would say higher education is not great on this they are generally the targets of hackers and things like that. When they have security breaches they tend to be incredibly careless security breaches things like storing private student data on an external hard drive that's in some locker that gets like it's things like that. So the amount of data that higher education has on their students is is quite extensive and they don't have enough security protocols. And so we actually have some suggestions and one of our publications around things colleges should be doing to be thinking about with securing their own data and then also securing the connections with any kind of vendor that they partner with and thinking about what the security and privacy looks like at that particular vendor. So it is absolutely a concern and unfortunately just the federal law isn't really good enough to help guide colleges to doing a really great job. Now that's understandable so I mean we talked a little bit about what the cause of these problems are right the algorithm bias as an issue and the idea of privacy intrusive machine learning as another concern. A lot of talk about that when we talk about algorithms it said that the discrimination by algorithms is really just a reflection of discrimination already taking place in real life. Right that machines aren't inherently biased it's the coding or there are some inferences being made as the system deploys. You know I wanted to bring in Prince here if you could sort of give us your thoughts on on how algorithms will reflect this disparity and if you could give us specific examples as well that have been brought up in the criminal justice perspective. Sure KJ. So I do just want to highlight causes and then I'll drill down on specific examples so with risk assessment tools in particular there are some of the causes of the disparities really are like the approach on the outset right. So the fact that a risk assessment tool is being used at all shows a reductionist approach to incorporating and understanding human experience right you're taking a range of experiences of why people may not have gotten an education or why they only have one parent or why their housing situation may not be stable at one moment in time but was always stable right and you're not taking account of that. And you're minimizing that breadth and depth of human experience to numbers to risk scores to needs assessment. There's a loss of nuance and individualization there. And I'll just talk about a few others before I drill down on a specific example. The second one with risk assessments is there's a monolithic understanding of the relationship between risk factors. So what does education have to do with family or community ties and what and as it relates to should I be free before you know before my trial and we're relying on kind of like monolithic recidivist studies that associate with the community of people like this have these problems. But it isn't particularized enough and I think we just see old understandings of problems that we're learning more and more about that that are not reflected in in these tools and misuse of the systems in the criminal legal system at all. And one example of this is pattern. So pattern is the prisoner assessment tool targeting estimated risk and needs. That's long acronym that the Department of Justice or the Federal Bureau of Prisons uses to assess the risk of all of its prisoner population. It did so after the first step back of 2018 and every person in the system has a risk score. And that score was based on a number of characteristics that are static and dynamic right one of the static factors and static factors of course have more weight than dynamic factors the static factor was age at first arrest. Okay, so you made a mistake when you were 16. Well now they were using pattern just a few months to go to higher archives who would get out of prison for the COVID-19 epidemic to depopulate prisons without any consideration of pre existing conditions without any assessment of health or prison conditions without the idea of understanding hey guards might be really the ones we need to look at not people in prison right who have had very limited if any contact with the outside world. And they used this system right at first to characterize in higher archives who would get out of prison during a pandemic that is not the use of this tool. We already see the problems of the use of the tool and just the context of early release, let alone using it to essentially prioritize who lives and who dies in a pandemic. And so these are kind of some of the, not just the issues inherent in the system but the way they're used for other adjacent reasons kind of as DKG said they become attractive nuisances that don't really solve any problems they just make us feel good because they're objective and they give us a short answer in a lot of uncertainty. No, that's that's helpful. And then you know when I also round out this conversation about exploring the causes and the sort of the reflection of discrimination we see in real life being portrayed in these systems. Iris if you could talk about the education field and sort of a little bit more about the causes to like why we're seeing these disparities in the system. Yeah, so two things here one. We have hundreds of years of systemic bias that have populated our campuses and so obviously the data that they're using is going to be discriminatory but not only that the experience that we know students in particular with students of color in general have on campuses can be very negative. And so I liked Daniel's point about pointing these algorithms in inside we'll talk about that with the solutions maybe but using your data to try to fit and then you're noticing for instance that if you have a risk algorithm about students that might be a risk and how you can intervene early, you might notice that they are over identifying African American students for instance as being less likely to graduate. Well, that is most likely true because it's actually true across higher education, but why is that it's because your campus is not supporting those students effectively right so the real solution comes with digging down on what's going on on your campus what your campus climate is and how you're supporting the students who are at risk or who are being identified as at risk. So that's one of the huge things another weakness I will say in these algorithms is that they are not built for unprecedented times. So, in a in a risk algorithm for instance, one of the things they use to see how well a student is doing and how at risk a student is is how much they interact with the learning management system which is the online portion of many in person classes. Well, now that all of these have have migrated to remote all the students are using the learning management system a lot more and so all these students who are at risk are now more and more being shown as being green or students that don't have any risk but of course all students are more at risk in this situation and so we see that the data pattern that may be held up before the pandemic are not holding up now. This is a particularly unprecedented time, obviously, but I will say that I think this is a risk across higher education with using these systems. If something happens on your campus that disrupts some of the historical patterns and the data, the output is no longer relevant and how do you catch that before you decide that this individualized person is no longer needs for advising support for instance. That's helpful. So we, I want to thank panelists you guys have done a very good job of staying with the flow problem cause we have a lot of audience members who've been asking questions about the solution they're clearly very familiar with the issues we have with these systems. I want to T up the conversation about what are possible solutions, both technical procedural and just frankly from a business model perspective. You know I have to use this opportunity to make the plug. Once again for comprehensive privacy legislation in the US, and we can look to the EU and see how some of the data protections and their privacy law, the GDPR contain restrictions on automated decision learning without any human involvement, and on the automated processing of personal data to evaluate certain things about an individual, or is there calling it profiling. Now privacy principles, you know, come up when we talk about machine learning, and about privacy protections need for transparency companies only clicking the data they actually need for the service of providing. Of course, use limitations. We also hear a lot about accountability and fairness. So, you know, I'm going to try to combine some of the questions that audience members have been asking about and you know I want to start off by asking, you know how do you manage or include racial disenfranchisement in, in these systems. And I guess you know I'll start with Daniel and we can kind of round robin it, but is there a way essentially to to when you know there's bias that exists how do you counteract that in the actual data. Well, again, the way that the systems are applied in society has a big factor on this right if you're using the system to try to figure out which police precincts are are doing racist police like more racist policing than others. I'm not sure if I know police precinct that's not doing any racist policing at all. But if you get what which police precincts are more racist and therefore need interventions. That's a very different application of these models than it then to say we're looking at this to figure out whether a specific individual should get to go home tonight instead of being kept in a cell. Right. So just the application alone is one is one question that you need to ask. You can also try to explicitly model what the level of bias is and attempt to counteract that mathematically. That's a pretty difficult thing to do and I'm not convinced that we've seen examples of that really well deployed yet. In particular, if you, if your system is asking a very simple question, like, is this person likely to get re arrested again soon. The answer statistically may very well be that African Americans are more likely to be re arrested soon. Right. And so if your system is designed just to ask that question. You can try to account for that. But the fact is the system outside of, you know, the justice system is the problem here. If you want to try to adjust for that, it's very difficult to say, like, how do you do it, right? You need to ask a different question than you're asking otherwise. There are some techniques that people have proposed that you can use to minimize sort of individual privacy harms from large data set gatherings. One example of that is the sort of differential privacy approach, which says, we're going to make our data a little bit noisy. And we're going to do that in such a way that no one individual is going to be worried that their data is going to stick out or be recoverable from this data set. We're going to make it noisy, we're going to aggregate a bunch of things together. We're only going to release the aggregated data. And so whether you participate or not, the system is going to come up with roughly the same answer anyway it'll be slightly more accurate if you're in there. But the fact of you participating or not isn't going to be, isn't going to leak your information. But that's only one piece of the privacy harm. The other part of the privacy harm is when the system is built and it measures a discriminatory environment and then turn your turn around you apply that discriminatory environment to the to the population again. People's privacy are going to be harmed in those in those situations too. So I mean, in, can you, and I'm glad you talked about differential privacy, I was going to talk about the sort of technical the technological tools you can use in these situations. Can you speak a little bit about federated learning and any other sort of privacy enhancing technologies that can be utilized as a way to address these issues and then, you know, I guess, in that sort of same breath is can I be used to, again, you're saying that the system is the issue that we're not asking the right question so can I be used to sort of counteract and separately, you know, what are some other privacy enhancing technologies. So, I'm a, I'm a data scientist right I, I am interested I believe in the scientific method I believe that the more information that we have the, if we structure our use of it well I think that's useful. I'm going to say that data cannot be used in a way that is, that is helpful. But the questions that we asked how we, those are the, those are the difficult challenges that we need to address. Federated learning is an example of a, you know, machine learning systems that say hey we don't want to learn too much about the main classifier classifier is like a machine learning system that sorts people or things into categories doesn't want to know too much about the individuals that it's calling training information from. So we'll have a bunch of smaller individual systems that learn some pieces of the model and then we'll aggregate those models into some metamodel. Again, this is useful in terms of making sure that an individuals personal data doesn't leak to the central aggregator, but if the impact of the system on the broader society is still problematic it doesn't necessarily address that kind of underlying concern. I'd be curious to hear Princeton irises perspectives on ways that the systems might be potentially used for good. And if there's ways that we can identify when a system is being misused and explicitly call that out. I want to note that if you are subject to one of these systems, sometimes you don't know that you are like a decision could be made. I was at a conference a couple of years ago where the title of the conference was the computer says no. I'm sure we've all heard that right the computer says no well what what's the computer who made that decision. How was it done. And if you've never been subject to that where a bureaucratic system sort of computer washes a decision that it made to you're like hey sorry like what can you do. It's objective. It's a very frustrating experience and so I want to know if if Princeton iris see ways that we can empower people to push back and challenge the systems when they find themselves subject to them. I'll let iris jump in first and then we go to Prince. I appreciate that. So, first of all, I think that the the appropriate use of these systems is really about augmenting human decision making and human interaction. So I will say that the reason all the training data is so bias is because people are biased so when an individual human makes a decision inside of bureaucracy very rarely are you able to challenge that decision effectively either and there's probably not a great paper trail about why they made that decision perhaps. So I will say that like just using these systems alone maybe it's slightly more opaque but I will say I think your bureaucracy and human decision making is definitely opaque and seriously racist and bias as well. If we think about judges making like to use a very important one like making bail decisions or things like that, we know that they're incredibly biased and horrible and so how do you try to make those better. It's not by replacing that with a machine that's learned how to make that decision based on the biases of all the judges ever right that doesn't actually fix the problem. So to turn back to my own wheelhouse of higher ed. So instead of having the professor come and tell that Latina students that she's not going to succeed in her STEM major that she wanted. You will you need to train the people on the ground to are interpreting and using this data to interact with the student in a that's supportive right so Daniel exactly what you've been talking about like it you don't say no you're going to fail at this you say how can. Here's I'm going to be real with you it's going to be challenging here's why here's what you need to do. And here's how we can support you in doing that right. So we've done a lot of work about thinking about how you can train faculty and staff to not only interpret the data because humans are incredibly bad at probabilities they think that 60% means it's going to happen right which is not true. So helping people interpret the data what the signals are telling them in the data and then also how to interact with the students themselves. So they're not discouraging students right so we know that people from different backgrounds interpret messages very differently. I'm a first generation college student who already feels like I don't belong somewhere because I'm not in a particularly welcoming environment due to my back or like, you know, because of my background. I, if I hear a you failed a midterm come meet with your advisor message that's going to be very discouraging to me and I may drop out. How you message effectively how you train the people in the frontline, and how you acknowledge that you can't just look at the output of these systems and say, Oh, this is definitely what's going to happen and hide your own bias behind the objective data right which is what we see in the interpreting of these systems a lot. I'm not even going to try to address enrollment management so the. Admission of students and the recruitment of students in the higher education, because that's got a lot more money involved and so it's much more complicated, but I will say that those are a couple of the ways we've been trying to help colleges think about how to do this differently, because for for higher education, using data is very important and this is a great resource for people to be able to use you just have to be able to use it well which means good training, testing the data, realizing what it's telling you and what it's not, and then interacting with students in a supportive and growth mindset sort of way. Prince what do you think. I now have to echo DKG and iris and everything that they said, something that really sticks out on that applies in particular to risk assessment tools. Back in 2018 the leadership conference with the coalition of about 118 other groups promulgated these principles about pre trial risk assessment tools and I won't go through all of them but there are two themes that kind of emerge that can help us tamper our expectations as iris is talking about about the outcome of the tools or as DKG is talking about restructure what we're using these tools for. The first is framing is important in the criminal context to preserve the presumption of innocence of a person. So an outcome instead of saying likelihood to recidivate likelihood to fail right and when you fail your failure, right, it should maybe be framed in likelihood of success, right to comply with these issues. So a recommendation, what would otherwise be a recommendation for detention should perhaps be a recommendation for an adversarial hearing right with robust constitutional rights for defendants. So instead of saying this person should be in jail right it's more like, hey, we need a hearing to really talk about this individualized persons situations, affairs, and maybe we can come together with an outcome and transparency right echoing defendants don't know what factors are weighted against them. So they can't defend the tools outputs they rarely see the tools outputs before the outputs happened happen they they don't know the prosecution's arguments against their pre release until the moment so they're being put at a disadvantage defendants are in trying to find out what the system is accounting for why it says I can't be free or I have to be surveilled while I'm free more than other people. And just trying to fight that in the system where you're already at heightened emotion and your life is already hanging in the balance that's too much burden on a defendant and that's unfair right so we need to reconsider the outcome the outputs of these tools and kind of change what the outputs mean right and for facial recognition in particular. I do. It seems like I'm continuing to highlight the problem but it's so important to know that I'd say in about 2017 Georgetown. Center on privacy and technology and MIT's famous MIT graduate, Dr. Joy Boulomouini they first broke respective stories about racial and gender racial and gender bias in facial recognition and just last December December 2019. NIST the National Institute for science and technology reported that most of the 189 algorithms from 99 of the top developers that voluntarily submitted their algorithm to them because they thought they were great and accounted for all these different demographics. And some of them falsely identified African American and Asian faces 10 to 100 times more often in Caucasian faces. That's just in December right so this is an ongoing process right that that needs accountability political oversight which refers to agency officials, explaining their uses of the tech and receive ongoing and continuous approval absent their abuse or overuse of these systems. We need technical oversight right which refers to just baseline verification and auditing standards, and that's more in the DKG wheeled house but we believe that. NIST if, if when they promulgate you know higher standards that they should be adhered to and continue to be reviewed on an ongoing process and community oversight right we've seen communities in the United States with the help of the ACLU on completely facial recognition technology being used by law enforcement, the communities don't want them right they don't want their elected officials and their agencies that are supposed to be in service to them using such a in an unsure technology you've seen bands in Oakland, San Francisco, California, Somerville, Massachusetts partly banned in Seattle and we also need to listen that you know maybe fixing the algorithm may not necessarily be the answer, not using it at all in certain places might be the right answer because there's, you know there's a reason why humans judges, sheriffs have made decisions and perhaps you know focusing on human training for these problems is better than machine training. Great, so we solved it all we're done. No, I appreciate that the thorough breakdown Prince you're right and I want to dig in a little bit more into the technical oversight that you spoke about and you know I want to bring in Daniel here there's there's been conversations about the idea of auditing algorithms and completely keeping in mind what Prince is saying and in some aspects, we shouldn't be using these systems at all. You know in the context where they will be continued to be used, and we're seeing it come up in legislation similar to representative Clarks I believe represented Clarks bill talks more about impact assessments which is more about anticipating. But starting with the auditing of the algorithm first that talks that looks at the product, the output. What are your thoughts on that if that if that is a effective measure, and then I want to hear from the industry's can actually employ that. Yeah, if, if the framing of the audit is, as you described it in a context where you're going to use these systems anyway. And then I think the audit is already starting out with one hand tied behind its back right and outcome of an audit needs to be able to include this system must be removed. Because if you don't, if that's just not an option, then there may it may be an unfixable system right you may the audit may reveal severe problems with it, whether it's problems with inaccuracy as Prince is rightly highlighting the facial recognition, or problems with extreme accuracy right, maybe the system is perfect. I'm not sure that would be a better system than system that's super inaccurate right. I don't think that it's a good idea that we actually have perfect face for perfect face recognition for all communities across the entire country, providing a centralized data source of people's locations people's contacts people social graph I don't think that's a great outcome either. So there's reasons to want to remove it even if it's not inaccurate. Right. So, so I think, I think you need to be able to say look an audit needs to be first off it needs to be conducted in the context where the results of the audit have some teeth. If they don't have teeth, then you've just ended up with like, you know, well, we did the we did the math and we found out that yeah we are racist society. Next. Okay, let's move on to our next agenda right like that's not a solution. That's not a, that's not a helpful auditing process. So, and I think Prince was right in calling for these audits to be done in an ongoing basis. This is particularly a concern for modern software, which is routinely updated. We know that software has bugs, whether those bugs are the kinds of inaccuracies that Prince was pointing out or just stupid divide by zero errors, which we've actually seen in in systems that are using the criminal context right there. There are genetic assessment tools that it turned out once you got access to the source code. In other words, you could see what they were doing. They were actually doing the math wrong. Right. And some people probably went to jail because the machine said, Yes, your DNA was definitely in the sample that was found on this, you know, in this particular location. Right. So, so, but then, but then the software was updated at some point and who knows whether those those bugs were fixed. We do we even know what versions of what software were used to produce the results that are found in trials going back a decade. We really know those things. So these audits need to be done in an ongoing basis as the software changes and as the training data changes right as these systems are machine learning systems. Presumably, you hope that they update them with new data sets as new data become available right you don't want to take a sample of data from 2004 when the folks who thought of this idea, you know, designing the system, and say that's going to be our new data that we're going to use in perpetuity as Iris pointed out, the world changes. And if your baseline data is from 2004. Well, the world in 2020 looks very different. And so you want the training data to be updated but as a result that means those audits need to also be reapplied. Right. If the training data changes and all of a sudden the system is providing bad outcomes, you need to make sure you can catch that. There's access to the audit right who who gets access to see it. Some proposals that we've seen the audit has to be held internally by the organization that's using this so that they have access to it. And maybe you can pry it out of them in some specific contexts in other recommendations, the audits have to be made public, you have to announce what systems are being used to the people who these systems are applied to do they even know that the system exists right or they just told sorry, your medical benefits were cut because, because we just have a new way of deciding it today, and we're not going to tell you what it is. So this transparency it needs to be an ongoing thing it needs to be visible to the people who are subject to it, and the people who are subject to it need to have the capacity to do the audit, a technical audit of a complicated system requires expertise and it requires actual testing. So imagine yourself, imagine that you were in a trial, and the prosecutors come in and say we have a fancy tool that we paid 10s of thousands of dollars for. And we took some of the evidence from the crime scene and we took some evidence from you, and we turned the crank on the fancy tool and it said 98% chance. Well, let's set aside what Iris's point, which is totally legitimate about the fact that people can't interpret probabilities. If you if you were faced with a tool that said we have a 90% 95% confidence that you are the criminal. And you know that you're not. How do you challenge that tool. What is the process, how can you do that who do you know is your is your public defender capable of requesting the source code and auditing to find out whether they're dividing by zero in the right place. How do we make sure that the audits are present and well funded to make sure that the appropriate expertise is routinely available in those contexts this is on going to be an ongoing social problem and we need to we need to actually plan for that. And I know and I appreciate that especially to talk about the need for to be ongoing and the need for resources and the expert, which would fund the expertise. So hopefully in ideally with the education field iris. I think a lot of these conversations we talk about theoretically and technically like what can be done to address a problem but how practical is it in the higher education. I mean you can see where I'm going right with the sort of, and we've talked about this offline how the automated systems of higher ed is using currently aren't super advanced. And so to now like put in resources into bringing in these experts to do this on an ongoing basis. You know how practical do you see that. And if it's not practical like what are the challenges and hurdles that need to be overcome to sort of bring bring us up to speed. So it's in higher education the internal expertise is absolutely not in for the most part there to do any kind of audit even a disparate impact analysis, I would say. This is a huge problem and it's a big reason why I put out a vendor guide that was basically colleges this is what you need to ask your vendors before you go into contracts because another issue that Daniel didn't touch on is that if you have a contract with a vendor, even if it doesn't turn out the way you think it's going to getting out of that contract is almost impossible for governmental entity. Who has already gone through all the procurement processes and all of these things. So you really actually need to know what you can ask the vendor to do upfront. If you haven't signed a contract yet they don't have any money you don't have any resources to audit the algorithm you don't have any resources to ensure that they can look at a disparate impact analysis. In my piece I recommended that people guarantee get a guarantee from the vendor that they will do a disparate impact analysis. So once the vendor they sign the contract and run the training data a few times. What we've seen is actually once vendors start running the data for higher education a lot of times the output is like laughable on its face until they really do a lot of work on the data interfaces and the data definitions and all these things and you actually actually have to work really closely with the vendor to get that right. So all of this to say creating a system where you where the vendors of these products have to do an audit every year with each college because generally they're only using the the colleges data as their training set. So it's individualized for each college at least that's what they say. So doing that audit every year for every client would be I would I would assume would be pretty would be pretty expensive and pretty hard. And I don't think they would like that at all. I agree that it's necessary and that something like that should be in place I don't know how it would work exactly in higher ed but I think it's a really important part of it and I, I wonder if they could do I don't know that's Daniel you'd have to answer this but if they could do some audit of the tool overall rather than having to do with each client. That might make it easier to have some kind of federal rule around this and then making sure that those audits had to be public in some way which they will not want to do because they're always worried about their black box which shouldn't really be a black box. And actually I've talked to a lot of colleges that have decided not to go with unders because of that they won't disclose enough of their of their algorithm for that that college and particularly the faculty who deal in data science to feel comfortable with that particular system. For Prince, are there any examples where are there jurisdictions that are actually sort of undertaking the sort of audits of their systems. And, you know, if not, again, what are the same similar question to Iris like what are the hurdles to implementing these types of, you know, audits of these systems that pretty you know have a pretty detrimental impact on on the lives of individuals. So, first I do want to highlight you know transparency is an issue as your question is suggesting kj. It took years of freedom of information act requests, you know, public records found foundational litigation to show us a clear picture of where facial recognition is being used, and how prevalent its uses what systems isn't using. And some of our partners have kind of really done that being counting work jurisdiction by jurisdiction, and that should not have been the case, right, especially these agencies that serve a public utility a public purpose must be transparent to the public I don't care if it's a private tool or not. So the procurement processes right should go into some kind of either legislative oversight or self reporting on, and there, there should be some kind of, there should be some kind of response right to if departments are found to be piloting technology or using technology that has not been previously disclosed right some remedy action needs to take place there that is also public. Um, and, you know, we are on agencies should also be forthright in promulgating their rules around use right on when groups did a FOIA request of the Secret Service, and the Secret Service revealed yes we're using facial recognition technology which is no surprise. Um, and they also shared you know the rules that they were using internally right to make those decisions and that was very helpful for us to know those should have been publicly available right there's they shouldn't have to have been been foy it and another issue is some of these tools are public. Public quote unquote when you think of a clear view AI right which blockbuster story that dropped in January about this company, collecting over 3 billion photos from social media websites that are publicly available and kind of the DKG's point their algorithm for facial recognition is very accurate um eerily accurate and in many cases and you know that company has worked with over 600 law enforcement agencies in the United States including the FBI, which the FBI itself has 20 to 30 times less images times less images than clear view AI which which is a private company so there's only legislation that does few things I mean I've kind of analyzed this and seen a continuum of four responses. It either goes from straight up banning the technology don't know how it's being used or it is being used we don't like it we don't know enough. The jurisdiction like to ban it moratorium and that's kind of where the leadership conferences right now saying, let's pause and study right let's find out what's going on with the tech. What are the legal structures in place to provide that outside look in the guardrails and let's stop before we know those things and restricted use control with controls it's like permissive use but as long as like the secret service example like as long as you produce some kind of guardrails about it or some kind of impact assessment on a regular basis if you self report. This might be okay. Some jurisdictions have taken this approach, and then I haven't. I mean the only places where I've seen the most extreme of no controls is just by happenstance right police departments are just adopting these people didn't know them and so there are no restrictions because no one knew it so almost in every case where people find out this technology is being used in their jurisdictions. They lobby for some kind of control or some kind of transparency. And I really think me just bring in legal perspective I really think that's the key here. Some of these systems as you said they're not going to be stopped they're in use. They're attractive. But what is the most that we can do since they have been used and ongoing auditing and transparency and reporting and some kind of political and community oversight and involvement are ways going forward. That we can work alongside the tech to bring about better results if that's possible. If they can't be abolished altogether. It's really clear for our panelists and for our listeners that you know I tried hard to sort of not conflate how sort of commercial entities are using this data and these systems versus sort of like the government, you know, it's industries that interact with government more so you know education, criminal justice and you know there are a number of other areas that we haven't even touched upon in this conversation credit right fair housing are other areas where these systems are having a detrimental impact and have a government perspective or government interaction, you know with, you know, we have about eight minutes left in the, the conversation and I feel like we've been kind of harping on how artificial intelligence is, you know the concerns with it and these systems but is there, you know, I'm wondering if there are any sort of positive uses of artificial intelligence that can be used to address some of these issues. Obviously, you know just to sort of end it on a more optimistic note, we've talked about a lot of the concerns the problems and the causes, a lot of them coming from systemic issues. We've talked about the solutions right you need a little more human interaction you have to reframe the actual question you're trying to get an answer to, but focusing on you know where we started the conversation about the role of algorithms, where can we play a role in solving some of these issues. And if they can't and that's, that's a very answer to but I just wanted to open that up and you know just going around the same round robin Daniel first Iris and then out prince. Yeah. So, I think, to the extent that an algorithm has been deployed for cost shifting purposes, as we've talked about. If people here when we're going to have to do expensive ongoing audits. That's, that's too much. Well, that may just actually need to be the adjustment to your idea of what kind of cost shifting is going on here. If the audits are expensive, and we need those audits in order to ensure that the algorithms that are in use are just and equitable and not privacy invasive. And it turns out that it costs too much to audit. Well, okay, you know the answer might be this is not an appropriate tool for the time. Right. So, so, so, you know, if there's a data system that's in place. And there is adequate auditing and testing for it. And that auditing and testing is not done by proponents of the system, but it is actually potentially adversarial auditing who have access to the data there. And we can see that it's, you know, it's asking a question and it's applied towards society in a way that we think makes sense. Right. Then, you know, those systems can be reasonable. If it turns out that those systems are to that it's too expensive to put the proper safeguards in place on the system, or that the folks who are offering it our private vendors and they say oh well you know this is our black box it's our it's our secret sauce we can't reveal it. Again, the answer may well be that's fine use your black box wherever you want to use it but when we're using it for a system that has this public impact is just not appropriate. You know, we're not going to, you know, a chainsaw is really good at at cutting through anything, but we're not going to encourage people to use chainsaws to mow the lawn in the public park it just doesn't make sense to do that right. And so there may just be places where the technology doesn't make sense because of because of these factors. That said, like, to say we're going to go cut the public park, you know, grass with scissors. That's also doesn't make sense right so there are some kind of balances that we can make here. And it requires us to be conscious and thoughtful and not jump at every sort of salesmanship opportunity that says hey, we can cut a bunch of costs and make it easier by making these abstractions by making by reducing as Prince said, the richness of human experience into a couple of costs, then that's an efficiency argument right if that efficiency argument isn't done right. It's a problem. And so we need to be wary of quests for efficiency at the cost of the social, the social contract that we're trying to enforce. So again, we want data we won't want to know how these systems work we want to know how our, how our courts you want to know how our education system we want to know how our commerce is affecting everyone. So these systems can be used to help us learn those things if we ask the questions in the right way and we're thoughtful about it and we take the appropriate measures. But we, but you know, if it's if it's hey we can do this thing quick and let us save a bunch of money. There's the red flags should go up immediately like think about who might be on the receiving end of that on the, and the, you know, how is that going to actually affect the people. So I couldn't agree with Daniel more on that in particular. I think actually these systems have had a lot of really positive impacts and different colleges and higher education, but not because they were put in place to save money or have a quick fix I mean this is a systemic change that has to happen in colleges and I'm thinking particularly about early alert systems so traditionally advisors in higher education would just deal with people who came for through their doors and asked for help. That was very much a self selected group of students, those were not necessarily the students that needed advising. What the systems can do when they're done well is help those advisors reach out to the students that are vulnerable who need their help who would not have otherwise come through their doors and we've actually seen that in the data where, if you like map how who advisors were after the system was implemented and who advisors are seeing after the system was implemented and how that overlaid with risk scores. It actually has a huge impact on students that are vulnerable in higher education and really it behooves the college to act on the data that they have to support the students that they've admitted to get to graduation because when you drop out with debt, you're in really big trouble and you're going to do better if you end up graduating. So, in that way it's had a really positive impact that is not universal. Because, as Daniel saying that can be seen as like a cost savings like you're somehow going to reallocate the same advisors to new students and that's going to save you money and time and all these things. That's not how this ends up working out the ones that have done this, the schools that have done this well have hired a ton more advisors have been much more intrusive about helping, and have really had a systemic impact all the way through their college to make a big difference. The most famous example of this is Georgia State University, where they've managed to make increase their graduation rate significantly over 10 years and close the gap between black and white students for graduation and with Pell eligible so low income students and people who are not eligible so is a huge success story, but they've done a ton more than implement the system. It's been a piece of their strategy that's been well informed for that. So, I think in higher ed, it can be done well, and when it is done well it's incredibly powerful and can be a force for real change and and and help, like our society really, but that is by no means guaranteed. I will say Georgia State is probably the exception in the way that they've implemented this. In a lot of cases it's just we are going to purchase a product we're going to put it, we're just going to like put it out there and it's going to fix everything and that's not how it works. I hate to be that person on the panel but I am going to say in the criminal justice context as we're seeing outside of our windows on the news every night and in over 4400 jurisdictions in the last week. There is so much structural inequity in the criminal justice system. I highly recommend that tools not be used right now if ever at all. The cost of an algorithm, you know, facial recognition system, misidentifying you or, you know, a bail hearing kind of a risk assessment tool, keeping you in jail when you're really low risk, you could lose your job. You could lose your standing in the community. You could lose your life. You could lose a whole lot of things. So to just put a algorithm on top of and say, Oh, we'll fix it over time with like a little bit of transparency and a little bit of like this and a little bit of that. The harm that is done to human beings along the way is irreparable. Right. And in that context, you know, as a leadership conference we're taking account of all of those harms that we've seen being done to people of color for centuries, let alone decades, let alone right now. And right now is not the time to be introducing these tools, these algorithms into the criminal justice system. I appreciate the diversity of perspectives and I think where we're coming from, especially if we're looking at a certain industry I think all these points make a lot of sense. So with that I want to close I want to thank our panelists so much for giving us their time. I want to thank the events and the communications team at OTI and New America for helping us put this together today and thank you to our attendees who gave us some of their afternoon to listen to this conversation. Thanks everyone.