 And welcome back folks. I'm Tim Briglin, the chair of the House Energy and Technology Committee. This is our second hearing. The morning of Wednesday, April 28. This is our 1030 segment and we're continuing our discussion of artificial intelligence and automated decision systems. We have three guests with us for this segment. Anthony Aguirre from the Future of Life Foundation, Jacob Appel from ORCA and Tom Adams from ORCA, and I'll let our guests introduce themselves and, and their background. We've got about 90 minutes from this segment, and there are a number of documents that have been posted to our website that Anthony, I think are under Anthony and Jacob's name. So for members of the committee who want to pull those documents up as we go through the testimony and obviously for members of the public who want to reference those as well. But thank you, Anthony, Jacob and Tom for joining us this morning we really appreciate your time. So I'm going to first turn it over to Anthony as I point to my screen as if I'm pointing to you but thanks for joining us thanks for being here. Yeah, good morning. Thank you for inviting me to testify testify today at the hearing. My name is Anthony Geary, and I'm a professor at the University of California at Santa Cruz and a co founder and co director of the Future of Life Institute, which is a nonprofit that brings together luminaries from the academic corporate and nonprofit worlds to research discuss and share insights regarding the impact of major society shaping new technologies such as artificial intelligence. In addition to my testimony today I'd like to enter into the specific into the record specific comments we prepared with regard to h 140 and h 250 263. So suddenly just starting just a few years ago machine learning and artificial intelligence systems are everywhere from driving directions to cars that will soon drive themselves, and including systems that recognize faces text, recognize spoken directions, organize news feeds, defeat masters and chess and go compose text and aid and scientific research. And in January of 2017, the Future of Life Institute called together a meeting in California at a cinema to bring together prime movers and AI and related fields to discuss an urgent question that this brought up. With the inclusion of capability and adoption in machine learning and artificial intelligence. How do we do. How do we make sure that AI is not just powerful, and not just profitable, but beneficial to its consumers users and society at large. And we set for this group a specific task to source formulate debate and hopefully adopt a set of principles that could guide technologists policymakers and others toward this goal. And this started by reading and synthesizing essentially all of the extant proposals for AI policy, something that could be done then, though not now and composing and debating specific possibilities in the weeks prior to the meeting, then discussing and refining those for several days in person, and, and talking them through, and this just succeeded I would say beyond our expectations, we were able to formulate 23 principles with excellent consensus and support across the spectrum. And these principles have drawn support from the heads of AI research at companies such as Google DeepMind, Facebook, open AI, Google Brain, Apple, and all more than 1000 AI researchers, as well as leaders in the tech industry like Elon Musk and Sam Altman. Those in academia from the late Stephen Hawking to Stuart Russell and Peter Norvig who literally wrote the book on AI, and many in the nonprofit and policy world. We're able to get this level of supporting consensus I think both because the issue is of great importance and also because many of these principles are almost self evidently desirable. We all want AI systems that are safe, that's principle six that are understandable. Those are principles seven and eight that respect our liberty and privacy and rights and support our shared prosperity, prosperity, mitigate risks and keep us humans in the driver's seat, and so on. But we all know that these desires are not self fulfilling. Safe technologies don't happen by themselves. Privacy is not automatically respected. Civic processes are not inherently immune to subversion, potentially dangerous arms races literal and figurative can start even if nobody wants them. Ensuring the benefit of technology takes active effort and participation from researchers developers policymakers such as yourselves and the public. Since 2017 such activity has rapidly expanded the asylum our principles were officially endorsed by the state of California and other sets of principles have been devised. A particular note are the OECD AI principles which have been officially adopted by the US. These are strong overlap heavily with the asylum our principles and future of life Institute endorses them. We do have some reservations which I'll return to momentarily. The legislation and other policy efforts addressing AI have been crafted a number of states, the US recently passed the very significant national AI initiative act, and the European Union is contemplating a major piece of legislation. There are lots of concerns regarding bias privacy judicial transparency and the like which must be addressed for AI systems to comport with our laws and ideals. And so among others clearly point to these. But in my view, there are two particular classes of danger indicated by the asylum our principles that often get shunted aside, because it's not in the interest of AI developers top much about them. The first concerns in a similar principle using a word that surprisingly uncommon nowadays, liberty and privacy application of AI to personal data must not unreasonably curtail people's real or perceived liberty. You can clearly see the trajectory toward a collection toward a collection of giant corporations that are awash and personal data with high powered AI systems designed to gather data, sell products and potentially manipulate individuals. If we end up in a system where each individual is responsible for protecting themselves against the power of these systems working to exploit them. It's going to be a very ugly picture. This power disparity will not resolve itself. And in my view can only be truly addressed by the power that the government wields in the service of the population. Closely related and to watch out for I think is the principle about non subversion, the power conferred by control of highly advanced AI systems should respect and improve, rather than subvert the social and civic processes on which the health of society depends. So that's one concern. Second, major concern, among many actually is that very highly capable AI systems, including but not limited to so called artificial general intelligence are coming. We really don't know when and provide great opportunity but also immense risks. As the asylum our principle state advanced AI could represent a profound change in the history of life on earth and should be planned and cared for with commensurate care and resources, that's principle 20. And as they called for high powered AI systems must be subject to planning and mitigation efforts commensurate with their expected impact. And in terms of the perspective, AI Commission to be established by age for 10 to anticipate that AI systems will become increasingly powerful and increasingly generalizable, and as such prevent present unique risks to society will happily support Vermont in this endeavor. And I look forward to any questions you might have for me today. Thank you Anthony. I just pulled up. One of your. One of the documents you posted to our website, and you refer to it generally. And then the beginning your testament, which is the, is it as a malar similar. Okay. Sorry, so, I need to say right, a silamar a silamar. Okay, thank you. You know, kind of a, one of the fundamental things that that that came out of this meeting of the minds, if you will. One of the questions that I asked in our, you know, earlier discussion today, the first hearing we had this morning was kind of in the realm of where the battle lines being drawn on some of these issues right now. You know, clearly there are, and I don't necessarily ascribe to the theory that big corporations are bad, but, but clearly there are battle lines being drawn where there are very large entities for profit entities that, you know, are, you know, today and in the future are going to benefit financially, significantly from, you know, the expansion of the use of, of AI systems. And, you know, one of the things that I'm really interested in, as we draw on the expertise of different folks that we bring into this committee is, you know, as we talk about things like implicit bias, what is the bias of the groups who are putting these thoughts together, whether it's future of life, whether it's Orca, whether it's the, you know, future of privacy forum, who's funding you. You know, the folks who are kind of behind the curtain in terms of supporting the work that is being done there. And, again, we all bring our own biases to the table. But, you know, I think that's an important fundamental thing that I'm trying to understand as to, you know, the folks who are sitting around the table who are come up coming up with, you know, these principles. You know, what is, what is driving the bias, whether it's, you know, kind of a passive implicit bias or whether it's something that they wear on their sleeve that says, I work for Facebook, or I work for, you know, clear view. I'm trying to understand what you know, who are the folks who have come together in your group, you know, to bring these principles forward. Yeah. So I can answer that certainly for the future of life Institute so the, the asylum are the process of the asylum our principles was essentially the, the team at the future of life Institute. So that's sort of the core team. So, so our, the future of life Institute website will exhibit the, the sort of core team of people who are actually operating on a day to day basis, and then the advisory board which is sort of consulted with occasionally So, so there are some AI experts on that board like Stuart Stuart Russell, Professor of I'm not sure if he's electrical engineering or I think I, well he's an artificial intelligence professor at Berkeley who wrote the main textbook with AI that's adopted in many classes. So, so he's, for example, an AI expert that is on the board but is not involved on an everyday basis. So, so the core team did the initial synthesis of sort of the galaxy of documents that was out there on AI policy which is rather small it's still rather small actually and sort of looked at commonalities between those documents in terms of with a specific focus of what do we do to keep AI, you know robust and beneficial. So the interest that the future of life Institute certainly brought to this was, how do we have AI work as well as possible for humankind and avoid the various pitfalls and dangers that this new technology can bring. So that's our, you know where we're a nonprofit with that, essentially as our, as our mission to to sort of safeguard humanity as a whole, and to try to increase the probability that these major technologies that are rolling out are just a broad benefit to society. So, so the, the process had that sort of calling, then a process of synthesizing lots and lots of candidate principles, sending them out to the attendees that were coming to that conference to that conference, the attendees to that that were also listed on the on the FLI website. But what we tried to do was bring together a sort of broad swath of people from the nonprofit world from the academic world and from the corporate world. So there are sort of leaders in academic AI there are the we tried to bring together the heads of many of the AI groups from the corporate world, and then lots from the nonprofit world of NGOs thinking about these issues of social impact. So I would say it was a sort of even split of those three groups, but the, the general sort of mandate was, what do we, what sort of principles should govern AI so as to make it beneficial to humanity and and robust against sort of dangerous side offense. So, there were sort of rounds of discussion and polling, and then that narrowed it down to a set of candidate principles that were pretty well defined that came to the meeting, further discussions at the meeting, and then voting at the meeting for sort of what level of consensus was on the various different principles. And so what we ended up putting into the asylum our principles were those that sort of were above some threshold of support from the meeting attendees. And I can't remember exactly what that threshold is but something like 60 or 70% of the attendees agreed with each individual one. And then after assembling that we essentially pass it around and said okay now are you willing to sign on to this group of principles as a whole. And so the end result is a list of people who felt comfortable, sort of signing on to that full list of principles. So, so that's sort of the process it went through everybody of course brings their own view and incentives and biases. Our goal certainly was to bring together, you know, the people who were going to have the most influence honestly on what happened in the future. And those are, you know, the high powered academics the people in charge of developing an industry, and the NGOs who were specifically thinking about the social impacts already. And the reason I asked the question is I've seen these principles referred to several places. I mean they're clearly having an impact, and just, you know, so thank you for that I was just interested in a little more background on, and how some of that came together. The other thing that I want to thank you for, and I have not had a chance to read through them, but it's really helpful for the committee, especially when we're considering a piece of legislation. We've provided some very specific feedback to both of these bills, and without asking you to kind of go through the granularity of that. I do want to give you the opportunity if there's anything, you know, at a high level you want to pull out there to, you know, to speak to the committee about, but really appreciate you providing that granular feedback that we can, you know, kind of go back and reference. Yeah, I think the what I, one of the things I mentioned was this general feeling that, you know, AI is advancing very rapidly, and more rapidly than even people in the tech world are able to react to it, let alone people outside the tech world. And I think there's a there's a danger in a lot of AI policy work that if you are thinking about the AI of today when you're formulating your policy, by the time your policy gets formulated and pass into law and goes into effect. You're already totally out of date, you know, in terms of what your, what you're worried about so it, it has to be forward thinking, and it has to be flexible and I think, you know, policy makers certainly understand this. And I think that's it's therefore important to really think about and consult with people who have a sense of, you know, not just what the technology is doing now but what's it going to be doing five years from now or even 10 years from now that's a hard thing to predict but I think there are certainly trends that you can see in the context toward more powerful systems more general systems we're certainly seeing that, you know, technically speaking that AI that used to be, you know, it used to be sort of funny because AI would do just one little thing, and then you try it on anything else and and it goes disastrously wrong and it's kind of amusing. It's not so funny anymore. Now that we have high powered AI systems where the same system is able to do a large variety of tasks, for example, the deep mind first built this AI system that beat the world champion ago and everyone was very impressed because that was, you know, significantly before anybody thought that that was going to happen, and it was kind of a classic problem of AI to play go. But then just a couple years later they beat their own system with a new system that taught itself go as well as taught itself chess and pretty much any other sort of closed full knowledge game like go and chess that has a specific set of rules that allow the AI to play itself so a single system can learn to play all these different games all by itself with no input or training data from the outside world. So that's just an example of the sort of growth of techniques that can do many, many different things when applied to different problems. And that brings a whole lot of issues that don't exist in more narrow algorithmic systems where the design for one specific task, and you can sort of monitor how does it do on this task. So that's something I think that that is worth watching out for and relatedly is is sort of liability for AI products. So if you have a product that does a particular thing, you might sort of check that it does the thing that it's supposed to do correctly, then you put it out to market and hopefully you've you've sort of anticipated the different use cases that people might put to it but you know it's the same product that's out there that's operating that you designed. And something that is concerning with AI systems is that they can go out into the world and then change, they can learn and improve themselves and that's a good thing they can you know respond to data that that they take on the fly and and start to increase their capability and sort of customization for a particular past. But that also means that the system that may be operating out in the world is not exactly the same one that you designed, you know, in development. How do you ensure that a system that may have done all the right things when it was sort of in the lab is going to perform well in the real world, both because the real world is different from the lab and because the system is different. So I think in terms of who is responsible when something goes wrong with an AI system, we need to both, you know, think about that question who is responsible, but also how is that was, you know, how is that responsibility going to be carried for a system that's evolving in the real world what kind of ongoing monitoring what kind of ongoing care and sort of oversight of a system once it goes out is there going to be. sort of the core of the two, the two points that we were making. Thank you. We've got a couple of hands up and I want to turn to other members of the committee. First representative Rogers and then representative be in touch. Thank you. Good morning. I have a couple of questions. My first one was, I'm curious in the asylum are did I say that correctly. In the asylum are meeting what kind of after or what was your experience of trying to bring in more diverse perspectives who might have, you know, different ways of relating to AI specifically racial diversity gender diversity people with disabilities. What was that process and and were you successful. What was the diversity on the group of people working. Yeah, I would say we, we tried hard and we're unsuccessful. So AI is a very, very unfortunately, unfortunately, like my home field of physics has historically been a very non diverse field. So we, we made a big effort to, to bring diverse audience, you know, a diverse set of attendees but I would say that nonetheless despite that effort. Given our desire to bring, you know, the heads of, you know, AI development for many firms that set of people was not a particularly diverse set of people. I mean, I'm just going to be honest. So, you know that that's not something we can, we can do anything about ourselves so we did make a large effort to to bring lots of voices but I would say that that's something that need that that needs more care as as these sets of principles are, you know, interpreted and put together. I would say that there was not in that room. Enough voices from, you know, non, you know, from around the world, there were like heads of corporations tend to be, you know, a large major AI corporations tend to be from the US and Europe, and, and that's where they're from. But the, you know, somewhat surprisingly I would say the principles turned out to be quite global in the sense of there certainly is a sentiment that high powered AI and the sort of wealth that it will hopefully bring is something that should be shared widely. I mean, I think there was a highly idealistic sort of sense running through the principles that you can see there. But that's not the same thing as having people say from from developing countries represented, you know, at the core of the meeting, and, and I think, honestly, they, they were not in the way that I would like them to be. So, yeah, that's my short answer tried. But I think need to do better, you know that that is something that tends to be done poorly in in many of these AI development circles, because of the poor representation of many groups in the, in sort of the technological group of people that are working on it. So, yeah, thank you for that question that that's just a great consideration to have. Thanks. My second question, I'm not sure if you're prepared to speak to this or not I saw that you had included the comments that it seems Jared Brown had predominantly put together on H410 and H263. I'm wondering if you're prepared or able to kind of summarize those for us, or if that would be something that it would be better to follow up. I think for the detailed questions of those it would be better to follow up I think with with Jared in the sense that he has, you know, I work with them on those but he has much more fine grained sort of legal, legal type of expertise and, and thinking about the nitty gritty of, of AI policy. He's sort of our point person for that. But I know that he's, you know, very interested in working with whoever would like to talk with them. Okay, thank you representative in touch. Thanks for the opportunity to hear you. And I got to apologize for jumping in and out of here at times but finally got got a chance to get back here. I think adopting principles and standards like the SLMR principles. Good, good idea and important, but I'm, I'm wondering whether they're really enforceable, and I'm thinking in terms of, you know, the Geneva and the use of chemical weapons. Governments come out may agree to these standards but then there's no real way to and enforce that if government decides to violate them. And in, in terms of AI. This is going on outside of government. Purview, I guess, in the private sector and the ability for governments to limit what is being done out there is really limited. So, what do you think about that. Yeah, I think these are sort of aspirational principles I would say that very much need to be translated into actual mechanisms for for governance and and those some of those might be purely within the private sector or, or, you know, policy or legislation. So I think there's a mix of things that it would take to actually bring these principles to life in the sense of governing how I actually operates but that will vary from from principle to principle so, for example, some of the some of the these sorts of, you know, we should spend more do more research into these sorts of issues so so that's an issue for, you know, funding agencies to take up or universities individually to take up. There are issues of you mentioned chemical weapons so so there's one regarding AI and weaponry, that's an issue for, you know, national governments to take up and NGOs and a little bit companies. So I think you're absolutely right. You know, everybody agreeing to these principles in and of itself is helpful. It's a positive sign to have, you know, at least social consensus around some ideals but but that doesn't stop people from pursuing the interests that they actually have when they conflict with those ideals. And that requires mechanisms to change the interests and change the incentives so that people do conform with their with their ideals and so that's what I see as the policy process. I think, though that policies need not be be perfect in order to be useful so for example with chemical weapons. It's true that that countries can cheat. And even though they're they've signed things saying that they're not going to use chemical weapons, sometimes do it anyway. But that, you know, sometimes cheating and doing it anyway is very very different than everybody doing it at massive scale all the time. And so I think similarly with all of these principles having a consensus or an astigma against doing the wrong thing is as good much better is to have policies that actually, you know, create the right incentives for people to do those things whether they're legal or financial. And even if the you know, even if there are exceptions and things go wrong here or there. But on the large scale, the principles much more adopted than not or, you know, these ideals much more followed than not. I think it's still a huge, a huge benefit. So I think just not letting the perfect be the enemy of the good is, is a saying that we have a lot that the future of life Institute because it's, you know, things are very far from perfect. Yep. Okay, well thank you. Thank you, Anthony, I don't see any other hands up in the queue so I want to turn to Jacob and to Tom and hear about Orca. Jacob and Tom, I only know enough about Orca to be dangerous and what I would read on your website but one of the bills that we're considering looks essentially at an inventory of systems that we have working in state government and to the extent that they are operating in a state that we should be concerned about in terms of how they, you know, use artificial intelligence or decision making systems that we want to better handle on as to what bias and things like that are incorporated there. And, but I'd love to hear more about Orca, the work that you do, and thank you both for joining us today. Thank you again and thanks to the whole committee for your time and for the opportunity to speak with you all. It's, it's a privilege so we appreciate it. My name is Jake appell I'm the chief strategist for Orca. I'm joined today by Tom Adams Tom do you want to introduce yourself. My name is Tom Adams I'm the general counsel and operations chief at Orca Orca by the way stands for O'Neill risk consulting and algorithmic auditing. The company was founded by Kathy O'Neill about four years ago, following the publication of her book weapons of math destruction. Kathy is a former math professor data scientist. Quant and now algorithmic auditor. Thanks Tom. And I'll sort of go through the remarks that we have prepared and certainly invite time to jump in anytime. And I thought it might be useful. First, I'll just say a little more what Orca does so we are algorithmic auditors and we have the very like fun and privileged job of getting to audit algorithms out in the world in practice. We're a consulting company so our clients are our clients hire us to audit their algorithms or the algorithms that they're using. And our clients include private companies as well as public agencies, municipalities. We work sometimes with attorneys general who are investigating the use of algorithms. So we work really on different sides of the issue. But the primary thing that we do is an algorithmic audit, which would be done where the client would be a company or an organization that's actually using or deploying an algorithm. And the audit means that we help that client work through whether the algorithm is operating fairly. Generally we're focused on questions that that relate to fairness. So that of course brings up a big question of what is fair and that's what we spent a lot of the time in in audits, determining. So that's a little background, who we are as a company and what kind of work we do, as far as the comments that that I can offer today. They're really like fall into two buckets. One bucket is, I thought I would say a little bit more about the principles behind our company Orca and the way that we that we do algorithmic auditing the way that we approach it. And then set of comments about the draft legislation, which we did have a chance to review briefly. And these are, I would think of as high level comments for the most part. You know just ways of ways of thinking about some of the, some of the things you're proposing. That sounds good. I'll just get into it. And I'll start off by talking through the three principles that we have at Orca, and these are on our website to. So you might have encountered these. The first principle is that context matters. And the simplest way to put this is that when we audit algorithms in context, we don't just audit an algorithm per se, we don't think of an algorithm as a few lines or a few hundred lines of code sitting in a Python terminal somewhere. We think instead of an algorithm is a process, you know maybe a decision process. It's being deployed by somebody it's being deployed on or with some, you know, subjects some end users. It's being deployed in a particular time and place, and it's being deployed like to make a certain kind of decision. The, the example that we often are an example we often give to just drive home the importance of context is think about an algorithm that produces a diabetes risk score. This kind of algorithm really does exist so it's it's an algorithm that predicts how likely any particular person is to develop diabetes in the next, you know, X years, let's say next five years. So you just think about a diabetes risk score algorithm. And that goes in the hands of your primary care doctor, and they used it to spot potential issue before it became a real issue and then they intervene early and they, you know, talk to you about diet or they start monitoring your A one C levels. That might be great and that might improve outcomes for patients and make people have better lives. You can use the exact same algorithm and put it in the hands of a health insurance company who might use it to in underwriting, like maybe they use it to price policies more highly for people who are likely to develop diabetes later completely hypothetical but you can see where the same algorithm in other hands could then be used to limit the amount of coverage people have or make it more expensive for them. And the point is that the lines of code that are that algorithm. Don't answer the question whether it's good or bad or what the ethical considerations are in using it the context is really critically important. So that that's why it's our first principle and that's why we don't audit algorithms, sort of in a vacuum we audit them in context. The second principle is we call putting the science back into data science. What we mean there is that data science, well we don't take things on faith. If, if somebody says this algorithm performs equally well for all genders, then we say show us the evidence that it performs equally well for all genders. We like to ask like specific and falsifiable kinds of questions and then and then submit them to tests. I feel like almost surprising that we would have a principle that we need to do science around data science, but the reason why we have it as a principle is because in practice, far too often, automated systems algorithms are just trusted because they are data like they, you know, a person can point to, we've looked at millions of historical records and trained an algorithm to predict X, you know, and we can show you a graph that it's predicting X with some level of accuracy, you know, whether or not that often suffices to put to rest questions about whether it's really right or accurate. And our thing is, or our principle is to not take those kinds of assurances that oh it's scientific, but rather to ask simple questions and try to see the data. And the third principle we have is that ethics cannot be automated. So these, the third one kind of ICS arising from the first two in a way. So in the field of algorithmic harm and algorithmic fairness there are some, there are some tools out there that would purport to like, make sure your model is fair at the push of a button. There are like open source tool sets tool kits available on GitHub that will do that there's like a fairness button. And we think that doesn't make a lot of sense, in particular because context is so important. We spend a lot of time in our audits, figuring out who the stakeholders are in a particular algorithmic context, and then we ask them what their concerns are. And once we have a sense of actual stakeholders concerns, then we can get to precise questions, precise fairness questions that relate to their concerns. So a large portion of the time and effort that we spend in an audit is getting to those precise questions and once you have them they're really not that complicated often they're like fairly straightforward statistical tests or like the quantitative analysis at that stage is not rocket science it's the hard work of getting to the right questions that takes a lot of time and an effort. And so for that reason yeah we have this principle because, frankly, potential clients would would love often to have a push button solution that they can check very quickly whether this model we're running is fair. We have to give them the bad news that that's not in general possible in our view. So, oh go ahead Tom. Well, flowing from that as Kathy has discussed is also the idea that artificial intelligence or algorithms are often used in these settings to avoid difficult conversations and Jake you're probably going to get to this later but. And so they are in a sense a tool for avoiding the automation, avoiding the issue of ethics and allowing the person who's applying the algorithm or whatever to to not have to answer the question of is this is this fair or is this biased. And that I think is part of the ethics cannot be automated concept as well. Thanks Tom. Since I mentioned our audits, I thought, and although I don't have a slide on the audit I thought I would take a minute just to explain very briefly like what an algorithmic audit is for us. In case of interest and certainly would invite questions on that if if you all want to hear more, but the basic question at the heart of the algorithmic audits that we do is how could this fail and what would that mean for you. And by this we mean an algorithm in a particular context. And by you, we mean the stakeholders and we take a very broad view to who the stakeholders are of an algorithm in context. To give an example, we would always include the organization that's deploying the algorithm there, for instance, like their legal department is a stakeholder, the data scientists who are building and fine tuning the algorithm or a stakeholder, maybe their communications department or stakeholder, but then also the customers are stakeholders. And also if there are. Oh, if there are various kinds or subgroups of customers who might have different concerns, you know, perhaps minority customers have a distinct set of concerns from other customers, then we think of those as separate stakeholder groups and we actually find representatives and we talk to them and ask them about their concerns. We record their concerns in a framework called the ethical matrix, which serves as a sort of living record of that discussion. And then, once we have a comprehensive, a reasonably comprehensive inventory of the stakeholder groups and their concerns, we can kind of talk of the whole picture, and start to do the actual work of ethics which is to identify where there are tensions between different stakeholders and competing concerns to start to weigh and and think through the tradeoffs in addressing those concerns. And as you could imagine, it's, it's just highly bespoke work. It, it has to take stock of the particular stakeholders in this algorithm in this context, you have to think about those tradeoffs in a fairly local sense. There's a bit of contrast I think to the like work around principles that Mr. Aguirre referred to earlier. We're often very deep in the weeds on like what does it mean for this stakeholder to have their concern addressed with regard to this algorithm. So, I'll just in light of time I'll stop there in terms of talking about our audits generally although as I said, I'm happy to answer questions about that afterwards if you have any. And with that, I'll move on to the few comments that we had around the draft legislation. I guess actually before I move on Tom do you have anything else to add on the first bucket. You said it but I'm just going to repeat it are guiding thought in doing the ethics, doing the audits and doing the ethical matrix is for whom does this algorithm fail and how. Thanks. I guess I want to chime in with a question there. And I'm going to. I'm going to demonstrate my, my, how much I've forgotten from my high school statistics course. But, you know, to the extent I'm looking at the, I think it's the third slide in your presentation. And, you know, about unfairness to discriminate against an individual on the basis of, and in our audits we focus mostly on statistical discrimination. And that seems to be a very kind of black and white, kind of linear way to, you know, from an audit standpoint to get at is something fair or unfair to different groups that are affected by policy or a program. And I'm thinking in state government right now since that's really the realm we're operating in. But basically taking a statistical look at does a program discriminate or treat unfairly a particular group. And there doesn't seem to be a lot of gray in that analysis and maybe there is. It's, you know, it seems to look at a statistical framework. And maybe if you're so far off an average or a median and I don't know how wide a dispersion there is there, you know, around, you know, the data that you're looking at but can you put a little more, a little more meat on the bone there. You know, as well as to what the audit says statistically as to what's fair and what's not fair. Sure, I can say a little more what we mean and in fact like that. Maybe that'll bring us into the comments on the legislation that was intended as a comment on the legislation. Yeah, I'll, I'll talk about that for a minute when we so I actually I just want to be clear on that, because the word fair fairness is used in our legislation and what you're saying is that the points on this third slide are essentially referencing the legislation. I wasn't sure if they're talking about how Orca conducts its business. No, no, yeah, we've moved into referencing the legislation. I beg your pardon. Okay, thank you. That's that's a helpful clarification. I should have made them clear in the slides. So but yeah it's it's a it's worth talking about so I'll just start on that bullet. So I did notice and like, I'll apologize in advance for not tagging these comments to a specific line in the legislation, but I, if they if they're not landing just tell me and, you know, and then we can figure it out. But I hope they make sense. So, there was a piece in the legislation that established something, you know, is unfair if it discriminates against an individual on the basis of and then there were a list of like protected classes I forget what they were, but they were race and things like this. And the point I wanted to make was that we have. We've sometimes come up against this very fine, or this, you know, this distinction between discriminating against an individual on the basis of some characteristic that individual has. And then a different thing is like documenting whether on average or looking across a wide group, there are differences between, let's say, you know, genders or between race or ethnic groups on average, and the nature of analysis that we are often on these algorithmic systems is like, you look at everybody all 100,000 or all million people who submitted an application, and then you try to see whether did, on average, women do less well than men. That question can be quite easy to answer, looking at a large data set. The other question about how an individual was treated can be very hard to answer because the models, the models that produce a prediction for any individual have many, many variables, and it's hard to point to the one variable that caused that individual's prediction. You see what I mean. So we've just often had more luck documenting unfairness or discrimination at the level of differences between groups, rather than trying to pull apart the many threads of what produced, you know, this prediction or this risk score for this individual. Understood. I got it. The other, the other thought about definitions was, I would, I would suggest to define algorithm or algorithmic process or algorithmic system, somewhat broadly and this gets to the principle around the importance of context. The algorithm, in our view, doesn't need to be written as computer code. It might be a combination of, it might be a combination of some automated pieces and some, you know, human levels of review or oversight to give a concrete example here. We were hired by the Washington State Department of Licensing to audit their use of facial recognition in the license issuance process. So, like many DMVs, they, they're concerned about fraud, they're concerned that the same person comes in and applies for a license under a different name. They use facial recognition like many DMVs around the country. We learned they use facial recognition to search, you know, they take this picture they want to search that the same face isn't in the their existing database under a different name. So, you could think very narrowly that the facial, the use of facial recognition there is a product it's actually a third party product that they license or purchase, you know the state purchases from a vendor, a for profit company. When they first called us to work on that audit. It was the initial focus was let's think about that little piece of facial recognition software. But when we thought about the context we recognized what we're actually auditing is the use of this facial recognition software in a system where that includes layers of oversight and includes other steps that the, that the customer at the DMV goes through when they step up to the counter and their picture is taken. In the end, the audit report was able to conclude that even though the software, the facial recognition software that they're licensing was demonstrably imperfect, like all algorithmic software is there. This was demonstrably imperfect because NIST, the National Institute for Standards and Technology did a wonderful benchmarking study of facial recognition softwares that were on sale by various vendors, and they documented carefully how they work differently for people with different skin tones for different genders and so on. So we had actual documentation that the algorithm wasn't perfect. However, we were able to keep in terms of the audit that their use of the algorithm was acceptable, because they had for two reasons one, they had sufficient layers of oversight that were able to catch errors made by the algorithm and correct them before they inconvenienced or harmed actual customers at the DMV. And second, because we were able to look at historical data and see, we were able to find a very upper bound for how many customers have actually been harmed or inconvenienced, you know, say in the last year, due to this kind of error, and we were able to find a very small upper bound for that, based on their actual historical data. So my point is, if my point is that those other features of context around the algorithm that they were licensing were really important to the audit and to answering the question of whether they were using facial recognition. Fairly, or responsibly in that case. Right, the difficult question was not being skipped or bypassed by the use of facial recognition. It was actually being addressed by the human oversight. Is that that parent Jake. Yeah, I think that's exactly right. So next, we had a couple of comments. The draft legislation. A bit of focus on procurement. So it seems like it seems like the legislation establishes a director who then gets to like a position of director who gets to then have some like supervisory role in the procurement of AI systems. If I read it correctly, then I would just say that seems like a great idea and a great way to exert some over the, you know, the quality of algorithms and AI systems that are in use by the state. So it just would encourage and support that approach. So the suggestion that we would suggest considering which is to make sure that any algorithm that is licensed or procured by the state should be open to an audit. There shouldn't be any language in the contract that like guards or shields or protects the algorithm from outside scrutiny. Again, like almost seems like why should that have to be mentioned, it has to be mentioned because the companies put that kind of language in these contracts routinely in our experience. So, and, you know, companies are allowed to have concerns over their IP they're not. And what what you're asking for is not to like disclose the secret sauce that makes their algorithm work what you would be asking for is like that they not prohibit reasonable how it's performing and those questions can usually be answered without looking at the inner workings of the algorithm itself you can often answer those questions by saying, like, let me send some sample inputs to your scoring system and you send me the outputs. And then, you know, if you make a set of sample inputs, you can design a sort of test, you know, you can test the black box without opening it by putting a certain set of inputs and observing the outputs. The point is in those procurement contracts. Don't let the companies weasel out offering at least at least. Like, in the case of Washington that was a third party software tool for facial recognition which we did not, which their contract said could not be internally could not be audited by Washington state, right could not be audit isn't the right word but couldn't couldn't be the data could not be scrutinized nor the IP. And so how did they get around that. Or haven't they. Again there's a real power dynamic in terms of, you know, if you have, you know, if you've got one option for a particular type of software that you'd like to deploy, but that that vendor simply shows the, you know, the state or the agency the hand says sorry, not going to happen. What is that power dynamic play out what happened with with Washington state, were they able to audit, were you able to audit on their behalf. We were able to have a dislike the the vendor was willing to speak to us on the phone as auditors, we didn't get into the line algorithm but as I've said like the, the other analysis that we did, which relied more on the State administrative data, the analysis that we did that showed, if anybody was harmed by this in the last year it was a small number and it was a minor harm. That analysis relied on the departments data, not the algorithm data. We were able to do the audit, because it didn't require opening the black box of the facial recognition code. It only required looking at how how actual customers fared when they were subjected to a process that included that facial recognition. Does that make sense. It does. And I think this is going to be actually a challenge with this legislation is the power dynamic between the customer and the vendor. But I mean, that's that's something that will encounter down the road, but I'm presumably you all are seeing this every day. It's a real issue and like the hope, you know, Vermont as a state, you know, hopefully you're a large sought after customer for these vendors and so have some leverage in that sense. It is like it's a it's a major gap. In our view, many of the kind of licensing agreements between these third party vendors and the companies or organizations that go and deploy their tools at enormous scale are like quite thin and and the acquiring entities really don't have the amount of visibility that they would like to or that they should have and the vendors are really like not answering obvious questions that they really should answer. I wish I had a better answer for you. Yeah. I'm sorry, go ahead. These are big issues that it's important that, you know, you are transparent in the use of them. So while they may say we won't share it to actually outsource decision making on crucial issues of hiring practices or family child services or or bail or sentencing without knowing those details is also, you know, high stakes issue. Representative. Yeah, so how much transparency and insight can we really expect because these algorithms are probably going to depend on very proprietary software and algorithms and companies may feel defensive about providing the details to to an outside entity. I think it is fair to expect some resistance or like, well, as I said, I think it is a challenge. I would say there may be like two potential bright spots. One is that not all but some vendors of AI products like third party vendors are themselves aware that there that concerns exist around fairness and bias and discrimination and so on. So some of them actually make claims on their own websites. That's like, our, you know, our tool is unbiased. The most basic thing is just to say, show us the evidence behind that claim. If they have a claim, you know, you can ask what it's based on. So that's, that's one thing. As I said, not every company will claim that its algorithm is unbiased but those that do you can ask. The second thing is there can be an objection that we'd love to answer your fairness question but we just can't open up our IP, you know, it's too sensitive. You can often find a way to ask a meaningful fairness question that does not force them to divulge IP. Maybe there's a different way of phrasing it or a slightly different way of getting at the question you care about, but like you can. I'll say, almost always, you can often ask, find a way to ask the question that doesn't require them to show you the lines of code, but rather just says, like, I'll send you a set of dummy applications and you send me the results right so I'm just saying there's, you can work on phrasing the question differently sometimes. You know, if, if you say that this is going to be fair in its application of whatever the model is, you should be able to show it, as Jake said, you should be able to demonstrate that that is the case that women will not be rejected at a higher rate than men if we use this tool or children of color will not be separated more readily from their families in child services than other children. If that's the case, if your model does that, you should be able to show it. So you're talking about looking at results as opposed to asking questions about how the algorithm operates. Okay, thanks. Another area of comments or thoughts on the draft legislation. We thought lessons from prior government efforts around the inventory of algorithms so we read about that in the, in the legislation. A similar effort happened in our backyard in New York City. We were not directly involved in it but, you know, since we were nearby, we had some observations that we, we thought we would share. The first is a suggestion to focus on high stakes algorithms rather than all algorithms, it might be distracting or even counterproductive to ask for all. So, let me just explain that. I think in New York City the intention was this was to create a comprehensive inventory of all algorithms or maybe they said automated decision systems whatever it was it was like, tell us about all the ones that are in use by all the agencies. I can understand the desire to be comprehensive absolutely, but in the end, I don't think that the inventory was complete. And I think the asking for all created an opportunity for two ways to like to two little loopholes or something to get out of the spirit of the requirement. That is just, well, we only have limited amount of time. This is too big of a request we just, we can't comply. The second loophole was, Oh, alright, I'll, I'll inventory, you know, five algorithms for you but I'm going to choose the real low hanging fruit in the sense of the ones that are not controversial, they're straightforward they're not high stakes. So, coming from the place we do in our audits which is like what are the, what are the biggest concerns to address. We would recommend, like, trying to focus somehow on high stakes algorithms. And I'll say a little bit more on the next slide about what what I mean by high stakes but anyway that's one point of feedback. I just want to interrupt representative Rogers hand went up and then went down so Lucy I want to give you a chance to answer or ask your question if you've got one in the meantime representative chase your hand is up so go ahead. Thank you. You mentioned, not all just focus on the, the vital ones. One of my first concerns that comes to mind is, even if there's a low hanging system, and it creates an extra two pages of paperwork that's required for one particular group. You know, women have to fill out extra paperwork or there's an extra delay for, you know, non white applicants or something like that. That's still an impact to society and still a problem that needs to be addressed. So, how would you suggest we capture those while not ignoring the the the ones that are causing people to be imprisoned at higher rates. I hope you can do both. And this all I'm saying is that if you had to choose one, and you could get one or the other, I would go for the prisons one first. That's all I'm saying, but I'm not trying to be glib like adding, you know, adding a delay adding extra paperwork for particular is, is a real issue and a real impact so I'm not at all trying to minimize the ones that are less high stakes than ones that are deciding whether people are imprisoned or not like they, they can still have somewhat high stakes. I'm just suggesting prioritization based on the kind of level and extent of harm that could be caused. Okay, perhaps I misunderstood your first statement sounded like you were saying just focus on those and don't worry about the other ones, but I appreciate your clarification. Thank you. Representative in touch. Yeah, I was just, I was just wondering whether there's a particular section of the bill that you're referring to when you, when you say, you know, focus on the important ones and don't worry about the low hanging fruit. I'm not sure how that applies to the bill we're talking about. It's entirely possible I could have misread the the bill or what was laid out as the inventory of algorithms or automated decision systems so if I did misread it I apologize. I'm not saying you did I just trying to relate what you were saying to build so maybe somebody else can help on that. I mean how I characterize this as when you've got thousands of software systems to go through focusing on ones that affect people's liberty, people's finances, people's livelihood, maybe there are other aspects but my sense is that these are things that are incorporated into the European Union framework definition of what high stakes algorithms are that that might be a way to focus the inventory that's actually done by ADS. I'm not saying and I'm not not even sure if Jacob and Tom are saying this, not saying that we should go there but this might be a way to provide more value from the inventory that actually comes out. Okay, I get it. Thanks. Thank you to answer the questions for me from now on that's exactly. Yes. And in fact, like your answer covered the last, the last couple of bullets that we had so that was high stakes, I think, as you said, Liberty finances, livelihoods are. If one of those categories is impacted or potentially impacted by a certain algorithm, then that sounds like high stakes to us. Mike I think your hand is still up from before but unless you got another question. So great. She might have a question yet. I'm sorry. I want to thank Anthony and Jacob and Tom for for your presentation on this as I said earlier today for our first group of witnesses this morning. We are kind of building a foundation. Essentially we're building a foundation of knowledge around this stuff as we move into more of a posture in terms of how we might might move forward. And so this is really helpful. And, you know, I think at a high level, Anthony, your organization's work in bringing together a number of stakeholders and thinking about, you know, some of the ethical issues around the work that you're doing today is helpful, particularly as we look at age four 10 and what we might ask of a, you know, a particular commission to do. And, you know, from my perspective in terms of the work that work does every way, every day in terms of auditing some of the stuff that has, you know, it's, it's a direct overlay, I think, and in terms of some of the work that we may be asking our agency of digital services to be in terms of the inventory of what types of things we have that are affecting how state government works right now. So that that's very helpful in terms of jogging our thoughts in terms of how we move forward with this legislation so. So thank you for your time, really appreciate it and also appreciate the documents that you've given to us that are posted on our website. It's helpful in terms of our record and in terms of my really shoddy note taking. So appreciate that as well. For members, we are adjourning for this morning, we're on the floor this afternoon at one 15 I believe. Matthew, if you could remind I actually I'll pull it up now I believe we have testimony that is scheduled for tomorrow or is trying to be scheduled that it's not on our. I'm waiting to hear back it will. So it has not been paid and that has not been updated to reflect it until I have confirmation that someone will be there. Okay, so for members, just for situational awareness, we may have a late morning hearing tomorrow morning, 1030 11 o'clock. And in spite of my best efforts, we are scheduled for testimony on Friday, late Friday morning and early Friday afternoon as well. So if you haven't looked at the agenda, lately, take a look and update your calendars. So, we are adjourned for this morning. Thank you again to our witnesses and season. Thank you all. 1030 10, not nine. We don't know yet. We don't know yet, but it'll be 1030 or 11 tomorrow morning.