 Hi, everyone. Thank you so much for joining us today. My name is Lauren Sarkeesian and I'm a senior policy council here at OTI, focusing on government surveillance and privacy issues as well as AI. For the past few years, OTI, led by my colleague, Sandy Singh, has been looking into how internet platforms develop and deploy AI systems and machine learning-based tools to curate the content we see online. OTI has also done work related to government uses of artificial intelligence, including the facial recognition and procurement contexts. We're very excited to have released a new report last week, which explores and compares the different mechanisms that internet platforms and governments can use to promote fairness, accountability, and transparency around high-risk AI systems. So, we tend to talk about many of these accountability mechanisms in a silo. Today's panel will focus on exploring some of these approaches and if and how they can work together to generate greater accountability around high-risk AI systems. We'll have some time at the end of the panel for questions, so audience members, please submit questions via Slido. So now, I'd like to briefly introduce our panelists. First, Professor Catherine Sharkey, who is the Segal Family Professor of Regulatory Law and Policy at NYU School of Law. Professor Sharkey is one of the nation's leading authorities on the economic loss rule, punitive damages, and federal preemption. Last year, she co-authored the Government by Algorithm Report on how federal administrative agencies are and should be using artificial intelligence for the Administrative Conference of the United States, or ACIS for short. Next, we have Dr. Christine Custis, who is the head of about ML, which is a program at the partnership on AI. Her work focuses on bringing together a diverse range of perspectives to develop, test, and implement machine learning system documentation practices at scale. Christine also leads Fairness, Transparency, and Accountability work at PAI. Finally, we have Spandi Singh, who is a policy analyst at OTI. Spandi's work focuses on content moderation, disinformation, algorithmic accountability, government surveillance, and privacy. In OTI's recent report, we covered a broad range of mechanisms that can be used to promote fairness, accountability, and transparency around high-risk algorithmic systems. This includes machine learning documentation frameworks, transparency reports, government procurement processes, audits, impact assessments, and more. I want to start off by honing in on some of these approaches and exploring what advantages they may have to offer, as well as what limitations they may have when we think about promoting fairness, accountability, and transparency around AI. Christine, we're going to start with you. About ML is a multi-stakeholder initiative that is trying to push for the implementation of machine learning documentation practices at scale. Could you explain a bit about what the about ML program aims to do and why you think this approach could be useful when we're talking about promoting transparency in high-risk AI systems? Absolutely. Thank you so much for the question. About ML, it's actually an acronym and I'm going to read it because it's so long and I want to get it right. It's annotation and benchmarking on understanding and transparency of machine learning life cycles. In short, this is just work that we believe is imperative because it is how we think we can operationalize transparency at scale through documentation. It's the way we believe we can action responsible AI through the documentation of the different phases of the machine learning life cycle. And so what we hope to do is set a new industry norm of documenting all ML systems as they're being built, as they're being deployed, changing this entire practice at scale. And a lot of times when you hear about the software development life cycle, there is documentation inherent in the life cycle that's well received, well known. Somehow in the machine learning life cycle, it's not so much. So we're trying to bring that to bear in that life cycle through research-based, multi-pronged initiatives soliciting feedback from stakeholders and baking ethics right into the process. Thank you. Thanks for that explanation. Catherine, the government by algorithm report had some really interesting insights around government procurement of AI tools, noting that just under half of the algorithms government agencies use right now are procured externally. Can you talk about the instances in which government procurement of external algorithmic systems could be beneficial to promoting fairness, accountability, and transparency in these systems? And also discuss some of the concerns that they raise or that sort of arise by relying on just procurement processes. Sure. And thank you. Thanks to Spandi for the invitation to join this panel. So by way of very brief background, I was part of a very large collaborative study that engaged NYU and Stanford law students and computer scientists. We brought together the lawyers and the technologists. I had colleagues at Stanford on the project. It was commissioned by ACIS, the administrative conference of the United States. That's an independent agency that tries to be a kind of fertile ground for the production of useful empirical information that agencies might be able to look to. So in a nutshell, we were struck by the fact that there were numerous commentators and scholars that were very engaged in the question of how AI should be regulated, particularly less attention actually on governmental use, much attention on how government might try to regulate in the private sector, and yet very little information about actual, whether fledgling or more well-developed uses within government. So we tried several different takes in this report. We did a canvas at the largest measured by number of employees, federal agencies, 142. We researched and canvassed. And of those, one of the interesting findings was nearly half had what we would call existing uses of machine learning slash artificial intelligence. So sort of popular opinions sometimes think that nothing's happening within government or government's way behind the private sector. So that was one finding. And then Lauren, as you alluded to, we tried to look, not only are they using these tools, we tried to have them classified by level of sophistication, and also by whether they were coming primarily in house or through outside procurement. And our findings were actually that 53% were coming from in house. So we can come back to that. But that's interesting in and of itself, because again, the procurement is an important piece and angle. But I think often missed is kind of what's happening with respect to internal capacity development within federal agencies themselves, maybe thinking about why this is happening, maybe thinking about some of the things that Christine alluded to in terms of getting some kind of accountability frameworks kind of ensconced within federal agencies. But there were 33% were being contracted for private from the private sector. So you're right, there's another 14% that makes up that external that are non commercial collaborations. And that's important too, because government is collaborating with nonprofits and with academic entities, but this roughly one third are being contracted from the private sector. And so again, you raise a very broad question, I'll try to be somewhat succinct, there are advantages and disadvantages, maybe I'll start with some of the disadvantages. Sometimes we have to be careful about thinking about off the shelf, private sector tools being imported in terms of governmental use, right? What I would focus on is the need to get the lawyer policy and technologists in the room together while developing these tools, oftentimes within government, they're very nuanced, complex tasks that are very policy dependent, and it's a little too late kind of expose to ask, do these tools meet our objectives for accountability and the like. So that's one piece that I would emphasize that sometimes there are going to be costs with respect to ensuring accountability and ensuring that these tools sufficiently meet all the nuanced policy goals within the agency, there can be costs of monitoring such tools and the like as well. On the plus side, a kind of advantage, and this was one of the features that I particularly like, there's much to commend this cracking open the black dots report, but one thing that I liked was a kind of theme or emphasis on how government potentially could use this procurement process to force some accountability. And so I do think that there are very interesting ways that through the government procurement process, the government can demand certain types of accountability standards. I also think there's a lot of discussion about how it will never be possible because of all sorts of barriers in the private sector with respect to trade secrets and patents and protections. And my response to that would be, well, with respect to dealing with the government, maybe as a lever, the government's going to be able to demand a certain type of access to be able to ensure the kind of accountability that we need for these types of tools. So I do think it's very worthwhile to think about the ways in which this process can be a kind of carrot, maybe sometimes stick, but a carrot for enforcing some of these accountability mechanisms. Yeah, thanks for that. And for going back a little and providing the background about the report, which was really just so crucial and provided the most insight by far into how the government is using AI. So anyway, yes, I jumped ahead a little bit with that question, but appreciate the background. Sandy, next over to you in OTI's recent report, you wrote about how mechanisms such as algorithmic audits, risk assessments, and impact assessments can be used to rein in high risk algorithmic systems. Can you talk about what value you think these kinds of assessments have in promoting FAT, for short, we'll say, around high risk AI systems and if you see any limitations? Thanks, Lauren, and thanks to all of our panelists for joining us today. So I think if conducted properly and transparently, audits impact assessments and other methods of evaluations can be valuable mechanisms for promoting FAT around high risk AI, both in the corporate and government sector. Some of the benefits of these approaches include they can shut light on the opaque nature of high risk AI systems, they can help evaluate specific variables that we are often concerned about like privacy, bias, fairness, and human rights. They can examine unintended harms of the system and help an entity create a roadmap for mitigating these issues. They can enable investigation of certain concerns in their ways by external stakeholders, such as impacted communities or civil rights organizations. And they can also help an entity determine whether their system is in line with their own internal policies or and or external standards or regulations. But I think in order for these assessments to deliver these benefits, we need multi stakeholder consensus on definitions, we need standards for these evaluations and so on. So I think it's been really interesting and great to see over the last few years how stakeholders from civil society, government and platforms have been debating and encouraging the use of these methods. But I think the key limitation I see in this space is that we haven't answered the fundamental who, what, when, where, why, and how questions, which are really important if these evaluations are going to be effectively deployed. So in terms of who, you know, who is conducting these evaluations for maximum accountability? Is it a third party entity? Is it the companies or government agencies themselves? Is it a combination of both? There's a lot of really great literature, we talk about this in the report as well, that if an entity is grading their own homework, really generating the kind of a the second one, what, what are we evaluating? Are we talking about auditing for privacy or doing an impact assessment on risks to pre expression or bias? You know, not all high risk AI systems have the same objectives or designs. So we might even need multiple different frameworks, depending on the context of which a system is being deployed. The when question is when, during the AI life cycle, should these evaluations take place? As we talk about in the report, many stakeholders have suggested that algorithmic impact assessment should occur before a system is deployed to identify potential harms. Others have suggested that audits should be taking place pre and post deployment, and they should continuously be taking place to ensure that systems are still functioning as intended. The where question is where are the results of the audits communicated and where is the oversight coming from? And then the why is what goal are you trying to achieve with these assessments? So I think we need clear answers to these questions in order for these assessment methods to be valuable in promoting FAT. Right now the use of these evaluations are all voluntary. So, you know, vague and unclear guidance is generally a disadvantage for an entity to participate in them. And I think a lack of standards and clarity also undermines credibility. There have been a number of freely interesting audits, for example, that have been conducted by journalists and researchers, but since there is no consensus around the methodology and the correct approach, the applicability of these findings is generally limited. Thank you. And switching gears a little bit, since there are so many different sort of accountability mechanisms to cover here, Christine, we're going to turn back to you and talk about sort of documentation frameworks. So there's there's numerous different existing frameworks for data and systems documentation in the machine learning context. These can these include Microsoft's data sheets for for data sets framework, IBM's fact sheets framework and the model cards framework that Google initially championed. These approaches share some similarities and differences. But can you talk about if and how these frameworks could be applied to a high risk AI system and sort of how impactful they can be? Yes, absolutely. So I'll just do a little bit of pitch here and I just want to encourage all of the folks that are participating today to take a look at the partnership on AI's website. It's partnershiponai.org. We have a reference document. We have a resource library and lots of great tools out there. And we'd love to hear from you if you ever use them and your experience. But I will say that throughout this machine learning life cycle, there are so many opportunities for record keeping for documentation throughout data specification and curation data integration one source with another the maintenance of that data keeping it fresh throughout the model specification and training and evaluation and integration and maintenance is all these opportunities to keep track of what is happening. Who did it what the outcomes are the impacts are to the system to the users to the impacted non users. Right. And so there's opportunity throughout the life cycle to do this type of record keeping and documentation which is beneficial to a high risk AI system because you know if we don't get it right we at least want to be able to go back and trace the error we want to be able to figure out the source of the harm. So giving ourselves this opportunity throughout the life cycle keeping track of just all sorts of administration minutia so that we can go back and check that and I will say you know you mentioned all the different types of tools and methodologies that are out there. Right now we're doing a meta analysis of tools and we've found 400 so far there are so many different ways to document an ML system you know and then ways that we may not even discover. So the way you choose is not necessarily as important as your consistency that cultural shift into record keeping and being transparent through your documentation. And then I'll also say that every way is not right for every organization for instance a small startup dealing in patient care in the UK may not want to use model cards it's very bulky you won't have one card you'll have multiple it's bulky you have GDPR to to consider you have privacy and security of personally identifiable information for patients. So it's a really great opportunity to sort of get out there and see what's available figure out the specific use case for the for the problem you're trying to solve and just find ways to sort of cafeteria style mix and match throughout the entire life cycle of your machine learning system different ways different approaches to documentation I hope I answered your question. Yeah that's helpful um thanks for that uh Spandi back over to you a number of legislative proposals in both the EU and the US have included provisions that would require internet platforms and government agencies to conduct audits or assessments of their algorithmic systems. What are the key elements that such legislation needs in order to appropriately address the risks and harms associated with high risk AI systems? Thanks Lauren so as you mentioned we've seen legislators everywhere from the EU to the US to Canada and beyond put forth proposals that include audits and tax assessments and so on so it's good to see that policymakers are thinking through these important issues but I think a lot of these proposals generally lack teeth for three reasons. First there's a lack of clearing consensus based definitions on terms like automated decision systems high risk AI amplification and so on and I think you know in a civil society context perhaps we may have a general understanding of what these terms mean but when you're trying to implement it into the law it needs to be a little more granular and this lack of clarity makes it practically confusing and difficult for companies or agencies to implement these evaluations. It also makes it difficult for external stakeholders to research and understand the scope of these methods. The second reason is a lot of these proposals lack clear guidance on how these evaluations should be structured and implemented. Some policymakers have proposed drawing on existing guidance around assessments from sectors like the financial sector where auditing is a very common practice. Auditing for discrimination in housing is also common and sectors like the environmental and aerospace industries also have pretty robust evaluation practices. I think this is a good first step but I think in order for us to really talk about the auditing of high risk AI or assessments of high risk AI we need to contextualize any sort of best practices we get from other industries and really get more granular and then the third reason is that these proposals don't really address the intermediate needs such as the need for funding to establish an external third party auditing or assessment landscape that is you know well resourced and legitimate and without this I think you know we can see a lot of different options play out but I think one of the most concerning is that we could see communities for most affected by high risk AI systems have to take on this external review function and they may not have the necessary resources to do this and they also won't be compensated for this work and I think in that way this can kind of further exacerbate existing inequities that these technologies can create. In terms of what good legislation should have one I think is really critical is good legislation should propose clear guidance around transparency around results so at the very minimum I think attitude subject to evaluation should be required to publish a summary of their findings and how they've addressed the issues in cases where audits are being conducted externally or assessments are being conducted the evaluators may need access to sensitive data and I think legislation can help establish these data sharing structures right now there are many conversations happening about for example researchers accessing internet platform data and I think that these these sort of conversations around how structures can be set up to assuage platform concerns around trade secrets and privacy can inform this broader AI evaluation conversation and then lastly I think legislation also needs to address the broader ecosystem that enables high risk AI systems to be so potentially harmful so for example in the US we need comprehensive privacy legislation to rein in data collection practices a lot of this data goes to training new models and I think we really need to institute clear and strong limits and to encourage platforms to promote user controls as well so that we're not just focusing narrowly but also in the larger ecosystem of things that could impact thanks bandy okay Catherine over to you next and we've all been talking a lot in various contexts about transparency in the acus report you talked about how transparency mechanisms need to generate actionable and interpretable transparency in order to promote accountability the report also mentions that administrative law works in tandem with an array of data and disclosure laws that in their current form can sharply limit transparency uh do you think that these kinds of laws need to be reformed and if so how great so um I want to go back if you don't mind just for a minute to contextualize again the question and go back actually spandy to some of your your second to last remarks about the various tools the audits the impact assessments and you came back to it here with respect to legislative um uh proposals because I'm very struck um I'll give a very quick sort of analogy um a prior acus field study I did went into federal agencies and looked at federal agencies that had gotten very um aggressive about preempting or ousting state tort law so they were coming up with federal standards and as part of that standard they would oust state tort law now it turned out in administrative law that agencies were supposed to be filling out what are called federalism impact statements so you alluded to like the environmental impact statements and some audits in the financial sector but I actually think that federalism impact statements has like a very close analogy here because it's a very policy inflected area we get into disputes over what is federalism what would a federalism impact be how are states affected by what the federal agencies are doing and I'll never forget the personal footnote I'll say as I went into a very high level uh rule writing person and policy leader within a federal agency and I started talking about these federalism impact statements and how could it be that a federal agency for example had just written a rule to entirely oust state law and yet said there were no federalism impacts right it was written into the rule and he turned around the way and showed me binders and he said do you know what those binders are each one was like this thick he said those are all of the required impact assessments that we're supposed to do every time we're doing rulemaking so you want to add a new binder there's an existing binder for federalism and it's like this size and you want to put in all this academic intellectual content about what standards there should be who should review it where's the oversight etc and so one might be at first at least within the governmental context a little bit humbled by how are we going to do this with AI how are we going to have not just federalism accountability which I've given a lot of thought to but now let's switch it and say how are we going to have AI accountability and I guess the first note that I would say is that notwithstanding how important I think federalism is and that I argued then and still believe it should have its right place in review of agency regulations we all I think could agree that sort of algorithmic governance is a sea change right so this is something really significant really important it should challenge us it should challenge those of us working in the real world those of us in academia like myself or in sconston thinking about administrative law and accountability frameworks this isn't just some like side area this is a sea change I believe federal agencies let's just take the FDA which regulates medical devices and drugs they are having a sea change right with respect to how they have to regulate the use of AI within medical devices and drugs what's happening internally how their rulemaking will be affected looking way down the future what will happen when we have virtual clinical trials instead of actual ones when we think about federal agencies they are increasingly becoming data aggregators themselves so spondies exactly right that we have to think about sort of a lot of our typical frameworks which I think can be imported into the space we have to think about it in a kind of transformative way so having said that right I would go back to some basics I do think that there are some administrative law frameworks so again to take the work that I've done in federalism accountability there are frameworks that we can set up that force agencies for example to conduct these impacts assessments with respect to AI and then doesn't just leave it there there are mechanisms of control right if we think about the office of information and regulatory affairs OIRA which does centralized review of agency rules and regulations they typically focus on significant ones sometimes defined as financially significant but there are ways that we could refashion right the use of machine learning algorithmic tools within rulemaking to constitute significance because and here's the good part right the good part of the story is these administrative law frameworks that already exist focus on things like accountability transparency reasons giving now what they mean in this context Fendi you're exactly right there's all sorts of lack of consensus need for further thinking but let's again go back to just simple basic administrative law like let's assume an agency starts deploying these tools within rulemaking they have to set things forward in a notice and comment rulemaking they have to put out to the public these things like the assessments they have to receive input from all of these entities both you know stakeholders random law professors whomever is interested in these to debate these kinds of concepts and I think that's really really important and I think that before in a way before we get to what other this question I'm not trying to necessarily dodge it but before get to the question of what other types of laws have to be refined we should first think about how this existing administrative law framework which by the way not only could force agencies to conduct this internally and have some kind of executive review through Olira but we have judicial review too and so the whole notion of judges at least in this rulemaking context sort of getting up to speed so to speak and thinking of what's acceptable reasons giving for the deployment of these tools will go a long way and I believe that it will become sometimes I feel like we get stymied by thinking about how the barriers are insuperable so just to come back again to something very concrete something I know quite well within the FDA you know the FDA is not only sort of the most powerful ex anti federal regulator in the entire world but it has the most data right it doesn't just review tertiary analyses of data and the capabilities of now getting more real-time ex post data with respect to say medical devices that are out in the real world etc leads us to a point I think that talks about how government at least already has the capacity to be imposing these kinds of accountability constraints on its own use and deployment of this data sharing with the public through the notice and comment process having internal executive branch and external judicial review and I think a lot could be learned through that process I fear sometimes the idea of a kind of wait and see approach the final thing I'll say is Spandi you raised and this question also raises really difficult issues about the interplay between ensuring kind of transparency with respect to reasons giving in this administrative law context and things like data privacy and also security and the only comment I'll say on that is that that is an issue that is going to cut across you know the entire federal administrative state so each right now each of our agencies have their own mission they have their own ways that they're starting to think about the deployment of machine learning and AI tools but this this like algorithmic accountability is one that's going to be cross cutting across the entire federal administrative state and I think a lot more thought has to be put into place as to how we're going to have kind of coordination among federal agencies so things like this panel are somewhat of a help things like efforts by a guest you know which is trying to share information between federal agencies anything I believe that kind of puts in the same room both lawyers and technologists to be discussing this issue and also puts in the same room people from government and the private sector and get them working together thinking as Christine was arguing earlier thinking about the life cycle at the very earliest stages right I've been in too many panels where some of the technologists will talk about how they'll go about developing and then down the line sort of administrative law or the law will say thumbs up thumbs down that is not the way this should operate and and hopefully we can collectively change the kind of trajectory of thinking along those lines yeah that's really helpful thank you and it sounds like our federal government is going to need a lot of smart people across embedded you know who know their stuff and as you said are um you know both lawyers and technologists working together uh as the acus report really emphasized so thank you um so I guess somewhat related I think a lot of you brought up various points related to um uh sort of high risk AI systems so in all of this we're talking about maybe not using sort of the strictest accountability mechanisms to to regulate every single algorithm but we're main we're often mainly focused on uh sort of the highest priority algorithms so high risk AI is what often the term we're using so there's this ongoing debate around how to define high risk AI I want to pose the question to all of our panelists here how do you think high risk AI should be defined and how can we as a community I guess between civil society and academics and our um private sector partners um work to reach consensus around the definition I'll jump in just for our first voice in the space I want to push back just a little bit on the categorization of risk um we do have with at the partnership on AI an incident database um and it's sort of a repository of harms where we keep track of any harm that AI has caused it's out in production it's out in the world and it has already impacted human beings um and these are you know range from it misspelled your name on an application to an autonomous vehicle had an impact with a human being so we have a repository of harms that we keep uh it is not subjective it's just a curation of news articles and other write-ups that have already been publicly made available the reason I want to push back just a little bit on which AI is high risk AI is because of the cascading effect of any machine learning system or AI system and the impact that goes well beyond deployment we talked a little bit in the beginning about how AI can impact users the people who log in and actually make use of the system can impact the engineers or developers who are in community in creating the system but there's also a very important demographic that we often leave out which is the impacted non-user I like to use it as an example you go to the bank and you apply for a mortgage and you get rejected probably no one in that bank can tell you what the algorithm was that made that decision and that's an issue especially if you don't have insight as a stakeholder as an impacted non-user of some algorithm or some mechanism or some collection of models you don't have insight into how that information was used and is kept so I just I think at some at some vantage point it's all high risk you know because there's so much happening uh not to use the the term in the title of your paper this black box we just need more visibility and transparency so I would say the the risk whether it's high or low or medium is not necessarily the way we would categorize the AI but I'd say it's the way we should categorize our uh visibility into it so it's high risk if we can't tell you what the heck is going on low risk if we know exactly what's happening but there's still some outliers some things that could occur so that's just what I'll add to the conversation in terms of the way we categorize the risk but of course I do believe that documentation is at least part of the solution uh yeah I'll just jump in so I'm fascinated Christine by this metaphor too of the cascading effect of AI so I feel I'm torn here because on the one hand um categorizing things as high risk is reminiscent for example to come back to the FDA of having like class three medical devices that are given the most scrutiny because there are devices that affect you know potentially life versus class one which are like band-aid so we need to have some kind of in the governmental sector we do need to have kind of different I think it's appropriate to have a kind of risk management type of strategy and yet I'm convinced you know listening to you that um and this gets back to the heart of the question that um that we might be better served and you propose one in terms of sort of how opaque or how explainable some sense you would have a metric of is this where are we you know there's new new an emerging field of explainable AI so we might have a continuum and if you are on the um end of more explainable AI then you put that into lower risk another way to do it would be to be really focused and again I'm going to come at this from the governmental uses but we might have very different demands I think we should have very different demands with respect to the uses if this is being deployed in a way that could for example take away someone's individual benefits or affect someone's you know right to a type of service versus you know some kind of mechanism for policing a new emergent health and safety risk I mean we might you know so we might kind of think about high risk with respect to the um type of use so we might think differently about in government uses that are in enforcement uses that are in like adjudication of individual benefits let's say and then things like the rulemaking process that I was alluding to um earlier the final note I'll say that brings me back a little bit I wouldn't I hadn't used this cascading metaphor but I worry in my space about governmental uses that too often scholars will draw a kind of line in the sand and say that if it's just being used for a supportive use we don't have to worry about the whole administrative law accountability framework I was talking about before but if it's being used for determinative like it's making the actual determination decision then we do and there's a lot of talk to about a human in the loop so long as there's a human in the loop we wouldn't be as worried I guess I worry and I've seen um and maybe the cascading metaphor is a good one that there are some uses initial deployment of these technologies that look like they're very mundane just supportive um not really affecting decision making it ever whatsoever and then as they learn and develop they start crossing that line and so I worry about the idea of these you know lines in the sand I worry very much about how there's a human in the loop but the AI has actually been making a determination and that the human really has no force to go against that so I think you've put your finger on something that's very important I think I would stress that we should think we should define these concepts sort of along a continuum um so as we don't uh inadvertently create sort of safe harbors that allow certain types of deployment of these tools to evade uh scrutiny the kind of scrutiny that I talked about earlier and I can just jump in there I think this kind of conversation is actually uh you know a really great example I think the broader conversations we should be having um in the space but um one interesting like this is a the question of how to define how to frame it is something we grappled a lot with when uh writing the cracking the black box open before and um an interesting a sort of discussion I came across was one that was happening in the EU where uh many stakeholders are starting to define high-risk AI as a system that poses threats or harms to fundamental rights and I think that ties in a little bit of what Christine was talking about and a little bit of what Catherine was talking about so you know some examples that were circulated were algorithms that determine creditworthiness algorithms that are determining whether an individual is sentenced to jail but then also algorithms that as Catherine was mentioning not just determine making determinations but that could pose an uh an issue down the line so algorithms that are used in police technology and depending who's uh sort of interacting with that technology is then sort of producing a harmful outcome um just to sort of like tie together some of the threats I think that um you know it's a really complex question of like how to define something that will so critically you know will have such a critical impact on how we move forward in this space um and I think an overly broad definition will you know likely uh require companies and governments to conduct evaluations of systems that perhaps are not of significant concern and then use valuable resources that you know could have been used elsewhere but uh to Catherine's point you know you can't always clearly define and understand the risk right off the bat you know systems change the situations change the context change but an overly narrow definition could also exclude systems that do make critical and high-risk decisions so that could in the future and so I think that we definitely need to continue these conversations but I would emphasize that the longer we put off these conversations and set to say that this conversation is easy at all it can be resolved you know over drinks one night but um the longer we put this off I think the longer entities that are perhaps using systems that we find concerning they are able to define the term on their own and they essentially can set the terms write the questions and grade their own homework and so I think it's really critical for promoting accountability around these systems that we start to move forward and you know put forth a couple more concrete proposals try to test them out perhaps because right now it's really a great in your own work situation thank you all um yes it is I think it's really a challenge but that this is a great sort of conversation as Fandy said um and sort of um an example of the bigger conversation that everyone's having that we need to have um and difficult to come to consensus um on this but um first of all I'd like to give the audience a reminder to please um use Slido to submit your questions uh Slido again is the box located on the right of the video and if you have any issues with that at all you can contact us at events at newamerica.org you can send your questions there um and so we will be looking for those um but in the meantime we certainly have more things we can discuss um so again I'm gonna pose another question to for everyone to weigh in on here um so I think as we've been talking about throughout and as I sort of started with we tend to talk about many of these mechanisms for promoting um fairness accountability and transparency in a siloed fashion we talk about um you know I'm all documentation practices very separately from how we talk about audits and they're often not part of the same uh conversation and um then we talk about transparency reports entirely separately wondering how we can connect those conversations and what are the challenges we'll start there and then I have a sort of follow-up I don't mind jumping in did somebody else Kevin you want to go I just had a very short response which is that I was motivated in particular I was really piqued by the the cracking the open the black box does a real service I think by um surveying and putting them all together in a document and as I mentioned in my longer remarks I won't repeat them here I was really struck by how a kind of administrative law accountability framework similar to one that I designed with respect to federalism accountability could really work here and what it did is it gave me food for thought with respect to you know bringing in things like the particularly the impact assessment because that's very analogous to the federalism impact one but um Christine's made me think a little bit too about sort of the documentation requirements and we had occasion I'll let her speak for herself but we had occasion she talked told me a little bit offline the other day about some discussions with GSA about how they might incorporate some of those things with respect to their procurement process so so to me there are some very easy ways right so if we bring in some of these tools into an administrative state of law you know framework and publish it etc to get these people maybe talking together about it so the report does a does a service you know by doing that bringing it together in one place and I guess the answer is each of us from our own individual vantage point to be challenged to think about what are ways in our own work that we can try to show not tell the need to do this that's awesome uh show not tell I like that um yeah absolutely Catherine we're talking with some folks over in GSA because you know they create those templates that government agencies use in order to procure what's that percentage was it from your study was it the 33% yeah 33 to procure that 33% of machine learning AI asset we're talking about well what language needs to go in that contract what do you need to require upon delivery so that's going to be a really interesting and exciting possibility for actioning transparency but to the question of originally posed by Lauren about how do we connect these conversations I'm going to quote Peter Drucker culture each strategy for breakfast or maybe it's lunch for at least one meal of the day culture each strategy and I feel like it's not about having these amazing tools that are super important or even this strategy or methodology or approach for documentation which is really important it's about a shift a paradigm shift in the culture of the way we develop and deploy these machine learning systems you know we all know from experience the way technology becomes pervasive it changes the way we do life the way we like imagine going around without your phone for a month like oh how do we do that right it just so it just shifts the way we engage with the world so just thinking of how as an organization we can focus on a culture shift where documentation whatever the cost if you've got to have a whole new department of generalists that cross model cross department cross algorithmic function are in charge of doing this life cycle documentation it can then inform at an enterprise level what you're seeing what's what records are being kept that's an amazing thing to do but it takes lots of time it's hard culture shift especially I believe in the federal government is difficult especially in a very hierarchical space like maybe the Department of Defense or some military groups so it's just a tough thing to do but it's at the at the heart of how we make this change and how we operationalize responsible AI and transparency yeah and I would just build on that I think part of that culture shift that we need more broadly in the space also needs to include trust and transparency I see a lot of times when stakeholders from different sides of the space want to talk about you know a certain type of algorithmic system for example but that trust doesn't exist between the relevant parties and as a result there's not any meaningful communication or transparency you know what we mostly get from each different side are talking points like this is what we think is happening like this is what's actually happening and I think if we're really trying to push forward meaningful multi-stakeholder dialogues that are actually trying to promote accountability you know in all aspects of the space then we need to sort of build trusting relationships with companies with government agencies and vice versa to sort of engender this notion that we're working towards the same goal yeah that makes a lot of sense thank you all so I think again building on that and like sort of working to bring together all these conversations we've obviously talked about a whole just within this past hour talked about a whole different a whole constellation of different methods I'm wondering if if there's any sort of gold standard mechanism that you all would be willing to put forth like or sort of put your finger on the scale for that should be used sort of a cross context again when we're when we're at least talking about high-risk algorithms is there one method of promoting fairness accountability and transparency that sort of reigns supreme do we need audits or external audits in every circumstance or in most circumstances or anything like that I'm wondering again we've talked about and there's probably a need for a combination of different mechanisms and it is very context specific I think especially as Christine was saying but I'm wondering if you all would put forth the idea that there is one that sort of needs to be there all the time I'm happy to jump in there maybe just to be a little contrary to that no I think like you mentioned there are and as we've discussed during this panel like there are benefits and disadvantages to each approach and one of the things that we my co-off and I really tried to drive in through the report are that you know many of these approaches can supplement one another if we did bring them together into a holistic approach for promoting FAT that's not to say that you know wine approach could not be championed but I think that especially just as we've seen during this conversation like procurement has so many advantages that maybe machine learning documentation doesn't even touch on that maybe audits don't even touch on and vice versa and they also have different limitations that they perhaps could fill and so I think that there are best practices we can learn from each sort of side of the conversation but I'm not sure that I would say that one of them necessarily raised supreme so I'll go out on a limb and say it's really apparent from my prior comments that within the governmental context I'm most drawn to the idea of requiring a kind of algorithmic impact assessment and how that could fit into an administrative law mechanism particularly with respect to thinking about the importation of AI into the rulemaking context so again I'm going bold by choosing and defending one but I'm narrowing kind of the context but what I like about that again and the thought that I've done today about the analogies with federalism is it will force kind of in real time and with differing views backed hopefully by some empirical data and some study and learning some of these definitional controversies that I agree with Spandi we don't want to let it we don't want those definitional sort of controversies paralyze us so again people can vehemently disagree with whether an agency for example should set a standard and should preempt all state tort law but I think we all could agree that they can't do that and say there are no federalism impacts and that was happening and then we had an administrative law framework that that assessed that so there's some low hanging fruit that we want to make sure we don't miss so we shouldn't allow for governmental uses in the rulemaking context to go forward without such an assessment and then we have tools that are already existing within that process to scrutinize now we'll get into all sorts of debates about when it comes to like court's review should it be hard look what should judges demand or a soft touch because it's technological areas it's not going to solve those problems and we have to continue to debate them but I think that one to me looks like a very powerful tool that we can what I like also about it is we can kind of put it into existing administrative law and judicial review frameworks for the governmental uses at least in rulemaking I'll second that I'll go for impact assessment you know that's super important thinking about the risks how you might mitigate risks that show up and being bold in the way we distinguish between issues and risks issues being things they've already happened they've already occurred it's already real it's already live it's already impacting people or systems so I think that's it but it yeah it's tough it's tough to name just one thing right because they're so interrelated I think about my time as a consultant and as an enterprise architect and the work we had to do to show agencies like OMB why we deserved this funding and we'd have to show all these architectural diagrams and how one system impacts another and we'd have to show all these requirements and traceability and how one affects another and then that on top of any type of budget information and the cost and how the cost could double or triple or you know what if COVID happens you know so there's so many things that affect so many other things but I think impact assessment is a nice if we had to require one thing especially across federal agencies that that would be a nice thing to consider just asking those questions up front thanks yeah I know it's a tough question especially since again as many of you have said it's often very context specific what mechanism we need and also many of them should be probably working together but but thanks for thanks for humoring me a little there and so again sort of building on that and Sandy you mentioned that there may not be one tool for every circumstance but that there are best practices for promoting FAT around high-risk AI I'm wondering are there best practices that you've seen internet platforms and sorry this question's for everyone are there are there but are there best practices that you've seen internet platforms use that you think could maybe be brought over into the government context or vice versa are there some you know practices that the government's using that haven't really been explored by a lot of private entities or internet platforms yeah anyone want to I'll give a real quick example there's a site it's it's a it's called design ethically calm they have a sort of a monitoring checklist and so I would say this really helpful thing is the idea of even though you have successfully developed and even successfully deployed that it doesn't end there just that whole concept of watching being aware making note monitoring checking in with users you know so just this idea of kind of like kids you know like it doesn't matter how old they are at any point they come and ask you for money right you're going to always have this model to babysit and to monitor and to watch and in that active monitoring you know there's this opportunity for improvement throughout the life of that model and any other model that is built based on the the the tenants of that model I can jump in so I don't say we do a lot of work around platform transparency efforts and we definitely you know encourage platforms to always be more transparent about their operations of policies I think over the last few years I've noticed that some platforms have started releasing more public information explaining how their algorithmic systems work in a sort of more digestible way so that the average user or perhaps the a researcher can understand the structure of these systems as someone who spends most of reading about this that's been really helpful and I I think the aegis report was really interesting because I was the first sort of like breakdown of a comparable sort of breakdown that I saw and so of course there are different sensitivities when it comes to the government and how much you can share but I do think sort of that transparency to raise awareness and generate common understanding in an explainable fashion around when systems are being deployed how they're being deployed what impact they can have that that's a it's I think that's like a first step best practice that I think should be applied holistically and I'll just add I mean it was alluded to in your report and I guess I hold out it's always good when there are bodies that are ongoing but they haven't received they haven't reached the end of their effort one can be quite hopeful but the effort by NIST the National Institute for Standards and Technology right they were they were energized by certain executive orders but as your report talks about they really have a desire to collaborate with both private sector and academia with respect to establishing sort of standards in this area I guess I'm curious if we're allowed to ask questions of our panelists right so should I should I dampen my enthusiasm for NIST I mean this is a this is a true question I don't know whether either of you has had much opportunity to collaborate work with them but it seems like as an entity particularly with respect to having the technological expertise and then if they're open to an infusion of legal policy input etc along the development of these standards it seems like that could be a very fruitful model so should I continue to be so enthusiastic and hopeful yes we answered the request for information and just talked about how you know some practical approaches to developing this template this guide this framework and then also the use and the audiences that might benefit from it so I say yes just because the conversation has started right in there a very pivotal organization within the federal government so you get points for points for starting the conversation so yes yeah I would plus one that we also submitted comments as part of NIST's recent consultation on AI bias and I I believe a number of other civil society organizations that as well so it definitely seems like that infusion is happening and yeah like Christine said like we're pretty early on in this conversation so I would hope that it's not too late that we've lost hope when I think that there could definitely be some meaningful work that NIST can do yeah thanks everyone so we've got kind of a kind of a tough audience question that just came in I'll go ahead and throw it to you all um so does the do you think the increasing use of AI and government and business seriously threatened our democracy I mean if Twitter can almost take us down anything's possible right so that's you know when I think about a question like that I think about uh do we want to forfeit um the evolution of technology or do we want to forfeit all the benefit that we could possibly get from some breakthrough just because of the risk the very real risks of that technology or that breakthrough being used for harm and the answer is no no we've just got to be thinking about risk we've got to be thinking about mitigation we've got to be thinking about all scenarios not just the good ones not just the ones we put forward to get the budget we want it but the ones that are bad and that could turn it on its side and so so no I mean it's possible but that shouldn't be a showstopper for us Catherine will agree with me we'll put we'll put stuff in place to make it possible to still move forward yes I you've you've intuited I do agree I think it's very worthwhile like you say to think about I often think about in the terms of the promise and the peril of the use of these technologies and it's definitely worthwhile to try to come at this in a kind of level headed way I also would second the idea as with all prior technological developments I think we need to take a kind of risk management approach right people I teach a variety of different things in the law school and sometimes people come in with this idea that to go back to the FDA again that their job should be to reduce the risk of medical devices and drugs to zero and we all know that would mean no life-saving medical devices or drugs so that's just not feasible we are going to have to think about we cannot hold sort of algorithmic accountability to some standard like that at the same time it challenges us because some people come up with metrics that technologies as long as they're better than human biases errors again that was alluded to I think Christine you mentioned in your the I don't want to call them the catalog the catalog of horrors or the or the catalog of harmed instances right you have an autonomous vehicle incident right so well if we compared that to automobile accidents right something like 98 to 99 percent caused by human error we would start to be very enthusiastic about autonomous vehicles now that doesn't mean in my opinion that we throw up our hands and not worry about these kinds of issues but it's really really difficult to figure out in a level-headed risk management kind of way how we should go forward but absolutely it shouldn't we shouldn't stymie the development because the promise side is real and so the way I like to think about it is it's like all kinds of risky technologies we need to think hard about about oversight and about sometimes these things are quantifiable and sometimes their values driven and we have to bring all of that into the conversation so it will be easy I would plus one that I would also say that a lot of the times with some of the technologies we grow to be concerned about the harms that they generate are we're in the moment unforeseeable and they accidentally we didn't predict but sometimes these technologies are also exacerbating pre-existing harms pre-existing inequities and so when we think about how these tools can interface with you know our democratic structures I think we need to you know do have a little foresight in the ways they could mess things up but also recognize our societies aren't perfect either so we need to be thinking about how can we improve the societal structures as well to ensure the democratic principles and so on remain strong and resilient as we introduce new technology and Catherine I'm going to tell our team that we should change the name of the incidence database to the catalog of horrors I think that's a much better a much better branding I do like that um so that was a sort of broader question we zoomed out a bit there with an audience question but to sort of zoom back into some specifics um sorry um just a second so um Catherine in the ACIS report you all outlined some sort of key next steps and recommendations I was wondering if you could let it give us a bit of an overview of what you all recommended and sort of what needs what needs to happen next for the government um yeah I mean a big piece that that I don't want to get too sidetracked but that I haven't uh hit on as much in my remarks is really the building of the internal capacity um and there are some challenges there right there are both um financial recruitment challenges that the government faces um but the idea again to return to our empirical finding that of the 142 major uh agencies that we were serving nearly half were deploying some form of machine learning AI and of the um use cases nearly half a little over half were being built in house um suggests that it's an interesting piece of this and I think the building of the internal capacity is only going to help with the kind of efforts that Christine and I were talking a little bit back and forth about about sort of the idea that those within government are going to be able to interface with outside groups concerned about these kinds of issues getting technologists and lawyers into the same room and the like but realistically or practically speaking I think um we encountered several examples again the FDA comes to mind because they were really out on the vanguard of this of kind of trying to build these non-commercial sometimes commercial but often non-commercial collaborations with academia in particular um with nonprofits the FDA offers you know various types of um you know research fellowship type positions to get people who are doing cutting-edge work in other sectors you know in academia and in other even in the private sector sometimes internally coming to the agency and I think we need to be um open to those kinds of things my own view is while the procurement piece is important for all the reasons that we've talked about that by and large I think that the that agency's given the sophistication and the nuanced contextual work that they're doing are going to have to develop the internal capacity and I think that that's going to um be a positive development so thinking about ways to encourage that I think is a is a key uh is a key theme there. Thanks for that um and I think this next question is related so I'll add it on here but I think that means that it likely is coming back to you Catherine um but we just had an audience question could you address the role of lawsuits in cases in which government entities deployed flawed AI um so that's an interesting one and one that we maybe haven't spent as much time talking or thinking about um will the concern about being sued lead to more careful deployment you think. So it's interesting um one of the things outside of the AI context as I said I alluded to earlier kind of administrative law accountability framework and I mentioned judicial review just to be clear there I was talking about judicial review when people challenge you know rules so let's take for example an agency that decides to issue a rule about doing kind of a retrospective review of its prior regulations and deploys an AI technology that is helping it to map out both what might be um kind of overly burdensome rules or things that they might want to revise and the like and they come up with this rule um and let's imagine someone challenges the rule that's what I was talking about when I was talking about judicial review and I have written about before and I do believe right uh academics differ about whether or not this threat of judicial review will induce agencies to do things differently and my own view it's hard to gather empirics on that there are lots of strong anecdotal evidence but my own view is that by and large yes it does incentivize agencies for example to think about things like federalism impacts or hear think about algorithmic impact assessments to do the job the best job they can kind of defining those terms thinking about the impacts putting it out there because there will be the scrutiny down the road um now the question raises a different angle which is you know actually suing government now it's something um there are other there are other groups and organizations that have focused on that piece and um and I haven't in my own work in this area I'm not averse again there's no there's not a kind of like inherent bias other than the fact that I was very interested and continue to be interested to explore kind of existing ways within agencies that they're deploying these techniques and there are um you know entities that come at this with a much more adverse adversarial and advocacy perspective that's either hostile or supporting the agency and given the work that I'm doing I haven't engaged you know in the former type but as a general matter and as a scholar who teaches tort law and thanks very much about kind of the deterrent effect of lawsuits I do believe that whether we're thinking about government or really increasingly thinking about private entities deploying these kinds of technologies I do think they're thinking about potential liability and that that does give them incentives for example if they're autonomous vehicle manufacturers they're going to get added incentives you know towards safety. It would be an interesting piece of this whole puzzle I'm sure maybe Spandi or or even Christine you know of you know organizations that are you know cataloging I've read about these as a I'm a consumer of sort of the secondary literature just my own research hasn't focused on the angle of suing government over use of these technologies. Yeah I can't say I'm a consumer of that content either maybe I should be but just to something you mentioned about incentives there is a sort of like a discussion around in the AI accountability space of whether developers of AI should be given sort of like subsidies or some sort of benefit to sort of develop more robust tools that match up with FAT expectations and I think and we talk about this in the report a little bit but I think you know that that kind of begs the question of do we think accountability is that kind of accountability is something that needs to be incentivized in that way or do we think that that's something that should just be an expectation from the ground up and that you know developers should be doing anyways. Yeah I mean that's an interesting question I think it comes in my mind it comes back to a little bit of an earlier question Lauren you asked me about sort of the positive and negatives of the procurement from the private sector versus building in house so I do think that a benefit within government at least of the building in house is there's there's a sort of they're already primed to be thinking about these kinds of accountability mechanisms given the administrative law framework that surrounds all of this in the private sector again maybe that would become as you said just sort of like a baseline norm so long as they're brought in at the earlier stages I mean again my worry is sort of a a buying but the sort of the example of horror to come back Christine that I have is sort of a governmental entity that's using this for a determinative use that might deny someone a benefit taking kind of an off-the-shelf tool that's already developed that for some other purpose that they know nothing about you know and deploying it so that's the kind of that's the example of horror it's not to say that earlier on there couldn't be sort of collaborations I personally think that it takes having some people within the agency to have the technological sophistication to sit in the same room as I said the in some ways I'll just come back to I mean I'm a law professor so maybe this isn't going to surprise anyone but I was actually really blown away when I did this piloting with the Stanford NYU students we did a cross-national zoom and this was you know pre-pandemic requiring this so everyone's much more facile even with the use of these technologies but we had you know PhD computer scientists law students professors all of us trying to talk together about the legal and the technological issues and it was very and just reading the kind of same documents and asking the kinds of questions now we had a selection effect we had law students who were interested in the technological side and tech people who are interested in the legal policy realm but I guess for me that's a little bit of a microcosm of a model for the future in a way in which we can have this you know cross fertilization I think each of us has been talking about well thanks for all of that everyone and yeah Christine I'm sorry Catherine I was going to add on the exact comment that you mentioned but as a follow-up uh yeah I think that it's not just the government who should be concerned about the the possibility of lawsuits if they're deploying these uh if they're deploying high-risk algorithms within a you know in a non-transparent unaccountable sort of manner that that create harm but anyway I think that that was actually our last audience question and so I think we may wrap up a few minutes early unless anyone wants to offer any sort of final thoughts on this very wide-ranging conversation okay well just wanted to give you all one last chance to add in anything that you didn't have the chance to say but um but yes I'd just like to first thank our panelists we really appreciate I appreciate all your insights today and I know I learned a lot I'm sure our audience feels the same so thank you all audience thank you all for joining us as always and and for the great questions and thank you to our new america events team for helping facilitate and put this on this was a great conversation I'm sure these themes and discussions will be ongoing so nice nice to hear from you all in chat today thank you