 Hello, my name is Benjamin Boudreau and I am a policy researcher at the RAND Corporation. My work primarily focuses on technology policy and includes, for instance, recent reports on the Internet of Bodies, on data privacy regarding COVID, mobile surveillance platforms, and on military AI ethics. Prior to joining RAND, I was at the State Department where I worked on international cyber policy issues. Prior to that, I completed my PhD in philosophical ethics and political philosophy. In addition to acting as a researcher at the RAND Corporation, I'm also a core professor at the Party RAND Graduate School, which is the nation's oldest and largest public policy PhD program, founded actually 50 years ago. And it is a PhD program that offers a PhD in public policy housed within the broader RAND Corporation. And my teaching at the Party RAND Graduate School has focused on ethics where I am the co-instructor of the core ethics course that's required for all first-year Party RAND graduate students. And currently, I'm also teaching a studio design course focused on some of the policy challenges associated with online social media platforms. And in addition to my teaching, I also have a role in seeking to integrate ethics throughout the broader Party RAND Graduate School curriculum. And the PITUN challenge grant I receive help foster those efforts to integrate ethics throughout the curriculum. And the main objective was really to identify opportunities to integrate ethical thinking into AI-related coursework and research at Party RAND Graduate School and to facilitate discussion of AI ethics across the entire RAND Corporation more broadly. And ultimately, our goal was to enable the students, many of whom will go on to be AI researchers and policymakers to consider the ethical implications of AI technology and to practice developing risk mitigations. And this objective is very much aligned with this broader mission I mentioned of Party RAND to integrate ethics as a cross-cutting thread across all of its various policy engagement streams. Unfortunately, in many cases, AI technology is research developed and brought to market without reflection on the potential harms or risks associated with the technology. And AI technology researchers and developers are not normally trained to reflect on the ways the researcher products might be misused or abused or in the longer-term implications of AI, especially for vulnerable and marginalized populations. And in addition, these technologists are often from homogenous demographic groups, and they aren't really prone to include a diverse set of perspectives that would help them identify potential risks. And I believe this is a systemic problem that we have sought to begin to address by identifying opportunities in the curriculum to inject ethics and execute an activity that required ethical reflection about a real AI application. And so in particular, we conducted three distinct activities. The first is we analyzed existing AI university courses across college and universities around the country to identify gaps and key barriers to integrating ethics into the curriculum. And we of course had a specific focus on the Party RAND graduate school. We worked from a dataset developed by researchers at the University of Colorado Boulder, so I want to give a special shout out to Professor Casey Feisler for her great work. But we went a little further and sought to identify other gaps within AI curriculum. And although there's a burgeoning effort to integrate ethics into AI coursework, we identified several key issues that are not currently discussed at sufficient strength. This includes diversity in AI, has to do with accessibility of AI systems, the human element in AI development, environmental issues related to AI, and military AI applications beyond lethal autonomous weapons systems. And when we looked at the courses offered at the RAND corporation, we found that approximately 19% already integrated AI ethics topics, but that's not enough and that there are opportunities for further embedding ethical topics into coursework through especially increased collaboration among professors and students. And so we used all this analysis to build an annotated syllabus that's designed as a modular model that professors can draw from to better integrate ethics into their existing coursework. And it also could provide a foundation for a standalone course or even independent AI ethics self-study. And what we did in this annotated syllabus was we presented a variety of topics and a breadth of readings that included an overview of key issues and additional references. And what we tried to do was initiate reflection by raising important questions and putting the readings in dialogue with one another. And they really are focused on three big questions, the first being how should we balance innovation and technical exploration with ethical considerations, especially those related to responsibility, equity, and well-being. Second, who's responsible for considering the ethics of technological developments in their application and how should this be done? What are the mechanisms for ensuring that critical ethical engagement is integrated in the development, testing, and application of new technologies? And what should be done when harms are not recognized until after development and products are brought to the market? And lastly, how do the cultural, political, demographic, and economic contexts in which these technologies are developed, designed, and deployed shape the benefits, risks, and harms? And how do they prompt the ethical, legal, and social responses to these technologies? So those were the major questions and we developed modules on privacy, bias, and fairness, diversity in AI, the environmental implications of AI, policy and governments, and a host of other topics. The third and final thing we did with our challenge grant was we wanted to organize an actual practical activity that brought students, professors, and researchers at the RAND Corporation into the discussion. So what we did is we piloted a concept of an ethics hackathon which involved a team-based competition between members of the party RAND Graduate School. And to conduct this, we partnered with the nonprofit COVID Alliance, which is a nonprofit coalition focused on developing a coordinated response to COVID-19. And it consists of epidemiologists, virologists, policy experts, medical informaticians, data scientists, and software engineers. And we worked with the COVID Alliance's research management platform, which is intended to aggregate and analyze data to be used for research that assists in COVID response. So the ethics hackathon included a dozen student participants, along with four faculty advisors, and we divided the students and the faculty into three teams. The work occurred over four weeks and involved an initial kickoff meeting, weekly office hours with the COVID Alliance, and a final meeting where teams showed the results of their efforts. Now the primary tasks and questions posed to each team were, one, what are the potential ethical harms of this COVID management platform, and how can these harms be mitigated through design features, visualizations, and use restrictions? And we put that broader question into two specific tasks. First, we asked the students to demonstrate the potential harms of data bias or other challenges with the platform and ask them to broadly reflect on the short and long-term ethical implications of unrepresentative data or how the platform might be used to contribute to inequity or social injustice. And then secondly, we asked them to propose and demonstrate an actionable recommendation to address those harms and to include, for instance, data visualizations or other technical features of the platform that would help end users ensure that the platform could be used equitably. And some of our outcomes were that we identified a range of data representation concerns with the Pratt platform, in particular with the under-inclusiveness of certain key demographic groups into the underlying data. We identified a set of data privacy concerns and the risk of potentially re-identifying users from the data that the platform ingests. And we identified a set of data use concerns that the data that the platform makes visual for policymakers might be interpreted or used in ways that potentially could have long-term and unfortunate implications to include targeting shutdowns to certain marginalized or vulnerable communities. And so what we did is we identified these potential risks and then offered some actionable recommendations and a checklist that the COVID Alliance should consider as they move forward. And in general, this was our pilot attempt at an ethics hackathon. I think it's a really valuable approach to enable students' ethical thinking related to AI and develop some of these actionable proposals and something we do hope to replicate in the future. And so that does turn to some of the future of the project. We will continue to work to integrate AI ethics topics across the party ring graduate school curriculum. We're going to continue to develop this ethics hackathon concept and continue to explore collaboration both with the COVID Alliance but also other external actors. And we're just going to continue to try to integrate ethics into distinct coursework at the party ring graduate school. Thank you very much. My name is Chris Berry. I'm a professor at the Harris School of Public Policy at the University of Chicago and I'm also one of the founders of our program in computational analysis and public policy. And hi, I'm Nicole Marwell. I'm a professor in the School of Social Service Administration also at the University of Chicago. And Chris and I are here to talk about our PITUN project, which is a course that we've been developing entitled Big Data and Public Policy. And just to get us started, I'll tell you a little bit about the broad overview of the course. The motivating question is how does the 21st century data explosion change the task of governing? The course is designed to provide students with a conceptual foundation for thinking about how they deploy data and advanced analytics in pursuit of the public good. And in particular, we're looking to integrate ideas from both data science and social science in order to pose questions and solve problems that are relevant to policy making and policy implementation. And we want to tell you a little bit about how we got to this to this point of working together on this on this course. So as I said, I have been involved in our joint degree program in computational analysis and public policy for a number of years. And one of the gaps I have seen in our education has been courses that unify sort of social science and data science. So we have a lot of students who are taking courses in parallel in both of these two subjects, but relatively fewer that bring them together. And in particular, in my area of interest, we have relatively few courses that bring together what I would call the traditional program evaluation approach or causal inference approach with the machine learning and predictive approaches. And so a lot of students learn these kind of techniques without really ever being taught about how they work together and what sort of questions each is appropriate for. So I had a long-standing interest in trying to produce a course that would teach students about how these ideas work together. And my entry to this project was rather different. I am an ethnographer by training, a sociologist who hangs around with people watching what they do and talking to them about what they do. And in my research, which is really a lot about the intersection between government and non-profit organizations in terms of the production of welfare state services, I've been noticing over the last few years that there is really a mania for what everyone refers to as data-driven analysis. And so watching folks in the field on the ground doing data-driven analysis, talking about data-driven analysis, I really started to wonder, what exactly are these folks doing? And what is it that they think that they're doing? And so this became a real interest for me. And when the opportunity arose that I saw the announcement for the PIT-UN grant, I decided that I wanted to try and figure out if we could do something that would address these questions. And I found Chris, Chris and I didn't know each other in the beginning, but Chris was heading up the PIT-UN activities at the University of Chicago. So we sat down, we had an initial meeting. I think Chris might have been a little skeptical of me, but we ended up deciding that we actually had a lot in common in our viewpoint and that we wanted to move forward with the application. And fortunately we were successful and we've been working on developing this course since we got the grant. Yes. And so I think the course is going to reflect these two different kinds of perspectives that we bring to bear. And that's really what makes it interesting. And so let's give you a little bit of an overview of the course and how it's going to work. The course begins with an introduction to data in public policy analysis and really the rise of this paradigm of, as Nicole described, evidence-based policy. So we start by asking, how did we get to this point? How did we get to a point where everyone has this mantra of evidence-based policy as distinct from past generations where there might have been more reliance on things like expertise, for example. And after talking about that, we talk about what constitutes evidence and knowledge, particularly in what we see as two dominant paradigms of analysis, which are the causal inferential program evaluation approach and predictive data science approach to public policymaking. So following that initial introduction, we move on to an initial section where we address questions of ethics. Now, even though we begin this early in the course, we are not segregating the ethics content into just one week, but rather we'll see it come up over and over again over the course of our quarter. We really want to lay a foundation early on in the course though about how to think about ethical problems in general, as well as exploring ideas about ethics that are specifically related to the use of big data and the use of big data in policymaking. One of the really important ideas that we'll be pushing forward in the course is that regulation may happen around the use of data, and big data in particular, and in relation to policymaking more specifically. But we're pretty sure that the technology is always going to run ahead of the regulation, and thus it's extremely important that students develop their own ethical compass for thinking about these issues as they're doing practice in the field. And so we're going to be looking at those issues consistently over the course. Oh, sorry, this is me again. So then as we move into the next section of the course, we're going to be looking at the issue of what we're referring to as datafication or data production. And what we want to do here is really focus students' attention on the many different ways in which new forms of data are produced and then become available for analysis. The course is really going to be interrogating this claim that some big data evangelists make, which is that the present day volumes and velocities of data actually allow a truthful and frictionless representation of the world. We will consider this claim as well as the many critiques of this claim that are out there and encourage students really to be critical consumers of the data sets that they might encounter or that they might in fact be producing themselves. The process of data production, and this is particularly my view, that process is at bottom a human process and a social process, and thus it's shot through with many social constructions and biases. And it's very important that users of data be aware of this fundamental reality and also be equipped to think about how to address it prior to the stage of analysis. So after talking about the data side of things, we then begin a section of the course that we call epistemologies of quantification. And this part of the course is really focused on understanding what kind of knowledge is produced in policy analysis within the two paradigms we're discussing, which is the causal inference approach and the predictive analytics approach. And we want to ask for each, how is knowledge produced within that framework? What kind of knowledge is produced? And what kind of policy problems and questions is that sort of knowledge useful for addressing? And along the way, we also want to talk about the limitations of each and when to use them or what kind of problem and maybe when not to use one or the other of them. And after having this discussion of the epistemologies and principle, we then move to the final section of the course, which is a more applied or interpretive section where we ask, okay, presuming that you have taken an appropriate technique to ask an appropriate type of policy problem, you're still not done. There's a lot of questions about interpretations of the results that you've gotten and what sort of claims are or not justified as a result of that analysis. And so we're going to talk about issues that relate to both disparate impact, algorithmic fairness and various ways in which there may be bias associated with our results, whether that bias comes from the data or the analytical side. And we'll consider these things from both a legal perspective and a technical and ethical perspective. And then we'll want to move on towards responsible communication of your results, including and particularly communicating the uncertainty about what you've learned, both understanding uncertainty yourself and communicating it to the likely consumers and policymakers who will use your work. That's the end of the formal instructional part of the course. And we then conclude with students giving their own work based on their own projects trying to apply a lot of these ideas to whatever the topical area of interest to them. That's our course. Just one more point I'd like to add in terms of the audience for the course. We struggled a bit about how to think about who the audience was. We ultimately settled on a sort of larger framework. So the course is open both to advanced undergraduate students and both masters and doctoral students. We're going to run the course this first time around as a relatively small seminar of about 20 students. We're really hoping to get a wide variety of perspectives in the room so that all the students can really learn from each other depending on sort of where they're coming from and what their strengths are and therefore to produce students who are really thinking very broadly about these critical issues as they think about a future in public interest technology. And as one final point we'd say that, you know, we began preparing the course by surveying courses that are related to this one at other universities including many of the Pitt UN member institutions. We now have our own syllabus that we look forward to sharing and getting your feedback on and we're very much eager for the discussion that will come at the upcoming conference and we look forward to talking with all of you. We'll be there. Thanks a lot. Hi my name is David Eves. I'm going to present about teaching public service in the digital age. Before I get going I just want to thank there are a number of people but maybe specifically Enos Marguel who's a professor at University of Constance and Tom Steinberg who's a former chief digital officer and a real pioneer of using technology and government in the UK who've been really close collaborators on this project with me. And of course I'd love to thank the Pitt University Network who've been very generous in funding this work and helping a collaboration with practitioners and faculty from schools all around the world. My own background is I'm a lecturer in public policy at the Harvard Kennedy School where I teach students really kind of policy and public administration students about what is the knowledge they need to understand about technology to be effective when trying to deliver services to create public goods. My history in doing this is I've trained all the Code for America fellows. I've done trainings for every cohort of the Presidential Innovation Fellows of the White House and numerous other groups around the world and I've really been at the intersection of public administration and policy and technology for the last 15 years. My own experience and that of many others has shown that there is a real struggle inside government on how to use technology effectively to deliver public goods and one great example of that is the Standards Group has a report every year where they talk about how successful technology projects are and they including government technology projects. And here you can see that the statistics are not fantastic. You know a huge number of projects fail or are challenged every year. You know a quarter I think actually number has dropped a little bit it's more like 20% now fail outright. That means like literally we have nothing to show for it and challenge means they went over budget they didn't deliver the value we expected. And here in the United States we see lots of that. I mean famously everybody knows the healthcare.gov debacle but it's really just the tip of the a much bigger iceberg which includes you know during the COVID crisis the number of states that had challenges with websites that would offer core public services to people who were furloughed or unemployed those falling over and you can imagine a parent not being able to get benefits not knowing where their money is going to come far to pay for rent or the next meal because the government wasn't able to deliver the service and they of course could not go in person and do that. Even just sharing data about COVID between agencies often sometimes resorting to faxing it around so you can imagine the data entries errors and you know even just policymakers not having the most up-to-date and effective data and maybe most tragically was this image which is you know two young students sitting in the parking lot of a Taco Bell because they've been told they have to go to school remotely and there's no broadband policy there's no way for them to get connectivity and so the only way they could do that was go free reload off of the fast food chains Wi-Fi as that's totally heartbreaking photo and what I'm trying to show with all this is that in a modern era every service by the government is at some level a digital service even the garbage collector who comes around and picks up the recycling the route they take is almost dictated by a computer everything we do is intermediate by technology and one of the Titans in our field Helen Marguess talks about how you know as policymakers and politicians as they try to make decisions those decisions are increasingly constrained by what the technology infrastructure is that allows them to make choices and so if you want to impose a new tariff or do something if the technology doesn't support it then it can't be done or it can only be done at vast expense so the mission of our project is to try to figure out how do we increase the number of public servants and public leaders with some minimum set like fundamental skills of digital and when we did a survey looking around to see how like the public policy public administration schools were doing in this space it was really not particularly positive as you can see here the number of schools that have a required course in technology and digital technologies is low and in fact interestingly the requirement from that from the certification agency NASPA that certifies public policy schools was actually removed a few years ago and I think very interestingly was moved because no one can define what digital was and I think if you go back to those earlier slides where talked about the failed projects it's because now digital is part of everything we do and so it's challenged us to figure out well how do we describe what the discrete skills are here that somebody needs when it's part of everything and that's what our project's trying to do the good news is is there's actually lots of being taught in the space and many schools actually are looking at these various issues but often they're doing like they have a data science program or a cyber security program so they'll do a deep dive the critical thing is how are we going to teach a minimum of each of these skills to equip our public leaders for the future again more good news there's a rich academic history of thinking about this in a public administration perspective that we're able to draw on so our approach has been quite simple we're going to build a community of practitioners and faculty members look at the research and then figure tied to practice figure out what are core competencies that public leaders need to have and then define those build a support network with teaching materials and community so people can go and teach those in policy and public administration schools and civil service colleges all around the world that work's been ambitious but incredibly fun here's a photo of people who are collaborating right now on some of this work and you can see they're coming from universities and countries from all around the world what has resulted after the first year of work is we managed to zero in on eight core competencies that we think are critical for leaders in a digital era and those you know those range from focusing on users so how do we understand what users needs are when we no longer see them face to face so how do we get into their shoes have empathy for them understand their needs and then design digital services for them all the way to how do we iterate how do we create you know governments where we don't create plans on budgets that are one year or five year cycles but actually cycles where we can learn in weeks or even days about whether something's working something something small and adapted over time like you see happening in the private sector I won't go through all eight competencies that you can find those on our website I'm teaching public service in the digital teaching public service digital what's been really exciting is when we announced these we did an initial form we had 200 people sign up now we have over 16 universities and civil service colleges in our network with actually this higher now probably about 20 to 30 faculty members who are active collaborators we've interviewed over 200 academics and talked to at least 40 50 deans from public policy schools getting a very positive reaction and and our mailing list now is over 2000 strong from 50 different countries just a multitude of schools that are that are reflected in that so what are we working on next really now what are we working on now now that we have these digital era competencies we're starting to build out a syllabus so a modular syllabus that will allow people to teach those here we're going to use that to pair with faculty who've already emerged that want to teach this in numerous colleges around the world including places like in the Philippines in Singapore and equip them so we probably even run a summer program next summer I'm kind of a teach the teacher for how to teach these competencies in your schools what we're really interested is if you're come at a civil service college or a public administration or a public policy school we'd absolutely love to talk to you learn more about what you're already doing in the space and see if there's anything we can do to support you in teaching this to your students we know that there's a lot that students like you know that are required courses so we're really trying to keep this to an absolute minimum and we recognize in some places this probably will just be an optional course but we'd love to talk to you nonetheless and see is there somewhere that we could fit this material in or is there someone at your school who's interested in talking to us for more information you can hit our website here or of course just send an email to any of the three of us on the PI so happy to talk to you but there's many people all around the world who could engage with you I hope this was fun and interesting look forward to hearing to you and thank you so much hello I'm delighted to be with you here today to talk about public interest technology research that's been taking part place in the southeast as a collaboration between Georgia State and Georgia Tech two universities that are part of the public school system in the state of Georgia I'm Ellen Zagura and I'm a faculty member at Georgia Tech let me start by highlighting the work of Beth Minet a region's professor in the school of interactive computing at Georgia Tech and Christy Seelman an associate professor in the school of social work at Georgia State these two together exemplify the goals of the public interest technology project that Georgia Tech and Georgia State embarked on over this past year in that they both come from two very different fields one where there's a excels at Georgia Tech that is in the college of computing interactive computing and then from Georgia State the area of social work which is a discipline that we do not have represented at Georgia Tech so this this is bringing together the skills and strengths of both universities this project is about COVID and in particular it's about trying to capture the voices of resilience and coping mechanisms amongst the LGBTQ community during this difficult time of COVID this brings Christy's expertise in understanding and supporting LGBTQ communities and Beth's expertise in doing user centered research and and qualitative methods projects so the objectives of this of this are to document strategies for not just surviving but also thriving during the COVID-19 pandemic and to do so among adults LGBTQ plus adults by by prompting them to take diary entries that allow a multimedia record of the ways in which they are responding to the COVID-19 challenges one of the key ideas here is to counter the narratives of LGBTQ plus people as being at risk and and and being deficit suffering from deficits but instead to try to demonstrate and document creativeness and strength of the population there's also an opportunity here by coupling the diary studies with other kinds of survey data to see how these strategies for coping differ based on generation race ethnicity status of health and disability etc and so on track to produce some really interesting insights in how to support and create resilient strategies amongst LGBTQ community under times of stress some of the prompts that the diary entries are responding to are you know positive terms like hope and adaptability sort of terms that that could be positive could be challenging like health and then identity a kind of persistent theme for LGBTQ plus individuals second project I want to highlight is very very different kind of project this one is joint between Mary Kim who was an associate professor in policy and government at Georgia State and has recently moved to George Mason University Mary and Calton Poo a professor in the school of computer science at Georgia Tech have collaborated on what's called a non-profit organization research panel so the idea of this project is to improve the quantitative evaluation of non-profit effectiveness that is what helps nonprofits become effective what are the management and over other overhead elements of their work and what is their actual and perceived social impact so overhead and impact and with a better understanding of non-profit effectiveness that will point to ways to improve the actual effectiveness of nonprofits pointing them in in directions for change and directions for best practice this is a data project this is a data heavy project in order to quantify non-profit effectiveness this team will draw on diverse datasets including those that come through public sources like the IRS 9090 form us census data and then surveys being administered by Georgia State George Mason and another collaborator one real highlight of this project is that the collaboration has already and recently produced a national science foundation research award in excess of eight hundred thousand dollars to fund the work of the of the co-pis here the third project I want to highlight is one that brings together computing and the criminal justice and criminology department at Georgia State so this is a this is a collaboration that started when Ben Shapiro was a research scientist at Georgia Tech in the interim six months nine months Ben actually has taken a faculty job at Georgia State so he's now an assistant professor in learning sciences so while this began as a Georgia Tech Georgia State collaboration it will move forward as a Georgia State Georgia State collaboration and the the idea and focus that Ben and Marie have is to look at novel ways to interact with crime data novel ways to to navigate and move around within crime data especially with a spatial and time-based component to it so to give you an idea of what the project objectives are here they're asking questions like how do new interactive representations particularly around the mobility of crime lead to new insights and what do those new insights about crime suggest about effective deterrence to crime allocation of resources and fair policing and as a somewhat of a meta question but key underlying both of the first two questions is what kind of information navigation tools are natural for criminal justice scholars and practitioners so these are domains criminal justice domains where information visualization and especially these novel new tools have the potential to really change practice and change research questions and research answers but that only happens if the navigation tools are comfortable and natural and familiar or learnable by practitioners so what's depicted here is two visualizations one is a traditional map based visualization of crime where there's a dot for each location where crime occurred the one on the lower right though is a novel representation where space is represented about not with a map but on a on a 2d grid space and then time is actually moving across the x-axis so if you take a slice vertically you have a point in time and a representation across space so Marie and Ben I believe that revisions to traditional visualizations are really going to make a difference in the way that crime and especially mobility of crime can be understood there are a number of additional collaborations that have been made possible with the generous funding and under the pit you and program I just wanted to mention two more very briefly one that's close to my work and that is a collaboration with Luisa Nazarino a phd student in public management and policy around internet access and covid and what is changing in terms of internet access during times of work at home and school from home and how is internet access the capability to connect to the internet changing as we see rapid additions to infrastructure to try to make it more possible especially for people who are in regions that are underserved by the internet to get online we're focusing our work in Georgia where we have counties with relatively low access rates as well as of course urban regions which are excellent access rates and then finally a collaboration between Bill Sable a professor in criminal justice and criminology and Jonathan Balak a phd student in robotics at Georgia who has expertise in video analysis so Bill and Jonathan are also looking at a topic that could be a covid topic but also could be relevant even in non-covid times and that is what is the difference between video and in-person contact between officers and probationers and how does that ability to meet in person change those interactions and what can be replicated well with video in terms of improving outcomes for probationers I'd like to close by sharing a couple of images from our kickoff workshop which happened in January on the what you'll see here is we had a very dynamic interactive get-together with I think more Georgia Tech and Georgia State faculty in one room together than had actually potentially ever happened in the history of the two universities we all shared our information about ourselves as well as what we were hoping to get and what we were bringing to the table and I just captured here the two information sheets that Ben Shapiro and Christie Seelman made and just thinking about how far these collaborations have come under the constraints of covid where many of these collaborators actually had never met before we got together in January and I have to say I'm incredibly proud of the work of all of these faculty and students across the two universities and I'm delighted to be able to share it with you today thank you