 Thanks so much for being here on Election Day. My name is Chris Bavitz. I am one of the faculty co-directors at Berkman. We have a great session planned here today for you on Election Day. A couple of housekeeping matters for those of you who have come to these lunches before. Please be reminded that we are live streaming, so say hello to the internet and this will be recorded online later. So bear that in mind if you ask a question or something of the sort. Please ask a question. We're going to talk for a little while but then we'd like to very quickly segue into turning this into a conversation with the rest of you in the room. And I think that's it in terms of nuts and bolts. So when we were thinking about what we would do on Tuesday, November 6th, immediately we thought about the fact that we have two extraordinary experts, not just sort of the most expert within the Berkman community, but really experts all around in Cathy Pham and Alvin Salahi in the development and deployment and procurement of tech within government. Both of them have had significant roles, which we'll talk about in a little bit in federal government thinking about a lot of these issues. And rather than I think reading your bios, if you don't mind, I might ask if I could have each of you before we kind of get into substance. Just introduce yourself, say who you are, what you do, how you came to do what you do, and then we're going to get into the conversation. So Cathy, can I ask you to start? Yes. Hi. Thanks for coming. I started my career in engineering and spent the last, probably spent about a decade at places like Google and IBM building consumer tech, but also building technology for governments. And then some of you may have heard in the US back in 2013-2014, there was the healthcare.gov failure, and several folks from private sector came to help stabilize that website. And thus, like the story is that that's the United States Digital Service was born in the US government. The real story is that actually maybe for about seven years or six years before that, there have been people trying to think of this idea of building out this tech entity inside government for a long time, and it took this like crazy fiasco to get people to pay attention and really establish something, which became the organization that I helped start and helped run, which is the United States Digital Service. And that became a team of about 200 technologists. I'll put that in quotes. It's engineers, designers, product managers, etc. who came and just worked with some of the hardest government technology failures around the federal government. But really, this is an area that I've been thinking about for a long time. I've done a lot of public service work in addition to a lot of my day job before that, so it was really just a nice fit to be able to apply that to basically take everything that was public service and tech and things I cared about and put that into one role. My parents came to this country as refugees, and for me, the fact that they let me into the White House and into government to just do this work and have that kind of influence was never ever lost in me and will continue to never be lost in me. And some of that will come through as we talk more. But for me, just the chance that regardless of any leadership or anyone in charge, the chance to really be in the room to help shape how we think about technology is really important. And then I came to Berkman because I really wanted to be around a lot of other people who think about law and history and social science and policy in ways that I never really thought of before and really challenged my thinking that way and think about what to do there. So I'm a fellow at the Berkman Center. I'm also a fellow over at Digital HKS at the Kennedy School, and I teach a class called Product Management and Society over there. Alvin? Awesome. Hey, everybody. I'm Alvin Solehi. A quick question. Obviously, it's a historic day in the United States. Show of hands. How many people have already voted? Very good. For people in the ceiling, that's actually a large population. Not everyone is wearing their self-congratulatory sticker. That's true. Some places don't have the stickers. How many people are planning to vote later today? Great. If anyone did not raise their hands, please come talk to us afterward. We can have a discussion as to why it's important to vote. So I'm Alvin Solehi. I am a research affiliate here at the Berkman Client Center. I'm also a tech advisor at the White House. Important for me to say that I'm here in my capacity as an affiliate at the Berkman Center today, not in my capacity as a White House advisor. So anything that I do talk about will be public information. So quickly, in my background, I started my career in government several years ago. I am an attorney by trade and started at the Advanced Research Projects Agency at the Department of Energy. I'm familiar with DARPA. This is basically Energy's version of DARPA called ARPA-E, where we invested in transformative, high-risk, super high-reward energy technologies. Incredible experience. From there, went to the State Department to work on expanding internet access to Africa, which was an incredible experience as well. Happy to speak with you folks who come up afterward to talk a little bit more about that if we don't get into it today. And then moved over to the White House in 20, what was it now? 2015. Back in the Obama administration, we were working there together in the office of the federal CIO. That's where I started. And in 2016, August of 2016, I authored the federal source code policy with an incredible team there. And essentially what we did was we closed a multi-billion-dollar government procurement loophole for software, which now requires that from this point forward, all contracts for custom-developed software must allow for that software to be shared and reused across all federal agencies, which is actually leading to a bunch of savings for taxpayers. Because as many of you might know, the government was essentially overpaying for duplicative software that it had already procured due to a lot of different antiquated loopholes. So we tried to close those. And as part of that policy, we also created a platform called Code.gov, which now essentially inventories custom-developed code across the entire federal government, as well as open source software to share it with taxpayers in an effort to ensure that you all can use it to leverage your own breakthroughs in innovation, to launch your own companies, and also to give back to the country through code. And hopefully we can get into a little bit of that during this discussion as well. I'm excited to be here. Thank you. That's great. So you've both touched on a couple of different specific projects, but I wonder if you could drill down a little bit more and just talk for those who haven't really thought about this before. What exactly does a startup look like or a tech development should it look like within the White House or within the executive branch of government, either the U.S. Digital Service as a whole or building the Code.gov platform? How does that operate? How does it set an agenda? How does doing tech development in federal government differ from doing it at Google or at a private company? Before I do that, can we also ask that you introduce your software? Okay. That's not as fun, but okay. So I'm Chris Bavitz, a clinical professor here. I run our cyber law clinic. I know some of the people in the room as clients of the clinic, we do tech law and policy work with students. And a lot of our clients, really all of our clients come from the world of public interest tech development. The idea of doing clinical work in law school has been around for 100 years where you provide free pro bono legal services to deserving clients on legal issues. And the classic model for that are things like landlord tenant clinics and family law clinics and other kinds of clinics that provide free legal services to deserving clients. The Berkman, client center than the Berkman center in 1999 saw fit to establish this program that I work in now, again, coming up on our 20th anniversary to sort of take that model and apply it to tech. And so we have a lot of conversations all the time about what does it mean to be a deserving client or an important public interest social justice issue that has a technology angle to it. So a lot of our clients are drawn from the private sector, although mostly nonprofits, startups that are building tech in support of a public interest mission. We do work with government organizations as well and do everything from day-to-day advising, drafting of contracts, amicus advocacy in courts on important tech policy issues, that sort of thing. I'm very interested in government tech, again partly because it informs our clinic docket, but also because I think we have been very focused at Berkman on two different things. One is sort of educating policymakers and those involved in law and enforcement, whether it's state attorneys general or state legislatures or the federal legislatures to educate them about tech issues that come up in the context of their enforcement or legislative priorities. And then we've also been thinking a lot about probably more directly relevant to the work you both have done, questions that I hope we'll have a little time for around government creation, procurement, deployment of technologies that may have at their core bias and other factors partially due to their being sort of black box technologies that we want to understand, whether it's a criminal justice product like a risk scoring algorithm used in pre-trial services in so many states around the country or other kinds of algorithmic tools that are used to distribute benefits and other things. How do we get government in the private sector, in the academic sector and civil society, talking about what these tools really do, how they work, making informed decisions very early on in the process of architecting new products to get out ahead of problems. So that's been a big part of the research work I've been working on. Tech development in government. And I'll expand it beyond just the federal government as well. So the U.S. Digital Service, I guess at this point is almost four years old, which is a bit out of maybe start up phase. But what that really looks like for us in the beginning was it was having a lot of air cover from a president that really believed in the technology. By air cover, it meant just having like the highest stake level leadership bought into this idea of bringing people with tech experts into government, which is candidly something that's really hard to do. We had a convening this past summer actually with digital service teams across the world, ranging from Mexico and Peru and Nova Scotia and the U.K., Australia, New Zealand, etc. And one of the biggest problems is bringing talent into a government, whether it's pay or having to move to a certain location in the U.S. D.C. and other countries of different cities, or maybe just being like the only one in the room that's trying to do what you're doing where everyone else is kind of stuck in like policy or bureaucracy. And it's a hard environment. So we were able to kind of have an outlet for people to come and do that kind of work. So I think that's kind of what it looked like to be a bit of a start-up to work on some of the highest priority. And by highest priority, we defined that as being touch the most number of people in the most vulnerable populations and could maybe be reused across different federal agencies. And we're now seeing different versions of the United States digital service pop up not only across the world. We've actually modeled ours after the U.K. but also in different cities around the country as well. And having different cities really and states actually think about this model also bringing talent into government so that you have people in the room who can make decisions around if we're going to buy a piece of software for the police department, what does that look like long-term? Versus just policy people making those kinds of decisions who oftentimes many times are actually well-intentioned and want to do the right thing but maybe just don't understand the tech enough. So yeah, that's a little bit of what but I also want to caveat that with the fact that I actually hosted a session recently at there was a public interest summit at the Kennedy School that David Eason and Vanessa Ryan Smith put on and I actually had a session called what can private sector actually also learn from the public sector. And there's also a lot of deep knowledge that the public sector understands about the people and the humans it serves that the private sector can definitely learn from. And as many of you see now, there's this tech reckoning of, oh my God, what we've done in tech. Sasha's in the room. I know Sasha's in a lot of work in Schneier's doing some running in that space. But there's also a lot that the, in my opinion, the private sector can learn from the deep expertise of the public sector and how well it just understands its people as well. Awesome. Yeah, I'll echo what Kathy said as well. So if we kind of take a step back and think about the startup landscape in general, startups in general, what statistics tell us over 90% of startups fail, right? That's a lot harder when you go into the federal government, which is essentially one of the most disabling environments on the planet. I mean, that's just a fact because you have agencies that are so bureaucratically entrenched over decades, and it is very difficult to catalyze culture shift when you are essentially fighting an uphill battle. So I think for the USDS, the former administration helped catalyze that shift, which was a big deal. Of course, it did, in a way, need some sort of catalyst sort of event, right? Like the healthcare.gov fiasco, which then led to this development of the most incredible SWAT team, the tech SWAT team of the United States government. For rossackcode.gov, when I came in, there were efforts to try to figure out how we can leverage open source software across the federal government in a uniform way. You had a lot of agencies that were kind of doing it in their own way, but there wasn't a uniform, consistent approach toward sharing government code with the public. And if you think about it, it makes a lot of sense if you're paying for us to build something, then you should be able to access at least some of it. And whether it's for transparency or for utility, it actually does make a lot of common sense. So we came into this project with this approach that we have to figure something out. But open source software, a lot of folks in the private sector are interested in it because whether it's based on a fundamental misunderstanding about the security aspects of open source software, or whether it's driven by kind of a profit-generated motive around not wanting to share code that your company has built for another entity. So we were also facing an uphill battle against a lot of strong lobbyists. And in a way, if we had approached this and some folks had tried this by saying we're going to create something too strong and that would just be dead on arrival. So instead, we found something that more folks could agree on, which was that there is this multi-billion dollar loophole where if you are building something for us from scratch for some reason in your contracts, you're telling us I can only use it for my agency, but I can't share it with any other agency. That doesn't make sense. If you had another company build something for Google, you would never tell Google, hey, we're going to do this. So from that standpoint, we basically asked a lot of companies to come to the table and we said, hey, we want to out of courtesy give you a heads up that this is essentially what we're going to do. And in law school, we always are taught to try to see things from both sides. So I thought maybe I'm missing something. So I asked them to come in and I said, look, we're planning on doing this, but please tell me how you feel about this. And almost all of the companies raised their hand and they said, you can't do it. And I said, okay, great, why not? And many of them said, because the government has never done it this way before. The reason many of you are laughing is because status quo arguments never win the day, or at least they shouldn't. In fact, maybe that's even more reason why you should change it if the government has never done it this way before, especially when you have a responsibility to make sure that you're spending that money efficiently. So I said, oh, I understand you all are here. You have a fiduciary due to your shareholders to protect their bottom lines. And so your job here is to make that argument. So we heard it and we can check that off. Let's move past that. As public servants, we also have a duty to the taxpayers to protect their bottom lines. And as far as I'm concerned, I think everyone in this room is a taxpayer too, so that duty extends to you as well. So if anyone here for us to pay multiple times for the same piece of software we've already built, please raise your hands. I would love to know who thinks that's fair. Of course, no one raised their hand at that point. So one person did raise their hand and say, all right, well, if that's the case then we're going to have to charge more the first time around, because we're not going to be able to profit as much off of it. And I said, okay, that's your prerogative. But as you all know, government contracts are a competitive bidding process. So you can charge as much as you want. You probably won't win but in fact, we can finally level the playing field for other small companies to compete too. From that point forward, we were able to put pen to paper. We wrote the country's first-ever federal source code policy, which gave rise to Code.gov, which we then launched, get this, five days before the 2016 presidential election. We just got it through. It was number one on Hacker News. Everyone was super excited about it. And then the election happened and everyone said, all right, Alvin, good job. It's time for you to leave. You know, it was a good effort. There's no way you're going to get through the last. And so I'm convinced that that would have been the reaction, no matter what, because no matter what, whenever you have a transition in administrations, there's a lot that you have to get through, which is essentially you have to stick around to help the new administration transition. It makes a lot of sense. There's a lot of work in history that you need to help them kind of transition toward. And so a lot of folks said, all right, a lot of people were leaving in droves. And our team, when we reframed it, we said, all right, if we're public servants, then we serve the public. And this platform has the potential to save potentially billions of dollars by cutting duplicative software. So we stayed. And of course, the first few months were difficult. We had to essentially make sure that our value proposition was as solid as possible. Once we demonstrated that this has the potential to save a lot of money for taxpayers, this administration was very supportive. Gave us the green light. We raised a couple million dollars to actually build a team. We built a full-fledged team of engineers, designers, comms folks, and code.gov was officially reborn. I kind of like to say that we were aqua-hired by the Trump Administration in a way. And so over the last year to two years, we've been running that program. And I'm proud to say that despite the fact that a lot of folks didn't even think this would survive, looking back two years later, when we first started the platform, we started with around 45 projects. Now we have over 4,000. And we're on track to save a bunch of money for taxpayers. So it's definitely worth it. And we're so excited to be able to continue moving forward. So I want to highlight attention that I was hearing between some things that each of you said. Kathy points out that she thinks there's a lot that the private sector can learn from the public sector in terms of how to do development. Public sector may be much more so than the private sector really uniquely attuned to sort of constituent's needs. And then Alvin says that working in government can be among the most disabling of environments to work in and talks about sort of a really rigid adherence to the status quo, which doesn't feel like it lines up very well with an environment that's ripe for technical innovation. And I certainly think that there's a line that I'll just parrot about government technology, which is that it's inefficient, it's bad, it's old. When you talk about government technology, you picture going to the RMV or the DMV in some sort of glowing amber colored screen from the 1970s that it's never been updated. Or you picture something like healthcare.gov, which again I think was viewed at the roll out obviously as having lots and lots of problems and you might attribute that, some might attribute that to something inherent about doing complex sector development in government. So how do we resolve this tension? What can the private sector learn about the way the public sector does tech development and procurement and is it actually as stifling to innovation as I think some might have us believe? I actually can wear both hats on this for all day. You actually, in your list of examples you can add to it the fact that the Social Security website has office hours. So if you go, if you try to access the ssa.gov website and try to log in between certain hours and though the night you will get the page of it's currently down for office hours. It just does not exist. You've helped me this before and I haven't tested it but the website is literally closed. The website is closed. You cannot use it. I didn't touch upon those parts but it can be advertising. In the US it's an $86 billion industry for tech IT. This was a number from a few years ago but 94% either fail or never delivered or never see the line of day or they just don't work. So that's really, really hard. I definitely don't want to underplay that. There's definitely a lot. The public sector can also learn from the private sector but I feel like I constantly hear that of how private sector is going to come and save government. But at the same time when I was on the ground with the Veterans Affairs or on the ground with the Department of Education, on the ground with some of these state social workers, etc. They understand their humans, the people they're trying to serve much more deeply than any of the teams I've ever worked on at Google. We did user experience testing but it was in my opinion can stay pretty high level and so to deeply understand how someone might use your newsfeed deeply understand all the different ways you might use your Google search and maybe not label some people as gorillas and deeply understand the human population. I think there's a lot that the private sector can really learn from how deeply, that's really what I mean by how deeply there are many people in government who may not know how to tie the technology piece to the actual human piece and make that actually work together into a piece of technology that actually works for the human but there's definitely a lot they do understand about the human side of what they build. Yeah. Yeah, I think the biggest distinction between the private sector and the public sector is that the public sector gets a bad rap. Sometimes it's warranted, sometimes it's not. In a quick show of hands, how many of you have ever had a frustrating experience with a government website? Okay. It's a lot of hands. How many of you have ever worked for the government? Okay. Not as many hands as the people who have frustrating experiences and I think this is actually a really interesting discrepancy which is that a lot of us experience government in a very particular way. A lot of us are very prepared to actually go in and help the government improve but whether it's because of various conceptions around government work or whatnot, a lot of folks don't actually take the leap to go contribute and I think it is one of the most fundamental privileges for us to be able to have experienced the government and to have contributed to the government and now, especially in this day and age, there's so many opportunities to be able to do a tour of duty in government and do your part. If you're a technologist, you can apply to the U.S. Digital Service, go serve for a year or two years. Join as a presidential innovation fellow. Join for a year or so. Go work for 18F within the General Services Administration. Or we understand that a lot of folks aren't able to uproot their lives and move to D.C. even if it's for a year. So we also have a lot of platforms like Code.gov where you can contribute back to government websites and different projects without ever stepping foot in a federal agency. I think the difficulty is really awareness. A lot of folks just aren't aware of the fact that there are so many opportunities. But it's not all incumbent upon the government to make sure that everyone is aware. I think it's also the individual interest to give back. And in a way, we have a lot of polarity in the United States right now politically speaking. And I think today is such a good forum for us to be speaking about this, which means that perhaps now more than ever is when the government needs you most. We need talented folks to come in and work. And we were talking about this on a call before our conversation, which is that no matter what, unfortunately, I mean this is kind of how politics works, after today, there will be a segment of the population that will be very upset or sad. And there will be a segment of the population that will be very happy and there will be rejoicing. But that doesn't come at the exclusion of good work no matter what the result is. And I think Kathy and I can attest to the fact that there are still a lot of good people fighting the good fight in government, doing work that they've been doing across administrations regardless of politics because the work is largely nonpartisan. And even though it doesn't get the press coverage that perhaps it deserves, it doesn't mean that the work isn't actually happening. So I think one of the messages that we also want to convey during this talk is that the government is still okay. And even though sometimes folks are disillusioned based on what's happening, we are still espousing good values and we have people who are very committed to the work that they've been doing for a long time. And some of these people are the unsung heroes that you'll never meet, but we need more of them and I think if you all are ever interested in contributing, there are a lot of opportunities to do so and I definitely would encourage you to do so. You picked on exactly what I was hoping to take us next which is to talk about politics and to just ask the question that I think, and I think you've answered about the extent to which technology development in government is a political animal or not, now that it sounds like in your experience there are at least components of it that really transcend the political obviously that's not true of all of them. Kathy, I don't know if you have thoughts on that. Definitely have thoughts. I also wanted to wrap up one last thing I thought of as you were talking and when we actually just say public service what does that even mean? And like now public interest is like a big thing, what does that mean? There's obviously government which kind of is a clear line, but then you also have when you refer to a consulting company and you're building for government is that public service and or if you're working for a big company as you've seen recently like Google, Amazon Microsoft, etc. and you have contracts for your software for the government is that also public interest and that all kind of is the government and in many ways if you really start digging you're like well Google technically was funded by NSF in its early days so that's also Google technically started with the government as well and so it's like all these ties that come together and I just start thinking like rabbit hole circles in my head so thinking about that so basically a lot of things just go back to the government regardless. I think it's nice, it's a nice talking point to say that everything we work on in the government is bipartisan and completely apolitical it's a talking point we've said in you know with the U.S. digital service as well but and many things are for example let's say 500,000 veterans are not getting access to healthcare and now we implemented something like a technical glitch that made it so these people can get access to healthcare that for the most part is apolitical but if you're working on something let's say at the Department of Education that makes it easier to find out which for-profit universities are preying on you and the leadership at the top no longer cares about that and so maybe you have to like re-divert your engineering power somewhere else like you can try and say that's apolitical but it isn't so there are lots of different cases of different administrations to bring in invent more refugees into the country and under a different administration these are all just real questions we have to ask and to pretend not for you all to pretend for us to not think deeply about them would be an oversight so those are just real questions that I think all of us have had to ask like what am I building how would it be reused at different points in time and how to really think about about that because the reality is it could be used for different purposes you can say the same thing about like the stuff that Palantir sells or any of like the policing software under different groups of people and for different intentions it could be used very differently bias aside right so I yeah I think and we can say that in some cases technology kind of is apolitical but it could definitely be used in different ways in governments absolutely I would echo that too I think that if you look at a lot of folks who have stayed across administrations a lot of the work that they're doing is most likely nonpartisan I think the easiest way to identify whether something is nonpartisan or not or at least when you're making the case to a new administration or to a new boss if you can find a way to really convey that your product or whatever it is that you're working on can save taxpayers money then the federal government should invariably support a mission like that even as what is the actual ROI on the investment that the government is putting in how feasible is it that those taxpayer dollars are going those savings are actually going to the right things all of these are considerations that I think folks do need to take into account but I think federal taxpayer savings usually wins the day when you're trying to figure out whether you can save a nonpartisan project or not I mean USDS was founded under a Democratic president in the Republican Congress and funded by Republican Congress so it's there was a case for saving taxpayer dollars in federal projects I want to bring in voices in the room maybe I'll ask one more set of questions which is a big one and this is my personal self interest I mentioned at the beginning that we're doing a lot of work at the Berkman Klein Center thinking about government use and adoption of tools that are algorithmic in nature artificial intelligence, machine learning really new cutting edge technologies can have significant and sometimes unintended consequences but I think it's particularly true with these new emerging technologies where the people who are in charge of procuring them sometimes even the people who are in charge of developing them haven't fully thought through their ramifications and there's a lot of debate going on right now about ways to ameliorate bias in technologies that are based on machine learning and that sort of thing if as Alvin mentions a lot of times tech procurement is driven by cost savings as the primary or at least a primary metric how do we educate people how do we get them thinking in a mindset that they really need to do an ethical impact assessment or something on some of these technologies that they're building is that something that's possible to do I think it's possible or at least I think that we have to believe it's possible and just strive for it and I mean that with no irony I really truly believe that we just have to believe that it's possible and think of all the ways we can impact that and that is I'm working on this thing called the responsible computer science challenge with Mozilla and Omidyar we have about three and a half million dollars to fund computer science programs so computer science is my background computer science programs to integrate social responsibility whatever that means for each entity into their curriculum and so to answer your question I think it starts at the people building the software the company just slipped my mind the company that sells the technologies to the police to the court systems yes, yeah it's like a group of data scientists who developed and they developed it with I'm sure wonderful great intentions but it's like used in different ways and they actually didn't take into consideration probably all the bias and unintended consequences so how do we how do we tackle that part of it so people building the technology so regardless of how good or bad people is able to buy the technology how do we retool and rethink about how we train the people who actually build the technology in my own computer science education there was like one ethics class and it was like a joke class about like scenario A and B is this good or bad and then you're basically done right or like company A did this and company B did this like which one was good or bad and it's like the class you take when you graduate and it was a joke so now let's rethink about how we build the technologies and then the second part of that was cool now maybe we have technologies that were built with some of this in mind like you said a lot of times even a joke in government the lowest bid is going to win right like proposals are complicated they're long sometimes they're written by like fresh out of college grads from these consulting companies and they're just like these bohemist documents and sometimes they're like either relationships prevail or the cheapest one wins and then once you buy it so we're bad at as a government we're bad at buying this technology and then we're bad at actually managing the projects so we have to get better at buying this technology which is a thing that is the people are actively working on now they're trying to change procurement in the government and then also thinking of ways to bring an expert in the room so that when we're buying a piece of let's say a piece of software for a court system there's someone who can actually look at it and say this works or doesn't work U.S.S. was only a team of 200 we've done a bunch of proposal vetting and just called out companies and like this actually doesn't do what either you said it would or you think it does this thing but it actually really doesn't it's going to cause a lot more problems or you're planning on shipping this in two months and it's going to fail and background checks are just still not going to happen for people and it sometimes just takes a few people in the room who understand how tech works so I think all that has to happen but there are people working at it from different angles and I just I feel like you have to just believe that it could happen and figure out how to put the right people there absolutely I think having people in the room who actually understand it is crucial and you know I was a few months ago during the NATO summit we had this panel we were talking about the blessings and curses of technology and a lot of the focus in the conversation was about artificial intelligence and obviously there are a lot of blessings that come with AI I mean you lead our ethics working group at the Berkman Center for Ethics on AI right? Ethics and tech and so there are a lot of conversations that happen in that room different things like you know positive for example from an attorney's perspective when you have a huge case load you can definitely lean on artificial intelligence and automation to essentially with pretty good accuracy tell you which cases you should probably take on that saves a lot of time, not positive but on the other side of things if you go into a courtroom and you now have artificial intelligence essentially determining whether somebody is guilty or not guilty then you have a lot of questions around ethics because at that point maybe the judge trusts the algorithm and believes that is subjective but what a lot of people fail to realize or sometimes neglect is that those algorithms were also created by human beings so in a way those are still subjective your hope is that they are the least subjective I suppose and then you go on to the other side of artificial intelligence kind of the dark side which is things like deep fakes how many people in the room know what deep fakes are show of hands ok not as many as I would hope so deep fakes and you all have probably heard of this at some point or maybe seen something like this it is essentially taking advantage of deep learning to transplant people's faces onto other people's bodies or even using their voices to essentially mimic somebody else so you might have a video of a politician saying something and it looks very believable it even sounds like them but it's not actually them or you might have revenge pornography being used through deep fakes this is very troubling this is criminal behavior taking advantage of really advanced technology where we usually see this technology used is in things like Star Wars in Hollywood where you have someone coming back and her face is on someone else and everyone is so excited because we're able to bring back actors from back in the day and it looks so real but there is always another side to it which can be used for dark and nefarious purposes and I think when it comes to these things it's so important to have experts in the room when you're making these decisions and for us even maybe as a global community to leverage our resources and best practices to determine maybe you know whether there is a set of guidelines that we should follow when we're trying to essentially cure the negative effects associated with new and emerging technologies great I'd love to get some voices in from the room anyone have questions, thoughts, concerns they want to share with our illustrious panel we've got Ruben playing the Phil Donahue role really interesting talk but I'd like to push back hard on one thing you just said you know with respect to the way to show something is bipartisan is by showing that it saves money and I'm first a personal confessor and I know about wasting government IT money I could tell you about the system I wrote in the 1970's for the state government that never saw the light of day because of you know an adjacent scandal so many of those cases right and you know the shame was it was an adjacent scandal in itself should have been a scandal but never mind I mean there are two ways I would push back on that first there was a flip in the way enterprises have viewed IT in the 90's where they went from thinking about it as cost saving to something that would save costs and you should save costs on to something in which you should invest and so I sort of just in that large scope I would push back on to it but the second way I would push back on to it is how that sort of puts an implicit bias into what something like the digital service might choose to work on you know there are many problems you know in the government that would be fixed if we spent more money and you know whether that's food stamps or whatever they're being called now you know or health benefits or whatever so how do you square that circle how can you or I'm going to ask this internally in the government does that really bias you to the certain projects that are in this narrow consensus of oh we can't spend more tax dollars you know or you know do you push back on that and do projects that might actually you know need to spend more dollars but work for the greater good let's go first I actually don't think and maybe I actually don't think bipartisan means less money so maybe that I actually don't think I said that so I actually don't believe that bipartisan means less money and it happens to be a talking point where okay one argument for bringing in tech talent is that sometimes having in-house tech talent might be better than hiring a consulting company for example but that's just like one small part of the argument and isn't the ultimate argument nor is it the decision on which projects to take on the USDS model specifically is a model that brings in people who are federal employees employed by the federal government who can go around and work on different projects versus going out and hiring let's say like an IBM or Northrop or any big company to build software in-house so it's just a bit of a different model it's like in-house I guess work versus hiring external talent but I definitely don't think that those two are directly proportional where all the decisions are made simply on what can save money period I actually think that would be a horrible metric and I would love to actually chat more about your thoughts on cases where more money spent on food stamps actually is better there's a lot of good work being done in the state of California on CalFresh and re-emping that entire system both saving money and increasing the user experience and finding more people who should be on services who are not on services and they found ways to just make the process more efficient versus throwing more money at the CalFresh system and they actually sued a contractor I'm sure like this happens all the time the contractors are sued they bid again and they still get the same contract again or something but the state of California actually sued a contractor because they failed and they actually were able to build something with less resources and funding that actually worked so just kind of a roundabout answer to your question I'll piggyback on that too and I'd say that I think that saving money is one of many factors I think depending on the product itself is a pretty hard factor but I would revise it to say that it's not necessarily about saving money but I would replace money with value and that as long as it's generating value I think it's important for what when you're investing and I think part of your premise was we should be investing more which I think based on your hypothesis investing is exclusive and saving money are kind of mutually exclusive I don't think that's necessarily the case I think that whenever you're investing you want to be able to generate some sort of return on that investment that doesn't necessarily have to be in the form of hard cash and taxpayer dollars but it can be some sort of value generated for taxpayers and as long as the value is being created I think that's usually kind of the calculus with which we approach a lot of the problems that we're trying to solve Other questions? I was wondering if you could speak more about the policing of potentially criminal activities regarding fringe technologies so I know you briefly mentioned Deepface but it would be great to learn more about what the government is currently doing here and ways in which if people are interested in the audience be it graduate students or young professionals could potentially get involved in helping the government Thank you I don't have any specific examples about government working on those particular things and again based on the fact that I'm speaking of my capacity as an affiliate here rather than the government but what I would say is as far as opportunities are concerned there are a lot of opportunities to work in agencies that are working on emerging technologies One of the agencies that works on a lot of emerging technology is the General Services Administration and so they have a lot of teams including 18F and others that are very open to having students come on for whether it's an internship or some sort of pathways program to start working on these different issues and their various teams are some of them are very ahead of the curve because we rely on them to essentially tell us whether we should be considering various things as we're kind of vetting emerging technologies same goes with tech teams within the White House the Office of American Innovation the Office of Science and Technology Policy the CTO's office, our office the Office of the CIO all of us are always trying to figure out specifically where technology is headed how we can get ahead of it if possible how we can prepare for it so that we are preemptively doing that preparation rather than trying to play catch up retroactively it is very difficult because unlike the private sector we do for better or for worse, and I think this is part of our discussion too we are accountable to 300 million Americans so it's not just our shareholders or just our board that we have to answer to so we can't really move fast and break things so to speak and I think that's a good thing I think that it's good and bad depending on what you're trying to achieve but overall I think it's net positive because it would be very irresponsible for us to move so fast that we're not taking everything into consideration and then as a result basically charging the taxpayers for our learning curve I think that would be very responsible from the perspective of government what that means is that we should be working more with the private sector to potentially take some of those risks in conjunction with our advice and potentially have more of these public-private sector partnerships that I think are absolutely invaluable so that we can ensure that everybody is at the table so we're not really solutioning in a vacuum here and with those emerging technologies it's absolutely critical but as far as opportunities for students are concerned there are so many of them and I think it comes down to awareness so if you are interested please come on over afterward and I'm happy to share more one resource to look up just because they're local to Cambridge is an assembly Sir Berkman Klein in MIT Media Lab had an assembly project last year called AI Policy Pulse and a couple of people on the team included Jack Clark from OpenAI, Gretchen Green and Amy Zing who are both fellows or affiliates just here at the Berkman Center and between all of them they have very deep expertise in engineering and law and wood making and open button policies and a couple other things but they spent a lot of time interviewing mayors and state legislators and I think other government officials on what kind of advising they need around just AI policy in general I think Gretchen is advising the government in Canada right now but they're doing a lot of really interesting work around ways to get in the door for various levels of government to advise them on making decisions around different types of technologies beyond I think starting with AI but beyond Sasha do you want to jump in? Thanks for a great talk this is a really interesting conversation my question is so we recently completed and launched this report called More Than Code where we interviewed over 100 technology practitioners who are working on the public interest and using different languages to talk about that one of the key findings of that report is nothing about us without us which is that people with lived experience of the domain area that you want to work on need to be at the table as part of the technology development process the whole time and another one of the key findings is basically the title of the report which is More Than Code which is that your team doesn't just need software developers you need a bunch of different types of roles my question is in the context of working inside say digital services you're working inside a government agency you're working for the state whether it's municipal or federal or state I'm wondering about the tension between the reality that people who are marginalized are experiencing you know the worst in any particular domain area so like people trying to get access to housing in the Boston housing market which is crazy and they can't get access because they're low income and it's a tight housing market and there's all this so people in those situations always have organizations that have a lot of domain knowledge about what's going on and what needs to be done to solve it but there's sort of this barrier between a nonpartisan government agency that's working to improve user interface design for access to services say the fact that the people with the most domain expertise are going to be already sort of organized and advocating for particular for particular policies so I'd love to hear you sort of reflect on that tension like how does that work we know we need to have people with lots of deep lived experience and domain expertise on the design team but if you're working from the position inside government it can be politically difficult to say well we know that we need to have someone from the tenants rights coalition at the table as we're designing our interface for access to section 8 vouchers or something like that so how do you deal with that and give some concrete examples I love this topic so much and if you haven't looked at more than code please go look up that work it's really really fascinating I am wholeheartedly excited about we have to bring people in the room with lived experiences it's actually another facet where private sector also is notoriously bad for it government is so in many cases far behind I've been told multiple times that user experience studies are illegal meaning it's not ok to go talk to your community and that sounds silly but it comes down to something called the paperwork reduction act which was well intended to make it so that government can't keep asking for more and over and over again and it gets morphed in weird policy circles and product teams until you can't go talk to people so this idea of just going out and talking to the people who might need section 8 vouchers or talking to the people who are waiting like 5 hours in social service offices while they have a minimum wage job those are things that for the most part no one does unless there just happens to be a team on the ground or like if you went and worked for the federal government you're like hey guys I have a great idea I'm going to go and sit with these groups it doesn't matter if I totally disagree with them politically it doesn't matter if they're totally right totally left because like you said housing topics are political in many cases food stamps can get very political some states don't want people to have more access to food stamps like there's all these questions we should ask but I think a starting point is getting teams on the ground that are like you know what I'm going to go if I'm working for the veterans affairs I'm going to go and sit and talk with some of these veterans service organizations I'm just going to go and look at and see what happens so we can go back to the department of health and like eight different VA hospitals and just see what people are doing and then watch doctors try and use the VA's healthcare system and it seems kind of simple but it just doesn't happen so there are teams that have been doing that now there are teams at the department of Ed who one of my favorite stories was a contract out for several years trying to launch this thing that tried it's not Obama have this initiative to open up And this team led by Lisa Galopter and Erie Meyer and a couple of other folks came in and were able to launch it in a few months over the summer. And their user experience testing was they printed out these mock-ups on paper that they drew, stood on the Washington Mall in front of the Smithsonian Museums and asked students what they thought of the interface and brought students in that way. But the Department of Ed just didn't naturally go and talk to students, even though it's catering to students. Or you might have people that are like, well, kids who are going to go off to college. So I know what they need. So I think the starting point really is just, yes, there are groups that are going to be political, but have product or engineering or design teams that know. You know what? This is like a typical product problem in general. There have been people who disagree with you always, who either love your product, hate your product, whatever. Bring them all in. Get all their opinions and figure out what to build. And you personally, not you, but like the engineer, the development team may personally have its own agenda for what it wants to build, but at least bring those perspectives to the table. We're not even there yet in government. I don't know, found a lever to broaden a process across many different agencies. Is there something similar that could be done with user testing, user involvement in the design process, highlighting, and ultimately, even requiring it, but sneaking it in there? I don't know. Yes, there are. And agencies like to copy each other. So for example, the Department of Defense did the hack the Pentagon bug bounty thing, which made it so that anyone can hack the Pentagon and find vulnerabilities, which is pretty common for any tech company. Really Google it as all the time. And then a bunch of other agencies are now starting to copy them. So it's really kind of showing a model that works for an agency and then having people really just from the grounds up out there hustling and being like, hey, this worked over here. You should do it too. And I really think it boils down to having just a few individuals in the house who can do it, like having one Alvin or one like a few people from different digital services and just like being out there yelling about this constantly and then people listen. We look forward to someday having UX.gov. There's actually, this is not quite the same. But there's, the USC has launched the US design standards, which is like a starting point. It's not even close to being UX.gov. But we definitely should have something like that. Hi. I was wondering that you describe the people going out and doing some form of what you might call participatory design or user experience, information gathering. What kinds of backgrounds do they tend to have? Because my experience is that this is the sort of work that doesn't come most naturally to computer scientists. And if they're not computer scientists, then how do you manage the interface between social anthropologists, for example, who might be employed in this role, and the kinds of accounts they give of what they found and how this translates back into actual requirements that can be implemented? Yeah. I laugh because you're absolutely right. Computer scientists are not trained to do that. So one of the biggest, I think, one of the biggest barriers along this is just the hiring process for all of this. How do we hire in for a UX role or an anthropology role or an ethnographer role, et cetera? And in many cases, there isn't that role to hire into in any government. There might be some governments that have thought of it, but for the most part, there aren't. Specifically, USCIS, we made it so that we could hire people with those backgrounds. So we have a design team or had, I left in March, of about at least 50 folks with that kind of background. People who are very well-versed in ethnographic studies, who are well-versed in focus groups and survey, in any number of ways to question individuals and communities to try and understand what they truly, really need. So we found a way to hire those people in-house, recognizing that you can't just throw an engineer out there and they'll just magically know how to get all the information you need, nor are they experts in doing that. I know that's not the greatest answer because it doesn't mean we've solved all the problems, but that's one way we've found to do the work. And I think this plagues a lot of private sector startups too, which is that leadership and designers typically assume that the types of people that they talk to or the types of people that they should be talking to. And so the issue is, the lazy way to do it is to reach out to your users and then say, hey, let's have a user testing session. We wanna know exactly how you've enjoyed the experience of this platform. Let's walk you through it and all that stuff. But you cut out a huge population, a segment of the population, because the people who are your users are already there. The people who aren't there, a lot of times are the people you want to come. So your users, you wanna target the person who came once and left immediately, but you'll never be able to find that person if you're just focusing on your users currently. So I think it takes a lot of progressive thinking. I think, like I said, it's not just a public sector problem, but private sector as well. You need to figure out how you can reach out to not just your own networks, but leverage other people's networks too and challenge yourself to be a little bit uncomfortable to find people who you would never speak with because those are the people who will teach you something that you never knew. Can I ask a follow-up question? Do you think that there are implications for how we train computer scientists? Implications for what, I'm sorry? Do you think there's implications for how we train computer scientists, computer science education? Should computer scientists be inculturated to be more comfortable going out and talking to people, having the means to do this, having some sort of methodology that is not currently part of many computer scientists' curriculum? So I think what I mentioned earlier is one of the initiatives I'm working on is to think about how we train, change the way we change computer scientists. I think the reality is sometimes people go to computer science for a certain thing and depending on personality, sometimes some people are just not great at those kinds of skill sets. I think the least we can do though is make sure what's ingrained in that culture is the recognition that the skill set is incredibly valuable. It's just as valuable as understanding data structures. It's just as valuable as understanding algorithms, theory, et cetera. The idea that you have to bring in the human component, the lived experiences, the user experience studies, and whether or not they wanna tackle that as their core competencies is up for debate, but at least making it known that this has to be part of how you develop software as well, whether it's you or you bring in someone who knows it. Because right now it's seen as a soft, fake, not real thing to many computer science curriculum, candidly. So how do we change that? Yes, we have to change how we train computer scientists. And I think that's a huge opportunity. Baking anything into a computer science curriculum, I think, is one of the best ways to really train folks early on instead of trying to correct improper behavior. We're even doing that with open source now where we're working with various schools to figure out how we can even bake something like code.gov into their curriculum to ensure that they're from the very beginning understanding how valuable open source software is. I mean, the internet potentially was arguably built on the backs of open source technologists who shared just this wealth of information that was completely basically decentralized across the country and across the world. And to not give something like sharing and collaboration through open source software, the credit that it deserves is irresponsible. And I think a lot of times computer science curriculum, they focus on how can you build something that a bunch of people will use and pay for. And it comes at the expense of how can you leverage this incredible network of computer scientists and others across the world to not only learn but to give back so that you can essentially take advantage of a diversity of perspectives. And I think that comes back to your question as well. Making it in early is important. Something I want to add but also kind of counter a little bit is I feel like the computer science curriculum teaches people how to build things a lot of people will use. And there's definitely this mentality that I'm going to go change the world. I'm going to use software and it's going to do so much good. And actually the money part is kind of, if you actually ask most Google engineers how much revenue Google brings in, they'll probably look at you with a total blank stare. Like the money part is kind of on the side. It's this general and deep belief that you're changing the world through technology when you're actually sometimes building things that are not ideal because you don't bring in the user perspective with the communities, et cetera. So how do we change that mentality? Two more here and here. Hi, my name is Uli. I'm one of the Neiman Fellows, meaning I'm a journalist and I'm working with the data team. So I'm very much in favor of the idea of publishing code. You're also publishing our code to be more transparent, how we get to our results. But we made the experience that not a lot of people reuse it because it's always very specific. So I'd be interested in how many private firms really reuse the code on DataGov. And is it more perhaps for the agencies to know or track better how they are acting and not repeating work? Yes, so thank you for the question. I think the primary kind of impetus for Code.gov in general was to figure out how we can take the corpus of code that we already have across the government and make sure that we're not spending money on duplicative code. And so that basically translates to potentially billions of dollars of savings. I think the second part of that is figuring out how we can train these agencies to release code that is useful to the public. And a lot of that comes down to good documentation, easy discovery, and also this concept of not giving the agencies all the say in what to release because a lot of folks don't actually know what others find valuable. But having this baseline requirement where you should feel encouraged to release everything and then let the community determine which ones are more popular or more useful. And then once you get that signal then potentially focus more of your energy and resources on those, maybe even have those be part of some sort of hack day or hackathon with the government and the public to make those a little bit stronger so that a lot of people can take advantage of it. Because we all know that not every single piece of code is universally applicable or useful, but at the same time there are certain sets that are very useful for government purposes, there are certain sets that would be very useful to the public, and then there are certain sets that would just be very interesting just from a transparency perspective. For example, NASA has open sourced one of its mission control frameworks. That's amazing. It doesn't necessarily mean you're gonna use it, but just to be able to go into NASA's mission control framework, look at the code, you can even fork it and reuse it as a data visualization tool. I mean, that's pretty cool. But if they were to say, how many people are actually gonna use this? And the answer was potentially not that many folks in the public. That might have resulted in them not releasing it in the first place, which I think would be a shame. So I think there are a lot of different public interests and one of many would be whether the public would use it or not. And are you even tracking if the code is to be used by private firms? Absolutely. Thankfully, with different platforms like GitHub and others, we're able to work with their API to really figure out how many of these different projects are being reused for and for what purposes. And then through regular check-ins with our agencies, we're able to ask them whether they are using other agencies' code and whether other agencies have used their code. So through that cross-referencing, we're able to come up with a more accurate set. Can you say how much is it, approximately? How much is what? How much projects have been reused? How many have been reused? So right now, we are in the process of calculating that. It's difficult when you have over 4,000 projects that have been released. Also, the program in and of itself is two years old. In the private sector, that's an eternity. In the government, that's a little baby. And so I think we are still in the process where we are putting in a lot of training and resources to our agencies to ensure that they are sharing their code responsibly. And once they really understand how to do that and provide proper documentation, et cetera, then I think the next phase of maturity is really making sure that we calculate those numbers as well. A small ad and maybe a fun side project for you is it's always interesting to see how creative people tend to use policies. So opendata.gov has been around for a while. And I think the wording was something like machine readable format. And there are literally things like JPEG image files of spreadsheets or something that are uploaded in the open data repository. You're like, what am I doing with a photo of a spreadsheet? But it's like machine readable. And so it'll be interesting actually to see if there are weird uses like that for the code as well. Absolutely. Time for one more. Yeah, thank you for the talk. That was really interesting. I was wondering, like at Berkman and at Digital HKS, there's a lot of talk about public interest technology, but at the engineering school, there's relatively little to none. And I was wondering what you thought needs to be done at Harvard or at universities in general to help solve that talent problem of people going into public interest areas, government or government adjacent? This is a very big topic of conversation around these parts right now. Well, it's one of the reasons I came to Berkman. I wanted to think more about this. So I'm like, why are my people not coming into the space? Harvard Seas is thinking about this for sure. People like Jim Waldo and Barbara Gross and the provost, I guess, of Harvard. And then Bruce Schneier also just released something on public interest geared towards technologists. So it's in the infant stages. If there's something you're interested in, like this is the time to like dive in and find all these people and figure out how to help build it out. You're right, it's just not an area that is the highest priority right now. Stanford just, they're gonna start a new course next. Again, these are all like elite schools, but like Stanford is starting a new course either next spring or fall that has the three professors in engineering, policy, and philosophy, teaching a class that will replace one of their core classes, hopefully trying to steer Silicon Valley in a different way. So like a bunch of things popping up. Ethics is like another kind of gateway into that a little bit, but all these things are like literally starting right now. I think ultimately it comes down to the schools themselves and the professors. I think a lot of professors and Chris, you can correct us if we're wrong, but a lot of them don't actually talk about this enough and they're not telling their students that these are even opportunities worth considering. At Harvard, there were a couple of undergrads who created something called coding it forward within the last year or so. And I think programs like that or a good opportunity, that program specifically basically matches computer science undergrads with federal agency. So they can go and do an internship within a federal agency, work on a tech platform and it be exposed to the fact that there are incredibly meaningful projects you can work on outside of the private sector. So I think if we have more programs like that, if we focus on the professors themselves and the academic programs, putting a premium on the government as yet another potential career path that is certainly worth exploring, we can definitely kind of catalyze that sort of culture shift too, but I think it starts from the top and then trickles down. The only thing I'd add to that is just say that academic institutions in general tend to be very siloed institutions and these kinds of considerations aren't necessarily interdisciplinary. And so I think there's a lot of work going on now. I can speak only to Harvard to try and bridge some of those gaps through people like Barbara Gross and others who are doing really interesting work. We have this TechTopia program that's getting off the ground now which is an interdisciplinary program with a cohort drawn from schools around Harvard, the assembly program you mentioned. So interdisciplinarity seems to be the key and breaking down some of the walls that have traditionally stood between institutions. And I mean, Chris, I think you can attest to this too. We worked on a project together in the cyber law clinic and we had a call where we had a bunch of different federal agencies on the line with an individual from the law school, a couple of students, and they walked away with it thinking, what? I mean, what? They were just blown away. Oh, absolutely, yeah. From a student perspective, this is a no brainer. The students recognize coming in that this is the future and this is the way we have to operate. It's getting institutions to function in this way, for sure. Yeah, absolutely, yeah. Join me in thanking Alvin and Kathy and go out and vote if you haven't already. Thanks so much, guys.