 Welcome to everyone. I'm so pleased to be moderating this panel on teaching public interest technology through a multidisciplinary lens. My name is Deirdre Mulligan. I'm a professor at the School of Information at UC Berkeley. And we have a terrific set of panelists today to talk about their work in a diverse range of institutions. Exploring the interdisciplinary nature of PIT training and education. So we're going to hear from about four different projects that are introducing theories and practices from different disciplines. For me, interdisciplinary teaching and research is both the thing that keeps me going every day and one of the most difficult things I've ever done. And I think that we're going to get some really terrific insights from our panelists today about both their triumphs and their struggles. I'm going to really briefly introduce everyone so we know who's in the room and then we're going to jump in and have, I'm sure, a very animated conversation. So, because Paul's here, Paul Ohm first, is a professor of law at Georgetown University Law Center, one of my alma maters. He specializes in information privacy, computer crime law, intellectual property and criminal procedure. And he teaches courses in all these topics and more as the faculty director for the Center on Privacy and Technology. Lydia Chilton, assistant professor of computer science at Columbia. And her current research is in computational design, how computation and AI can help people with design innovation and creative problem solving. Christopher Gronson is a distinguished service professor at Carnegie Mellon. Chris teaches classes and geo information spatial. Damn, I missed that up. Data visualization and at the intersection of design innovation and analytics. Very importantly, Chris was previously at 18 F and office within the General Services Administration started during the Obama Administration, managing digital services consulting strategy and design projects for the federal government, which really provides him some novel insight into some of the work that we do here. Next, me here is sheer Sharga is the technology and policy clinic director at Princeton. He runs Princeton Center, it's the first of its kind interdisciplinary technology policy clinic that gives students and scholars an opportunity to engage directly in the policy process. And he came from active practice and consumer protection and tech policy at the state AG's office. And next Stacy dogan a professor of law at the University School of Boston. And Stacy is a leading scholar and intellectual property competition and technology law who's been really instrumental in building interdisciplinary and instant inter institutional collaboration the areas of law technology and entrepreneurship, and has been teaching interdisciplinary class with some colleagues and computer science or an algorithms in the law. And as you can see we have folks coming from a diverse set of backgrounds, even within the Academy but then also with some really interesting outside experiences that I think really inform the way in which they think about the importance of interdisciplinary work. There's nothing like trying to do this work out in the wild to drive home the need for collaboration and interdisciplinary perspectives. So I want to jump in and start with me here. And say, you know, why are you teaching the course that you're teaching. Why did you identify the need for technologists to inform policy debates what is it that that really drives the work you're doing. Thank you. That's that's a great question. So I, you know, my background is as a prosecutor and I was litigating a big case against internet service providers in New York. And I discovered that a lot of the kind of fundamental technology questions were open. There were things that were hard to understand that you needed expertise of different kinds to be able to help answer these difficult problems of how do you ensure that there's access to broadband how do you ensure that companies are being truthful about what they're saying. And so when an opportunity came up at Princeton CITP to run a clinic with computer scientists and help them and other interdisciplinary people address these problems and learn together. So that was a great opportunity. I decided to jump at it and working with Ed Felton, who's the founding director of CITP, he was the one who sort of conceived of this idea. It's been a wonderful ride of working with students in individualized projects. I really enjoyed that. So Chris, same question to you. Why do you think interdisciplinarity is so important to the work that you did at 18 F and the teaching that you're doing now. Yeah, no, I think that's a great question. I think I think interdisciplinary students or people with different backgrounds is really critical because if we're talking about public interest technology we need to be thinking about how digital services and other ways in which government engages the public is used by a variety of different people. And so what I found in my classes is that the most creative ideas often come from those with diverse perspectives. And we're fortunate that we have at Heinz College, both a public policy school as well as an IS school, but we also get students from the business school and the design school. And a lot of our work has to do with, you know, really doing a lot of user research so they have to interact with people that are expected to use government services so that will take disciplinary lens really lends a hand I think to making sure that we're, you know, as much as possible seeing the problem we're trying to solve from a lot of different angles. So at the risk of looking like I'm meeting with the men, I actually am going to ask Paul next. And it's not because I'm leading with the men but it's because Paul is going to round us out with his perspective from the Federal Trade Commission and I think a bit of his connection to OSTP and from that policy lens, why do you think interdisciplinarity is so important to building the public interest technology field. Yeah, I spent a year as a senior policy advisor at the Federal Trade Commission I did interact a lot with OSTP I also spent four years at the Justice Department as a computer crime prosecutor so similar to me here's background I mean the work of the Federal Trade Commission in particular is all about new technology and technology platforms and just pick up a headline from the last, you know, 10 years and you'll see that born out. When I was there I had the privilege to work alongside two different chief technologists which was still a little bit of an innovation at the agency, and one that continues to today although a little bit in disrepair under the current administration. And it's just amazing how a little bit of technical interdisciplinary engagement still goes a long way in the world I mean one thing I tell my law students I get law students at Georgetown who don't have a technical background. I say you know the world doesn't have enough people with formal training. And so we need to find ways to give you a little bit of interdisciplinary engagement, and you'll be surprised how many opportunities will open for you. I hunger for the world 10 years from now after all of our collective efforts work out where my law students are actually not going to be able to get those opportunities, honestly because we're going to bring together subject matter experts with the lawyers with everyone else. We have to like the world is fundamentally broken. And this is one small way we can take a step toward fixing it. So, Lydia as the most junior among the group here. You know you're obviously part of the hope for the future of building this field as an assistant professor. And what is it that you know obviously I think is academics we all know there's some risk in doing interdisciplinary work right there's a lot of encouragement to kind of stay in your lane. And I'm wondering what has brought you to center interdisciplinary teaching and research in your work. Oh, yeah, great question. So, what I fundamentally teaches computer science students how to make apps, the website so less that people will use and will want to use and there's an art to this. It involves the human centered design process but rather just thinking of something and just building it and then see if it works you test things along the way you come up with a lot of ideas that anyway this is a general process there but there's a general dissatisfaction although that process works well. There's a criticism of Silicon Valley which I think it is true is that it's, if you look at things that are being designed in the tech world as people who are in the tech world. I understand that there's a broader set of people that we can and should be designing for, and I realized that my students, who largely just had computer science backgrounds. They, they needed some help with that. And I think it's just there's more opportunities we've done an awful lot of the low hanging fruit and just like, Oh, take something that happens in the real world, make an app for that like social networks, that stuff will be done and there's, there's, but there's, there's much more out there and so we wanted to figure out like, how can we find other people who know some of that so we partnered with journalists and urban designers of architects who often work within communities and for communities and then had to adjust the human centered design process so that we could make use of that expertise. So that's what I'm excited about. I think maybe the same as others. There is this huge gap between tech and largely non tech and they both benefit from being able to cross the wires. So that brings me to you Stacy and law schools are kind of notorious for being kind of these silos unto themselves we don't generally have PhD students we don't have postdocs our students are kind of like laser focused on acquiring the knowledge they need to be lawyers and so are a little less likely to venture off and take courses and other departments. And yet you have been really actively developing some of these interdisciplinary classes, both in your role as an individual faculty but I think also in your role as dean for academic affairs. And I wonder if you could talk about why you view it so as so important to the public interest technology field and to kind of the vision of a law school of the future. You're muted. I had to be the one to do that huh. So my thoughts about this pickup I think where Paul left off I think we're hoping that our students are going to go off and become leaders in as policymakers as regulators as as lawyers. And in order to do so effectively, our law students absolutely need to understand the technology that they will be engaging with and that and the developers of that technology so one of our principal goals really is to increase the fluency across the province to increase the ability of lawyers and policymakers and regulators to engage with talk to and understand the developers of technology that they are both regulating but also using as tools to address social challenges. The fluency has to go the other way as well. And we, I think as a society and we've seen this over the last couple of days in these panels. We've been fetishizing a certain type of smarts in technology right better faster, you know, quicker smarter right. But the people who are studying computer science and other fields of technology in our society don't necessarily have a sensitivity to the ethical legal and regulatory environment in which they're innovating. And so part of our goal is to increase the communication across fields not just between law and computer science but also to get our philosophers involved our ethicist involved in these conversations so that we're educating these interdisciplinary communities of people who are interested in the whole range of challenges that technology can help to address but also can help to confront us with. So one of the things that there seems to be quite a bit of commonality is the use of experiential learning as a place to help people kind of develop these multiple fluencies or to get some experience both collaborating with people from other disciplines and to gain some knowledge from other disciplines. So I'm wondering if each of you and we're going to start with Chris could talk a little bit about the kind of methods you use in your interdisciplinary effort and so I know it includes experiential learning, but my guess is it also includes design methodologies and other things and kind of what's the connection between the method and the sort of interdisciplinary fluency you're trying to develop. Yeah, sure so so in our class the policy innovation lab. Well first it's a short class it's only seven weeks and the goal that we really explore with the students is how, how do you identify a really big challenge and figure out small pieces of that challenge that then you can start to fix through the identification of the problem you're trying to do. merge a lot of methodologies. We use an agile methodology to help the students figure out how to set up and structure their teamwork. We use design thinking and human centered design to help them always think about who the government product or service that they're trying to design, ultimately benefits and what the impact that service has. And then we use other prototyping methods and deployment methods to think about how to create a real tangible impactful method so for us really the goal is showing them how all of these various pieces work together to ultimately solve a problem and that's a pretty. I think critical learning objective of the course is to really through that experience working side by side with an actual government partner understanding how they can actually make a meaningful impact in government through the combination of these multi disciplinary practices. So here I'd love you to pick up on this question and particular you're a lawyer you come from you know a practice and you're sitting at Princeton which doesn't have a law school. And the clinic actually has a history of both having lawyers as visiting fellows and also having them lead the clinic. So I'm wondering if you can talk a little bit about like what's the teaching methodology there and what's important about the legal skills you bring into that non legal environment. And I should just clarify that there's the center which has existed for a long time the clinic is a new project which has started last year with me. But you know one of the key things I came to Princeton because it didn't have a law school right the two me lawyers in technology to Paul's point you know when you're a litigator and you and you investigate a company. And you say where did the company go wrong right and you don't want to get after them when it's all over. You want people when they're making these decisions to have the ability to think about the impact that decisions are having on our communities and for them to have that responsibility to actually engage in the policy process and what you see is that students computer science students feel inhibited because they feel that there's this strange language of law and regulation and we can't break that down we don't have a voice there. So one of my key things is to try to bring that in and bring projects around specific policy problems, so that students understand that they actually do have a voice that they have something to contribute that they have the ability to make a difference and I'll illustrate briefly one one thing that I'm doing is we have this case studies section where we bring in a real world policy problem and we bring faculty, graduate students fellows postdocs there's no hierarchy it's sort of everyone coming in, and we spend an intensive session just understanding the problem and then in the second part of this proposing solutions to the policy problem and then if there's something more to go on. We develop further projects on it so this is a way to get people in very deep I think sometimes in interdisciplinary work we try to show the breadth of everything there is and that's just, you know, then they basically become lawyers which is not a good thing. They have to be you know they have to understand that how to have a voice in the particular area of expertise that they're good in and that that's one of the goals. So Lydia I'm wondering in your teaching. Do you have is it problem centered or client centered. You know we've been hearing about clinics which often are very client centered. What approach do you take to kind of center people on as you were talking about kind of the human problem oriented and what do you do there. Yeah, so you've actually run the class twice and we learned a lot from from the first version. One thing that we did well is pick a. We try not to use the word client we don't like any of that sort of capitalist language and yet we haven't quite found anything different but like a focus area just like we hear said you, there is this tendency to go broad and there are so many problems but how do you solve a lot of problems and how do you solve one well and then you knock down the other dominoes and we actually really struggled to find a focus area because none of us the architect the journalistic computer scientists didn't already have like an area where they were doing this and so public libraries. I think someone had just read Eric Kleinberg's book about eight libraries are palaces the people. I was like, Okay, let's do that and you know New York has a lot of libraries and has so we found them partners within the Queens and Bronx public libraries to work with that could help answer questions give guidance give some of that depth the students a lot of reading to get themselves familiar because no one was an expert on libraries going into this although everyone had some amount of personal experience with it so it wasn't quite like the law where city average computer scientists has no concept like just don't break them. The idea but anyway we had to they had to deepen that but finding an area that we could apply technology to that was already in the public interest was I think critical to us to having helping students develop public interest technology that was probably our biggest challenges of what is this new term that just got invented. How do we do what is it what isn't it can it involve government kind of doesn't have to be you know mutual aid and so we're like hey let's partner with someone who we believe is already doing an awful lot of work in the public interest that said libraries do not have a perfect history they've definitely discriminated in the past before. They're probably still doing it someone in the future but but by and large they have a. Pretty good reputation and do still do a lot of good and it's a place where technology has not penetrated as deeply as it could without there's probably a lot of opportunities where we could help. Here so yes working concretely but still we had to define those those problems I think we didn't go in with a particular case study or particular person or problem which is shyed away from some of those because we kind of didn't really like a lot of the things that were being surfaced to us. By by people in the space a lot of those problems where we built this thing and we don't know no one's using it can you help and I'm like that's not exactly where we want to go we want to make sure that we have a problem and a group of users. That's specific and solvable. So so that's a restarted and libraries are nice because they're already basically about information technology parks and recreation I think would have been harder there's just less of an entry point there. But we can now I think identify more fields where we could go in build up that expertise and find PIT opportunities. So Stacy and Paul I'm going to start with Stacy and the BU law and MIT Arrangement is interesting to me in that it kind of keeps lawyers and technologists in their traditional roles right the law students are offering advice. The technical students are providing the innovation and I'm wondering if you could talk a little bit about the strengths and limitations of that particular approach. Why you know what what do you think is useful from the public interest technology field building and what do you think are some of the limitations. Yeah. Now it's a good question and it actually allows me to reflect on Some of the similarities and differences between that context and some of the other contexts in which we've been doing interdisciplinary work across on technology. So the the partnership with MIT consists of two clinics a technology law clinic and a startup law clinic in which our law students under the supervision of lawyers, clinical instructors Provide legal advice to student innovators at both BU and MIT and the the the virtue of this I mean there are many virtues but one of the virtues is that The student innovators before they go about doing their potentially risky Activity can kind of come to us and get counseling about how to proceed in a way that minimizes risks. So there are legal risks associated with data scraping that with Encryption research, you know, they're all sorts of Legal challenges that historically MIT students had found themselves in hot water about and so it's empowering in many ways because the clinic allows them to get counseling before they go ahead and And do their work and try to In some cases work with the potential target of the research To collaborate with them to move forward in a way that promotes knowledge without subjecting the students to To legal action but you're right. It does preserve this kind of lawyer client relationship on between our students and and the innovators on We've been involved in a number of other classes and projects that I think provide more of a bridge so I teach a class called law for algorithms that involves Half law students have computer science students and gets these groups of students working together on sort of creative new technological approaches to solving some interesting legal dilemma, whether it's a privacy challenge Or a challenge that their housing authorities are facing in trying to cut back on the the the transaction costs associated with getting people off lists and so forth. So, so that's an attempt to get people working together and then a third thing that I just want to mention is Ziba Cranmer who is involved in pit UN she runs an organization within BU called spark, which brings students on technology computer science and other other students together with clients to use tech tools to solve interesting challenging Questions and she's partnered with our human trafficking clinic and the Massachusetts Attorney General's office On a student developed app for detecting and reporting victims of human trafficking. So, you know, these are opportunities for true interdisciplinary collaboration across departments in ways that, you know, really promote the development of public interest technology solutions. Yeah, it's interesting to think about all the different opportunities and many of you know I was the director of the first law and technology and public policy clinic out at Berkeley. I guess starting 20 years ago now, which makes me feel really old. And it is interesting, you know, the challenge of we I had PhD students from engineering and from the iSchool come to do projects, you know, we help libraries with should we use RFID can we write a good RFP that helps us get what we want Oh, the city is thinking about using surveillance cameras what empirical work along with regulatory work might we do to kind of help them view it through social justice lens, but there are a lot of like interesting challenges in taking law students for engaged in legal practice and adding engineers computer scientists, etc, and keeping kind of the expert relationship and thinking about how you do that and teach them good lawyering skills right so there's some really interesting challenges. And so Paul I wanted to turn to you. I know you and Stacy have been looking part of your PIT UN supported initiative at how computer science and data science and the law are building out curricular innovations. This is what you're doing but kind of more broadly and I'm wondering if you could tell us about, you know, what are the range of ways you see people trying to innovate in this space and how much do they keep people in roles, versus try to do things that are more interdisciplinary rather than multi disciplinary. But it's, it's been a great process and I'm so thrilled to be partnering with Stacy and be you on and so we had 55 people it was friends and family but we asked friends or friends to come as well and we are of course we're going to have an in person convening and it turned into a big zoom party. So we talked through not only pedagogy but we also talked about how do you get tenure how do you get hired how do you convince you know you're dean to fund whatever crazy off the wall thing you want to do. Our stated goal was a white paper and and we really wanted to kind of dive deeply into the question and so it's lawn CS for a reason we thought by cabining it to those two fields, we could then go really really deeply. Pedagogy specifically, you know, I'm not very prescriptive I wrote that chapter I'm not very prescriptive about the right way to do it. I think 1000 flowers are blooming in really interesting ways. And so we found people who did bring the two groups together and they play their roles we found people who brought the two groups together, and they mixed their roles they learn how to do the thing the other knows. And I would have teaching has always been I teach law students I teach law students how to think like techies. And so let me just say a little bit more about my approach and the methodological question you asked a few minutes ago. I think it's interesting that here and Lydia, and I are all going to say the same thing which I think is still kind of a little counterintuitive which is, there's such a premium on breath in law schools and probably in CS departments where we're going to teach a little criminal law and we're going to teach a little bit of criminal law and we're going to teach you some corporate law. Whereas all of us with lived experience in the world know that there's something to be said about depth as well. And what I've tried to do in my law classes that teach technology is I try and absurdly absurdly go too deep, go deeper than seems rational. So for example, in my technology a privacy class, every student these are law students, they get their own virtual machine I teach them the Linux command line. I teach them this is going to be a deep, deep cut that only a few of you will get. I teach them how to use all of the flags to the cut command in the Unix command line and if you ever look at that that is about as esoteric. And the reason I have them go through that painful exercise is because lawyers are really good at that lawyers are really, really, really good at being an inch wide and a mile deep. And I saw this when I clerked for a judge. I saw this when I summered at a patent firm. There are lawyers out there who know more about a particular technology than any technologist, because they've just read every patent ever read on that question. I asked them about one thing just a step removed, and their hope they're useless but if you ask them about that. And I think that's an important thing to remind law students who want to go into pit UN and probably computer scientists who want to go into pit UN, which is you shouldn't be afraid to have deep knowledge. Because if all you do is off yourself out as you know I can skate the surface and hold my own and hopefully my own discipline knowledge will carry me through that's not enough like I think you need to learn to dive as deeply as possible and I'm glad that I think I heard my hair and Lydia say the same thing. So I'm super interested in like what we think we need to do in order to kind of build bridges and I, you know, it sounds like Paul you're saying that some of this depth and like really going in the weeds and someone else's areas you know, you know my own thinking I don't feel like my law students need to I do want them to understand technical standard setting I want them to be able to read a technical standard I want them to have. Oh how white we use you know a man in the middle proxy to actually look at what information is leaking from something. But it's really interesting to think about like what competencies, even if we're thinking about, you know, is it about architecture is it about code is it about where the things are developed and who participates like what we think and it's We don't have a cannon right there's no cannon in this area and so it's always really interesting to think about what what, what do we think we need to do to kind of like reach across the aisle. So I want to go back to this question about some of the challenges with experiential learning and I hear you mentioned the kind of different cadence of the academic cycle and being involved in policy processes. I'm Chris I've also heard you talk about that a little bit. And I'm wondering if each of you could talk a little bit about you know when you're trying to either work with a client or engage with a process that isn't on the academic calendar. How you're finding ways to kind of push things together to make that work, because many of us want to have an impact in the world, you know that we're trying to help our students find their voice, get their feet in this work and so those connections are important. But we know that that's hard and I'm wondering how you're managing that. Why don't we start with me here. Sure. So I think what we've done is on some of the projects say and we've given comments to the California AG on privacy law we've given comments to the FTC, and that's proceeding on a particular cycle and we have to get people together to contribute to those projects. And it's works. Okay, when it's an expert an area of expertise for people and they're just contributing what they already know, but when it's a new knowledge when you're trying to create some new data was trying to gather new evidence. It's hard to like create that cycle where you're ready to give that data over and plan longer term. And so that is one of the things I'm working on is to develop methods for beating sort of pipelines of evidence gathering that then can be available to the policymaker. When the policymaker is confronting these decisions. Same question to you. Yeah, well so I was just trying to think about it when when me here is speaking I mean I think like one of the lessons we've learned is that. I mean, I guess to your point like if we really want impact in the world we have to meet our partners where they are and so oftentimes that means that we can't necessarily hold to the schedule that maybe I originally anticipated when we started the class so we provide a fair amount of flexibility to the students in terms of how they set up their sprints when they conduct their demos with the partners with again the realization that we really want impact so if the goal is that they're going to make a meaningful and and a demonstrable difference in the world that we have to make that the priority. I think one of the really exciting things that we've been able to do to broaden that is for the most promising projects, you know whereby at the end of seven weeks we have an MVP that actually looks like it might start solving some sort of a problem. We've been able thanks to a new America to take those projects and actually turn them into full time summer fellowships. So the model that we're hoping this leads us to is that we can incubate ideas in the lab, we can then lead to a good summer full time experience where they continue building those ideas into real world open source products. And you know we're testing out this right now then for for some of even the best of those good ideas those actually can then become essentially the capstones for the master students so hopefully through that process again we're we're finding their way right way to fuse like an academic schedule with with still leaving lasting impact in the world. Thank you Paul. I think the tech policy clinic has a fair number of staff and fellows that are probably pretty important to some of the continuity, but could you talk a little bit about how you manage the kind of different cadences between the quote unquote real world and the academic. Yeah, sometimes things just line up really well. And I'm not I've actually not taught in the clinics but I've worked with the clinicians enough to know what they face along these lines. So for example, I believe the copyright office is doing its triennial 1201 review, and it happens to be that they had like a December 7 deadline so as a dream come true at all the law clinics. I bet they'll get a lot more law clinic participation I bet they'll get better results because of it so this is my open plea to all the government bureaucrats please look at the academic calendar when you're setting your schedule. I mean I think the other one is probably similar to what Lydia experiences with the libraries which is sometimes you could be cagey about picking a client that isn't so deadline driven. I work a lot at Georgetown with nonprofits and the nonprofits say, you know I've got this persistent data problem and it means I don't know if the intake is matching the results. And so there's no deadline there's no you know we're not litigating we're not walking into court with them. They can imagine they can tolerate it if we say you know for the next three months we're going to be in summer you're not going to hear from us. And so it might be nice for someone who's never set up one of these clinics to try and do a little mix of that in your docket. Because if the clinics are the heroes of the world and I'm looking at all of you who do this. I mean a lot of the time you're just working without your students, and you're just, you know, being a full time lawyer, running a small law firm, except all your lawyers are now in final exams. I think that's just the plight of a law school clinician and probably pit you and technologists clinicians as we expand this model. Yeah, my husband can talk to you about that. If a student gets sick and you have a deadline but not pretty. Okay, so I want to encourage folks who are in the audience to start shipping in some questions. And I want to get in the weeds a little bit more about like the contents of people's classes as well as who they're teaching so some of you are teaching graduate students, JDs, maybe PhDs. I do spend most of my time with graduate students I do teach a large undergraduate class to some of you are teaching mostly undergraduates. And I'm wondering if we could talk a little bit about what, what are the actual contents of some of these classes Paul was talking about, you know, teaching them how to use But I'm, you know, wondering Lydia, could you talk a little bit about like what's the meat of what you're teaching and your audience I think it's undergrads but I'm not sure. So, what a funny question so the computer scientists are almost entirely undergrads, and they're very a plus driven. When we have graduate students and journalism architecture on pass fail. So you already have a line incentive alignment problem. Luckily, everyone's been motivated enough by the problem that it's fine but everyone's on a different schedule they start on different days they end on different days. Different holidays, the middle of the semester. I know all these are boring problems but they're all things that yours truly has to fix. So, that's a reality of things that you just have to deal with. So, who is it and then. So when we talk across the first time. I think the biggest problem was there was sort of a fight between the disciplines because we all thought our way was the best and just kind of merged them but the way because we all do design in some way shape turns out they were more different than they were similar in terms of trying to get things done so. And so there's a lot of just gaps of information for us to fit in. I didn't realize that architects design but don't build the things and so we design and when you use a totally different process you have to do when there's the reality of this has to get built. And so that's what actually made us lean towards. Okay, if we're going to be building technology we're going to lean towards the framework of what technologists typically use and just sort of append things on to to that in terms of what we teach and that then we had to teach the human center design process to the architects and the journalists. This was another problem because Chris, I know your experiences but architects don't seem to like design thinking even though it has an awful lot to offer and so I have to sort of give a very compressed 10 minute lecture on this is what prototyping is this is what testing is so that now and the groups are mixed and so the people who already know it can can can fill in I found that that is really important is to have when you have mixed teams they will fill in each other on topics so that has has worked but without some sort of grounding you know for example, user journeys is something that architects have never heard of the idea of you don't just interview people and it's like, what did you do today, you're a librarian, what do you do when you woke up and then and then and then just to see what their actual flow of kind of the boring mundane because those were a lot of the problem sits no one thinks to tell you like oh my God I hate filling out form XYZ but when you do that sort of journey towards you know how did you help this patron fill out their taxes and citizenship, that's where this this good information crops up so we had to teach that and sort of reteach that a couple times because it's just where we found that we get a lot of the best insights. I'd say those are some of the teaching challenges. Another I'll say is just like misalignment of words and particularly human centered design uses a couple words that are very Silicon Valley capitalistic so the main one that I kept using the idea is figure out what's already out there so you don't repeat it and you steal the good ideas and leave the bad ones. But that's just not so but so we change that to precedent analysis when I think actually makes a lot more sense and that's what architects do of course you're not going to build this you have to research what other things have been built before and what's been done so you don't repeat any of that so same idea just had to align the terms but that's a big deal to as it turns out use the wrong word and you will turn people on or off, or it'll do something you don't intend them to do. So lots of that continuously realigning people on a singular idea of what the process is incorporating multiple terminologies and methodologies but mostly aligning I would say to to one framework for consistency. How about you Chris. Yeah, I think I think a lot of what Lydia said resonates with me so so most of my students are master students. You know we, we kind of throw a lot of tools at them that I found that I had to learn at 8 to enough so things like GitHub, Trello how to set up a con bond board using slack mural. These are really great collaboration tools and there's various levels of comfort with them. But I think to Lydia's point a good portion of our class also has to do with human centered design and really building empathy with with, you know, the problem that they're trying to and those frameworks I think are easiest learned just by experiencing so that we do a lot of workshops. We really encourage them to embrace the ambiguity we actually wrote a policy innovation lab playbook which was a collection of lessons learned both my own as well as that of the students in terms of like how to actually get through the class, functionally and and how to get through with a more positive experience kind of setting yourself up for some of the, you know that some of these processes are just going to be a little unusual and every class is a little bit different we try and take on a different subject every time the partners are always different so. But I think from from what I've heard from the students they really like that and they embrace it in the end because I think it does start to feel like a real world problem, a real challenge. Interesting so the ways in which you come to understand your client the ways in which you manage projects I guess some of it. Coming from a law background, I teach I actually use a fair amount of design methodologies but I also spend a lot of time with computer scientists. Everybody has different conventions right like as a lawyer you really and particularly when you're working with an institutional client. I'm going to turn this over maybe I'm here and Stacy to talk a little bit about like who is your client and really understanding who can speak for the client and understanding the client's goals and you know getting students to understand what their job is what questions they get to answer and what things are really at the clients discretion with the client you know I'm very different than in design right very different ways that you come to understand your client it's not an intake interview right you do very different strategies so it's super interesting to think about how you sensitize students to the strengths and weaknesses of different ways of knowing right how do you actually come to understand an organization its goals etc. I'm here I'd love to hear your thoughts. Yeah, it's, you know, I was thinking when you was mentioning that we have a project and, and reflecting what Paul was talking about working with a client who perhaps as a long term horizon so we have a project with the Wikimedia Foundation, and to deal with the issues there they have on the loud voices on their platform. And as I think about that one of the key things that we've tried to have the students do and the other people working with us is to understand the client's perspective. You know, as, as a former defense lawyer and a prosecutor that's sort of a natural comes naturally to me but it's just, you have to understand what the client cares about, and what's driving them. And I think sometimes in the policy in the tech policy will we think of solutions to impose on people, and we don't necessarily sort of ask for what why do you do it the way you are what do you care about how do you think through these problems and so. And I've heard feedback from clients who say just doing that has been fantastic, because no one's actually sat around, you know, with a sociologist and a computer scientist and a security expert and you have, and just ask me these questions from very different perspectives, and that itself is a learning experience for the client so I think there's a, there's a net positive there in terms of engaging with the client. And to just reflect on one point that Paul you brought up I think there's the key to me is that you, you have to develop ways to communicate across boundaries so while you're very deep you have to learn how to share your deep expertise with others. And so this kind of client interaction gives that opportunity for students to learn how to communicate their deep knowledge with another person who has deep knowledge. I'm going to turn it to Stacy next and drawing out this insight from here this sometimes putting yourself in a room where a bunch of people who don't have your same assumptions they don't have your same training. And all of a sudden, they start asking questions right like when I teach law to non law students they're like, why'd they do that, right like that was a dumb way to do it. And law students once in a while you'll get that right but often they're like okay let me write that down right. And I think the lawyers, the law students often bring that to the computer scientists they're like, what that's shocking, you know why would you think that was an okay thing to do right because I like, it's in a different moral way with them. And I'm wondering if you could talk a little bit about you know the population you're teaching and some of the kind of core content. Sure, I think you know I have some reflections both about the class that I teach but also about the clinic and following up on the here's point and, and your initial question Deirdre about sort of the institution, sort of that the interests of the client versus the sort of broader or policy oriented goals of the clinic I think one of the main reasons that we have this technology law clinic is because MIT initially the institution really felt like there was no place for their students to turn to get advice that was really focused on their own desires and needs and concerns like students who were not being kind of captured by the the goals the ideological goals or other goals of a particular clinic. And they didn't have the ability to turn to like the general counsel's office at the university for for legal advice so that clinic is very much focused on the student as client the student you know very student client centered goal which is a really interesting experience for both sets of students the law students and and the student on that in my class the sort of the law for algorithms class that I teach I do think they're a really interesting kind of disconnects in the expectations that the students come to the class with I think you know the law students do have this kind of different set of expectations. I think I hear this a lot from lawyers and technologists the law students assume that technology is going to be able to solve a problem that there will be a tech solution a perfect tech solution to the problem and that the technologists assume that the law you know consists of a neat set of rules right that of course can effectively lead us to the optimal solution and so one of the interesting areas in which we really engage is trying to bridge that gap in in sort of different assumptions know you know whatever technological solutions might be available are themselves highly imperfect and getting both sets of students to really kind of be prepared to interrogate and to have the language and the analytical skills to kind of interrogate the solutions that are provided by each set of disciplines I know that doesn't quite answer your question directly but but but but it is the you know what one of the challenges that we face. It's great. So Paul I'm going to let you close us out and I want to ask you to reflect on something in particular that I think draws on what Stacy was just saying and I think isn't that aren't you credited with that maybe the right the lawyers think that felt felt and third law is third law. Yes that's right. I actually haven't found that at Berkeley everybody's very skeptical about everybody else's capacity to fix problems so I don't know if that's like a Berkeley thing but some skepticism but I am curious about one of the things that I do think is really useful in my own teaching experiences that the lawyers will often come in thinking you know like they look like they look at things through a legal lens and you know you come from a social justice or a public you know advocacy and right where I did right like you look at a problem and you're like oh well is this a public relations issue is that like what lever can I pull right can I redesign the market. Do I need to go get a law passed. Can I get a really good article in the New York Times right how do I move the ball forward and one of the things that I think is super valuable by bringing law and computer science students together for example. I think they get a deeper appreciation of the solution space it all of a sudden becomes broader like oh I could protect privacy that way or oh we could use some encryption over here or maybe you really want to you know redesign that database or whatever it is and I'm wondering a little bit about the broadening of the understanding of the solution space and the role that that plays in building this public interest technology field. Yeah that's such a wonderful question and it's something I think a lot about I mean so computer scientists and I should have said at the very outset I was an undergraduate computer scientist and I'm a programmer and I defended networks. You're taught early on that there is an objective measure of your work and there is right or wrong and and you're also taught I mean I'm totally caricaturing here you're also taught that the world is sort of a meritocracy where if you're smarter and a better coder and your students will win at the end of the day. Lawyers are taught the opposite right lawyers are taught that it's unfair power structures all the way down and what you need to do is learn how best to like find the persuasive hooks into this in this power system and what you and one of the things which. I think we've Paul has frozen. I think what he was going to say is one of the important things here is helping them kind of get some of the sensibilities that come from the other fields professional training and orientation right like our fields are so rich. There's there's the substance there's the methods and then there's like perspective about what problems or we're solving our orientation towards problems. And I think by the one of the benefits of bringing people who are studying in different home departments together. Is it it can really rock their worlds right and my undergraduate class and data science behind the data humans and values you know one of our early classes. We do the politics of artifacts right and then we talk about data and and capture and having these students the computer science that they're like oh my god you just blew my mind data isn't data right like it's the product of all of these decisions and a human action. And and the social scientists are like yeah was that novel was that something new. But you know bringing them together is just so important for that reason it's like really profoundly eye opening to realize that other disciplines come with their own perspectives. And that thinking through those other the lenses of those other disciplines can help you think differently about what problems you want to solve what problems you can solve. And what goals you might think about achieving so I think that's really valuable. All right so our last two minutes I want to give you each. An opportunity to say here's the question you should ask me that she didn't and here's what I think about it and I'm going to start we're going to go in alphabetical order with Chris. Oh sure all right I'll be quick I I would love to I guess maybe the question that I would have loved to have discussed was how can we all collaborate more. And I don't have a perfect answer that but please let's collaborate I'd love to learn I you know what I can find online is always so fascinating if we can find ways to really leverage what each of us is doing in the classroom and share our shared experiences in our own multidisciplinary format I think that would be wonderful. Terrific Lydia. One of the struggles in our class we're largely designing for low income US people and designing technology for rich people is actually easy they all have an iPad they all have a phone but there's a variety of access to digital tools. In with with lower select not everyone has broadband for example people don't have a desktop and there's just a greater variety there and that has created quite a design. Challenge for us and I think that's knowledge that we could if we got together we could sort of share like we actually a lot of our students built texting services. That you could use from a smart phone or a dumb phone or anything so that was an interesting solution type we came up with. Me here 10 seconds. The thing I would like us to talk more about which we have and we continue to do is to talk about systemic bias and how we can address that all of us together that continues to be in Lydia you touched on that. Stacy. I'm interested in thinking about what the baseline is how we get to a common baseline of understanding so that we can get our students to actually work forward on together on problems. That's a big challenge I think in this kind of teaching. Terrific. Well thank you all so much for all you share today and for the work that you're doing to move the PIT field forward through your educational research efforts. Thank you.