 Hello, everyone, and thank you for joining us in this Public Interest Technology University Network webinar, in this case, Building a Cybersecurity Clinic. Today, we will look at MIT's student-based cybersecurity clinic, which is a grantee from our network challenge in 2019 and 2020. We hope that the conversation today can help our viewers test and build out public interest clinics in their campuses, engage local and regional organizations as the clients and hopefully identify best practices in their clinical education model. This is something that is really important for us at the Public Interest Technology University Network in New America. And as we continue to grow the field, we're more than happy to share these successes with you all. Before I introduce our panelists, just let me again remind you that you can drop questions in the Q&A and the chat. And we will be actively monitoring them. We want to hear from you as much as you can. So please keep the questions going. So just quickly to get us started, I'm going to introduce Larry Suskind from MIT, he's a Ford Professor of Urban and Environmental Planning. And I'm also going to introduce Zhang Wuchun, which he is a PhD student in Public Policy and Planning at the Department of Urban Studies and Planning at MIT. And lastly, Avital Varal, who is a Master of Engineering and Student and Computer Science at MIT. They will talk about what is the MIT Cyber Security Clinic, the EDX course that they developed and partnered with other schools. We're going to talk about that for a couple of minutes. Then we're going to talk about working with clients, in this case, municipalities and hospitals, and this whole student experience. And lastly, we have our last version, which is the clinic's work with one of their partners, which is Mass Cyber Center, on creating a minimum baseline of cybersecurity for municipalities, where we're going to be joined by Stephanie Helm, from the Director of Mass Cyber Center, MassTech. With that further ado, Larry, the floor is yours. Thank you so much. It's a great pleasure to be here with everyone. We're very grateful for the support from the network and we've had a chance now to run our MIT Cyber Security Clinic several semesters. So we have some experience to report on. And while we're sure that other universities, colleges will go in their own direction, and if they want to do work on cybersecurity and work with cities and towns and states, we want to share with you what we've learned so far. Next slide, please. Yeah, one more. Thank you. So we run a clinic, which is a course. It's a semester-long course. And all the students from all over the campus who are invited to enroll in the clinic begin by taking four one-week modules that are pre-prepared and are online. So each week, the student works three, four hours on a module on their own time. But at the end of that week on Friday, the class meets and we talk about the experience of the module the students had that week. And we do the same thing for four weeks. And at the end of the four weeks, students take an exam, multiple-choice exam. And so far, there hasn't been a student who hasn't passed it because they've paid attention and done the work necessary. But it's very important that students pass this exam because the next thing that happens is that they join in teams and the teams work for public agency or hospital clients for the remainder of the semester for let's say eight weeks. And I want to be able to say as the head of the clinic, look, you're getting students who are prepared to do this work. They can come and help you over eight weeks, prepare a vulnerability assessment. So your agency or your hospital will be able to anticipate the kinds of attacks that might be aimed at you by hackers in the world at large. And we can help you think about the things you have to do to reduce your risk or your vulnerability. And I want to be able to say to city agencies when we sign them up as clients, don't worry, you're gonna get students who are prepared to do this work for you. They've taken an exam at the end of the four weeks. It's true that the four week course is also an edX course that anyone, anywhere in the world can take for free. And you'll see in the chat some link where you can look at that edX course and anyone can take it. And any university could have its students start or prepare ahead of time by taking this course before students go to work for public agencies or in our case also for hospitals. We have both undergraduates and graduates that together in the class and the teams, each team has undergraduates and graduates. Nobody's required to be a computer science major because we are looking at what we call defensive social engineering moves that cities and hospitals can make to reduce their vulnerability to attack. We use very specific templates and tools that students learn about during those four weeks and then they apply them in working as a team with an agency. And we have a memorandum of understanding, a kind of contract with each public entity that a team is working for. We don't charge the public agencies for the work but we have a contract and the guarantees that we will keep confidential any and all materials and insights generated during the semester. Next slide please. So the clinic starts with this material that students learn on their own and then we meet to talk about and then they work in teams. Now during the time they're working in teams, we also meet every week and each team gets to talk about the problems they're having, executing each step in the process of preparing a vulnerability assessment. They need to gather information. So we have question asking tools and formats and then when the information comes in, they have to begin to imagine what the answers to the questions are from the information that's come in. And with that information, they prepare a draft assessment using a format that is the same in every case. They give that to the client. The client says, oh, wait a minute. Oh no, who did you talk to? How did you get that? And we work with them because we only wanna give them a document at the end that the client is gonna find useful within the city or amongst their staff. And so we have a process of reviewing the draft assessment, their comments on it and producing a final assessment which only goes to the client. We won't answer questions from the newspaper, from the media, nothing. We only trying to be useful to the client. And there's 17 questions in a format that has to be answered for us to produce what we think is an adequate draft assessment and then final assessment. All of those questions for anybody in the cybersecurity field come from what is called the NIST, N-I-S-T, larger framework which is used in the United States and elsewhere as well. And we've boiled it down to these 17 questions which ask what have you done to prepare if you've had attacks, what have you done during and what will you do during an attack? And then what's your plan for how you're gonna make sense of what happened after an attack? And the NIST framework that we work with has this before, during, after framework which carries over into our 17 questions. When we started adding hospitals this semester we had to augment the question-asking process and Rebecca, spirit who you'll meet in a few moments is a student who has helped us from her own professional experience, develop a further version of our question-asking process so that we can work for hospitals as well because they have a variety of additional things to worry about. So that's the general frame. Next slide, please. The four-week course has a lot of background that students need to prepare. And so we explain to them what others have done research about and we summarize research and we run certain hypotheticals in the material in the edX course and the four modules and we get people to understand what critical urban infrastructure is why it's vulnerable, who are the attackers, what are the lines of attack, what has the federal government been doing, although our focus is more on state and local government. Then in the second module, in the second week students see examples of ways of communicating with the potential client. In fact, they see video of other students communicating the request for information well and badly. And they see what happens in those two different versions of trying to get information from the client. And then we require students to interview other people besides the client contact to fill in certain background to understand the context. And again, there's video of students talking with others in the city or in other agencies that the city might work with. And when that goes well, they see what that looks like and when it goes badly, they see what that looks like. And then in the final segment of the preparatory edX course, they see a team of students trying to prepare a vulnerability assessment and what happens when you present that to the client. So there's lots of background reading and material all summarized in these short video components and then there are self-tests that students can do to make sure they're learning what's necessary so that when the final exam comes up at the end, there is no problem that everybody has been able to pass that. I think there's one more slide or not. Yeah, so the point of the clinic is both to prepare students for a career or work in cybersecurity, particularly in the public sector or in the not-for-profit world. We also try to provide a service at the same time, which we believe universities have an obligation to do. So the direct client interaction becomes the clinical learning opportunity for students. We believe that it's possible to systematize the question asking that goes into reducing a city agency's vulnerability to cyber attack, but the specifics, is it a transportation agency? Is it a health agency? Students might know something about the substantive area because of what they're studying or they might not. We try to make the team, depending on which students are in the class, include students with different backgrounds. They may be from the urban planning field, they may be from the computer science field, they may be from the management field. We try to have a student team that knows something about the substantive questions that the agency works on, but really the most important thing is that they're learning what are the problems of trying to help a public agency anticipate cyber attacks and do things organizationally that mean that they reduce their risks. And the kinds of risks that we focus on have more to do with what people do than they do have to do with the failure of hardware or software. Yes, we have lots of computer science background amongst the students on the team, but the questions are about what happens when people in a public agency haven't been trained to take cybersecurity seriously and they open an untrusted email attachment and that releases malware. The malware releases ransomware, all the data of the agency is encrypted. Now they have to decide whether they're going to negotiate with terrorists, pay a ransom or not, and we show them what they should have done before that moment comes. Because if you don't have backups of your most important data, the problem of deciding what to do is way harder than if you already imagined what could happen and you have system of backups, you have a clear understanding of the risks you face. So this is what we mean by defensive social engineering. That's the orientation that we take in the course, a defensive social engineering, even though people have lots of technical background, we're saying that the big risks have to do with what the least educated and prepared person in an agency does or doesn't do at a critical moment. And if there hasn't been proper training internally, if they haven't thought about what they're gonna do when and if there's an attack, if they haven't clarified who has responsibility to take action when and if there's an attack, it's too late when there's an attack. And so we try to help our clients by asking questions and then giving them feedback on their answers compared to what best practice looks like to us. And when we discovered that we were lucky in Massachusetts where MIT is that we have a state agencies that thought about all this way before we did and it's been working and you'll hear more about this from Stephanie Helm before we're done today. The state's been thinking about what should cities and towns do at a minimum to prepare to deal with cyber attack. And we've been trying to learn from and work with Stephanie and Mass Cyber and we're actually now spinning off some research for them so that students at MIT who work with the clinic can then also develop research activities of their own. Avital, who you're going to meet and have met is writing a thesis on some of the material. I'll let her talk about it in a minute. But the clinic is a focal point for teaching, training and service. And it then is a launch pad for students on the MIT campus to spin off other research activities with faculty. And yes, there are some other courses on the campus. There are lots of technical courses on cybersecurity in the computer science department. And the Sloan School gives courses on cybersecurity in the corporate context, but we're the focus for a clinic that prepares people to work on cybersecurity. A question came in asking about my background. I do not have a computer science background. I'm lucky I have terrific students like the ones you're gonna meet in a moment who know way more than I do. I got into this in the beginning because one of my doctoral students was very heavily into this field and he wanted to move from the computer science side more to the defensive social engineering side. And since I study the way in which public agencies and organizations make decisions, how they think about risk, how they manage risk. I work on climate change, that's a risk. I needed to learn a lot from my students and from other faculty who I brought in to help to develop the online segments of this course. So when you look at the edX course you'll see other faculty from the computer science side at MIT. But no, this doesn't require a faculty member in my view from computer science to be the lead in the cybersecurity clinic. You could do it that way, but in the model we've adopted because of my limitations, we've done it in this particular way. Let me stop there and maybe pass things along to, I guess, Jung or Rebecca or Abhi, I'm not sure. Which of you want to start talking about what it's like working with clients? I can't take that. Yes, please, Abhi. And to all of the other viewers, I'm looking at your questions and we'll get to them at the end of the webinar. Abhi, tell all of yours. Yeah, thank you. If we could move on to the next slide. Thanks, so Rebecca and I are going to spend a little bit of time talking about where clients have been in the context of this clinic course and sort of the main takeaways that we've gotten from working with those clients. So we've mostly been working, or a part of the clinic is working with municipalities within the New England area, which is our regional area as we're based out of MIT, with a particular focus on the areas of greatest risk within those municipalities, which are often critical infrastructure and essential public services. So things like, you know, fish fast systems for the fire department or payroll systems or other systems like that that provide essential services and are often left unguarded or not sufficiently well guarded. And then our goal as Professor Susskind talked about is to support municipalities in achieving, you know, in making themselves sort of not the lowest hanging fruit in terms of being vulnerable to cyber secure, to cyber attacks. And we do this through this vulnerability assessment process, which as we've discussed before, is not so much based on specific hardware or software or penetration testing or sort of that sort of thing, but rather examining processes and organizational structures that ensure that everyone has a baseline of understanding about cybersecurity and about what to do in order to not make themselves as much of a target. And then in addition, we sort of want to balance the security needs of these organizations with the known constraints that they face in terms of resources, because municipalities, especially smaller municipalities often have very limited budget, very limited staff. So if we suggest, you know, expensive solutions that are expensive in terms of time or in terms of money or both, you know, we don't necessarily think that that will be followed through. So we want to provide not quick fixes, but, you know, things that are, you know, process changes or organizational changes that would make a real difference in terms of their cybersecurity readiness rather than jumping directly to, you know, like really expensive capacities. So that said, I want to introduce Rebecca who's going to speak next. Rebecca is a graduate student at MIT who has also worked as part of the clinic and has led our move into working with healthcare delivery organizations such as hospitals and expanding the clinic's work in that area. And she has a bunch to say about that process. So I'm just going to hand it over to her. Thanks Navi, really appreciate it. So a lot of the expansion into healthcare delivery organizations, hospitals, clinics, it builds very neatly off of working with municipalities. There was, I think, a surprising amount of similarities considering hospitals are for-profit, municipalities are not, although that is obviously a key difference. But there are a couple of areas that are significantly different that we had to pay attention to when creating a questionnaire or working with our clients. The first is the area of greatest risk, right? So hospitals themselves, healthcare is considered critical infrastructure based on CISA's designation of the 16 critical sectors. So, which is great, but there's a particular piece here which is patient safety, right? That if something actually ends up happening in over the course to a hospital, for example, a patient's not able to get the critical, you know, life-saving care that they potentially need. So we had to keep that in mind when we talked with our healthcare clients to say, have you actually thought about what would happen in X, Y, and Z circumstance given how important the services are. The other really different piece is the nature of the sensitive information. I know many folks are familiar with HIPAA, right? As well as just in general, right? That your healthcare information is something that is private to you. And that comes with extra security controls and risks that a hospital would need to follow. So as an example, when we were working with our clients, there were sometimes a little bit of back and forth on, oh, you know, we set up things given our limited resources, right? Some of the things that Avi just spoke about, but we accidentally put data together that shouldn't have been put together, right? And it was potentially an easy fix, a conversation. So it's those sorts of things that we're trying to kind of uncover. And the last thing, which is more of a trend. So if you're setting up your own clinic and you're looking at just general sort of technology trends, this is us basically introducing something that is definitely on the horizon for many hospitals, which is the internet of things, in this case, medical devices, but it might affect whatever sort of clinic you decide to set up. And in this case, it increases your attack service, meaning you have more things connected to the internet. There's more of a chance for someone to use that to launch an attack, get into an environment or move within a hospital's network. And that was a very interesting conversation. In our experiences, we didn't find that it was wildly applicable yet because a lot of hospitals haven't necessarily fully converted into buying internet connected devices, but it's good to start having the conversation early and think about it from a prevention standpoint, rather than in a retroactive cleanup. So that was a very sort of interesting piece of working with hospitals as well. Let me go to the next slide. So we're gonna, we also like to go through a little bit of the themes that Avi and I have experienced both as students and as TAs working, working with clients. Avi, you wanna kick this off? Yeah, so the first thing is somewhat of a hopeful thing, which is that we see a really sincere desire to improve security and to take security seriously, at least in all the clients that we have worked with. There's a lot of municipalities out there and a lot of folks working in those municipalities who feel really strongly about their job and who really wanna make sure that they're doing the right thing, but find themselves unable to do so sometimes, partly due to a lack of information and partly due to a lack of resources. And something that we have feedback we've received from clients that has been really heartening in the context of doing this clinic as a mechanism for service from the university to its sort of regional area is that providing them with this vulnerability assessment, both highlighting the things that are lacking, but in particular highlighting the things that they may be already doing well, is something that is really helpful for IT directors and other people who find themselves responsible for cybersecurity to go and argue with their, at the municipal budget meeting or in the places where resources are decided and to point to this assessment and say something like, well, we're doing this thing right, but we're also looking to make improvements in these set of areas. And here's someone external to us who's like pointing these things for us. So we think of this service that we're doing in terms of providing this assessment from a third party that can help change things within inside of the organization. Yeah, and to kind of pick up from there, something that was previously alluded to on the sort of client engagement slides is for anyone obviously who's starting a clinic, a lot of this, even though we're talking about technology has a lot to do with people in process, right? There are things that have to do with accountability, roles or responsibilities, knowing who's in charge of what, at what time and understanding the way things work from a process perspective, even though our brains normally think, oh, it's a tech fix. I think it's a really important point when engaging and starting a clinic that it's often the people element, not necessarily the tech though, end up moving things forward. Another important theme that we did discover over the course of working at the clinic was folks had trouble defining their area of greatest risk and in your clinic, it might be the areas of greatest priority, but in this case, because of cybersecurity, we call it risk. And a lot of folks, even though they're working in a city or a hospital, they didn't think about the difference between a water treatment facility, right? And their IT infrastructure. So it was a really interesting conversation. And again, it's just bringing something up and having the conversation because no one had asked the question before. We also found that a lot of folks, they did have plans, right? They did actually think about these things pretty in-depth, but they never tested them, right? And being able to say, it's really important in cybersecurity to have a tabletop exercise, but again, just in general, to actually go through and test all the hard work that you've done. So you could see it actually works, whether with data backups or doing a simulation, it was really eye-opening for some folks. And the other thing, which I know many folks have seen in the news, is that the accountability between those within the organization, municipality or hospital, and those who sit outside of it was also very vague, right? So who is accountable in the event of an attack? Where does my data go, right? Some sort of basic questions because most folks just think about their own organization and they don't necessarily think about interaction outside of it. So that was an important piece that we did. Key thing we didn't see. The last piece is we really wanted to, we realized that there's a lot of low-hanging fruit to kind of move further research and engagement forward, right? So one of them was that roles and responsibilities piece, making sure people know what they're, you know, basically documenting down what they're supposed to be doing, at what time, knowing how they engage with stakeholders. We have an example where the hospital, one of the hospitals that we worked with, they thought that their vendor patched the servers and their vendor thought that they patched the servers, right? So all it took was a conversation for something very, very easy, very low-hanging fruit, but no, it was just assumptions that ended up putting them at greater risk. So we had a lot of impact there. And then the sort of last piece, which I think is a good transition is, even though having these conversations and these engagements is very important, there are these resources that would be able to move things forward and allow municipalities and hospitals to be able to work on their own so that things live on after we're gone, right? And one of those things that we found was having clear implementable security standards. So I'm gonna pass it off to Jung, who's gonna talk a bit about some of the standard of care work that he's been involved in. Yeah, thank you, Rebecca. I'm Jung Wu. I work with Prasari Suskind on various things, including running the cybersecurity clinic the last year or so. And I think, going back to the point about the clinic being the focal point for teaching, training and service, I found it really fascinating how different research topics emerge from the clinic. And one of these topics is the standard of care, as Rebecca mentioned previously. So in addition to helping municipalities and hospitals, the clinic aims to expand the outreach by doing more in-depth research on these topics that come out of the volume assessments, what they feel vulnerable, the gaps, the holes that we find, the things they might be able to implement right away but may not be able to. So we started discussion on the importance of standard of care for municipalities and healthcare systems, whether municipalities and local agencies should have a certain level of cybersecurity standard in place. So we hope to spell out the minimum cybersecurity protections that should be expected from these entities, such that of training, whether employees in the city are required to take certain basic measures, such as training on phishing or other common risk factors, or whether there is a predefined incident response plan, knowing who has the authority to shut down the system when there's an attack, who has access to critical systems and who to report to during an attack. All of those things I think fall under what we believe as the minimum. And the clinic has established partnerships with state and local agencies, including the Mass Cyber Center and the Massachusetts Municipal Association. And without these organizations, I think the clinic wouldn't have come so far. They've been great advocates for the clinic, not only to help advertise the clinic's availability to potential clients, cities and towns in New England, but also work with us as collaborators in the research areas like the standard of care. So the clinic partners with the Mass Cyber Center and the Cyber Resilient Massachusetts Working Group on refining the minimum baseline of cybersecurity for municipalities that they developed. So the Mass Cyber Center and the Working Group released a municipal cybersecurity toolkit in October, 2019 to help municipal leaders assess the cybersecurity posture of their municipality and figure out next steps for protecting their municipal infrastructure against cyber threats. So I think we can move on to the next slide. And we are very privileged to have Stephanie Helm with us today who leads Mass Cyber Center. So I'll turn it over to Stephanie to talk about the Mass Cyber Center, the Working Group and how the work with the clinic came about. Well, thank you, Jungwoo. And thank you all for having me here today. This is a really valuable partnership in the work that we're doing. The Mass Cyber Center was established in 2017. I came on board in 2018 and we created the Cyber Resilient Massachusetts Working Group. Meets monthly, it includes state agencies, representatives from some of the local governments as well as some representatives from private industry with the goal of trying to improve the collaboration on resiliency as well as the planning efforts for the state. So one of the topics that came to the top of the heap early on was the challenges facing municipalities. Massachusetts has 351 cities and towns. There's no overarching county government as there is in some other states. So there's no structure that sort of put, consolidates folks within regions. That's all 351 different standards of resources available to them and different IT architectures. So therefore each city and town is sort of uniquely facing a cybersecurity challenge. So we kind of embarked on how did we, how to get a better handle on how we can be helpful to the municipalities in the state. I think the first thing that we did was created a subworking group that was really focused on municipalities and really tried to engage them and try to understand what their problem is in trying to improve cybersecurity. The next thing that we did was we partnered with the Massachusetts Municipal Association to conduct a survey of municipalities. We took about 12 to 15 questions just so that we could get sort of a sense of what the shortfalls were from their perspective and what they thought might be useful. And one of the things that came out of that survey was the pointing to the fact that probably about 10% that answered the survey had an incident response plan. So that meant probably about 90% did not have a plan. And to me that was sort of like a good way to start because if you understand the planning process you know that you will have to create a team and you have to ensure that you weren't using the same language and you have some documentation involved. So we felt like the planning would be the first step. And so we've wanted to create tools for the municipalities to be able to use in order to support their cybersecurity efforts. Jung will mentioned that we created an online municipal cybersecurity toolkit. That toolkit was a place where we could post resources that would be able to point cities and towns into the right direction. But it really was sort of a whole listing of things that they could do, particularly with regard to ransomware or other things that might be relevant to cybersecurity for the municipalities. So this sub working group actually came up with the idea of creating the four goal areas that would be the aspiration for every municipality to be able to say that yes, in these four goal areas we have something for our town that could provide what we would call the foundation or the minimum baseline. The four goal areas are an employee training program, a threat sharing program of some sort a cyber incident response plan. And then the fourth area was general best practices such as managing passwords or improving your architecture for some way. But it would be the four goal areas that a town could at least internally do their own assessment that says, do I have an employee training program? Yes or no? If I'm yes, then I've met the minimum. If I'm no, I haven't even got to the foundation. And so that's kind of the approach that we're using. And we were very delighted to have the clinic partner along with us as we were embarking on trying to raise awareness for this minimum baseline of cybersecurity and hopefully improve it. The clinic was already individually engaging with one or two or three municipalities during the course of a school semester. This would allow the clinic to kind of broadly help us statewide improve this program and also to be able to see where is it that municipalities are finding the most difficult time in meeting some of those goals. So I don't know if that gives you the overview that you were looking for, Jungwoo. Oh, that was perfect. Thank you. And I think we can turn it over to Abhi, give us a little update on our progress so far in working with Mass Cyber Center and we're hoping to expect more results in the summer. Yes, thanks, Jung. So I, along with some other student graduate students at MIT have been working on continuing this research project. In particular, the research question we're asking is how can the state and Mass Cyber Center in particular help at risk municipalities reach a minimum baseline of cybersecurity? And so we've come up with a sample of Massachusetts municipalities broken up by region of Massachusetts as well as city and town sizes. So we can get some really small towns and some bigger towns as well. And we've been asking them questions around their current progress towards the minimum baseline, the sorts of challenges that they face. And in particular, what resources and or interventions from the state would be helpful and how the current resources and the toolkit measure up to their needs. So we've only, we've just gotten started with this process and we don't have too big of a sample size yet. We're expecting more results by the end of the summer, but preliminarily just as a person who's been conducting this research, I think we see a need for more inter-municipal cooperation, as Stephanie mentioned, we don't have a county structure that would allow especially some of these smaller municipalities to really like pull together their resources and their knowledge around this issue. And so a lot of them feel kind of lost in that they say, well, I want to do the right thing. I know that I need to have employee training or I know that I need to make sure that all of my employees like know what to do in the event that they get an email that looks like a phishing email. But the way that they like figure out what to do seems a bit ad hoc in that they will ask people that they know, often like other people in IT roles at other municipalities and say like, hey, like when revenue are you using for this or how are you handling, you know, this problem? And so we definitely see a need for a more centralized like source of information about this and, you know, perhaps like in person or Zoom meetings between these different people who are in similar positions and find themselves like solving the same problem but they don't know that each of them is solving a similar problem. So that's sort of our very preliminary results but we're hoping as we're continuing this research and talking with more folks where we'll have some more actionables that will put into some sort of report that will, you know, evaluate a little bit of the state of this issue and the progress towards the minimum baseline as well as resources that the cities and towns themselves have told us that they would like and that they need. If I could add on, Abby, I think it's an important point to say it's nice to have a third party who's a neutral in this whole process asking the questions because you tend to set up the expectation like if the state is asking, what do you need? The answer is always going to be more money, more people. Okay, well, I don't know that there's a one size fits all that we can do for every single municipality. So it's kind of nice to be able to try to get below that request for sending more people more money and try to get more deeply into the issues without that expectation that you're going to be able to deliver on it. That I think is very helpful because then we can take all that body of work back and then we can kind of correlate to see what that is that might be most helpful. And then another point that Abby brought up that I think is valid and we've kind of recognized it at the center, very much when we, at the Mass Cyber Center we actually did have a contract to put on workshops for incident response planning. And we didn't run it as one big workshop for the state. We did it in the Homeland Security Regions. So we broke it up into the five regions with the hope that people who attended the workshop would meet their neighboring towns or cities and start to come together physically with a kind of a regional geographic connection. The problem was a pandemic hit and those workshops ended up being virtual. The good news is we got the products and so you'll see the products in the templates and the other information are on the website. But we did hear back from the people that participated that they were looking forward to doing something more regionally focused. So we've been working within the working group, working in the working group with the executive office of technology, services and security to see if we can facilitate connections regionally and have IT working groups that are regionally focused with the outcome hopefully of those local working groups building trust and relationships amongst themselves. And then that also allows us to have a better communication methodology where if something is happening in one particular municipality, we can alert people locally or we can at least get information from them to share with other working groups. We're adding that. Yeah, and I'll add that broadly speaking, we find that the people that we've been talking with so far are quite savvy and definitely think that this is an important issue and that often they're the person who is advocating for more resources, more work on cybersecurity in their own organization. So I think that's really encouraging to see. Yeah, go ahead. No, I was just going to say to Alberto, I'm eager to respond to questions that have been raised and the terrific questions, a lot of them about nuts and bolts, a lot of them about sort of philosophy or pedagogy but you're going to be the master of ceremonies. Yes, right now. I've been actually going through all the questions and we have a lot of questions regarding security issues with and how are you protecting for them? So I'm going to start with that and then I'm going to cycle through and please, I will direct the question that some of you but feel free to pass it along to someone else. And so the first one that I'm actually going to bring is the last one that we heard from Bo Yuan. Sorry if I mispronouncing. So it says that when engaging with clients, it is possible for students to make mistakes. So is there any legal protections for students? And I'm actually joining that question with another one and talking to, so you can talk us a little bit about that. Do you sign an NDA or just verbal commitment? I heard that you do memorandum of understanding. So can you walk us a little bit on that process? I think Larry. Yeah. So I started with the conversation with the MIT General Counsel. When we got the grant and we wanted to design the clinic, I said, if students are going to be working for cities and towns and hospitals, what liability do students have? What liability does faculty have? What liability does MIT have? How should we handle that? The first point that then this is MIT's response. I have no idea what another university might say. MIT says, students sign nothing. I cannot ask students to sign anything in terms of working with the client. That I, as the faculty member teaching the course, which is a course for credit, I have to sign the memorandum of understanding with the client. The client is typically a city or a hospital. It's the CEO or the COO of the hospital. It's the head of an agency or someone asked what is the reaction of the CISO, the chief information security officer in the places that we work. The answer is they don't have one in the most of the places that we work. They have an IT guy who's been saddled with cybersecurity in smaller places. Even in some larger cities, it's the IT office that's been given this responsibility. And so either it's the deputy mayor or the head of the IT office that we sign an agreement with. The agreement says we will produce a draft assessment. We will send the draft to the client. Based on their responses, we will submit a final assessment. The assessment will include a review of the answers, yes or no, do you meet the, what we would argue minimum standard on each of these 17 areas? If no, why no? And what do we suggest are things that they should do? We are not gonna take those next steps and do the thing. So a question from NIST in our outline of questions. And yes, we're happy to share the document with the list of questions and the question asking templates, everything. We're happy to share all of it with everybody. And Jung will say something about how we make that available to you. But the question would be, do you inventory all of your computer systems? All of your information systems? Yes or no? Do you have an inventory of them? Do you check that inventory? If there are things that need to be, as you heard from Rebecca, things that need to be patched or repaired? Who does that? Do you do that regularly? Do you have training for your staff? Not just when they're newly hired, but continuing training. Do you have an incident response plan? Have you tested it? Do you have cyber insurance? What is that insurance policy? What does it cover? Do you know what it covers or doesn't cover? Do you have you rehearsed the implementation of your incident response plan? Yes or no? Have you been attacked? Yes or no? What did you do when you were attacked? What have you learned and changed as the basis of that attack? These are the kinds of questions that students are asking. They're not just asking the client, but all of the staff and all of the related agency staff that you need to ask the questions of to be able to come up with an answer. And then in the draft assessment that students provide, we give our answer, our reasons why, and there's a section with suggestions for things they should do. Every student on every team has some responsibility, but it is student team that prepares the draft assessment. Every draft assessment is reviewed by all of the students and the faculty along the way. In the end, the assessment draft, I could have a mistake in it in that we should have checked no because we should have interviewed somebody else, but they're telling us who we should interview. We're doing the interviews, we're summarizing what they say, based on what they say, we're checking yes, no, spelling out why and making recommendations. If they get the draft from us and they say, we don't think you should check no in this category, we'll say, well, here are the reasons we said that, what's wrong? They said, no, no, it's just not good for us to have a report that checks no here. We say, well, we want this to be useful to you. We can check not clear, we can say we don't know if this is yes or no because we thought it was no, but you've given us some reasons to be concerned about that. We'll give you that in the document, but we're not gonna change it and say something we don't think is true. We could be misled by not doing enough interviews, we could have misinterpreted maybe, different things that different people said and drew a wrong conclusion, but in general, there's no individual student responsible for the draft and the final draft has gone through the client. So MIT says the statement that we signed at the beginning needs to say, here's exactly what will happen, here's exactly what we'll do. MIT is not gonna make any corrections for you in your systems and it's not gonna draw any final conclusions for you about your systems, you're gonna have to do that. So MIT general counsel said, well, you Larry can sign this and say that's what we're gonna do and that's what it says in the memorandum of understanding and there's no money changing hands and there's nobody that report goes to except to the client and if we use it for research, we scrub any reference to the actual place and then we use the data in our analysis. So MIT felt there was no liability for the students, they said they felt that I could sign that as myself as the person teaching it, I file that memorandum of understanding with my department but thus far that's been the process, there haven't been any issues. I don't think there's any liability because we are not physically changing anything in the place. We are giving suggestions and our reasons why and all the decisions are up to them. Thank you for that Larry. Again, this is just one case and here at the public university, the public interest technology university never wanted to push it to create more spaces like MIT has done. So I'm more than happy to connect you with Larry and go over those documents if that is needed. I wanna switch over to another question and I think that Avital or Rebecca might be the answer for this one. Sorry, before we move on to the next question, I just wanna add one quick point. In the, we provide a link to the online course in the chat, I believe. So the courses usually run most of the year. So as you've heard, the course is four weeks long. It's generally open for about nine to 10 months. Right now at this current state, right now it's currently closed and it will reopen in August. And once you, it's free to join the course. So once you join the course, you'll see some sample forms that are provided in the course. And as Professor Suskind explained at the very beginning, we have various simulations showing students at MIT engaging with client-like situations where they would go through these various processes and one of those is signing the letter of agreement or the Memorandum of Understanding. And we have a sample form there that everyone can download. So you can use that as kind of a platform to design your own and think about how you could use it when you create your own clinic, whether it's cybersecurity or not. So just wanna point that out. And the course is again opening in I think I believe August 20th. So right now you won't see a start date but it will begin in August and it will go on for another nine or 10 months. And John, we'll follow up on that. Where can people find the NIST diversity assessment and the survey questions? So John, I don't know whether we can do this right now but I have no problem posting a link to the 17 questions, which are in a memo that I think you might have handy. But also anybody can email me after this event. It's just my last name, S-U-S-S-K-I-N-D at MIT.edu and say, send me this, send me that and we will do our best to respond. The things that we have are a model of the memorandum of understanding, the questions that are gonna form the basis of the assessment that are boiled down from the over a hundred questions from the NIST framework down to 17. And you can see the format that we use that students fill in. The survey, the questions that we ask in the beginning to get the data from our clients that we then use to put the interviews together that supplement the survey that then allow us to fill out all the questions. So we're happy to share all of these materials. We're not assuming everybody will do the same thing but it will give you a basis to start with. You may not wanna focus on the defense of social engineering side of cybersecurity. You may wanna not work with public agencies but rather private agencies in which case there are different kinds of things to be concerned about. But we're happy to share all that material. Harry, and I think a really good followup question on that is when you were creating this, what are the most half tools, the equipment, the software that you needed to include to build the lab? And this question comes from Hassan Zemir. Yeah, well, it's gonna sound maybe incomplete but we needed to make the online courses because I needed to supplement my meager background and certain technical questions with what other faculty from other parts of MIT could help me prepare. So we produced, video produced this four week, four module program. And because of the grant from the Public Interest Technology program, we had the money to go to MIT and say, we're not asking you to pay for it, we're paying for it. We're producing it and then it's gonna go free on edX. So you're not gonna sell it after we make it. Once we had that training material in one place, we use it every time. I don't have any other costs related, I mean, yes, there's a TA or really a partner, if jungle weren't here, I would have to have another person with his background capability and skill to help me manage the clinic because we offer this every semester and I'll come to the question of how do you recruit new clients and who does that between semesters? But I need a very capable graduate teaching assistant and that would be different in different universities how that's financed. My department helps to support this because the clinic is a course in the department. But other than the TA, because we've been online, I haven't had to pay the costs for students to travel to a city where they might have to stay overnight for two nights. So they have to rent a car or stay at a hotel for two nights and do 20 interviews. Now everything's been online. So there've been no cost associated with travel to places. There's no other equipment. We're not doing some of the things that cybersecurity consultants would do where you need various forms of various kinds of software programs to test or simulate to show a client the things that are not working in their place. So Stephanie knows more about this than I do but we're not doing that. So the cost of this clinic is my teaching time, Jung's teaching time and the preparation of the initial materials. That's it. Now the additional research that has spun off from the clinic that we're doing with Stephanie, that's a kind of thing where you need to raise research funds whether from inside your department or outside but to actually do this clinic. And now what we're saying because of the initial foundation support, this online course, and there are several universities around the country that are using the same first four weeks at no cost. And if they want the students to pay the $100 to take the test, fine. If they don't, they don't have to. But other places don't have to invest in creating all of that teaching material. It now exists. And after that it's faculty and graduate students to supervise the student work on a weekly basis with the clients. So there's not other costs. There's no equipment that I know of. I'm not using any. But that's because of the focus that we've taken. If somebody else may need to invest in other specialized supplementary teaching or software or other programs, we don't have those costs. Jung, am I skipping anything? No, I think you're right. I mean, we are exploring different ways to improve our ways of keeping the data. But again, that's not a separate huge cost that we're expecting. We are able to work with the resources that MIT provides backed by MIT firewall and things like that. So no, I don't think, and I think that's because of the focus that we're taking, the social defensive engineering focus. If your focus is more technical as far as using certain software for cybersecurity per se, then I think that might be a different story. But yeah, that's my understanding as well. Why don't you just say one more word about what we had to do to promise our clients that we're taking good care of the data that we're collecting while we're doing the assessment? Yeah, so we also outlined those promises or the words that we would live up to when it comes to keeping the data safe in storage and transit, there are certain things that we definitely abide by with the students and everything that we exchange or receive from the clients, we always use Dropbox with dual authentication. And we don't send anything in the email, we don't receive anything in the email, always goes through Dropbox. And so there's always this single channel of exchange of sensitive material. We absolutely are very careful about confidentiality and anonymity. The report is only shared with the client, no one else. The client has co-copyright, so the client is able to share the assessment with people within the city, with the leadership in the city, but it doesn't go out of the clinic or the client can decide to use the report in the way that they want to, but it doesn't go out of the clinic with our discretion. So I think that pretty much covers everything, although if I'm missing anything, please feel free to add. Thank you, John-Boo. So here's another question that's changing a little bit of the topic, but how are clinics and assessment received by those public institutions? Larry, you already talked about this a little bit about how chief information security officers, and well, sometimes that there are no chief information security officers, but how are they received then and how is that relationship done over the course of the project? I'm also gonna ask Rebecca, I'll say something, but then I'd like to ask Rebecca to speak about this because this was the first time we're working with hospitals and it took a long time to work out the basic understanding, the memorandum of understanding with the hospital, and we had to deal with the leadership in the hospital, and she can speak to that in a moment. In terms of dealing with the cities, it's not like dealing with a company, right? If I'm going to a private corporation, they have a CISO and they have a CEO and a COO, and if I were gonna have an agreement with them, all three of them would probably have to sign whatever the letter of agreement was. If I'm going to a city, a major city in New England, and Stephanie knows this better than I, it's not clear who speaks for the city on these issues. They have somebody who's responsible for cybersecurity, they're not often labeled the CISO because they don't wanna pay them what they'd have to pay them to be a CISO to compete with CISOs in the private sector. So it's the cybersecurity responsibility added on to the IT person's job description and that is the top person with regard to these issues on the technical side. Often we have to go through somebody in the chief executive's office in addition. So we might have an agreement signed with the chief, with the chief information technology officer of the city, but it has to be counter signed by the deputy mayor or whoever's responsible for finance for the city. So they're both our client, but it's not signed by the mayor. It's not signed by the head of the city council, but it is signed by, and so when the report comes in, as you heard from Rebecca, it goes to the person that signed the memorandum of understanding. They wanna use it in many instances to get the city council or the mayor to please approve a new budget line for cybersecurity. Our report thus far, excuse me, thus far has been helpful in that regard. Someone in another question asks, what are your measures of success? If the person who hired us says what you gave us, allowed us to now begin to get the resources and to mobilize the mandate to do the specific things you said we needed to do more of, we think that's a measure of success. We can't make those things happen, but if they can use the report because of, as Stephanie said, our sort of neutral or third party status, oh, it's not just a self-serving statement from the IT guy that he wants an assistant. It's a serious document that says you, we gotta do more of this. You gotta get more training for your people. You've gotta do a different kind of risk assessment. You've gotta have new backup systems and maybe they're gonna be in the cloud and that's gonna cost you money. And you never thought of spending that money before. And it's our report that's helping them make that case. That's our measure of success, but so far the reaction of the people who have to make those budgetary and managerial choices has been, oh, this is a serious thing, even though it's done by a group of students in the university. Rebecca, you wanna add what your experience was dealing with the hospital leadership? Sure, so the hospital leadership because of the regulation, right? So they were familiar with, we know that this area is important and they had people who were dedicated to security, but they were very, very strapped because it's a cost center, right? So they're not revenue generating from a tech standpoint. They're not bringing in new patients. So that was something that we had to be really mindful of. So we had really good like stakeholders to work with who we work with them on how to craft a really good assessment, you know, to Professor Susskind's point. We're trying to make a case for them to get more resources to have the conversation. We did have an opportunity to speak to the CEO of the hospital. And once the letter of memorandum is signed, so kind of skipping the initial engagement piece though, they, there's some sort, there's bias to want to help, right? Because why take the time to talk to people if you don't want to do anything in security? So you end up then kind of establishing a better relationship and that kind of serves at least in some cases as additional buy-in because they're kind of aware that, oh, we've engaged this clinic. I should at least do something with, you know, the fact that we've had this partnership now. So that was also, I would say pretty interesting. The other thing, and this is probably widespread, right? It's not just hospitals. I work with teams for municipalities as well this semester in particular. One thing that we do, which is nice and would recommend doing for other clinics, whether it's cybersecurity or not, is a lot of times folks have a conversation that requires money and then look, it's like, oh, I have a solution. Can I have money for it? And the answer is no. And what we sometimes end up providing is, okay, this solution didn't work. Have you thought about X, Y and Z in security? It's called like a compensating control or something that it does not the best solution but is a solution. And allowing folks to have the resources, to have the conversation of alternatives has been really helpful and it helps us establish better relationships. So I think it's a really important point as well, broadly, not just for hospitals. Berta, does that answer at least something? I think so. I think so. So just rephrasing the question by Hassan as well, do you have a specific best practice that you would say that it's a top one when dealing with local municipalities and your invasive hospitals? The Stephanie's team and her task force have had to deal with the question of minimum necessary and best practice. I've avoided the term best practice at every corner because in whose eyes, really, so what we're saying is at a minimum, if you have not assessed what your most important data resources are and you don't know what it's gonna take to replace them if they're lost in a cyber attack, you're in a bad place and you can do better. How much of a cost-benefit analysis or a risk assessment should they do of all of the potential loss of all of their different data? I can't tell them that. But what we say is what's gonna be the biggest problem? What's your most valuable data source? What's the hardest to recreate? Don't you think you should have a backup? What have you considered? What are the different options, as Rebecca was saying, about how to back up your most important resources? Well, you don't know what your most important resources are if you haven't answered the kind of questions we're asking of what are your greatest vulnerabilities, and risks with regard to your data? And then we raise that in the context of insurance and we say, what are you gonna do if you're attacked and you're asked to pay a ransom? Now, most cities would say, the FBI and the National League of Cities say, don't pay ransom, you're just encouraging bad behavior. Yeah, but if you know the story of Baltimore and they were said we're not paying this $80,000 outrageous ransom request, fine. They lost their data of the huge portion of city financial data, cost them $16 million to recreate those data. So should you have paid the ransom request? Maybe. Now, if you had had backup and if you had had an incident response plan and had practiced it, you might not have been attacked and you might not have had to pay the $60,000 or the $80,000. But at the point in which someone's asking, what's best practice? Should you pay ransom? All we can do is say, have you thought about what you'll do if you lose control of your data? If you haven't even thought about it and you haven't thought about what the most critical data are and what the most costly would be to recreate and you haven't considered your backup options, you're not prepared. Best practice, honestly, I don't know what to say. I have lots of colleagues who offer their services when someone's attacked and negotiate with the hackers to try to reduce the ransom request. And it does turn out that you can reduce the ransom request even though you don't know who you're talking to and you can't talk to them directly. A lot of success around the world, negotiating with ransomware attackers to reduce the amount. But is it best practice to negotiate or to have a phone number or an email address of a development who's gonna do that for you? I can't say that. So I'm sharing my anxiety about the term best practice in this context. This is why I'm so attracted to what mass cyber has done, which is say, what's at a minimum that you should be doing? And here's the tools for doing it. So what's your excuse? That is, thank you for that. I think we've run out of the questions and I'm really thankful for all of your panelists and the great job that you're doing with this project. I would like to have just a final question. This one is to Stephanie as a process. What has been the major barrier for creating this minimum baseline for cybersecurity? Cause you do, I'm sure that you are targeting public and just within it, but what has been the biggest challenge on that? Well, there was a lot of trying to put yourself in the shoes of a municipality and trying to figure out if I was told this is the dictate from on high, whoever that on high is. What's a reasonable expectation of an under resourced and overworked municipality to be able to meet? And so the working group worked really hard to break it into just four areas because you could say, well, the CIS controls cover a lot of this or NIST covers that CIS controls and NIST are way above most of everyone's head at the point of the people that we wanted to help. So the first step was to try to make it the minimum and make it approachable. Once we determined that that was about where we wanted to go, now it's just explaining what each of these areas are and ensuring that people understand the value of the goal and why they should make a decision as a leader in their community and a leader for their town to try to at least make the minimum in each of the four goal areas. So I think that barrier is trying to get contact, a level of trust with these communities and explain to them this is trying to help not to, it's more carrot than stick. It's voluntary on your part. There are things that you can do that are free that just take your time that you can potentially improve your resiliency or make you at least less vulnerable than you might be today. So the barrier would be, I think, communicating, establishing trust, making people aware that the program even exists and why it exists. I think that's the biggest thing. Alberto, I know there wasn't a question about this but I wanna loop back around to the point we started at and to the whole point of public interest technology, I mean, we've chosen one way of working on public interest technology and we're very thankful that the foundation was willing to provide resources for us. But I just wanna emphasize the fact that it's my personal belief that universities and colleges, however strained they are from all of the internal, external pressures and forces that they face, have social equity responsibilities. They have social, public responsibilities and the notion that while we're educating people and doing research, we're helping the world. What do you mean? We're already meeting our social responsibilities. I would say, no, those are the things you wanna do and have to do. There's other things that you ought to do to be helpful and I would put one category of these is cities and towns around us need help and all the people that are served by those cities and towns need help and we can contribute. There can be clinical learning opportunities along the way but I really believe that public interest technology involves colleges and universities doing things that they don't already choose to do because they want to because it's part of their core mission but it's because they ought to do that to help places that don't have assistance. And we have the option, we could do a cybersecurity clinic and focus on corporations and ask them to pay and we would have that money coming in and we could support students with it but that seems to me, antithetical to the idea of public interest technology and so- Thank you, Larry, for that. Thank you so- I wanna underscore the public interest portion of what we're doing and that's all. No, no, of course. And I mean, that is something that is part of the debates that we foster within the university network and on that definition of what public interest technology is and where should it be focusing on. So thank you for that position. And I said that that was the last one but I really wanna bring it back to Rebecca and Navital as a student. This will be the last question. Can you give me what was your main learning from working with clients outside of the university as students, what was the thing that you would highlight? Yeah, so- I'll show you Rebecca first. Sure. So just so you know, I spent seven years in industry in the private sector so I do work with a lot of clients. I think the difference here is that one of the main sort of takeaways that I had working with clients is that everyone was I guess very friendly and very engaged and very resource strapped. And like that dynamic was something that I think was both new and important to learn, right? Cause if you're gonna be sort of in a public interest field especially if you're working in security. So for me, like one of my main takeaways was how to be better partners with folks that you may not know very well but you have a similar goal, right? So those communication strategies was really important, understanding their perspective and being very differential to folks who, if they've been in the same job for 30 years, right? I should be listening to them and just making sure that we are focusing on the partnership side of things. So I think those were sort of the highlights as a student as opposed to a TA. Avi. Yeah, I think I'm gonna echo that. I took the, when I was a student in the clinic I was an undergrad as I was in a different position than Rebecca was. And I think it was like one of the first opportunities I had to really engage with like someone who I thought of as like an adult and someone who was like, knew a lot more than me in a lot of ways but like really engage sort of a professional relationship where you're trying to build up that mutual partnership and really like provide a service but at the same time, really listen to what people are saying and how they came to the conclusions and they came to the state that they are in right now. And so sort of like developing those professional relationships I think is a really good skill for students to develop in general. And a good way of developing those is through practical like field-based courses like the clinic. Well, and I will close it off with that. Thank you for talking to Rebecca. Thank you Stephanie, John Wu. And thank you Larry for having this webinar with us. This will be again posted on our YouTube page from America. I'll see you on the next one. Thank you.