 Good afternoon and welcome to policy at DEF CON. I am Heather, I am your room host and this is demystifying hacking for state government officials. A few announcements before we begin. This talk is being hosted on the record. As a courtesy to speakers, please make sure that you set your cell phones to silent. We will have questions at the end. Please use the microphone. Because this is recorded, we wanna make sure that everyone can hear it on the recording. As a reminder, the photo policy prohibits taking pictures without the explicit permission of everyone in the frame. That is not something that everyone in the audience has done. So please, please be careful. And with that, let's get started. I will let Lindsay Fulson introduce the panel. Thank you. Hello everyone. I know we're the last panel of the day, so we'll try to keep it brief and not too boring. We're here to talk about demystifying hacking for state government officials, which has been a collaborative effort of members of the National Association of Secretaries of State. Very quickly, I want to introduce my panel. My name is Lindsay Fulson. I'm the deputy executive director for the National Association of Secretaries of State, which is the last time I'll say that. NASS is our acronym. I have with me Jack Cable, senior technical advisor for CISA. Michael Ross, deputy secretary of state and chief of staff for the Iowa Secretary of State's Office. Jason Engels, founder and CEO of Engels Information Security and Brad Manuel, chief information officer for the Louisiana Secretary of State's Office. I know this is an on-the-record session and so I wanna note that only my remarks may be attributed to NASS and each panelist's remarks may be attributed to their individual organizations. So very quickly, what we're gonna talk about is background and history on demystifying hacking for NASS members and those they work with. The idea of Michael Ross here to bring a hacker con to NASS impacts another efforts of these efforts, state perspectives and lessons we've learned and then we hope to get the whole room involved in some brainstorming and discussion. I wanna very quickly talk about who NASS is. We are a membership organization representing secretaries of state and in a few cases where there's not a secretary, lieutenant governors across the country. Secretaries of state, I mentioned this this morning, we're not the US State Department, we represent the secretaries of state in each individual state who have many roles and responsibilities, not just elections as many think is that their only role but about 40 members of NASS service their state's chief election official so that means they are legally, it's a legal designation where they oversee election administration in the state and what exactly it means varies dramatically state by state. NASS's role is to serve as a medium of exchange for the secretary of state offices. We act as a conduit of information sharing between the states and their federal governments, partners, other partners and even more importantly, our most important role is sharing ideas and information, lessons learned across the states and territories. As I mentioned, secretaries do more than just elections and we have initiatives in the major areas of overlap across the secretary of state offices. They all do something a little bit different but these are the areas where they tend to have similar responsibilities across the state and so that's where NASS focuses its attention, elections and voting, cybersecurity, business services so that's registering and renewing businesses in the state sometimes providing information and resources to businesses, particularly small businesses, state record keeping and archives and more. So why have we been focused on demystifying hacking for members as an association? As you all know, the 2016 elections heightened attention of cybersecurity and election administration. Previous to 2016, cybersecurity was a thing that all secretary of state offices were focused on in some way but that dramatically increased after the 2016 elections when there was nation state targeting of the internet connected infrastructure that relates to voter registration. In 2017, elections were designated critical infrastructure by the Department of Homeland Security and that really changed the attention that was given to election cybersecurity by the federal government but also by the public and that led to increased security research attention on elections and on secretary of state offices including the launch of the voting village here at DEF CON. We'll get into this a little bit. Let me not get ahead of myself. We'll get into this a little bit in just a minute but as many of you all know that the initial relationship between NASS and its members and the election community as a whole and the security research community got off to a rocky start. As I know happens, I'm learning here happens with just about every single industry and so we began focusing on relationship building and education between the election administration community particularly NASS members and the security research community. I started with NASS in 2018 and that was one of the first things I was tasked with and so I just wanna provide a little bit of background on the work that we've done to bring us up to today. So one of the first things we did is look to good models like where is there a good positive collaborative relationship and everybody pointed us to medical devices and so we got in touch with and the cavalry, Bill Woods and brought him out to NASS. We got connected with Jack and several other independent security researchers and initially it was an education effort, right? We wanted to learn, NASS wanted to learn more about who hackers are, what you do, what motivates you and so initially it was an education effort that kind of morphed into a focus on networking and relationship building. We started hosting meet and greets between the cybersecurity staff and the secretary's offices and some folks in the hacker community and then that really led us to where we are today which is really bringing hackers to NASS, trying to bring a sample of what is going on here to our conferences and that's why we're here today because we're really interested in hearing from you all on good ways to build off of those efforts. I'm gonna ask Jack to provide a little bit of perspective on the background and history from the hacker side. Thank you, Lindsay and first of all, thank you everyone for coming on 5 p.m. on a Friday night. I know we're the only thing between you and happy hours so we'll hope to keep this engaging and maybe chat with some of you there but as Lindsay alluded to really, there has been a storied history between hackers and the elections community and happy to say now we've made quite a bit of progress, something we couldn't even have said say five or six years ago where I think especially after the 2016 election there became a recognition of how crucial security research was to helping build security into our elections and something that has been reflected through the work at NASS, through the work elsewhere as well, we see it in the DEF CON community as well but I also saw this myself where I come from a background as a security researcher later ventured into some of the policy world but my foray into elections was when I was 18 and registering to vote for the first time in Illinois and came across a pretty serious vulnerability in the Illinois voter registration database as someone who didn't know much about how elections were administered in Illinois took quite a bit of discovery to find even someone to report that to and through the likes of CISO was able to eventually find a contact, eventually was able to get that vulnerability patched but through that saw really the benefit of collaboration once I got in contact with the Illinois Board of Elections they were very grateful for that but also the necessity for there to be strong pathways between security researchers and election officials to ensure that when vulnerabilities were found security researchers were able to make that known so since then we've had some remarkable progress we're now up to by my count five states including Iowa, Ohio, North Carolina, Idaho and Minnesota who operate vulnerability disclosure programs allowing security researchers to report vulnerabilities they've found in their systems the majority of major voting machine manufacturers also operate vulnerability disclosure programs so we still have progress to be made but really we are light years ahead of where we were even a few years ago and I'm excited to see states and election manufacturers like really welcoming security research to help secure their systems. Thank you Jack and I think we're gonna continue to see that number growing here in the very near future on states with vulnerability disclosure programs. So I'm gonna pass it down to Michael in just a minute but after Michael's first trip to DEF CON Secretary Pate who's here with us was our president at the time and he was hosting our summer conference coming up it was 2021 so pandemic summer conference already really busy and Michael gives me a call and says I wanna bring DEF CON to NASA at the summer conference and so I'll let him talk just a little bit about why and what we did. So I woke up one day and discovered we were part of the critical infrastructure and I had no clue what that meant I had to Google it and honestly the federal government didn't know who ran elections and so they had to kind of figure that out too and so we started off pretty rocky and then I woke up another day and a nine-year-old kid had hacked the Secretary of State's computer system and changed the election results in less than three minutes. That was a little bit of a surprise and so we weren't quite sure that was quite accurate and so just hearing that we had the opportunity to Secretary Pate allow me to come out to DEF CON and it was really kind of suspiciously and kind of in a guarded way and what the heck is going on and who are these people and what are they doing and how did they hack it in less than three minutes and but what I discovered and I came back to Secretary Pate is that what I thought the White Hat hack with that community was was not what they were I found them to be patriotic they aren't anarchists I mean maybe there's one or two anarchists out there in the crowd but the majority of you guys were really elections are important and you guys see things differently and so we just kind of wanted to go that next step we had breakfast with Bo or lunch with him to kind of discover a little bit more about you guys and as we started to kind of trust that we started to look at what can we do because we have both cybersecurity which is the critical part but we also have to reassure the public that elections are secure and that is also a critical piece and I think that the White Hat hacker community plays a part and helps us do both of those things So based on Michael's idea we created an event at NAS that we now call hacking demystified and I called up as soon as Michael came to me with that idea we got on the phone with Jack with Bo with a couple of others who are organizers here at DEF CON and we're like you know Iowa wants to bring a hacker con to NAS what do you think we should do? You know recognizing that most Secretary of State offices didn't have representation at DEF CON folks had never been here and their idea was like let's start with some of the fun stuff let's start with some of the low hanging fruit like less intimidating things and so we brought a mini lock picking village soldering village we had IoT hacking village we had just opportunities for NAS members their staff and the folks we work with to get a taste of what goes on here and you can see some of the fun activities that happened at our first event in Iowa up here on the screen and then you know we got a lot of good feedback we got a lot of constructive feedback we learned a lot from that event and decided we wanted to build on the success and up next was hacking demystified 2.0 but I want to ask Jason next so Jason's company is a corporate affiliate of NAS and they have been incredibly supportive both financially but even more importantly operationally with running these events they've blended so much volunteer staff support to making this happen and so I just want him to talk a little bit about the importance of these events and why you've been so supportive Sure thanks Lindsay and thank y'all for being here so we got involved in NAS as a sponsor and we're going to Iowa for the conference and there's a hacker conference going on during NAS we got to be part of that so we got to be part of it and then we're thinking what do we do about next year when we go to Baton Rouge because we're a Louisiana based cybersecurity services company and so we do a lot of reach response and we thought might be helpful to do something about ransomware because people hear about it in the news and I kind of think of it as like the digital equivalent of an AK-47 you can do a lot with it so we wanted to make sure that people understood the basics around how that happens and what happens after everything gets locked down, et cetera and we were able to think through and produce kind of this narrative with everybody's help we were able to put it together but more important than that narrative for us was the opportunity to get universities involved and to get like students who were in cyber majors to come in and participate that again was something that we picked up in Iowa with the universities contributing labor and support in Iowa we were able to do that with some of the schools in Louisiana and what I saw there was kind of this wonderful combination of government, academia, industry and interested parties and it was a really positive experience for everybody folks got to make connections and discuss things where there probably wasn't a lot of overlap before they realized where that might lie afterwards and how they might be able to connect and start discussing ways to help protect this critical infrastructure that underlies democracy and Brad's office, the Louisiana Secretary of State's office was again I think president and host of this conference last president and host of this conference and so Brad was one of the key planners anything you want to add about the event in Louisiana? So Jason kind of stole a lot of my thunder there but you're good so anyways when they brought it to us and the topic became ransomware it fit very well with where we were because in 2019 Louisiana was struck with ransomware statewide pretty heavily during their school board and it all occurred right around an election which created a narrative that was false so we wanted to take and demystify it worked perfectly, demystify the thought process behind ransomware and understanding that there is a structured response to any type of incident and then also bring in like Jason had stated bring in the education portion bring in volunteers all over the place security researchers and just keeping that line of communication open is really what we were trying to push forward and I think it was a complete success. Agreed and I do think Louisiana is the most collaborative event we've had where we had several hackers there we had aspiring hackers we had a lot of different folks involved in that one and in this scenario we did introduce a vulnerability report from a security researcher so that folks would have the opportunity to understand kind of how that would be handled if you have a vulnerability disclosure policy versus if you don't. Jack do you wanna talk a little bit about that? Yep so really had two main goals with the event the first as Lindsay mentioned was to put attendees in the shoes of hackers and I imagine many people in this room have been the scenario where you find vulnerability that's really serious and you know it needs to get into the right hand so it can be fixed and how challenging it can be without having a defined point of contact not knowing who to disclose it to because every day that this is out there is another chance that can be exploited so putting participants in the shoes of security researchers to understand the value of say having a vulnerability disclosure policy being able to have a channel where you can report it being able to have a legal safe harbor that assures you that no legal action will be taken for the good faith reporting of a security vulnerability I think really was crucial to kind of helping attendees understand the importance of that and then the second part was to really show that a lot of these vulnerabilities aren't these kind of crazy complicated attacks that only kind of top experts can understand but the truth is that majority of vulnerabilities attacks are quite simple leverage on sorts of vulnerabilities that are repeated all over the place so being able to demonstrate that and kind of in combination with the fact that applying a few relatively simple defenses can help block off entire paths of attacks we hoped would be able to kind of help paint the picture that security isn't some insurmountable thing but rather by taking some pretty basic steps can eliminate especially with ransomware which is financially motivated targets that the most simple vulnerabilities can prevent a lot of that. Thanks Jack and yeah, while many secretariat state offices developing vulnerability disclosure programs and getting to that level of maturity is a positive thing we also wanted to show like even if you don't have that and you get a vulnerability report it doesn't mean panic, right? These are some of the ways that you can work through this and Jack did a great job working with states on that. We also after the success of this event really saw a lot of interest from others the National Conference of State Legislatures for example reached out to us and that sat down with them and helped them plan, you know, get started on the planning of an event like this we've seen states take the idea and run with it and so it really is evangelizing a bit and so our most recent hacking demystified event was at this last summer conference we were in Washington DC and as we were planning hacking demystified 3.0 and trying to pick a theme one thing that we could not ignore was that we had many new secretaries of state and some of those folks were new to the elections community and their staff as well so we didn't want to, you know, get into a more complicated topic we kind of wanted to go back to basics and we chose behind the breach for this topic and Jason really helped kind of lay out the strategy for this so I want to turn it over to him to talk about it. Yeah, so we were trying to really tell a story about okay, you hear about data breaches let's talk about the various different ways that they can happen let's talk about the kind of impacts that they have let's talk about how people can identify ways to respond, how to reach out for help and educate essentially folks on the risks of a data breach happening and then what to do about it and we had scenarios where we had we of course, because LLMAI is the new flavor of the month thing that everybody wants to talk about we had to talk about a scenario that involved that and we also had other examples of email phishing and just your typical types of entry vectors for bad guys to get in and wreck havoc and just being able to again have government, academia, industry and other interested parties participate in various scenario parts was very helpful, I mean I thought that people got a lot out of it and they were able to, this time I guess the thing that I would say was a little different was we had folks moving between stations almost like a spoken hub system so they go to the station, they go back to like center base they talk about what they heard or what they saw and then that would lead them to the next section and so whereas like Hacking Demystified 2 was more of a serialized thing where people went through these stations three was more of a hub and spoke type thing I think that people probably got more out of that because they were able to essentially pick your own adventure but it was somewhat controlled in the sense that you were able to get some information part of the scenarios, go back, discuss that put the whole picture together and then kind of add to that over the course of it. Thanks Jason, yeah this one was a lot of fun in terms of the activities and we had a lot of secretaries involved we had a spear fishing competition which was the competition was fierce actually Brad down here was one of our winners one of our master spear fishers Okay so I wanna move along so we can open up the room so really quickly I'm gonna ask States to talk about just kind of in this light some of the things that you've taken and run with either either leading into this or as a result of some of these collaborative activities so I'm gonna turn it down to Louisiana first and just to talk a little bit about your collaborative approach to resilience in Louisiana. All right so this event basically has opened up doors for continued communication whether it be with third party vendors or educators or researchers it's opened that door for us to have that conversation it's allowed us to realize the need for having internal security as well as someone else looking out a 24 seven sock. It's opened up conversations for even within NAS on other states that's been a huge effort for us to be able to call on another state to say hey are you seeing something like this? What are you reacting to right now? Maybe I'm seeing the same thing it's it's been great on that effort as far as opening up doors is the biggest thing for us. Awesome and then Iowa has been one of the states that's really led the way in the NAS community at embracing security research and so will you just talk briefly about that? So we've been encouraging other secretaries of state but also even within the state of Iowa other agencies to put together vulnerability disclosure policy to invite researchers to come and to look for vulnerabilities. I think two reasons why I think that's important. Number one we've had the feds come in to do a pen test and we had the Iowa guard come in and do a pen test and then we had private company come in to do a pen test and then we opened it up to researchers and you guys find vulnerabilities. I'm scratching my head how come you guys find things? I think you guys your wire different. You see things differently, your puzzlers you have different perspectives and I think that's important. And then second we want to reassure the public because if I'm government and I say government says we're good some of you are going to say oh yeah government says we're good yeah it's you guys and I think for us to be able to say but we're also inviting white hat hackers researchers to come in. I think also helps to reassure some people that yes not only is government not only are these military and those folks helping out but also researchers are and so we just went with the bug bounty program it's the I think the first secretary of state but the first agency in Iowa we've had positive results there's been a couple of things that they found that was out of scope so it didn't affect us but they were some fairly major ones for some other agencies so we passed it on to them and also encouraged them to do a bug bounty program and I think one of the areas that we're probably looking at next is about half of the counties in Iowa don't have an IT and or the janitor's kid is the IT director and it's probably does a really good job and so part of our challenge is how can we help those counties without an IT infrastructure and I don't know how we involve or work with with your community but I think that might be part of the solution you know down the road how to do that we are doing another bug bounty with three counties and the results are going to go to all the county IT directors and we're our suggestions if three counties get the same vulnerability you probably should do something immediately and but again I see that challenge in the future how do we maybe work with you to figure out how do we assist those counties we have some counties 5,000 people more dear than there are people in the county and so they don't have you know 100 200 300 thousand dollar budget for IT and so I think we need to kind of figure out how do we work with you and with others to assist them in that. Okay, I know everyone is ready to get the party started so I'm gonna quickly go through this part and then we'll open the room up so lessons learned and recommendations for folks in the room coordinate directly with the folks who are implementing and overseeing systems right I think that it's not always easy and you don't always know where to go but I encourage you if you're looking to support government with your security research try at least to coordinate directly I think that both sides will find it a more productive relationship on that note it's really important to meet each other where we are I think that when we say that the relationship between policy makers broadly and the security research community or the election administration and the security research community got off to a rocky start there were just a lot of misunderstandings on either side and not understanding each other's role and you know it's really important to meet each other where we are and have conversations and get to know each other and respect each other's expertise and be willing to learn something from each other and then of course some of you are probably thinking like I don't think hackers in the election community have a perfect relationship no, that's not the case collaboration does not eliminate disagreement we are still gonna make each other nervous but it does increase the value of the work that you're doing and so at this point we're gonna open it up and we can go down the line again if we don't get a lot of questions but I do wanna note we didn't, this wasn't our focus today but demystifying goes both ways just like policy makers or policy implementers or election administrators don't understand exactly the work that you do you all don't exactly understand the nuance of the work that we do, right and so try to understand reach out to the people who can give you that trusted information who can break it down for you explain it to you NAS is always available to connect folks with their own secretaries of state we're always there to give the national 101 on how elections are run in the US and what a secretary of state is so please be open to having the work of government demystified for you just like we're very open to having your work demystified for us and with that I wanna open it up first we can take questions but I also put some prompts up here where we wanna have a bit of a conversation so some things we wanna think about are what are remaining misconceptions you think that we have about your work and what's the most important thing that you wish government officials knew about what you did or if you're a government official who wants to add to the conversation here so how do we continue to improve collaboration from the NAS perspective specifically we've done these three events we've done a lot of networking we'll continue to do work but kind of what's the next step in this process we wanna hear from you all so go ahead and please use the mics thank you hello hello yeah I'm from Baltimore like it was really well publicized but 2019 Baltimore got affected by a really significant ransomware attack took down significant portions of our IT system and then there's been routinely ransomware attacks against medical databases and a whole host of other things and I think in response to that or in the midst of that Larry Hogan made a pretty significant investment in creating a director of cyber resiliency with a decent budget attached where they can sort of run like there's a contract for like a red teaming company to go in and sort of like try to maybe also run essentially try to influence the IT system to increase its resiliency but with a dedicated like significant budget to do that as well and I'm just kind of curious from y'all's perspective is that something that you see in the conversation at other states potentially making similar investments thank you I'm from Baltimore as well so I'll let the states weigh in on this one right so Louisiana has taken effort since 2019 we also our school board was hit with ransomware across the state and the state identified that we needed some sort of dedicated cyber response right so the Louisiana Cyber Security Commission was created and under that Louisiana National Guard is an incident response team is kind of what it's built as and then the election there's an election security committee on that which I sit on that as well so that was one of the responses and each year legislation is passed to present more in the budget and just this year that commission was in executive order but just this year it is now legislated and mandated so we're taking a serious effort at that we understand that it's needed and we keep putting our money where our mouth is on that front I just and yeah I think it's a great idea I think maybe that's an area we might be going into I think one of the things too that you need to know is that elections in each state are probably a little different and even within the state of Iowa we have the federal elections but we oversee the elections we also have counties our voter registration file we're in charge of that but the counties have to do certain things by law and so we've had to kind of work in that area we've worked with our state to get in point detection on every piece of equipment in the auditor's office we were able to do that working with our state OCAO and so I'll bring that up to our state OCAO about putting together or seeing if there's any contract possibilities to actually putting together some red teams to try and do that so it's a great idea Anyone else? Who's up next? Trevor Timmons So you mentioned that you brought in folks from universities to kind of help these sessions trying to bridge the divide maybe between government officials and the cyber security research community I mean have you seen some examples where that may be paying off in the future or do you see some potential for where that might go in the future? Go for it, Jason Yeah, happy to talk about that so everybody talks about the job shortage or the lack of qualified talent in cyber the number you hear today is there's 700,000 open jobs in cyber and we all know, well I mean having done this for a little while I think most folks know that the real challenge is not so much that there's not enough talent it's that there's a huge barrier to entry to get qualified, to get into jobs so folks go into cyber great, they've got a degree and nobody wants to hire them because they don't have the magic number of things they don't have that CISP yet for the entry level job so what we found was that helping so bringing university into this picture and having real discussions about election cyber security with them in the room helped them understand what they needed to be preparing their students for when it came to the job market so that was one thing just getting a good look at what are the real challenges what are the positions that these folks need what are the skill sets that are necessary to support cyber risk management for the election vertical so to speak so that was one big win and then I think also just having their voice sort of helping to inform where they see issues in their own academic environment where they see problems there's a lot of commonality between different industries there's overlap and taking the time for folks to get to the common ground of okay we have this problem too what did you do about it because maybe you got an idea that we don't and we'd love to hear it because we might wanna go apply it to our side that was very valuable yeah and I think another key point in involving students in this is just teaching them about the space right as was mentioned earlier a lot of folks in the cybersecurity space that have these patriotic motivations and really wanna help improve lift the collective resilience of the US and kind of what better way to do that than support the cybersecurity of our election administration process but elections the way that we run elections in the US it's a very complicated system right we have the 50 states who run their own elections and within the states they're mostly administered at the local level it depends on the state but so you have 8,000 plus local election jurisdictions and every state does things a little bit differently based on their own independent or their own individual states needs and state culture and learning about that and being able to interact with the folks who are running the elections I think has students have really enjoyed it I always get really great feedback from the professors who bring their students out there and I hear from the students who have come who have come to those events asking like how do I get involved what do you recommend in terms of internships to get involved in this space and so it's exciting elections are a challenging space but it's an exciting space and it's great to see the folks really wanting to pour into it with their skill sets others? Yeah, go for it I'm going to jump in there a little bit with what he just asked I'm with the city government in Oklahoma run the state's cybersecurity grant one of the things we looked at in implementing those grants was something that Texas A&M came up with and that is starting a sock that supported the local city governments and so we are in the process over the next two to three years of building socks in security operation centers in each one of the major state institutions with the idea of getting students that foot in the door to get them that experience get a CISSP you got to have like five or six years of experience you're not going to get that in college but you have all these companies out there wanting that thing so now you're going to be hiring somebody that may get two to three years working as a sock analyst and one of our private universities actually did this internally and over the course of the time they were able to build up and get some really good they started actually getting more than just sock analyst jobs they started going out and doing the pen testings and that kind of thing and so I think that's where the states can really start working with their own institutions you know the state education stuff and start building that cyber talent pool so I just wanted to toss that in there Great, thank you and I know both states here have done a lot of work with their local universities on you know both getting them involved and just the demystifying process as well, right? Others? Okay. Are there other NAS demystifying sessions planned and if so how do we get involved? Great question, so no, not yet we've typically so far done these at our summer conferences so our last one just happened last month and in our July summer conference and we plan two conferences per year but open to ideas and folks who want to get involved my contact information is here so feel free to reach out to me you know like I said we've done these three events and we're kind of looking for is the best approach continuing this model maybe doing something a little bit different moving forward so I'm happy to hear from others and have ideas and also states are doing some of this stuff themselves right and state events that Iowa does there what left of boom event and so you know even whether it's NAS events or getting you connected with some of our members we'd be happy to hear from you. Are there other questions or feedback? Okay, go ahead. All right so when you guys are interacting with folks in the cyber research community right and they say hey they look at some tech that you've got which is gonna be a lot of tech right because you have to have a lot of voting machines and they say hey I found something for you how difficult is it for you to actually implement a fix is it as easy as a phone call or are you like now you're talking to vendors and sourcing and talking to people about funding like is this a big turn or a pretty easy turnaround? So I'll give kind of a quick overview and then if anybody wants to weigh in they can but you know everybody's mind goes immediately to the voting, vote casting and tabulation systems which the manufacturers of those systems play a big role and I think the question that you're asking there but what our members have been more focused on and I think for good reason which we could talk about is are the internet connected systems that support election administration so we're talking voter registration systems, election management systems, websites that share election information and so their vulnerability disclosure policies when a secretary's office has them that's the scope of those and those are the systems where they're able to implement changes whether it's working with their vendor or in-house developers but does anyone else want to weigh in on that? This doesn't probably answer your question but one of the things that we need to do is reassure the public and so what we've done in the state of Iowa, secretary paid is put together with the legislature after each election we have the vote counting machines, we have the physical ballots and we actually go in and we do a hand count of in each county one precinct and they just you know with their finger one, two, three, four, five and they match it against the machine and so again we have confidence, we want to kind of ensure confidence but we're also again, we have the VDP so that if you do find something and we want to be able to see that and respond to that and work with that but again just with the machines and not every state does it but I think most of the states do they do an audit to make sure that those paper ballots match with those machine counts and I think the majority of the time I'm not sure if I've heard it where they haven't matched so we have pretty high confidence from that level. Yeah it's really important to understand the difference between the internet connected systems that support elections and these vote casting and tabulation systems, really the latter have a lot of, you're able to implement a lot of resiliency into the process through things like paper and auditing that the paper ballots or the paper trail so that you know, two instill confidence but also to ensure accuracy, right? The internet connected systems, those are not connected to the voting systems and they're separate but they're still very important, right? They're still very important to the process and they're what we know has been the target of adversaries in the past and so it has to be an effort for both of these things and I know we're almost out of time so I really want to go down the line one more time just in the spirit of collaboration, any last minute thoughts, encouragement for the folks in the room, lessons learned that you want to share, just quick note, we have two minutes to go. We'll go to Jeff. I'll keep it short, I'll say really just get involved. There's so many opportunities now whether it's participating in a vulnerability disclosure program or a bug bounty, if your state doesn't have one get talk to them, encourage them to set one up. There's all sorts of other opportunities on getting contact with Lindsay at NAS, I'll plug at CISA, we also have a number of opportunities to get involved. We have a session tomorrow where you can provide feedback on some guidance we're putting out on secure by design. We put out a request for information on open source security as well so please do get involved, get your opinions in and we can collaboratively do better here. I think I just want to say we appreciate the research community, you know there's the landscape is changing I think for you, for us, we're trying to learn and I think one of our goals has been how do we work towards cyber maturity? We know we're not there, the minute I say we're there one of you guys is going to prove us wrong so we're just moving in that direction of cyber maturity and continuing to improve and continuing to learn from you guys and from experts in this field. I just want to say that thanks to Mike's leadership Lindsay's continued shepherding of this project, you're seeing what you see is a multi-year project that has made some impact and has helped inform and educate and bring together people to collaborate on a problem that overlaps multiple disciplines. And I just want to second that and as well if you do want to get involved, please contact NAS, they can get you to whatever state that you represent and then as well if you are a student and you're looking to get even more involved, look at scholarship for service, it's not just for federal service, you can also relate that to state service as well. So that's one way to get some education paid and be involved at the same time.