 a very pleasure to introduce Matt Bernard and Alex Harderman. Matt is a PhD student and Alex is a professor at the University of Michigan. Both of them do security research, and they have a focus on e-voting and its security implications. Today, they are going to talk about Recount 2016, an uninvited security audit of the US presidential election. Please give them a very warm welcome and a huge round of applause. Thank you. There we go. Hello, everyone. I'm Alex Harderman. It's wonderful to be back here at the CCC. I've spoken in recent years about other areas of my work, like cryptographic attacks on TLS, ZMAP, and internet-wide scanning. But today, we're coming to an issue that has been for many, many years, very central to my heart. And that is ensuring the integrity of elections. In 2016 in the United States, we've seen some unprecedented things during the presidential election. And I and Matt and others have had the great privilege of taking part in what is essentially our own kind of hack of this election. A last-ditch attempt to find out for sure whether the election outcome was genuine. This is our first time speaking in front of giving a full-length talk about the recount process and what we found. And I can think of no better place to give it than here at the CCC. Thank you, Alex. So as some of you may have heard, we recently had a presidential election in the United States. It was a contest largely between Hillary Clinton and Donald Trump. And as you probably know, Donald Trump won. What you may not have known is that this was kind of a surprise. 538, which is a US election blog, had Hillary Clinton winning 71.4% of the time. The New York Times gave her an 85% chance. Sam Wang at Princeton gave her a greater than 99% chance and promised to eat a bug if she lost. And just generally, the general consensus was that Hillary Clinton was going to win this election. So what happened? As many of you probably know, she didn't win the election. And here we have Sam Wang on national television eating a bug. So what did the result of this look like? Trump won the electoral college in the United States. We aren't a direct democracy. We have this thing called the electoral college where every state gets a proportional amount of votes based on its population. And the election is decided. In most states, the candidate that wins the majority of votes in that state gets all of their electoral votes with two exceptions. So Trump won 304 electoral votes. Clinton won 227. However, Clinton won 48% of the popular vote and beat Trump by 3 million votes. So this gives you an idea of how the electoral college can sort of, it causes some indirection. Donald Trump, of course, insists that he won the electoral college in a landslide, which isn't quite true. In fact, he had the 48th best victory in the electoral college in US history right behind Harry S. Truman. So a little overblown there. But what happened? There are sort of two explanations here. Why were we so wrong? Why was this such a surprise? One is that the polls were just systematically wrong. Nate Silver at 538, Sam Wang, and many others had just said, you know, there was higher uncertainty in the polls than we anticipated. And this seems reasonable. The other explanation is that the election was rigged, which is a position that prior to the election was largely supported by President-elect Trump. This is probably unlikely, but nevertheless was rigging feasible. How would we even know, you know, how are we going to determine whether it was just the polls were wrong or whether or not there was actually rigging? So with that, I'll give it back to Alex here. So elections in the United States are really quite complicated to pull off logistically. We have a massive scale with about 200 million registered voters. Elections are highly distributed because we operate in a federal system. Most of the work of running an election happens at the state, the county, even the local municipal level. And there's a lot of complexity to the process. Our ballots are sometimes ridiculously complicated, loaded with local races unique to each jurisdiction. And above all, our elections are very sensitive to latency. So whatever we use to count them, we need to get the results basically on election night where people just find it unacceptable. So in the country, there are more than 13,000 different voting jurisdictions that are making decisions about how to implement elections. They're running elections on election night in more than 187,000 voting locations or precincts. And our technology? Well, we have 52 different models of voting machines in the U.S. And the choice of which to use is left up to the states, the counties, even in some cases, individual cities. All right, so here's a map just to illustrate a little bit of this diversity. You see all kinds of different systems, optical scan, DRE paper ballots used in different parts of the country. Here's just one state. This is Arkansas. And you can see just within a single state, the variety can be extremely large. Here's one of those long complicated ballots I mentioned. This comes from Dallas, Texas this year. It's an example of the ballot. This is just page one asking you for your party and your presidential choice. Here's the second page. And here are the six other pages. So if you can imagine trying to count a ballot like this by hand would be extremely time consuming because you'd have to go through each individual race. The fastest way to count ballots is just to sort them by who was selected on a particular race and count the stacks. Instead, you'd have to go through each of the 175 or however many questions on this ballot individually. And you can see some of them are even a paragraph long describing a particular referendum. So we use two primary styles of voting machines that those 52 models of election technology mostly fall within these two styles. These are optical scan voting where the voter fills out a paper ballot and runs it through a machine that scans it right in front of them and spits it back out if the voter has made a mistake. Remember how complicated these ballots are. We need the computers to help us. The second style is called direct recording electronic or DRE voting. And in this style of machine, the voter casts a vote on a computer terminal. The computer records the vote in an electronic memory. And sometimes, but not always, the computer prints a piece of paper that the voter can see that is saved for later that makes a physical record of their vote. And for years, many scientists, including me and including Rob, have been criticizing DRE voting machines in particular because they're fully electronic. Anyone who tampers with their software can change every record of the vote. Let me give you an example of that. This is one of the DRE voting machine models used in the US, a machine called the D-BOLD AccuVote TS that 10 years ago was the most widely used voting machine in the country. I was part of a study that obtained one of these machines from an anonymous source and reverse engineered it to do an uninvited security audit. What we found was really pretty devastating. One, these machines, well, they use a removable memory card before the election. Most voting machines do. And this is to load the design of the ballot into the machine. After the election, the card is removed, it has a copy of the election results, and that's what's counted on election night. Well, the first thing we found was that anyone who had temporary access to that removable memory card could corrupt the files on it in a way that would exploit buffer overflows and other vulnerabilities in the voting machine so that when the card was inserted before the election, malware would be installed onto the machine. The second thing we found was that that malware, well, let me show you, here's an election we ran as a demonstration on our machine. George Washington versus Benedict Arnold, the traitor of the American Revolution. Well, on our machine, infected with our malware, no matter how many people vote for George Washington, Benedict Arnold always wins. That's because our malware is just another application running on the computer and it can access all the same records of the vote that the software running the election can. So every time someone votes, our malware goes and modifies those records, the electronic records of the vote, to make sure our preferred candidate wins. It even has a nice UI here. You can pick who wants, who wins, and by how much. Anyway, real malware would likely be more surreptitious. So are US voting machines secure? Well, we know in this one case, this one DRE, that the machine was not. But what about other machines? Well, many different models have been studied by academics and other independent security researchers. And in every case, the machines have been found to be susceptible to malware infection from their memory cards that can lead to the machine giving incorrect results. This one a few years ago, I turned into a Pac-Man machine because there wasn't any more science to do. But it's not just the DREs. Wait, wait. It's not just these DRE voting machines that are susceptible to malware. The optical scan machines can be infected too, and their electronic vote records can also be tampered with in just the same way so that the results that are reported electronically on election night are wrong. The paper will still be right, but the electronic results, the ones we hear about immediately will be wrong. And this has been true for just about every model of optical scan machine as well. In fact, every US voting machine that's been subjected to rigorous independent security testing has been found to be susceptible to the use of malware spreading on the memory cords to introduce a vote stealing attack. So, how would we use this to hack an election? So there are three main ways that we could hack an election. We could alter election night results. This would undermine the credibility of the election, even if, especially if it was detected, or obviously it could get the wrong person elected. We could perform denial of service attack. We could prevent people from voting. We could make the lines at the polls really long. We could prevent votes from being counted. And then there's general political interference. Let's say that a candidate that I don't like is running, I can go about a cyber campaign to basically ruin their chances. And we've seen all three of these in practice. In 2014 in Ukraine, there was a massive cyber attack, one part of which was to try and change the vote totals. And another part of which was to prevent the votes from being counted at all. And in this election, we've seen multiple cyber attacks against political campaigns, specifically the Democratic National Convention, voter registration databases in Illinois, Arizona, and a few other states. And then John Podesta, the campaign manager for Hillary Clinton, also had his emails hacked. And in almost all of these cases, the US government has formally attributed these attacks to Russia. So we know that these kinds of attacks happen. But what about other attacks? What about invisible attacks? Let's say we wanna influence the election in some other way. To do this, there are really three main challenges. We need to overcome the diversity centralized technology that Alex just mentioned. But this is pretty easy. Because of the electoral college, elections very frequently come down to very few states and a very small margin of votes. To give you an idea in this election, you would need to flip the result in two different states, both of which had margins within 1%. So that's not really that big of a challenge. The second challenge is that how do you infect the machines? And when you ask poll workers this and election officials, their first response is, the machines are not connected to the internet. As I'm sure many of you know, this doesn't really matter. All voting machines are programmed through centralized election management computers, oftentimes which are connected to the internet. And so all you have to do is put a virus on this machine. Then every memory card that is programmed with that computer, every voting machine it goes into will become infected. So you effectively have a county level hack that's pretty easy to do. But how likely is this? In the United States, many jurisdictions outsource ballot programming to these small businesses. For instance, in Michigan, 75% of counties use two companies, both of which have fewer than 20 employees. This is governmental business solutions, one of them. And as you can see, they don't even have TLS enabled on their website. They're pretty small, like I said, a small business. GVS services, I believe, four or five different states. And this is their warehouse. You can see some AcuVote TSX is there. Basically, they're pretty public facing. It's pretty small. It wouldn't be that hard to infect machines at a very high rate. The third challenge is these paper records. 70% of US votes have paper records. And this paper is sort of akin to, in most critical infrastructure systems, we want to have some sort of fail safe in case the electronics go down. So for instance, in airplanes, they have physical compasses in case, for some reason the navigation system doesn't work. Paper records are effectively the same thing. The major caveats to paper are they are slow and inexpensive to tally, as Alex mentioned, whereas memory is fast and cheap. But the major strength is that every paper record is verified by the voter. Either it's an optical scan ballot that the voter fills out and checks before they feed it into the machine, or it's a paper audit trail produced by a DRE that the voter can look at before they push the big red button that says cast your vote. Whereas on memory cards, there's no way to verify it at all. That's just regular computer memory. So really what we need to do is we need to audit the memory cards with paper records. And this would trivially defeat most attacks. Unfortunately, as I mentioned, 70% of states are recorded votes on paper in 2016. This is up significantly, and is largely due to efforts by Alex and several other people over the last 10 or so years talking about election integrity and no ways to prevent fraud. But unfortunately, unfortunately, most states never look at the paper. And you have this great way to defend against an attack, but you never use it. So in short, hacking an election is actually much, much easier than you would think. All right, so let's put this together. How would you actually hack a US presidential election? Well, step one, before the election, use pre-election polls like Nate Silver provides to identify the states that are likely to be closest, that are going to be within a percent or so. Step two, target some large counties or their service providers, and compromise their election management system computers. I'll leave it as an exercise to the attacker to find out how to compromise the election management system by, say starting by emailing Sue. Step three, use the compromised election management system to spread your infection to the individual voting machines. Developing an attack for one of these machines is not terribly difficult. I and others have done it again and again in the laboratory. All you need to do is buy one government surplus on eBay to test it out. Finally, step four, your attack steals votes on the computer, but no one ever looks at the paper because as we'll show you, most states just throw away that paper without ever looking at it. So coming back to 2016 and the presidential election, we knew on November, on November 8th, excuse me, not December 8th, on November 8th on election day, that hacking was possible. And at the end of the day, when the election results came in, we saw that the result was extremely close, was surprising compared to the polling. We knew at that point that there had been cyber attacks of an unprecedented nature in American politics aimed previously at interfering with the election. This was all before election day. And we knew that it was feasible for an attacker to change votes on enough machines to have stolen the election result. Shockingly, at least shockingly to me and many other people, even under these circumstances, approximately zero US states were going to look at enough paper ballots to know whether the computers had been hacked. This is a major gap in our system. I previously had believed that the paper would provide a fairly strong deterrent, but if even in 2016, we're not going to look at any of the paper, well, it might as well not be there. So at this point, there were five weeks between election day on November 8th and December 13th, which is a deadline under federal law for states to lock in their electoral college votes. If there was fraud that had taken place, it had to be exposed by this date in order to have any hope of changing the outcome of the election. So what to do under these circumstances? I and other election integrity advocates got together and started discussing some possibilities, but we didn't really have a good solution. Was there any way possible to make the states actually examine the physical evidence they had in a way that could potentially detect a cyber attack? So the first question, and really the beginning of this whole process for me was on the 13th, on the Sunday after election day, I was on a flight from New York to Los Angeles when I got a frantic email from Barbara Simons, another election integrity advocate. The email said, can you join a phone call in 15 minutes? It's with the Clinton campaign. Hillary herself might be on the call. Well, what do I do? I'm on a flight. So I find a way to work around the airplane Wi-Fi's attempts to block Skype and so forth and find a way to dial into the call. It doesn't work well enough for me to talk on this first call, but I can listen in and I can hear what they're saying. Clinton herself isn't there, but Podesta, her campaign manager and many other top people from the campaign are on and they're listening very, very carefully as Barbara and others are describing arguments why Clinton should request a recount in several key states. So candidates under American law have this ability. They can demand a recount depending on the state as long as they pay the requisite costs, but wouldn't Clinton do it? So there were a few concerns that the campaign brought up. The first one and the most aggravating to me is they wanted to know what evidence we had that the election had been hacked. Well, wait a minute, that seems completely backwards. The purpose of a recount of any examination of the paper ballots is to gather evidence that the outcome was correct or not. Without looking at them, we don't have the evidence to begin with. All we can do is say that it would be possible for an attack to have changed the result without leaving any visible evidence. Another concern was how to pay for the recounts. The cost was likely to rise to the millions of dollars, but maybe most important and most aggravating was they were concerned about backpedaling from the concession, about tiptoeing away from having conceded the victory to Donald Trump. And for Clinton in 2016, this was a particular concern because there was this famous incident during the third presidential debate where Trump refused to say yes, he would accept the election results. And Clinton reacted to this by saying that something truly horrifying, it's a direct threat to our democracy. Well, there are two things he could have meant, right? One was he wouldn't accept the result when settled, right? That he wouldn't concede power to the victor, that is truly horrifying. But the other, which is kind of a rational thing, is well, I don't want to accept the results until all the evidence is examined to make sure it wasn't rigged. Well, she was committed to a position that not accepting the election result immediately was a threat to democracy. And so any kind of tiptoeing away from that would have been a political problem. All right, so that was the first conversation. We had a number of conversations trying to convince the Clinton people to move ahead with recounts. But it didn't seem like we were ever going to get there. But then we had a new idea, this time from David Jefferson, another election integrity advocate. His idea was, his wonderful insight was that actually in many states, any candidate can request a recount. And although you might think we're a two-party system in the US, there are third parties. In fact, we identified one third party, Jill Stein, the green party candidate. And Jill, we had a conversation with her culminating in a talk on the 22nd where she, unbelievably, was willing to help. Jill has a long history of being a champion of election integrity. She was on the ballot in most states. She ended up winning about 1% of the vote, not really all that much, but still enough. And most importantly, she was willing to help. This was an absolutely pivotal moment for the entire effort. But then that same day, at about six o'clock in the evening, we also had what may have been the biggest setback. And that is in New York Magazine, there was a story published about the conversations that we were having with the Clinton campaign. And this was based on information that had been provided to the press in violation of everyone's confidence by a person who was there. And it was a second-hand account that got most of the facts wrong. In fact, it quoted, it said that I and others had found persuasive evidence that the results may have been hacked. Well, wait a minute, that's 180 degrees the opposite. We want to check and see if there's evidence that the results have been hacked. We know it's a possibility, but that's all at this point. Well, what to do? If this is the thing that I have to respond to. And my phone started ringing off the hook at that moment and continued to ring off the hook for weeks. In fact, I just checked my voicemail yesterday and I have a call a month ago from my congresswoman. I'll call you back on Monday, Debbie. So what to do? So what I did to try to not have to answer the press in a way that says, well, we don't really have any evidence end of story was I stayed up all night with Matt and others and wrote this medium piece that some of you may have read in which I present the case for recounts that the only way to know whether an attack occurred is to look at the evidence. And if we're not going to look at the evidence in 2016 by looking at the paper, when will we ever a candidate needs to act now? So that kicked off what really has been an ongoing media circus around the recounts. And unfortunately, this narrative of is there some kind of secret evidence or not has been very difficult to get around. But I think we're doing our best. And in this talk, we'll show you why the recounts were essential and what evidence the recounts are able to provide vis-a-vis the election result. So what happened next? On the 23rd, the day before Thanksgiving, the same day my medium post went live, Jill Stein announced that she was going to lead an effort to recount three critical states. In just two days, a crowdfunding campaign that Jill and the Green Party led raised more than $5 million for the recounts. That's more money than she raised for her entire presidential campaign. 160,000 individual donors. It's absolutely incredible how many people care passionately about the integrity of our elections. The next question, now we have the money. We have a candidate. Where to recount? So in order to figure out where to recount, we decided to sort of take attack, like I mentioned earlier. If I were an attacker, where would I hack to have the most result for the least amount of effort? And as I said, there are a few states in the United States that had very small margins. Three of these states were Wisconsin, Michigan, and Pennsylvania. All three of them were projected to be won by Clinton. All three of them were won by Trump. They combined through 46 electoral votes. The electoral margin was 37 votes, I believe. All three had margins less than eight-tenths of a percent. And in total, the US election was possibly decided by the 78,000 votes in these three states, the 78,000 march. And moreover, we had to find states where we could actually do something. Michigan and Wisconsin allowed candidates to petition for recounts. Pennsylvania requires a court order, but we thought our chances were pretty good. So in Wisconsin, to start us off, the margin was just under 23,000 votes. That's about 0.7, 0.8%. The technology there is almost entirely Opscan, and every ballot that is cast in Wisconsin has a paper record. And as I mentioned in the law, any candidate can demand a recount if he or she can pay the cost. And each county in Wisconsin decides how to do the recount. You can recount all the votes by hand counting all the paper, or you could just run the ballots back through a machine. And so in Wisconsin, that decision is left up to the counties. In Michigan, the margin was much smaller. It was about 11,000 votes, 0.23%. And by the way, Michigan is also where we live, so it was sort of convenient. The technology there is entirely Opscan. And again, all votes use paper. And the law states that any aggrieved candidate can pay a statutory fee in however many precincts he or she wants to get a recount. And then the state board of canvassers, which is the state committee that kind of decides, that runs elections, decides how the recount will be done. And finally, in Pennsylvania, the margin was a bit bigger, 44,000 votes. Unfortunately, in Pennsylvania, they predominantly use paperless DREs. So 70% of their votes are cast without any kind of paper, record, or backup at all. And the law there is very, very complicated. It says that three citizens from every precinct that wants to be recounted must post a bond, swear that there is fraud, and that's how you have to get a recount done in Pennsylvania. And bear in mind, there are tens of thousands of precincts in Pennsylvania. Not tens of thousands, there are 9,000 or so precincts in Pennsylvania. So that's 27,000 people we need to swear and pay and et cetera. Fortunately, Pennsylvania does have a clause that states that an automatic recount has started statewide if the margin is less than half a percent. But unfortunately, the margin was just shy of that. So why not other states? One of the criticisms in particular from our president-elect was, you picked three states that I won that Clinton was expected to win. Why aren't you auditing states that she won that I should have won? Why not other states where Donald Trump allegedly has heard tales of fraud? And so the three he pointed out, Virginia, New Hampshire, California. New Hampshire, the deadline to get a recount done passed six days after the election. And as Alex just mentioned, we didn't get recounts started until over two weeks after the election. In Virginia, it's actually illegal to recount an election if the margin of victory was greater than 1%, Clinton won by almost 5%. And in California, the margin of victory was 30%. I'm pretty sure Trump wouldn't have won California even if there was hacking for him. And all of this is to say that Trump could have also initiated recounts. Every candidate is entitled to initiate recounts in the United States. So if he really cared about voter fraud, he would have initiated recounts. So we're finally ready to kick off the recounts. And I'll let Alex tell you about that. All right, time to let the recounting begin. So the recounts kicked off the day after Thanksgiving, November 25th when Jill Stein filed a petition in the state of Wisconsin to begin a formal recount. And this is just two days after she announced the recount initiative. The other states would soon follow in days to come. But at this point, there were only about two and a half weeks before that December 13th deadline. So it was an incredible scramble. President Trump, of course, as you might expect, was not a fan. And I say that having looked at elections and election security in countries around the world, and I can say that no one who has won an election ever wants to see the election technology that was used to vote them into office criticized. I'm sure the shoe would be on the other foot if Clinton had won and Donald Trump was alleging that the election had been rigged, but we have what we have. So Trump alleges, among other things, that this is all just a scam to help the Green Party. He and his lawyers would continuously oppose us in legal battles in all three states in an attempt to stop the recounts from beginning and halt them once they started. Now, Matt and I, at this point, basically just played a supporting role while others did the real work, including an army of Green Party volunteers, more than 10,000 volunteers who worked at individual recount centers across the states to observe who provided legal assistance on the ground, who provided assistance in filing petitions in Pennsylvania. And also, and critically importantly, we had excellent lawyers. We had assistance from the firm of Emery Shelley, Brinkerhoff, excuse me, Matt, and Abbotty who are specialists in litigation for topics like civil rights and election law. About half of the firm seems to have been involved in the recount efforts in one way or another, and over the course of about a week and a half, they initiated more than a half dozen different court actions in three states, in state and federal court in support of these recount efforts. We also got expert testimony from an army of leading electronic voting experts who rose to the occasion. So what does a recount look like? Well, it's actually not all that exciting. It's a bunch of people sitting around at card tables looking at stacks of paper. Matt and I went and acted as recount observers in Washington County, Michigan, where the university is, and actually got to be a part of the process for a little while. So basically what happens is at each table, they recount one precinct at a time. You have two people who are employees of the county who look at each ballot. They count the ballots, then they divide them into stacks according to who the marked candidate was, then they count the stacks. All the while, there were observers from any political candidate on the ballot who wants to have observers present who are watching to make sure everything's right, and they can raise a challenge if they see any kind of error. So that's it. It's a laborious process. It took about three to four hours to recount a precinct while we watched of about 2,000 ballots. But with parallelization, you can see there are a lot of tables there. This is the end of the day, so most people have finished and gone home. You can actually recount this process fairly quickly. It's nothing like the Florida 2,000 recount, which used this awful ancient punch card technology and lasted for weeks and caused endless litigation. But a recount also looks like this, endless litigation. Well, unfortunately, in all three states, as I said, we had opposition from either the Republican Party supporting the Trump campaign or in some cases from the state governments themselves. And we had to go to federal court, to federal appeals court, to in Michigan to the state Supreme Court in an effort to try to have the recounts proceed. Recounts also look like this. This is in Michigan, where Donald Trump's lawyers succeeded eventually in halting the recount before it was completed. This is of Green Party protest in front of the state Supreme Court, asking them to rule in our favor. Recounts also look like this. As I mentioned, there was a media circus. This was the news just about every day. I at one point, it's just so ridiculous. Had a camera crew, a photographer, a writer, a lighting guy, another photographer, a sound person and a director following me around in an elevator as I was going from court to court. And it looks like this. This is Jill Stein in Philadelphia, where in Philadelphia we went to the federal court to try to convince a judge to give us a recount of the entire state, or at least of select counties on constitutional grounds because the state law didn't provide any mechanism that we could use to directly petition for it. So recounts also look like this. Comradery, people doing what they think is right, many, many people working together. But ultimately, the recounts ended. And Matt's going to tell you what the results were when the election was decided. So as you know, and as we sort of already discussed, Donald Trump won. The electoral college on December 19th submitted its votes. And what's more to say, right? No state actually finished a full hand recount when all was said and done. Wisconsin, almost all of their counties were hand recount, but as I mentioned, their counties could decide whether to do hand or machine recount. And all told, about 400 votes changed. In Michigan, the recount ended after about three days, again, due to opposition and legal arguments over the term aggrieved. So what did we learn, or what are the takeaways here? Wisconsin was a statewide recount, it did complete. 51 counties counted by hand, and then 21 didn't, more or less. There were 11,883 ballot corrections found in the recount. And this was things like poll workers finding stacks of 20 or 30 ballots left over that they forgot to count on election night. And by the way, that's more than half of the margin. So had every one of those corrections been for Hillary Clinton, the results in Wisconsin would have flipped. But as I mentioned, they weren't. Only about 397 votes changed place. And all told, we didn't find any significant evidence of attack. In Michigan, like I said, it halted. 10 counties actually finished recounting. 12 other counties started, but didn't finish. And there are 83 counties in Michigan. Over 2 million ballots were recounted, which is 43% of the original cast ballots. We're still waiting for full data. There was a net change of 1,651 votes. That may change as more results come in. But in general, no evidence of an attack there either. In Pennsylvania, the recount never really got off the ground. It was defeated in federal court. Only one county actually even started to recount as far as I know, out of 67. They recounted by hand, and they only recounted 143 of their 228 precincts. They have not published results, but presumably there wasn't an attack because we didn't hear about it, hopefully. The lesson in Pennsylvania was that it's really hard to get a recount done in Pennsylvania. Other things we learned. In Detroit, Wayne County in Michigan, there were huge systemic issues. In 37% of precincts, they found more votes for president than there were actual ballots cast, which is kind of a headscratcher. And nearly half of the ballots in Detroit were not eligible for a recount. With this basically, there's this notion of recountability, where in order to recount a ballot, poll workers have to show that the chain of custody from the end of the election until when the ballots are being recounted was upheld. And this includes things like poll book logs, tamper-proof seals on the bags that have ballots in them, and so on. So if you're an attacker, all you have to do, let's say you hack a machine and you wanna hide your tracks, all you have to do is go pull a few seals off of these bags or tear open the ballot bags, and no one will ever find out that you hacked anything. So even when you find evidence that maybe something went wrong, nobody looks at it. Using the available evidence that we did get from the recounts, we tried to sort of extract information and draw conclusions. What we would really like to do is what's called a statistical risk-alignity limiting audit. Or basically you hand count a very small sample of ballots randomly chosen across a state or across a county or however you would like to do it. And you hand count them, and you can show statistically that the hand counted result verifies the statewide result. This is significantly cheaper than a full state recount. But unfortunately, most state laws, and in fact I think no state, actually allows risk-alignity audits except maybe one. What we actually wound up with, of course, was incomplete non-random county level samples. So what do we do? How do we draw conclusions from this? Well, we can rule out some attack scenarios. For instance, statewide fraud. It's unlikely in a recount that every single ballot was hacked or that a ballot or a ballot in every precinct was hacked. What about other scenarios? So we're still waiting for a complete analysis, but the idea is that we wanna test the likelihood of a probable attack, of catching a probable attack. So our model is an attacker compromises some subset of counties at random. We presume that the attackers won't have the ability to choose where they get to attack, modulo a few things. And basically we sample counties at random and change 10% of the votes until we have enough to flip the election in that state. If the randomly selected counties were recounted by hand, then we consider that attack detected. So what we find is that in Wisconsin, we ran a million simulations of this attack in every state. And what we find is that in Wisconsin, you only need on average seven counties to throw the result. And fortunately because Wisconsin had almost all states or all counties do a hand recount, the chances of being undetected are pretty low. You have a 5% success rate there. In Michigan, there was a much tighter margin of only 11,000 votes. So you need an attacker needs much fewer counties to rig the results for, and because the Michigan recount didn't actually finish, the chance of the attack going undetected is higher than it was in Wisconsin. Albeit it's still not that high, considering only a small fraction of counties were actually recounted by hand in Michigan. Pennsylvania, there was a wider margin, so you need more counties to throw the results. And as we mentioned, only one county, only part of one county did a hand recount. So the chance of going undetected there was very high. So what did we learn overall? The recounts support that the election outcome was correct, that we can, I can sleep at night knowing that Donald Trump won the election and having no doubt. There is not strong evidence that there wasn't fraud or that there wasn't a cyber attack though, because recounts were incomplete because there's sort of an imperfect way to do this, to verify. So, yeah, like we said, cyber attacks already happened before. We did get a congressional investigation out of this. President Obama has ordered an investigation before he leaves office. And we got intelligence agencies to agree that there was hacking that did happen that probably influenced the outcome. But overall, we can sleep at night, but we can't completely rule out fraud. So the recounts didn't complete. What I thought at the beginning, as I said in my medium piece, was that, well, the election probably wasn't hacked. What do we learn from the recounts? Well, we learned that it even more probably wasn't hacked. We get additional basis for confidence in the election. But at the same time, along the way, we found that hacking an election in the US for president would be, well, even easier than I thought when I began this process. We have the vulnerable machines, of course, we knew that before, but how central the points of attack are, these small companies that control voting machines across many states and are such small businesses, well, that was news to me and many other people. Also shocking is how unlikely states are to look at any of the paper, even in a surprising and close election like this. Apps into recount, probably none of the paper would have been looked at. But with the recounts, at least we had some, enough to gain some amount of statistical confidence. Even if a candidate can force a recount, and this is probably the most damning thing about the entire experience, well, there are opportunities for the apparent winner to try to stop it. There are many, many opportunities, and in fact, they're quite likely to be successful. Whoever wins is likely to do this, Republican or Democrat. So unfortunately, although we have some amount of physical evidence we could go and look at, we probably will never get the chance. What we need in the United States quite badly at this point is some specific reform to the election process. We need this because even if the 2016 election wasn't hacked, the 2020 election might well be. We're facing increasingly powerful and aggressive state level attackers in every country, including attackers targeting the US. We need some effective defenses to keep them from undermining our core democracy. First, it's common sense we should be hardening or voting technology to make it more resistant to attacks to buffer overflows to basic security vulnerabilities. But even this, it's not sufficient. This raises the bar to an attack, but we need to make sure that we're getting a physical safeguard, a common sense kind of security audit of backup by having a paper record in place for every vote. We have to go from 70% to 100% and there's still a number of states that need to take action to do so. But most importantly, we need to make sure that every state actually makes use of that evidence. And to do that, we need not recounts, recounts are a hack. We need statistical risk limiting audits to take place after every major election. These are a cheap and easy to use mechanism that states can use to make sure the electronic totals are right by looking at enough of the paper. States need to act now to make sure that these are going to be in place, all of these reforms in time for 2018 and 2020, the next presidential election. But it's going to be an uphill battle. Unfortunately, because of our distributed federal system, states individually are likely going to need to act. I think we're unlikely to see federal legislation to improve electoral integrity under the coming Trump administration. So thank you very much. We have some time for questions. And we have about, all right. Thanks a lot, Matt and Alex. We have about 10 minutes for questions. So if you have a question, please line up at one of the microphones in the room. And I think we'll just start off with microphone one please. I do not believe in paper voting so much. Why? Because I lived half of my life during Soviet occupation and I have seen many paper votings that ended 99.9% for Communist Party. My question is related to the financing and then methods of pressure, how to make governments to make elections better. That's very pity that I had to miss your previous talk here about Estonian voting. So I remember what it was. Professor Haldermann's team was invited into Estonia by a political party which later was seen at least asking money from Putin. I don't know whether they got or not, so I am not sure. Could you please come to the point? Yes, I'm happy to answer that. Yes, yes, absolutely. And then we compare the definition of terrorism. May I answer your question, sir? So let me address first the point about paper is also vulnerable and that's absolutely true. We have a long history through all use of paper in democracy of paper also being hacked. The key is that we have an opportunity now because we're combining computers and paper to get the best of both worlds, a digital record of the vote and a physical record of the vote as long as we check to make sure they agree. If we don't check that they agree, then we might as well just have the one or the other. Also, I'm not sure that we at the University of Michigan could hack into all the paper ballots across multiple states sufficient to change the presidential election. But I'm pretty sure my undergraduate security course could have changed the outcome of the presidential election this year. It really is that bad. Thank you. How to make the voting more transparent. Sorry, can we please just have one question per person? We have lots of people who want to ask questions. I'll be happy to talk to you one on one afterwards. Just a quick heads up. If you really have to leave right now, please do so as quietly as possible so we can finish this Q&A. And also, if you're asking a question, please make it precise and short so we can ask as many questions as possible. Mike, too, please. I want to ask if it wouldn't have been easier to get your hands on the hardware and the software and just check for any, if it has been compromised. I mean, pay some guys off and get your hands on. Well, we did check, we did try in Pennsylvania to get access to the hardware. In fact, because that's often the only record of the vote is the one that's made in the computer's memory in Pennsylvania. That would have been very difficult to complete a forensic analysis in the time available in only a couple of weeks before the deadline. And it's not really as direct a way of checking as looking at the paper if you have access to paper records, too. But absolutely, I support forensic analysis of the voting equipment, too. I think that's just another kind of physical evidence. Okay, perfect. Mike, four, please. Thank you. In the history of the world, to the best of your knowledge, has it ever been a case where the election process was rebooted because it was proven to be compromised? There's a first time for everything. I think we're gonna squeeze in a question from Mike, three, please. Okay, this risk-limiting, hand-counting approach, does it have some requirements that will make it unfit for other countries than the U.S.? Does it, I'm sorry, that makes it fit for other countries? So this statistical risk-limitim approach, does it have requirements that are unique to the U.S.? No, no, risk-limiting audits can be used just about anywhere there's paper to count. And it can also be used, it's not strictly to the first pass, the post-voting method that we use in the States, it can be used for any of the methods that just about anywhere uses. Okay, thank you. Okay, I think we have a question from the people watching the live stream. Single Angel, please. Actually, we have two. The first question is, why is there not always done a recount? Good question. It is a really great question. It should always be done, some kind of risk-limiting audit, anyway, looking at enough of the evidence. But in part, it's because the law hasn't caught up to technology, right? Back when we were only voting on paper, there wasn't, there was a hand count already of every ballot before we had any kind of voting machine. But since the introduction of technology, and especially such hackable technology, we need to update the law to make sure some amount of physical auditing happens every time. And we have a second question, which as in your own opinion, wouldn't make sense to get sensible laws in all the states? To get, pardon, can you repeat that? Wouldn't it make sense, in your own opinion, to get clean laws in all the states? So it's more sensible to understand if there is a rigging or not? Certainly, but one of the issues is the United States is a federal system. So the states ultimately have most of the rights over what they can do. And so there are things like the Federal Election Commission that do try to, at a federal level, enforce policy. But it's much more complicated and difficult, constitutionally even, than just saying all votes will be done this way. Believe it or not, in the United States, the Constitution doesn't even say that the states have to have an election to decide who their presidential votes will go to. They can just pick it by whatever method the legislature decides. So we really do have to go state by state. I'm glad they at least ask the people what they want. All right, Mike Five please. Hi, election reform is itself a political process. So I wanted to ask how your efforts to improve the technical aspects of actual voting interact with the wider issues that other people are struggling over, such as voter suppression due to felony convictions, positioning of ballot boxes, and all the rest that's well-known in the US? Well, there certainly are a lot of issues with elections in the US and in many other countries. And as you say, those are a number of other ways that people's votes are sometimes being suppressed or are not being fairly counted. Our expertise lies in the technology, and so that's the primary area of focus for me and for Matt. But we often do get to work with other election integrity advocates who are concerned with and work quite often in this wider spectrum of issues. But I agree with you, the politics and technology here in election security are so unfortunately intertwined. I don't think it should be a question for anyone whether our votes are going to be checked to make sure they weren't hacked. We should all want to make sure they're checked, whatever our political affiliation. But unfortunately, there's always resistance from whoever the apparent winner is. Thank you. Okay, microphone six please. Thank you very much for your talk. I've got one question that you didn't seem to approach was the question of voter suppression. And in particular, hacking can mean an awful lot of things. There were an awful lot of machines in Detroit that just didn't work or were miscalibrated. Could you approach that and how much can you contribute to fixing that part of the system? Well, we did sort of, we touched on some of the systemic failures, particularly in Detroit. And I think that the solutions to those kinds of failures and to things like voter suppression due to lack of funding in a district are solved by the same problem. They have the same solution. You provide funding to buy better voting machines and so on and so forth. Beyond that, it's much more complicated and I'm not really sure. Okay, all right, microphone two please. Thank you for your work and talk. Moving away from the concrete Donald Trump scenario, you said most or to all winners oppose recounts. So the question is if they can be confident that they would be confirmed as a winner, why do they oppose the recounts? Well, it's a good question and this has been my empirical observation in every country where I've worked on election security and I think the reason is because even the mere suggestion that they might have been elected unfairly, the mere possibility that a recount would expose any problems. Well, it's worth the time and effort of their lawyers opposing it to try to make that possibility smaller. I think people seek to move past an election so quickly after the votes are counted the first time that there isn't time to think about it to check the evidence to dot the i's and cross the t's and that's that demand for instant moving on is part of the problem. It really makes the whole issue so much more difficult to deal with technologically. Thank you. Okay, so we only have very little amount of time left so please be quick with your questions, Mike One. Do you think there's value in end to end verifiable elections and cryptographic techniques to ensure integrity? Absolutely part of the solution but the complexity of the system already is so difficult to deal with. I'm not sure local election officials will be able to add anymore. We need to make it easier. Right now audit is the best way to secure election. Okay, unfortunately we are out of time so please give another huge applause for Matt and Alex, please.