 Thank you, and thank you in particular for being here at such an early time in the morning. I know that by CCC standards, this is the crack of dawn. So I'm Alex Halderman. I'm a professor of computer science at the University of Michigan in the U.S. And the work I'm going to be telling you about today is primarily collaborative work with others. I have to thank in particular my students, Drew Springle, Travis Binkenauer as a Cure de Rumorech, and our collaborators, Jason Kitkat, Hari Hirstie, and Maggie McAlpine. This work would not be possible without them. In fact, three of my students who have worked on e-voting research with me are here today. Eric Wistra as a Cure de Rumorech and Drew Springle, if you guys could please stand up. All right, applause for the students. They really do the work. All right, so e-voting is something that's been an interest of mine for about the last 10 years. And it's an interest of mine in particular because it sounds like it would be something so wonderful to be able to use computers to count votes securely, to be able to vote over the internet with all of the convenience that the technology brings. Perhaps we could reduce costs. Perhaps we could increase participation. At the same time, e-voting raises some of the most difficult challenges in the field of computer security. And so I see it as a motivating example and motivating problem for advances of all kinds in cryptography, in systems construction, and usability. E-voting is a really hard security problem because of its unusual requirements. We need two things in a secure e-voting system above all else. One of them is integrity. And by integrity, I mean that the outcome of the election matches the voter's intent. This is a relatively weak definition of integrity. Let's just at least say that the right guy wins. This means, of course, that votes have to be cast as voters intended and that the election result has to be counted as the votes are cast. But the second requirement and the reason that this is more complicated than other problems that we more routinely solve, things like banking online and making purchases in electronic commerce, is we also have this requirement for ballot secrecy. The secret ballot, which is one of the most important technological advances in the history of election technology, the secret ballot, the thing that protects you from being coerced into voting a certain way and protects us from you selling your ballot. The secret ballot says that no one can figure out how you voted, even if you try to prove to them how you voted. This is what we want to prevent people from being coerced and prevent them from selling ballots. And the reason that e-voting is a difficult problem is largely that these two properties, integrity and ballot secrecy, are in tension. Many defenses that we might normally try to use to increase integrity, things like we do in electronic commerce to send people a receipt or a bank statement or to do accounting where we have just a big table where all the money in and out is totaled and we make sure they're in agreement. These things are very, very difficult or impossible to implement if we want to maintain a secret ballot in a strong form at the same time as we're trying to preserve integrity. So we need very different mechanisms to achieve an e-voting system that provides these critical properties. Now, of course, that hasn't stopped people from building electronic voting systems. And many countries around the world use e-voting or are starting to try internet voting. And I've been very lucky over the past few years to have been involved in some of the first hands-on studies of these different systems. For instance, in 2007, I was part of a team at Princeton that did the first hands-on security analysis by an independent party of an e-voting machine used in the U.S. This thing, the D-BOLD AccuVote TS, was the most widely used e-voting machine in the United States at the time. The manufacturer was very secretive about the technology. They assured people that, oh, of course, it's perfectly secure, but know how it works is a secret, you can't know. Now, of course, that's very rarely ever a good sign. It was after several years of debate without knowing the facts that finally a whistleblower gave one of these machines to our research group at Princeton. And some of these stories, I mean, you can't make them up. I mean, after seeing Citizen 4 last night, I mean, this feels like something out of the movie. Now, I had to go and pick up the machine and receive it from the source. And it's so ridiculous. I had to drive up to New York City and double park my car outside a Times Square hotel. I then went into the alleyway behind the hotel where a man in the trench coat gave me a black leather suitcase containing the voting machine. Anyway, we spent a summer working in secret to reverse engineer it, and I don't know what the big secret is. It's basically a PC inside a fancy case with a touchscreen, a removable memory card to load the ballot design and unload the votes. Anyway, we figured out some interesting things about its security. So it turned out that it was just a normal operating system kernel and an application that read and totaled the votes. And by doing some funny things with the memory card, you could cause the machine to replace its software without any cryptographic checks with whatever software you wanted. So we came up with an application called the stuffer that you could load into the machine. It would present a nice UI on the touchscreen, let you pick who you wanted to vote and by how much. It would then proceed to change every record of the vote because this machine just maintained the votes in electronic memory. In that way, you could say have an election between George Washington and Benedict Arnold, the famous trader of the American Revolution, and whenever we would do this election, Benedict Arnold would always win because we had tampered with the machine. This is so easy to do that even a bunch of, in my case, naive grad students could pull it off in a few weeks, thanks to the computerization of the vote. We also discovered that thanks to the computerization of the technology, we could create a voting machine virus that would spread on those memory cards from machine to machine to machine in the course of a normal election cycle so that someone who had just a few minutes alone with one voting machine could change the election outcome in an entire state. That's really the danger of e-voting to me is not just that tampering is possible. Tampering is possible with paper ballots too. It's that tampering, because of the power of computerization, can be much more wide-scale with a very small conspiracy and also very hard to detect. But this isn't the only study that's shown problems in e-voting. I was part of a study that the California Secretary of State, Deborah Bullen, in the middle there commissioned also in 2007 that reviewed every electronic voting technology in use in California. We studied machines from three manufacturers, Hart, Sequoia, and Debalt. Believe it or not, each of these machines is based on source code in the hundreds of thousands of lines of code, too complicated to possibly be secure. So it was no surprise that all of these machines were susceptible to vote-stealing code and also to a tax that would let election officials violate the secret ballot and find out how everyone voted. As a result, all of these machines were decertified and banned from use in the state of California. But it's not just a U.S. problem in Europe, too. People have been experimenting with e-voting. And one of the seminal studies in Europe was conducted by my friend Rob Holmkriep, who's sitting here in the front row, round of applause for Rob. So Rob and his collaborators studied the NADAP ES-3B machines that were introduced in the Netherlands. And they discovered that by just easily changing an e-prom chip that they could do in, I think they demonstrated in less than a minute, they could cause the machine to steal votes, to be dishonest, even to play chess. Inspired by Rob, I and colleague Ari Feldman took a machine that's still in use in many parts of the U.S. and a few years later converted it into a pretty good Pac-Man machine. But it's not just the U.S. in Europe. India, too, uses electronic voting. It's both the world's largest democracy and the world's largest user of e-voting. They have these homegrown, very beautiful, very simple embedded system voting machines. It's too long a story to tell the whole thing today, but about four years ago now, an anonymous source, a whistleblower, gave one of these secret government-made machines to this man in the center, Hari Prasad, to study. And he called me and Rob, and we went to India and investigated it. As I say, it's too long a story, but the long story is short. This ended up with Hari being jailed for a while. Rob and I almost being deported from the country. And just recently, the Indian Supreme Court ruling that they have to introduce a paper trail so that Indian voters can have a reasonable assurance that their votes have been securely counted. But internet voting. Internet voting is even harder than voting on just a stand-alone machine in a polling place. Because internet voting, you have the problem that the voter is using their own machine outside a protected environment where they might be vulnerable to being coerced, to having their user names and passwords stolen, to imposter sites claiming to be the real voting system, to malware on their machines, even to botnets that have already infected large numbers of machines compromising the outcome of the election. And it's not only that. The server, too, has to be able to resist denial of service. Remember, an election takes place during a fixed period, so you can't say, well, our system is down this week. We'll postpone the internet voting portion until next month. That's not going to fly. You have to worry about insider attacks on the server, about remote intrusion, even about more advanced attacks like state-sponsored threats. How many countries do you think might want to affect the outcome of a major country's national election? You're probably a growing number of sophisticated states with that desire and ability. But at the same time, internet voting systems are more difficult to study. You can't just rely on someone to bring you a machine in the middle of the night when the going gets tough. You also can't hack into the server during the election. You can't ethically do that because as a researcher, as someone who's trying to improve the state of democratic technology, I can't possibly justify anything that might risk interfering with the conduct or the outcome of the election myself. And so we've had to look for other opportunities to study these things in other ways. And one of the best examples that I've been able to find so far has been an incident in 2010 where Washington, D.C., decided to introduce an internet voting system. It was a government grant to build this thing. And it was for use by military and overseas voters sending in absentee ballots. Now they did a lot of things right. They did it as an open source system, hired some of the, some really experienced web developers. They even called in security researchers and asked us, well, how should we build our new internet voting system? And the researchers all said, no, no, internet voting, it's too dangerous. We don't know. Don't do it. But D.C. did it anyway. But as maybe a compromise or maybe to tell us to, you know, put our money where our mouths were, they decided to hold a public trial and say that the week before the election, we're going to hold a mock election and anyone in the world who wants to try to hack in and show us how vulnerable this is can do so. Well, it's not every day that you're invited to hack into government computers without going to jail. And so I got together a team of my students and we decided to take them up on that. So here's what their system looked like. You'd log in, nice pretty web interface, download a ballot, fill it out in a PDF reader, upload it again, and that's it. Thank you for voting. Tell your friends on Facebook and Twitter. Nice and shiny. This is anyway. So the week before the election, these guys, Eric Wister and Scott Walshok and I got together in my office one night, stayed up pretty late, reading the source code that they had published in a nice GitHub repo for us. And we read the DC source code and it was maybe a few hours into it, three, four o'clock in the morning. We were looking at this procedure here. And this is Ruby on Rails, which none of us had ever seen before. But we were able to sort of figure out what was going on. This highlighted line here. The voting system is using GPG to encrypt the uploaded ballot so that it would be secure, secret, until it was time to count the votes when they would be moved to another machine and decrypted with a private key stored there. The problem turns out to be right here. They're using double quotes instead of single quotes. That was enough to let us hack in and steal all the votes. So the problem is that this allows a script injection attack because this particular library they were using, the version they were using, just dissembled a string and used the system call to pass it to bash. And it was good in that it sanitized the base name of the file. But it didn't sanitize the extension. So if you used an extension that had some bash commands in it, why they would be executed at the shell under the user permissions of the web server process that was accepting ballots and running the election. This was the first example we tried and it worked. Anyway, we also found on the same network some other interesting devices, including a series of webcams that didn't have any user names and passwords that were in the data center. So here are the machines that were running the election. Here are the workers. Here's the security guard. He doesn't know that we're hacking into the server. But this was actually very useful. As we could tell by monitoring these guys, whether they were suspicious that anything was amiss. And in fact, we saw a market change in their behavior, their posture, after eventually they found out that we had gained control of the system. They were not too happy. Anyway, I'm getting just a little bit ahead of myself. Finally, it was time in DC to attack the system. So we waited until five o'clock when I knew from the surveillance videos that the citizens typically went home for the night. At that point, we started using this shell injection vulnerability to execute remote commands. In fact, we built a kind of simulated shell that would do the right thing to compile something into a ballot, upload the ballot, execute the command, leak it back out via putting something in a public path on the web server and basically make it look like we had a command line. Anyway, from there, we proceeded to attack the system, playing the role of a real attacker. So you're a real attacker. You've hacked into the voting system. What is the first thing you do? Does it steal all the votes? Well, no. In fact, the first thing you do is steal everything else you can get your hands on that might let you get back into the system. You establish persistence. The second thing we did was steal all the votes. And we replaced them with our own ballot where each candidate was a writing candidate, an evil robot or AI from sci-fi or the movies. Who would the computers vote for if they were in charge? At that point, we rigged the system to replace any new votes with votes of our choosing. We added a back door that would reveal the secret ballots of anyone else who voted. And we cleared the logs to try to hide all the traces. Now, we just had one more dilemma, which is that we had gained complete control of the system. But the real election was then about a week away. And we wanted to let DC know what happened. But we didn't want to just call them on the phone because we thought it would be really, really interesting to test how well election officials could detect and respond to and attack during a simulated election. And there's never been a good example, a good study of how this would play out. So instead of just calling them up, we decided to leave a, we thought, not so subtle calling card. So what we did was we changed the source code to that thank you page at the end, the thing that says, tell your friends you voted on Facebook and Twitter. And we added a few lines right here. This would, after a few second delay, cross the voters computer to start playing the University of Michigan's football fight song, Hail to the Victors. So it was still almost two days before the election officials noticed. And it was only when someone else called them up and said, the system seems fine to me, but I don't like the music that plays at the end. It's distracting. So at that point, I think they looked around, realized they were all fans of a rival football team and had a terrible sinking feeling. Anyway, DC ended up doing a smart thing. They did not use the internet voting system to receive votes. Instead, to help absentee voters cast votes in a timely way, they let you download a blank ballot, print it out, and send it back in the mail. So by doing that, they eliminated most of the risk and met the voters who needed to get their ballots back in time halfway. All right, so this is the best study we've had so far. And it brings us to the topic, the main topic of today's talk, internet voting in Estonia. So Estonia is a really, really interesting case. And I've been following for years what has been happening with Estonia's internet voting system. Because Estonia, more than any other country in the world, has deployed and utilized internet voting. So for those of you who don't know where Estonia is, including most of the Americans in the audience, it's right here. So you can see that it borders Russia. It's also a member of the EU for you Americans. It's also a member of NATO. So Estonia is actually quite a technologically developed country. They're a leader in the government. They're doing lots of interesting experiments with ways to provide services to the people over the internet. So it's no, within that context, it's a little bit less of a surprise that a country like Estonia would be experimenting with internet voting. And yet Estonia has done more than just run a few trials. Over the past decade, they have now conducted, I believe it, seven elections over the internet. Including in the most recent election in May of this year for European Parliament, more than 30% of all votes were cast over the internet. So that's just incredible. There's no other country that comes close to relying that heavily on the internet for votes in national elections, 30% and more. And yet whether the Estonian system is secure is a question that had not been adequately answered. In fact, there was not until we got there anyway an independent international study that examined in detail the technology and its security implications. And because of this, many other countries, including my own, had people looking around, wow, Estonia, look at that. They're voting online. Why can't we do that? Wouldn't that be great if we could vote over the internet? So I wanted to know, and my students wanted to know, and many of my friends, has Estonia actually solved the problem of voting online securely? Have they defined their system in a way that it would answer the kinds of realistic threats that major countries face in conducting elections? And what can my country, what can all of our other countries, learn from Estonia's example? So I was looking for many, many years for an opportunity to go to Estonia, to meet the people there, to study the system, and to try to answer these questions in the context of e-voting research. So finally, I had this opportunity in October of last year when I was invited along with a team of other international researchers by the city of Tallinn government to come to the country to talk about our experiences with e-voting and to be an observer in the national election to actually see the process of the e-voting system being administered. So we were official credentialed election observers. We got to tour the data center where the servers conducting the election were housed. We got to meet many of the system developers and I interviewed them extensively, including people like Tarvi Martins, the father of the system, who actually gave a talk about it at I think the 25C3 years ago. Tarvi is one of the lead developers. It's his baby. And so it was a great pleasure to get to talk to him one-on-one for an extended period to spend a day with him and to learn how the system worked. We also got to examine the source code because Estonia just last year released for the first time partial source code to their voting system. They've released the source code to the server. The client is still closed source. They're asking you to install a closed source program on your computer. They say this is to prevent people from building dishonest look-alike software or at least to make that harder. We also got to review, and this is a really interesting thing they do, we got to review dozens of hours of video footage that Estonia post during the election of them doing all of the pre-election server configuration of them doing the daily backups in the data center. Really interesting that they provide this and it gave us the ability to fill in a lot of the gaps, the blanks in how the system was actually being run. So the next thing I want to show you is what the system looks like from the voter's perspective. So if you're a voter in Estonia, you go to cast your vote. You have about a week to do this prior to the in-person election day. And you log in from your computer and download an application. And there's only one thing in this picture that might look unusual to other people. And that's this, what's that? Well, Estonia, one of the really interesting things they've done with computer technology is their national ID cards are all smart cards. And in fact, every citizen in the country has one of these IDs that has a pair of RSA keys embedded into this chip. And you're able to use them both to authenticate to web servers using HTTPS TLS client auth. They're one of the only countries where that's widely adopted. Or to sign documents and they have a legally binding accepted by the government e-document format that accepts signatures from these cards. So many countries have attempted to do things like this, but Estonia is unique in that this is actually widely used. Many, many people, a large fraction of Estonians use these cards to bank online. They use these to file their tax returns. They use them for accessing healthcare services online. It's actually widely used. And I think it's fantastic that they've been able to roll out a national PKI and see it adopted. So it's no surprise really that they based the whole design on the voting system on the capabilities and functions that these cards provide. So as an Estonian going to vote online, the first thing you do is go to an official website and download a client application. And this is available for Windows, Mac and Linux. Then you install the client on your computer and you proceed to interact with it using your smart card, your national ID to cast your vote. So here's how it works. So the first thing that you do is you run the application and it asks you for a four digit pin. This pin is required by the smart card in order to activate the authentication and signing functions and let it use the keys. After that, it connects to the election server, authenticates via TLS client auth, figures out where you live and sends the ballot to the application. You just get to pick who you want to vote for in the app. There's a new application for each election, by the way. At that point, you click cast your vote and the election software proceeds to do some crypto. So here's what it does. It does two things. First, it's going to encrypt your ballot choices using RSA and using some padding randomness that's been generated by the client. Then it's going to take that encrypted ballot and sign it, digitally sign it, using your national ID card. So the second step where it digitally signs it results in the signed ballot and the signed encrypted ballot goes up to the election servers where it's stored until it's time to count the votes. So that's the voting process in a nutshell, very simple. Next though, Estonia has a feature that's been introduced just in the last few years that lets you do something that they call verification. So the last thing the election client software does is it shows you this QR code. And the QR code contains two things, a ballot ID and the random coins that were used to pad the ballot before RSA encryption. Using those things, you can use a smart phone application and this is now available for iOS and Android to scan the QR code. The app then queries the election servers and the servers send back an encrypted ballot with the signature stripped off. So this is claiming, the server is claiming that this is the ballot it received from you. At that point, using the random coins that it got from the QR code, the smart phone app can try to brute force your ballot choice. It can try each possible choice until it finds one that encrypts to the same thing. Okay, then it finds a matching one and it displays the resulting candidate. As a safeguard against coercion, this validation procedure can only be done up to three times and it can only be done for up to 30 minutes after the ballot is cast. Otherwise, someone who wanted to coerce you could just say here, show me your verification. Interestingly, as another safeguard against coercion, you have the ability to replace your vote as many times as you want up until the in-person election. So you just do the process again and your new vote takes the place of your old vote. Only the most recently cast is counted. All right, so that's the verification process. Now how does counting work? Counting is interesting in the Estonian system because it's basically just trying to do the cryptographic analog to the double envelope absentee ballots that are used in many countries. Basically, with a double envelope absentee ballot, you have an inner security envelope that contains your ballot. Then you have an outer envelope that contains your name and signature. When it comes time to count, they strip off your name and signature after checking that you have only voted once and you're entitled to vote. Then they separate the inner security envelope, shuffle them up so they can't be matched to the names, open them and count the votes. Well, in the Estonian system, they're doing something very similar. So the election servers store up the votes until it's time to count. And then they take those signed encrypted ballots, strip off the signatures and burn them onto a DVD. So they take this DVD and use it to move the encrypted ballots to a physically separated air-gapped machine called the counting server. And only the counting server has access to the private key that is used to decrypt the ballot. So they're encrypted all the way from your client to the counting server. And the counting server can decrypt them and see the votes, but it never sees the signatures that identify them. So in that way, they attempt to maintain the secrecy of the ballot. The output of the counting server is simply the election results. They're added in with in-person votes and the winner is declared. So that's the Estonian process. And it was really interesting to figure out quite how it worked. There are no published descriptions in English that cover the entire thing prior to our study. We had to actually ask people and review the source code and all of that in order to get a good idea of what was really happening. The next question, though, after you understand how the system works is what threats does it face? And really, we've reviewed already some of the problems with internet voting, you know, insider attacks by dishonest election officials, people coercing the voters, malware on the client, but who else would want to attack such a system? Well, Estonia brings some particular examples to mind because, first of all, Estonia very notably was hit in 2007 by some of the earliest examples of what many observers considered to be state-on-state cyber warfare, when they suffered large-scale denial of service attacks against national infrastructure from groups associated with Moscow. Secondly, just this past summer, Ukraine had its post-revolution election. And during that election, there were widespread attacks against the election infrastructure itself. Now, that election wasn't conducted online, but the tabulation process, the process of bringing together all of the results from across the country relied on networked computers to receive votes and publish the totals online. That process was reportedly attacked by groups thought to be linked to Russia as well, who attempted to discredit the election. And I read, even attempted to cause it to release the wrong results. So all of this became public this last summer. I mean, this leads me to believe that the right threat model for an internet voting system has to include sophisticated state-level attackers who might want to influence the national result. And for a country like Estonia, an EU and NATO member that borders Russia, there are probably many sophisticated state-level attackers who might want to say in its future alignments. All right, so with that threat model in mind, let's evaluate the design of Estonia's system. And there are two components of the Estonian design that just by reviewing the design you can tell are implicitly trusted components. And in security, when we say something is trusted, we mean basically that if it's hacked into, we're shit out of luck. So that's what we mean by trusted. These two components are the voter's client and the counting server. And let me tell you why both of these are potential vulnerability points or serious vulnerability points in the Estonian design. Well, let's start with the client. So the voter's client in Estonia potentially could be compromised by client-side malware. So here's a simple design for malware and something we actually implemented in the lab. To implement these attacks by the way, we've reproduced the complete Estonian system in our laboratory using their server-side source code, their documented procedures, and by reverse-engineering the client to change it to talk to our servers and use our keys instead of the official ones. So we set up a complete mock election in the lab and we have VM images on our website if anyone wants to try playing with it in their lab. Anyway, so imagine you have a voter's client and you're able to get some malware onto it. That malware could just basically tap in to the election client process and steal the voter's pin as it's typed in during the real election. Then later, the next time the voter puts in their national ID card, say to bank online, that malware can just invisibly in the background use that stolen pin to cast a replacement vote. The voter never finds out, the vote is changed and the attacker is able to steal one vote by that procedure. Now there are two big questions here, how to infect clients and how to defeat that verification app. So how to infect clients, well, we have to leave this a little bit to the imagination because we didn't have a pre-existing botnet infecting thousands of computers in Estonia to play with, but other people do. And so one way, that's one way you could easily imagine changing thousands of votes. Another way would be if you were, say, the NSA, you have your stockpile of ODES, you just attack some popular website or application used in Estonia and thereby infect people's clients with the malware. A third way might be to smuggle in malicious code with the official voting application, which is something we know that everyone who's voting online in Estonia is installing just prior to the election. Anyway, there are various ways you might infect the clients. Then how to defeat the verification app. Well, this turns out not to be all that complicated because thanks to their anti-coercion schemes, remember this tension between integrity and ballot secrecy, thanks to their anti-coercion schemes, the verification app can only be used for 30 minutes after the election is over. After the vote is cast, so all we need to do is wait. If that doesn't work, if that's too suspicious, we could also try a hybrid attack involving a malicious Android app and compromising the voter's client. Thanks to the convergence of these platforms, it really isn't so hard to believe anymore that someone could simultaneously and in a correlated way attack both. Anyway, there are various ways to do both of these things. We can move on and look at the server side too. Now the server side, the Achilles' heel of the system is this counting server. It's the only thing that gets to manipulate the unencrypted votes. And nobody ever sees those votes, they just see the output. So if the counting server lies, right, if the counting server lies, then it can just arbitrarily say what the election outcome is. But they tried to make it pretty hard to manipulate the counting server. It's air gapped, it's built before the election. It's sealed, it's in a safe somewhere. We have to imagine that it's fairly difficult. So somehow you need a way to tamper with the performance of code on that machine. Now we experimented with and actually built a tool chain to do this and compromise that machine even with the security procedures they have in place. So our tool chain basically draws on the idea from Ken Thompson's Reflections on Trusting Trust, which says that even if one system is secure, right, you needed some other system to build it. Right, and you needed some other system to build that one. So if you follow that chain, eventually you'll come to some place that the attacker has access to. So in our investigation, we found out that the way they build that machine for the counting server, it runs, I think, a Debian variant, and that is installed from a DVD. That DVD is burned in a separate development machine that is built before the election and actually downloads a fresh copy of Debian from the web and burns the DVD. So let's assume we can step back a few levels from the counting server and compromise that dev server. It is on the internet. It is built before the video recording starts at the beginning of the election. Let's say we can get some malware on that. So we built a demonstration where that malware infects the installation CD that's burned, uses a root kit to lie about the SHA-1 hash of that ISO image because they are checking it. And then that compromise DVD proceeds to install some backdoor code into the counting server when it's built. At that point, changing votes is really easy. All we have to do is intercept some code in the counting server that uses an attached HSM to do the decryption of each vote and basically look at what comes back from the HSM and replace it with a vote of our choosing. This way it takes approximately the right amount of time. The HSM decrypts the real votes, but the results are fraudulent. Okay, so these are two attacks that rely a little bit on some capabilities that we don't have, right? Access to ODes or a botnet or insider access, all things that the real attackers do have though. But still, it might be possible that Estonia's operational security is so good that these advanced attackers would have problems compromising the machines. So how good is their operational security? This was a big question in our work and something that we spent a lot of time reviewing the videos, doing the interviews to find out. All right, so the president of Estonia, this guy says that their security is better than Google's. This is the standard that they've set for themselves. That's good, that's something to aspire to. Let's see how good their security actually is based on the official videos that they published during the election. All right, so here's Tarvi Martin's. Here above his head is the Wi-Fi SSID and password for the network they're using. All right, here they are building some of the software and the servers and the configuration for the real machines. Let's zoom in on their whiteboard here. Oh dear, okay, so they're using some Windows Shareware, they're downloading over HTTP in order to write the configuration files for the servers. This doesn't look good. All right, let's move on. Here's another computer. They're testing out the client software in this screenshot. Here, let's zoom in. You can see the desktop pretty well. Wait a minute, what are all these icons on the desktop? Here's some poker sight, a BitTorrent client. I think this is pirated music. Oh dear. All right, so this is not a clean, secure machine. I hope they're not doing anything important. Oh my, all right, here they are signing, digitally signing the official voting client that they're going to ask everyone in the country to download and install on their computers. Oh my gosh, this is like the most dangerous possible thing to have the real client binary on a potentially compromised machine. So if someone compromises this machine, they can get their malware into the official voting client and distribute it to every Estonian voter. All right, what else? Oh, later in that same machine, there's Tarvy's name. I think this is Tarvy Martin's personal laptop. Oh, all right, here they are setting up one of the servers. They're logging in his route. You can see the keystrokes. Here's someone entering their PIN for their national ID card. Here in the data center, this is really useful. This big key here, the big key there is the one that opens the data center door. Anyone have a 3D printer? All right, so this doesn't seem to be the level of operational security that we would need to defend against state level attackers. But I don't want to be too harsh on them. This is the level of operational security that's typical in a government IT system. But this isn't just a government IT system. This is determining who the next leadership will be. This is a national security critical system. All right, here's one more thing that happened. So they had some things go wrong after in the election, in the last election, including that at the end where it was time to copy the official results out of the counting server, the DVD burner wouldn't work. And so they looked around, well, how are we gonna do this? Well, we have to copy them off and bring them on to another system to digitally sign them and publish them. Well, so Tarvi Martins pulled the USB stick out of his pocket and put it into that counting server. You know, the only thing that knows what the real votes are, and then plugged it into his Windows laptop and signed the votes and published them. So here's what appeared on his Windows laptop when he plugged in the USB stick. And you can see that it's been all over the place. It has a talk about the voting system he gave on there. This is not a clean USB stick. So yet another potential route from malware to get into the counting server. All right, so this was our assessment that the Estonian system had serious potential vulnerabilities, especially against a state level adversary. And there was no way that the operational security they had in place was going to resist it. And we were in a bit of a predicament because we had been to Estonia the previous October and told the election officials basically that we had these concerns and then we went and we confirmed them to ourselves in the laboratory. So now Estonia had another election coming up. In May of 2014, we knew this information. What would we do with it? We knew the election officials were already convinced that the system was just fine. So we decided to go public with the information and unfortunately, because we all had a lot of other projects going on, it got pushed off until more close to the election than I was comfortable with. And we went to Estonia just about 10 days before their next election to tell the public what we had discovered. So we flew, we got to see Tallinn, beautiful city. We set up a hacker base camp in a nice big Airbnb place with I think the whole team present. And we proceeded to call a press conference and announce our findings. We put up a website that summarized them as I have now for normal people to understand. And we published a detailed technical report that was later published at the ACMCCS conference. So the reaction and understanding the reaction requires understanding just two things about Estonian politics. One, that there are these two major parties, the Estonian Reform Party and the Estonian Center Party. The Reform Party, which is currently the running the national government, loves the voting system. It's their puppy dog. It's one of their sources of national pride. They want to market it to the rest of Europe and show how outstanding and modern Estonia is. The Estonian Center Party, which has previously been in charge of the national government, is currently the minority party, but they control the city of Tallinn, the basically the city state within the country. They hate the voting system, possibly because they keep losing elections. The parties out of power always hate or are willing to criticize the technology and the parties in power never are. Anyway, the Center Party has criticized the system for a long time and the Reform Party loves it. Unfortunately, every media outlet in Estonia seems to be closely affiliated with one or the other of these two parties. And so everyone covered the potential results, the potential attacks we found, but they either covered it saying that this proves that the elections were fraud or there's some people attacking the voting system because they're working for a rival party. So we've landed in the middle of all of this and it was rather incredible, but not so much fun. So it was major news for a week and at the beginning of every nightly news program, I flew home on the plane with people in the rows next to me reading newspapers with stories about this. It was the weirdest experience ever. We also got to meet the election officials in a very formal meeting with them and their lawyer and Tarvi Martin's thanked us very much and told us that they had already accounted for all of this and there was no problem. All right, we also got to have some drinks with the security people working for the election who were very convinced that everything was all right because they told us the right people kept winning. I asked them, well, what would happen if by some horrible, horrible error, the wrong people won, fired all of you and just kept running the system exactly as it is. And they sort of had frowny faces and said, that would be pretty bad. But later though, Harry Herstie, one member of our team who is a very large Finnish man and known as a prodigious drinker, went out for serious drinking with this very nice Russian fellow who is the head of security for the election operations team. During this dinner, I am told each man consumed two bottles of vodka after which nothing can be, nothing can be hidden from the truth. So, Harry reports that by the end of this evening, he had drank that root password out of the head of security. And here he is coming back. All right, so one last thing to wrap up. So the president of the prime minister of Estonia went on TV and said he had Facebooked us and we were working for his opponents to discredit the system. Because apparently Jason Kitkat was friends with someone in the mayor's office. He was also friends with the prime minister's minister of finance and had an outstanding Facebook friend request with the prime minister but apparently he never accepted it. Finally, we had some very interesting, I've never been attacked on TV by the prime minister of a NATO country before, especially for whom my friends were. Finally, we had some interesting official responses online from the Estonian election authorities. They say that the verification app detects all bad behavior. Yes, all right, we've talked about the verification app. They also say why steal votes when you can steal money if you could do all of this? Well, all right, I don't really buy that either. But the most surprising thing was Estonia's Estonia cert came out with a blog post. You can find this online in English also. The title of the post is E-voting is too secure. Oh dear. Nice people who care about computer hygiene have no viruses, they say. In practice, computer risks have been eliminated in the voting system. There, my team is there, not because of technical savvy, but they're politically suitable but technically incompetent message. Oh dear. Well, I don't think we're going to have much luck convincing Estonia to change their voting system. However, we can take away some lessons for other countries. The I voting system is in fact not secure against the kind of national threats, state level actors that could potentially be targeting modern countries conducting national level E-voting. It's a national security issue, not an IT problem. And so if you're even thinking of implementing such a system, you need to be pursuing a totally different threat model and level of defense. Politics, unfortunately, as we saw, can obscure major technical problems. And in your country, if your country is considering adopting such a system, please be wary of that. Our recommendation is Estonia should discontinue their internet voting system until their fundamental security advances. But I hope I don't have any... I don't have much hope for them. But just to conclude, here's what I see as the fundamental problem with internet voting. That we want voting systems where you or I or our friends or Tarvi Martins or Vladimir Putin or the NSA can't just hack in and change the election outcome. It's as simple as that. We want democracy. Major fraud needs to be at least as hard as it is on paper. And no technology so far can assure that. For this reason, my take, even though I know about all of the promising research along these lines, is that it's still going to be decades, if ever, until internet voting can be secure enough for use in major national elections and not without fundamental advances in computer security. Thank you very much. Thank you very much, Professor, for this very scary and very entertaining talk. So, we are a bit over time, but fortunately, we have a break in Sal 1 after this talk. So, if you have any questions, please line up behind one of the eight microphones. Yeah, number four, please. Thanks for the talk. You explore how you can change, attack the counting server, but have you looked into whether you can do this other server, whether you can attack this other server, because these signatures are stripped, if I remember correctly. Yes. So, the other server could just burn arbitrary encrypted votes on the DVD. Isn't that right? Yes, so that is indeed another vulnerability. We concentrated just on the counting server, because that was maybe the, what we considered to be the most interesting version of this attack, but there are other places you could potentially be attempting that to. I think that would have a little bit more forensic evidence. We do also know, though, that in terms of the front-end servers, we know that these servers used in 2013 were vulnerable to heart bleed, of course, only discovered months later. I suspect they were vulnerable to shell shock as well. It's really a problem if you're implementing a system like this, no matter how careful you are. Question from number three, please. Yes. So my question is, was there like any testing of the counting server? I could imagine that, for example, you could just produce a big heap of sample DVDs, where you know what the vote is, and then run these on the system, maybe even on election day that you say, here are 10 sample DVDs, here's the real vote, and we do a random order and run them several times, and then see if all of the sample DVDs are fine in any order, then the voters will also be fine, something like that. Yes, so they don't have any procedures like that in place. I think they're thinking of adding such things, but there's subtleties involved there. If you're building such a thing, you have to make sure that there's no way that the malware can detect whether it's an audit or whether it's the real vote that it's counting. And because of, say, side channels or secret knocks, you could plant some signal in the sample files that would cause them to be counted correctly. So you have to be very, very careful in the design of such a system. My worry is not so much that none of these problems can be corrected, as that correcting all of them perfectly is going to result in a system that's too complicated to conduct and to administer. It's just going to result in a Rube-Goldberg system. I think that's why the system is as it is, because they had to make compromises in order to build a system that was cheap enough and easy enough to run. And so, yeah, we can think of some improvements to each component, but closing all of the holes sounds extremely difficult, at least to me. Question from number four, please. Thank you very much for your talk. It was really inspiring. And I wanted to ask what you think on the question whether such systems could ever evolve as being secure enough. As I see the developments that take place in security are always triggered by hackers, not only, but the security holes are being filled as they're detected. So it's like a race between the two that make systems evolve. So I wonder if it can ever be secure. It's a great question. And there's a really promising area of research on this, something called end-to-end voter verifiable voting that's based on some advanced cryptography. The idea here is basically you want a system that ensures your vote is cast as you intended, that it's counted as cast, that all votes are counted as cast, and that every voter can confirm these. One way you could get this is just by publishing everyone's name and how they voted in a newspaper, right? But of course, we don't want that. It's not a secret ballot. But by using some more advanced versions of cryptography, we can get that too, believe it or not. It's really counterintuitive that you can have all these properties. So there are such systems in the works. And if you're interested in trying to hack on these things and make them better, looking at these systems is a great place to start. But they're not ready for prime time. There are lots of questions about usability, about the security of the protocols, about the complexity of the implementation, about whether they could be run properly at a national scale. So there's hope. There's hope coming from research. But I still think that it's at least a decade away, and that's if things go well before it's ready for prime time. It's a great question, though. May I ask another short question? How would you think that end users are going to be able to trust in such systems as they're not developers as we are here? They have no means of verifying that there is no fraud happening. That's really an open question. Just thinking from the American context, I don't know how voters are going to react when their favorite radio pundit goes on the air and says the vote was the election is a fraud. And some nerdy cryptographer goes on and says, well, no, I can prove it because this property of this, you know, this mixed net shows that this and that. You know, I don't think we know how to solve that problem yet. And getting confidence and rational confidence is one of the big challenges in any election technology. Thank you very much. Thank you. Number one has a question. So if these smart cards will sign TLS transcripts and documents on demand, are they really confident that no TLS handshake or document could ever pass as an encrypted ballot? There are separate keys for authentication and signing. And so hopefully they have thought about that attack. But that's an interesting question, Adam. That's not something we looked at. In fact, we said that the security of this PKI was actually outside the scope of our study. But Tarvi Martins is also the father of the national PKI. So you could talk to him about the security of the implementation and operations there. It seems he may not be entirely open to the question, but thanks. Number five has a question. Well, it's also like similar, like even if there were significant advances in cryptography and like how would a citizen be confident in the system if a person is not a cryptographer and can't prove it? And the other thing is like when you're a kid, you're told we have democracy, but when you grow up you realize that there are lots of like skewed results of some votes count more than others and so forth. So you have to be a political scientist to understand like how the votes are counted and you have to be a cryptographer to understand that the count is accurate. Like does any one person like have all that knowledge? I know, I know these are very difficult questions and I don't know the answers. I can say that my belief of what I personally think should be the case is that voters should be able to trust the election outcome without having to trust the election officials or any specialized cast of people, including the nerds or the cryptographers or the political scientists. It should be possible, it ought to be possible for someone to go with their friends, with their group, with their club, with their party and observe the election process and establish rational confidence. It should be designed like that. I know that that's a problem and in fact there are many ways we can try to use technology to give people ways to increase confidence including means of electronically auditing elections on paper ballots that there's been a significant amount of research on that let you, without advanced crypto with just some basic statistics get additional confidence that the election outcome was right. So there are things we can do with technology but I don't have all the answers. Number two, please. Hi, I'd like to ask if you're familiar with the workflow that Bitcoin works do you consider a decryption and encryption and decryption mechanism like this to work in an electronic election? So some people have talked about building voting schemes based on some variants of the Bitcoin protocol. I think it's an interesting thought. I don't know that we want to trust the outcome of national elections even to the Bitcoin ecosystem because it could be that say the outcome of a US presidential election is worth more than the Bitcoin economy. I don't know if that's true or not yet but it's something to think about. Anyway, I think the idea of a distributed public ledger as used in Bitcoin is potentially a very interesting idea for future voting schemes. But again, it seems pretty far out there compared to where the state of the art is now. Once more, number two, please. Yes. Well, you found quite a lot of different problems with their voting system. Could you evaluate what proportion could be easily fixed and what is rather inherent to the system in itself and would need a major re-engineering of the whole thing? Well, the inherent thing about all of these systems is that they rely on the correct and secure operation of some code that's just running in a machine where people can't see the votes. You can't verify the votes with your eyes with your hands because they have to be processed in secret by a computer. And that kind of black box property is something that leads directly to the proposition that if that computer is somehow compromised, say through a supply chain attack or malware coming in through one of these reflections on trusting trust style attacks, then it can lead to compromising the election results. And that's a hard thing to fix. That's the thing that the end-to-end voter verifiability crypto eventually will try to fix. The little things, the bugs, the shell injection. Yeah, of course. Those are easy problems to fix. But in reality, when you're operating an election there's some big vault in one of the packages your stuff relies on that was patched last night and you don't have time to retest and re-audit all the code. You're faced with the choice of either shipping something that's been untested or shipping something that has known vaults. So I don't know how in our patch release cycle-based security world we can deal with problems like that. I think that's something where, you know, we really need fundamental advances in the way we conduct ourselves with computer security before we can have a good solution to it. So overall you're saying that there is no hope that there could be such a system and just pencil and paper still beats it as far as the required properties of the elections are concerned. Pencil and paper has the very nice property that you as a voter can tell how your vote has been recorded and that other people can observe the counting process. Now we have hundreds of years of experience with fraud with paper ballots. So I'm not saying that there's nothing that technology can do to improve on that. We can do stuff. But I think that the most promising ways to improve on paper ballots don't start by throwing away the paper. They start with the paper and then add some other technology that's able to track, to record, to help to audit that paper record to make sure that we can be confident in the result. Thank you. So maybe two or three more questions. Number six, please. Yeah. Do you think it's possible to design a system that guarantees both the integrity of the votes and the votes to be anonymous? Because I think those two completely don't go together. There's certainly intention. These E to E verifiable crypto systems attempt to do that. But as I say, they have other problems with usability, with implementation that haven't all been solved. That's certainly the goal, but that's what makes it a hard problem. And I think that's what we'd all like to be able to build. We don't know how to do that in large scale in practice yet. We're working on it. But that's not a guarantee that we'll ever be able to comfortably resolve those problems. I just want to quickly plug, shameless plug, for my students' talks and my colleagues' talks about other work we've been doing, by the way, that you might also enjoy hearing how we are using our ZMAP scanning tool we built to study Heart Lead, how we bought a TSA naked scanner on eBay and found all these attacks against it. And how we're building, along with EFF and Mozilla, a free certificate authority that's going to be our attempt to see the entire web get encrypted. So shameless plug, thank you for staying. Number one, please. I want to thank you for the talk, but also for sharing your research in the form of a free online Coursera course, secure and digital democracy. So I know you don't want to toot your own horn. I'm doing it for you. Now, to make it a question after all, will there be next edition? Oh, well, thank you very much. Yes, I do. If you're interested in learning more, I have a free online five-week course on digital voting technology available on Coursera. It is free. It will run again soon. In fact, I'm in the process of releasing it, essentially as an online textbook form of where you can just go and watch the video lectures at your own pace. So if you'd like to see the 10 lecture version of this talk, you can find that at Coursera or find a link to it from my home page. Thank you very, very much for bringing that up. Thank you all. Before we come to the last question from number four, I'd like to remind you if you leave, please take your trash with you. And you can still come up and ask the speaker after the talk. So please, number four. My understanding of democratic elections is that they have to be free and private. So even if all the problems you mentioned would be fixed, wouldn't there still be the problem left that voting at home is not ensuring that principle? So imagine some family member forcing all the other family members to vote after what he wishes because they cannot vote in their own cabin. I agree with you completely that it's a very, very hard problem assuring the voter a safe and coercion-free environment when they're voting remotely over the Internet. So, yeah, you can easily imagine a spouse or an employer coercing someone into voting a certain way. And Estonia's approach to that is interesting. This idea that they'll let the person cast a replacement vote, that makes coercion harder, but it doesn't rule it out. The coercer could take the person's national ID card until the election is over to prevent them from casting a replacement vote. They could wait until the last minute and then force them just before the election closes to cast a replacement vote. I do believe that this is one of the hard problems and it's one of the compromises that are made in our democratic principles when we decide that Internet voting is the way we want to go. Maybe there are technological approaches that can try to improve upon that, but I'm not very confident that there are. I think that's an open problem where the answer is likely to be that coercion is simply part of what you get in exchange for the potential convenience of voting online. Thank you. Thank you very much. One round of applause please, one more. Thank you.