 Hello everyone. Thank you for joining us. Next up we're going to have Dr. Philip Stark giving his presentation and I'm just going to read off his bio real quick to introduce him. Dr. Philip B. Stark is professor of statistics and associate dean of mathematical and physical sciences at the University of California, Berkeley. He works on interference and uncertainty quantification in many applications including the census, elections, information retrieval and internet filters. He also studies foundational questions in the philosophy of science and statistics. He developed risk limiting audits as a method to check election results which are now in law in six states and required by pending federal legislation. Stark currently serves on the Board of Advisors of the U.S. Election Assistance Commission. He has testified as an expert witness in a range of civil and criminal cases on issues including antitrust, elections, employment, equal protection, food safety, intellectual property, product liability and vaccines. Thank you very much and let's hear from Dr. Philip Stark. Thanks a lot. Thank you. I'm not sure how to make it. There we go. That might do it. Not quite full screen. Sorry. Excuse me. Thanks. It's a pleasure to be here. Thank you very much for coming. I thank the organizers for having me. Sorry. Is there a... Like nobody dealing with attack here but I guess this will work. Sorry. F11. F11. Function or just F11. Let's try again. Ah, thank you. Awesome. Okay. I'll skip that. The work I'm going to talk about, I mean some of it is work and some of it is just blathering but the intellectual content, a lot of it is joint with a number of collaborators, some of whom are here in the room. Since time is short, I'm not going to list them right now but lots of people. So the question I'm going to talk about is how can you tell whether the reported winner of an election actually won? We have all kinds of vulnerabilities in our voting systems. Did the reported winner really win? And by that what I really mean is did the person receive the most votes among the votes that were cast sort of consistent with the intention or the action of the person who marked the ballot and tried to cast the ballot? So how do we start to think about this? So right now in our country we largely have a procedure based voting system. Sorry, the mic. Is it better? Okay. Sorry, I wander. Right now in our country I would characterize what we have as a procedure based election system rather than an evidence based election system. The evidence that the reported winner really won is election officials saying I used certified equipment, I followed procedures so forth and so on. Trust me, the outcome is right. And the analogy I like to use for this is this is kind of like a brain surgeon saying I used a sterile scalpel, I followed the procedure. Trust me, the patient is fine. And what you'd really like to do is look at the patient to find out if the patient is fine. I think we need to be doing the same thing for elections. All right, so a lot of things in election integrity that have involved auditing have been largely looking at checking the equipment to see did the equipment function correctly? And the problem with that is it's a false dichotomy. Everything functions with some error rate, right? There's always going to be some stuff around the edges whether it's random noise or voters not marking their ballots properly or a box of ballots not getting scanned or whatever it is. So checking that the equipment function correctly doesn't really mean anything. It's not a bright line and instead I want to shift the conversation to checking the outcomes of the election. Instead of checking equipment, we want to check outcomes. Okay, so when we're doing this, there are different ways of organizing elections involved trusting different people to do different things. A lot of what's going on right now in our elections in this country involved trusting procedures to have been followed by, sorry, instructions to have been followed by voters, procedures to have been followed by poll workers, software not to have been modified or altered or to be buggy in the first place, election officials to do things correctly, many, many third-party contractors to do their jobs correctly that are crucial jobs ranging from programming voting machines or configuring voting machines, sorry, creating them, distributing them, programming them, configuring them, aggregating results from voting machines, posting them in some formal way that produces a formal result of the election. That's a lot of trust in a lot of different people. What we'd like to do is to be able to trust fewer people to do things that are more directly inspectable. So part of what I'm talking about is how do we reduce the need for trust in this? So any way of counting votes can make mistakes. Every electronic system is vulnerable to bugs, configuration errors, and hacking. And so the question is, did this stuff that is almost inevitable, is it material? Material in this context is, did it change to appear to win? So whatever happens, stuff will have gone wrong. Was it bad enough to change the reported winner? So it turns out that paper is really a wonderful medium for conducting elections. It has a lot of desirable security properties that right now make it, in essence, essential in conducting secure, verifiable elections. It's tangible. You can count it. You can keep track of it. It's tamper evident. It's human readable. And so if you want to alter an awful lot of it, it's not one person sitting at a keyboard somewhere. You need a lot of accomplices on the ground if you're going to do that. So that makes it harder to sort of hack a paper-based system, at least the paper portion of it. Other portions of the tabulation if you're tabulating things using scanners is another story. All right, so none of those things are true for fully electronic systems. All right, so the fortunate thing is if we do have a reliable voter verified paper trail indicating how each individual's preference is, there's a way to check whether the reported winner really won, even if calculating the reported winner involved a lot of technology, software, people, so forth and so on. Provider we keep track of the paper trail. Now, one thing we could do is just completely recount the paper trail by hand, if the paper trail is trustworthy. That's expensive and it's not something we want to do routinely, but it is ultimately a fallback that we can rely on. What we'd like to do is be a little bit more strategic, use some statistics in order to look at less than all the paper when that is possible. So if you're willing to accept a small risk of not correcting an incorrect reported outcome, then typically you can get by looking at a tiny fraction of the paper that was cast in the election rather than all of it and still get strong evidence of who really won without having to recount the whole thing on the assumption that the reported winner really did win. If not, what you'd like to do is proceed to a full hand count and use that hand count as the basis of correcting the reported outcome that the electronics have reported. So a risk limiting audit is one way of formalizing this notion. A risk limiting audit is any procedure for looking at the election results and the ballots that has a known chance of correcting the reported outcome if the reported outcome is wrong and never makes a right outcome wrong. So far, so good. This is like the crucial thing. The assumption is the outcome is wrong and if the outcome is wrong, you want a big chance of correcting it. If the outcome is not wrong, you want to do as little work as possible. So that's the goal. The risk limit in a risk limiting audit is the largest possible chance. Thank you. It was my squelch. I'm popping. A risk limit is the largest possible chance that the procedure won't correct the reported outcome if the outcome is wrong. The way it corrects the outcome is by doing a full hand count. You can think of a risk limiting audit as an intelligent incremental recount that stops as soon as it's clear that it's pointless to continue and doesn't stop until it's clear that it's pointless. This is not looking for a smoking gun. Thank you. Thank you. Is that better? It's looking for affirmative evidence that the outcome is right. So you start by assuming it's wrong and look to disprove that hypothesis. All right. Now I still have to hold this up because I'm not... Okay, I'll keep talking. All right, so what's the rule? The rule is audit until you have strong evidence that the reported winner really won and that could mean keeping going to a full hand count if you never end up with strong evidence that the reported winner really won. Does that work for everybody? Okay. All right, so it turns out, as I mentioned before, that very often this doesn't involve looking at that many ballots in all. And the best analogy I can give you for that is tasting soup to find out how salty the soup is. If you want to know if the soup is too salty, you stir up the soup and tasting a tablespoon is enough. And a tablespoon is enough whether it's a one quart saucepan or a 50 gallon cauldron, right? You don't need to sample a particular fraction of the soup. You need to sample a certain absolute amount of the soup. And the act of stirring and then taking a tablespoon is taking a random sample of soup. Stirring is incredibly important. You can't just stick the spoon in without looking. That's a haphazard sample, not a random sample. But if you do the stirring and then sampling, you can get a very accurate impression of how salty the soup is from a sample that doesn't have to grow like the population of soup grows. Similarly, you can get very strong information about who really won an election by looking at a relatively small fraction of the ballots in a large election. Maybe a larger fraction for smaller elections. The amount that you need to look at depends on the margin because the smaller the margin, the less room there is for error. Similarly, if you are extremely intolerant to salt, the sample of soup might need to be larger than a tablespoon to determine that the amount of salt is below some threshold, right? Let me push that analogy a little bit too far. Okay, so another point is this should really be routine no matter how big the margin in the contest is. It's not like, you know, otherwise someone can engineer an incorrect large margin and avoid scrutiny. We don't want this to be contentious. We want this to be routine. It should just be like measure twice cut once. It's count a little bit more than once and certify once, right? But once is not enough. So here's just some pictures to try to ground some of this and something that did actually happen. The upper left panel there is a screenshot from just a little online tool for conducting risk limiting audits. It's not meant to be a production tool, but it's been used for a bunch of pilots around the country. The dice there are translucent dice that Ron Revest who's pictured there standing up on the right gave me. I'm very proud of these. These are my cryptographically secured dice. The fact that they're translucent is actually a security measure. It means that if somebody loaded the dice, they needed to load them with a material that had the same index of refraction as the host material. And we're generating a lot of entropy. It's 20 rolls of 10-sided dice to get this thing going. So the idea is you can't game the audit. Nobody can guess which ballots or groups of ballots are going to get sampled. And this is actually conducting an audit in Napa County. This was back in 2012. But you kind of get a feel for what's going on. Stacks of ballots in the room, rolling dice, running suit around our number generators, plugging data into a little web tool. All right. So what do you need in order to be able to conduct a risk limiting audit? You have to have a voter verified paper trail. Any jurisdiction that has paper can do a risk limiting audit. You don't need new equipment. There are states that have been investing in new equipment that can make risk limiting audits easier, faster, involve touching fewer ballots, so forth and so on. But it's not necessary. What is necessary is voter verified paper trail. You need to ensure that that paper trail is trustworthy. That requires ballot accounting and a lot of other security measures. And what would like those to be done in such a way that election officials can demonstrate to the public that things were secure, not merely assert that they were secure. So some kind of auditing of that process is actually necessary for the public to have reason to trust the result of the audit. Okay. You also need, in order to be able to draw a random sample of things, you need a list of the things you're trying to sample from. In principle, you could dump all of the ballots in an election into an enormous cement mixer, stir them up for days and days and start pulling them out at random. It's hugely practical and actually as it turns out, trying to do mechanical, entering mechanically that way doesn't work very well. There are notable failures from the draft lottery in the 1970s where that didn't work so well. So having a description of how the ballots are stored is incredibly important, but that should be something that jurisdictions do routinely. You know, it's like the public service message from when I was a kid. You know, it's 10 o'clock at night. Do you know where your children are? And the message here is it's the day after the election, the ballots are. And election officials should. They should be able to tell you how many they have, how they're stored, et cetera. Then when you're actually conducting it, what you're doing is manually inspecting randomly selected paper ballots. How you use those data, there are a variety of ways of doing it that have different statistical properties, different levels of efficiency, but there are a lot of ways that you can do this sampling, all of which are workable. You can sample individual ballots. You can sample random like all the ballots that were cast in a particular precinct, or all the ballots that were scanned by a particular machine. You can draw an unstratified sample. You can stratify, meaning like drawing independent samples from different counties or from votes that were cast by mail versus those cast in person, et cetera. You can sample with or without replacement, and you can work out how all of these things work. Ballot polling audits are a particular way of using the data. They're kind of like an exit poll. You take a random sample of ballots and if that sample shows a large enough majority for the reported winner, that can be strong evidence that the reported winner must really have won. That's going to depend on the sample size, how many ballots are for the winner, how many ballots are non-votes, so forth and so on. It's kind of like an exit poll except ballots have to talk to you and they have to tell you the truth, which people being interviewed at exit polls don't have to. Another way of using the data that you get from looking at the ballots is a comparison audit which involves comparing human interpretation to how the equipment tallied those very same ballots. So there are the physical batches need to match the reporting batches for the process. You need data export from the voting system. The demands from the voting system are much higher, but this approach can end up leading to large statistical efficiencies where you don't have to look at as many ballots as you would for polling audits. So these are being done now routinely in Colorado and soon will be in Rhode Island. There's pilots in, I think, eight or nine states now. There's probably been close to a hundred different counties that have done this. Is somebody close, not quite, north of 50 anyway? Yeah. Well, not counting, actually, Colorado by itself accounts for a large number. So there are laws now, I believe, in addition to Colorado and Rhode Island. Is it Texas, Virginia, and then California has a pilot law? Is that right? Okay. Yeah. Texas failed? Oh, I thought it passed. Okay, my bad. All right, I'm going to talk about something else now briefly, which is ballot marking devices. These are right now extremely controversial in the election integrity community. We're actually fighting with each other about this, even though we agree about a lot of the ingredients. So you can think of a ballot marking device as something like an electronic pen. You have an interaction with a touch screen or a rough or an audio interface or whatever it is, and the device ultimately prints something on a piece of paper that's supposed to be your choices in the contest. That thing that's been on the piece of paper might look like a regular ballot. It might only contain your selections and not contain other candidates or what you didn't select. It might or might not contain a barcode or a QR code. All of these things are problematic if for no other reason that the QR code can be a vehicle for hacking the device itself, typically attached to a USB bus. It has... BMDs have some desirable properties. Among others, they can present ballots in many languages. They can adjust the font size to meet the user's needs. They can provide an audio interface and a sip and puff interface and other things that in principle enable more people to vote independently. In practice, it turns out that some of the popular ballot marking devices on the market that have been tested in a number of states have been shown to be essentially unusable by people with disabilities of various kinds. The fact that the idea that this is the miracle cure for accessibility is a little bit overstated. It is probably the most accessible solution we have right now, but it's not great. What if the BMD malfunctions? By malfunction here, what I'm really concerned about is the voter did something on the screen or with a sip and puff interface and indicated his or her choices, but what shows up on the paper is something else. The usual rhetoric around this is well, it's the voter's responsibility to look carefully at the printout and if there's a discrepancy, if that doesn't reflect the intention to go and ask for a replacement ballot and start the voting ritual over again. Research done by some of the people in the room indicates that voters do not actually review the paper, that those who look at it look at it for seconds, not for the minutes that would be required to do this. That people's recall is not very good even of choices that they may remember whether an issue was on the ballot that they actually voted. Experiments that were done with paper printout from touch screen voting machines, not ballot marking devices, but direct recording electronic machines showed that people would generally not notice errors when they did look, if they did find an error, they would generally think it was their fault, not the machine misbehaving. So at the end of the day, if you catch the machine in the sense that it didn't print what you intended, you might or might not notice, that it's misbehaving. Correcting your ballot isn't enough to ensure that the outcome of the contest is right. There needs to be a feedback mechanism that's going to inform the local election official that this machine is misbehaving, it needs to be taken offline, and moreover, if it's misbehaving, others probably are too. Do you have any confidence in the outcome of the election at all? You don't know which votes have been affected, you don't know how many, you don't know in what way. If the election official is going to take complaints seriously, then the whole election is subject to a thud attack, a fear, uncertainty, and doubt, a kind of a crying wolf. And if they're not going to take it seriously, then there's no mechanism to correct problems. And regardless of a problem is detected, there's an issue. The inherent problem with the ballot marking device is it does not create any evidence that the voter can take to prove to someone else that there was an error, of the ability to present the evidence to somebody else that's the fundamental issue. Okay. All right. This is basically what I've already said. Okay. So this property, this idea that if a voter gets evidence of a malfunction, the voter should also have a way to present public evidence that there was a malfunction. There needs to be a dispute resolution mechanism in a technical sense around being able to tell the difference between a legitimate complaint and crying wolf. And the current design of BMDs and the wrapper around them does not make that possible. People have said, well, you could photograph or video or this or that. Well, maybe. First of all, it's illegal in a lot of places and it's not that hard to edit video. The naive thing would be you make your selections, video yourself making the selections and then video that and then change the selections and print something else. It would be pretty easy to spoof something like this. Conversely, one would like to use voting systems where if the local election official has evidence that regardless of what malfunctions might have happened, the outcome is still correct, the local election official ought to be able to prove that to the public without revealing any information about how anybody voted beyond what's already on the ballots. So this first property we're calling contestability that if you have evidence you should be able to contest what's going on and the second is defensibility. The local election official should be able to have evidence to defend a correctly conducted, a correct outcome. All right. So the problem we have right now is that ballot marking device output might not match with the voters indicated. It might not be what they intended to reflect their actual interaction with the screen. As a result of that, risk limiting audits of elections that are conducted on ballot marking devices could confirm the wrong winner because there's no way for the audit to close that gap between what the user did on the screen and what actually was printed on the paper. So right now there's some of the debate in our communities around whether parallel testing would work that is having election poll workers or other staff at random times during the day with random collections of votes and seeing whether they were recorded accurately on the paper. And I'm not going to go into this right now but I don't believe that this is feasible for statistical reasons and for logistical reasons. The amount of testing that would be required would preclude doing any actual voting on the machines. It would just take too long. Okay. So as a result of this current ballot marking devices can be hacked undetectably to alter outcomes. So let me talk about what software independence is. So software independence was introduced by Ron Rivest and John Wack. It's the idea that an undetected change to the software or hardware that's running the election should not be able to produce an undetectable change in the outcome. So everything should leave breadcrumbs. It should leave a trace that you can go back and tell that something went wrong. So software independence is one way of saying that. Strong software independence says on top of that you should be able to reconstruct the correct election result without rerunning the election. So that means not just tamper evidence but resilience, the ability to recover from faults. Okay. Risk limiting audit we've spoken about already. It's the idea of looking at enough of the paper to get strong evidence that the reported outcome really is right. Or if the evidence is not strong enough you've done a full manual tally and you know who really did win. Evidence based elections are this idea that we should move away from relying on procedures to insisting on convincing evidence that the reported outcome is right. And we can do that with a combination of strongly software independent voting systems, handmark paper ballots optically scanned, rigorous chain of custody for the ballots and then risk limiting audits of the result. That kind of closes the loop. Auditing of the integrity of the paper trail as well. I'm probably out of time. I should leave time for a question or two. So I'm going to shut up having left one of the favorite topics here alone. Apologize. Are there questions? I can't quite hear you. I'm sorry. How do we stand? Where? We're in a mess. No. Aside from things like the social hacking that you just observed we still have roughly a quarter of the country on systems that don't produce any paper record whatsoever. The states that are moving away from paper lists to paper, many of them are putting in ballot marking devices for every voter instead of trying to use handmark paper ballots where possible so that we have more direct evidence of what voter intent was. There are relatively few states that have decent audit laws. There are probably no states that actually have as a legal requirement really serious chain of custody of the ballots and demonstrably secured chain of custody of the ballots. There are some places that have good practices, some that have worse, but they're not all kind of mandatory. So we're in trouble. What do we do? I think I kind of just said the list. We need paper, handmark paper with some accessible solution for voters who can't mark a paper ballot by hand independently. We need rigorous chain of custody of those ballots and most of that is relatively low tech. We know how to do this. It's kind of a solved problem technologically but we seem to lack the political will. Yes. People in the audience. What can you do? You can talk to your elected officials. You can be a poll worker. You can volunteer with your local election official. I got to get off the podium here. Thank you.