 So our next lightning speaker will take less time than we just took setting this up is Maggie McElpine. She specializes in a whole lot of things relating to voting. She worked on the Estonian case. She has worked on risk auditing metrics. But really, please. Alright, thanks very much, guys. So, I don't seem to be projecting, guys. What? Oh, just drag it? Oh, no. We'll get there eventually. Well, basically, yes, I've worked on the Estonia eVoting paper with the University of Michigan, and I've also advised the California Secretary of State's office on their post-election audit pilot program of 2013, and I also work in election security, and I've run transitive audits specifically in multiple states, but I've also done a risk limiting audit in Colorado. So I sort of accidentally became perhaps the person who scanned the most ballots anywhere for transitive audits. Maximum was 160,000 one time, and we learned a lot of interesting lessons from that. So I think my previous speakers were really great in kind of covering some of the basics of how risk limiting audits work with the statistical analysis, and the transitive audits though are kind of interesting. Maybe people aren't familiar with those. It's basically running a black box against a clear box, which is commercial off-the-shelf scanners, and then we run it through a program called OpenCount, which kind of is a little bit like running it against a second voting machine. Not literally, but because we can't go into the voting machines by comparing the two, we can basically determine the accuracy of the results. But there are some pitfalls to that, which is not to say we shouldn't do them. There's also a lot of really interesting benefits. It'd be really useful if you could see my slides right now. That would help a lot. Yeah. Yeah. So, unfortunately, that means I can't see my notes that were on the slides, but I'll just have to wing it. Well, now you've had a preview. So like I was saying, there's the transitive audits, which are a little bit lesser known, but they're also very exciting. So let's go into very quickly why we need to do audits. They basically, as other presenters have said very nicely, help ensure the public trust. Paper ballots provide a publicly acceptable forensic artifact of the voter's intent, which is also anonymous. And basically what we're advocating is that there should always be an audit done at the end of an election, regardless of the difference in results. And this is regardless if it's a landslide, because if it's a landslide, it's very easy and low cost. You only have to pull a few ballots, relatively speaking. And if it is close, then it should be audited to keep people from being annoyed. So it also kind of, if you don't have it every time, and for example, one problem you see a lot with recounts, is that the loser can get shut down legally by the winner on whether the recount even happens, which is a bit of a terrible loophole and also fosters sort of an us versus them mentality. Why are we doing a recount this time or not that time or an audit this time and not that time? So we should just do them all the time is what we're saying. The third, the lesser known problem, perhaps because it's a little bit less sexy, is that public interest is stuff like the Senate races, the presidential races, the federal races. That's what gets everyone out to vote. But a lot of the money is actually down in the ballot. So like ballot measures, you know, $30 million bridge is going in. There's a lot of interest for a certain number of people in how that ballot measure goes through. So audits should be across multiple races too, because even the less popular ones. So as I was trying to say before with the risk limiting audit versus the transitive, I think the risk limiting one's been very nicely explained as the statistical analysis you're checking if the results are, if the winner won basically. That's not the same thing as a transitive audit. So what we do with transitive audits is we take a commercial off the shelf high speed scanner, rerun all of the ballots, and it has to be all of the ballots. Then we run it through a program called open count. It's more expensive up front, but it has some benefits over risk limiting audits. They're not contradicting one another, for example. There's no reason to do one or the other. They're not in competition. But you can basically audit all of the races at once. For example, you can compare races, and basically they become one problem, which is like if somebody were to change the paper, the cancelling error thing. So this slide I'll skip through quickly. A lot of times people will say we should or shouldn't do things in the US based on things that are happening internationally. It's a totally different animal. America has one of the most complex fragmented voting systems in the world. That creates a false analogy. For example, one thing that's going to make auditing more complicated is if we start adding ranked choice, for example, because then you've got to audit those races, and then you've got to drop those races off, and then audit the next thing. And that can cause all sorts of craziness and confusion. Transitive audits don't have as much of a problem with that. So we're going to say why you should scan ballots. There's actually a lot of upfront benefits. And in general, auditing improves public trust. But you shouldn't take the following... These are cautionaries. These are just things to think of. They're not meant to be don't do... Transitive audits don't do auditing with a scanner. So here's an example. One is the Xerox bug discovered in July 2013. It is basically an image enhancement thing. Now here's the problem with scanners. They're not dumb devices. Your camera is not a dumb device anymore either. They don't just take what's there. They correct images. And this is a problem, because actually one of the hardest things when we started doing Transitive Audit Pilots was to actually get the scanner to not be smart anymore. It wants to be smart. It wants so badly to help you out. And you've got to tell it, beat it over the head until it stops helping me, because you don't know what I'm doing. Forensic level auditing, you basically want to turn off the quality, turn down all the features. And for example, scanners will try to help you out, and one thing they'll do is correct the images. Do you see what changed? We can go back and forth a bit. Hint, it's in the first line and the third line. So those sixes became eights, which we discovered a problem. The reason this one was discovered, which is fun, is because it was used on architectural things. And that messes with the dimensions of a room if the six and eight get switched. So that was a big problem. That's an example of a bug where a scanner gets too smart for its own good and tries to help you out, and then gets it wrong. So the other one is... So the reason why they kind of do this is because of legal documents are scanning thousands and thousands of pages. You basically want to... How do I say this? It's for the human eye. But when we're doing a transit audit with forensic level, we're not trying to please the human eye. We're trying to get a computer to read it, and human eye and computer see very different things. So one thing it'll try to do with a lot of legal documents is clean up the image. One thing it'll do to clean up the image is remove hole punches. What do hole punches look like? They look like little black circles in the left margin. What also looks like a little black circle in the left margin. We lost an entire population of one of our earliest pilots to an error where it was so random when they would take out the vote, we literally could not reconstruct it and had to redo the whole project. And that is a pretty big investment of time and energy, especially if you're doing a pilot program like a state. So be aware of that. That's one of the features you have to turn off. Other might be things like image correction. So it'll straighten the lines for you and make them kind of non-euclidean as a result. And that'll destroy your timing markers, which tell you what precinct this all is. That's what those timing markers around the edges are doing. They're telling you what precinct, what race this is for the computer, not for the human eye so much. Another thing that it would sometimes try to do was, what was it? Oh, this is just a third one. It's not in my presentation. But when the scanner tells you to clean it every hour, you should do that. Because you'd be amazed how many people just blow that off as like a fuddy-duddy sort of rule. But the black marks begin to streak down from the vote ovals. And they paint the roller. And the roller will roll over the vote to the ovals. And we actually lost an entire population once to the scream marks from the ink, because the software could no longer determine which bubbles were filled in comparatively, because the stains got so dark. Okay, I'm getting on 10 minutes, I think. So, let me see. So basically, I'm not trying to say don't do them. I'm trying to say that there are some things you should be aware of. But they're really, really valuable. I'm a big fan of transitive audits. I'm only going to talk about them a little bit more, because risk limiting has had its cheerleaders today. But you can compare races. We've had software where you can, for example, now if you do the transitive audit, now you'll have the entire population in high forensic digital form. You can actually run risk limiting audits from transitive audits from the digital population. What you have to do, though, of course, once you have the digital population, is check that the paper reflects the digital image. And that's one reason, too, to use the digital images from the voting machine if it creates its own. You really do have to do a second one, because otherwise you're asking in the computer, is this right? And the computer says, yes, I'm right. So the image that's taken from a voting machine isn't necessarily the same thing as checking it with a transitive audit, too. Which is fun and exciting and valuable to know, because a lot of, I think, vendors are going to start claiming soon that you can do transitive audits just with the image they capture, but that's really just asking the computer itself already. The same question. I think that's really it. I think I can give my time to the next guy. And thank you very much.