 Hello, my name is Harry Hirsten and this is talk about Senator Bill 43, New Hampshire, Windham election, forensic election audit we conducted in May and published a report out in mid July. First of all, we were a team myself, Professor Philip Stark from UC Berkeley and Mark Lindemann from Verified Voting. How we were chosen was that I was appointed by the Secretary of State and Attorney General of, jointly between Secretary of State and Attorney General of New Hampshire. Mark Lindemann was appointed by the select man of the town of Windham and we jointly chosen the third person to be Professor Philip Stark. So first of all, why we are talking about election machines at all, this is a little bit of my personal story. I started hacking election machines in summer of 2005 and I got involved by being invited by election, back then election supervisor, Ion Shancho of the capital of Florida, Tallahassee. He was responsible for the widely popular, widely publicized 2004 recounts and when the Supreme Court stopped his effort to find out the truth what happened, he felt that he needs to know more and he wasn't fully appreciating the claim that voting machines cannot be hacked. So he wanted to know the truth how the voting machines work. After the work we did in Florida, I was participating in a number of state commissioned voting machine studies. One of those largest ones is from 2007, Everest by Secretary of State Ohio. It is a sad fact that the study we made 2007 is still valid. Most of the same systems are still in use, very few of the systems have had software upgrades and you still literally have the same software up and running which we published the vulnerabilities 2007. Even the some vendors which have published a software upgrades to their system, those software upgrades are very seldom put in use and deployed. So we have a United States elections are run with a lot of very old systems and I have been doing similar studies around the world outside of United States. I'm also a co-organizer, co-founder of the voting machine hacking wheel chair at DEFCON and also with the election integrity foundation. More recently, we received an Emmy nomination for both of our documentary movies by HBO. Back in 2006, HBO documentary hacking democracy was the first one to light to the general audience the problems in voting technology and then the new one kill chain, the cyber war on America's elections. We received an Emmy nomination on that just few weeks ago. Our mission in DEFCON, voting machine hacking wheel chair is not to prove that voting machines can be hacked. Every single machine we have in a room, every single machine is in use in the United States, but every single voting machine has been hacked and there are no vulnerabilities. Our mission is educational. We want people to be able to see the truth and experience the technology, the old technology themselves. If they have concerns about the software, if they have concerns about hardware, experience themselves how the systems actually work. There's nothing which makes me more happy than when local election officials come to hack the very machine they are using in their jurisdictions under the election and this is the first time when they are allowed to do so. So this is something where they ask the first and often last line of defense, they need to have that knowledge, they need to have the experience, and so that they also understand what kind of mitigations needs to be done. When you look the voting systems, we have a lot of things where we can lower the risk by using mitigation strategies, better physical security etc etc. So our mission is to educate people and before we started the voting machine hacking village, there was a very small number of privileged people who had been part of these election system studies. We have single handled 100-folded number of people who have first hand experience on the machine by just having this room and having people to be able to experience the technology and find the truth. Now when we go to this extremely heated 2020 election along with 59 scientists and election security experts, we sign a letter in 16 November where we underline that extraordinary claims require extraordinary evidence. And while we have seen a lot of claims, we have been not being found a single occurrence at that time where these vulnerabilities either are real or they have raised in the level of possibly changing the outcome of the election. We keep on looking every single time a new claim is coming along and we have access. We always keep our eye on and want to study and want to research any claim with the election technology. In the 2020 election, like the audit which we are going to be talking now, there are instances where the results were inaccurate but like in the case of the New Hampshire audit, which we're going to talk, the inaccuracy didn't change the outcome of the election. The same people get elected even when there was a discrepancy in a number of votes. So we're still largely behind this letter and the text but we keep on looking and we have to remember how long are the lead times from a evidence and be able to look in this. So we keep on looking and we keep on developing strategies to mitigate against the risks because even if the discrepancy didn't change the outcome of the election, there's always a lesson to be learned to improve the election. We have to improve our elections a lot. Little bit about New Hampshire elections. New Hampshire is using handmark paper ballots. No ballot marking device is no electronic voting. Everything is done with handmark paper ballots, which is the most secure way of voting. From the election purposes, New Hampshire is organized to 320 towns and wards and 123 of those count only by hand. New Hampshire elections in the smaller jurisdictions can count with hand without using voting machines. Largid jurisdictions require voting machines. 197 use those and this is normal in the United States. Every state has their own election laws. Some states have very complex ballots. There even the smaller jurisdictions need to have voting machines to reduce the error rate. But with humans we are slow and error prone so we are not optimal counting machines for ballots. The combination should be using a voting machine, which you don't trust and then verify the result with a mandatory risk limiting audience. In New Hampshire, right now New Hampshire is piloting with a risk limiting audience. The process in New Hampshire is that the candidates can request a recount if they think that the result was inaccurate. And recounts are very common in New Hampshire. There were 16 recounts requesting conducted after the 2020 election and seven after the state primary. New Hampshire also has a safety mechanisms in place. One of the many safety mechanisms is ballot rotation, which means that the parties and the candidates appear in a different order in a ballot. So in one town it might be Republicans all the way left and Democrat next and the other on the right and the next town over where it's different ballot it is. Other is first and then Democrats are central etc. So those all are in a rotation. The same way if there is a race where you have a multiple candidates per party and in multiple jurisdictions then the candidates are rotating when the parties are rotating from left to right then the party candidates are returning from top to bottom. In this recount, hand recount, which started this whole thing motion, this was the largest numerical discrepancy between the machine count and the human count and that was 300 votes. And because New Hampshire didn't have a mechanism to find the explanation why there was this 300 vote difference, that required the state legislators to take an action and pass a new law authorizing this audit, the forensic audit. So I have a typo, it's SB 43. So this was a reaction to the Rockham County District 7 context and that context is a eight candidates vote for any four. That's the format. In New Hampshire when recount is done, it's customary that all candidates gain more votes and this is because people are voting by filling the ballot inaccurately, maybe circling the name or underlining the name or whatnot, making the mark in a way that the voting machine didn't recognize that as a vote, but the human looking into the voters intent clearly understands who the vote was for and this is the mechanism why the hand counts always tend to have more votes to all candidates. However, this time in the hand count each of the Republican candidates gained about 300 extra votes. The three of the of the Democratic candidates gained about 25 votes and the woman who asked for a recount, she lost 99 votes. So there was a two different discrepancies. First of all, it was not clear how it's possible that in the hand count she lost that many votes, but also 300 votes was more than the traditional in the context of the election. 300 was a high number for extra votes. The whole election was a 10,006 ballots so that gives you size and this audit was starting from this local down the ballot race. It had nothing to do with the presidential race, it had nothing to do with the top of the ballot tickets. This was a single discrepancy found in the recount in this one local race only. This is what the ballot looks like. Again, this may be giving a better idea what when I say the ballot rotation changes the names of the order of the political parties from left to right and then if the same candidates are in multiple races, then from top to bottom rotating the name of the candidates. You also see that the names of the Democratic and Republican candidates are not on the same line and this was on purpose. The reason why this is done this way in New Hampshire was that they want to make clear that the voters understand that the candidates are not running against each other. You can pick any four, you are not bound to be making the choice between two candidates who are in the same row. So this physical setting of the candidates, having the Democratic candidates one row below, was to try to make sure that the voters understand how the ballot is constructed and don't have the misconception that the candidates would be running against each other. This is the voting machine used in New Hampshire. It's an acu vote precinct optical scan machine. It's a very old model. We go back to that in the moment, the technical specifications. It is also widely used in other states. So this is not only in New Hampshire. This is widely used in all the states of New England, but also in other ways. These have been used previously in large states like California and not anymore widely used machine. Just because it's old doesn't mean it's bad. First of all, this machine was originally developed in 1986 and it's believed that it was first time used in elections in general election 1990. It started to be marketed in 1989 and the first election where it's believed to be used was in Minnesota. This is extraordinarily old technology. The CPU of that is Systemon Chip. It's a microcontroller version of V20. It's a VNEC V25. It's a pin compatible with Intel 8088 and instruction compatible with 80186. So this is the first IBM PC era technology. That chip cannot address memory more than one megabyte. That's the maximum size of a memory. This is really that old. In this particular system, there's 256 kilobytes of RAM. That's all. This is not scanning an image of the ballot. This is a technological optical marker recognition, which means that the scanner is looking for a landmark, landmarks, timing marks and finding the vote targets from the cross sections of those marks. Inside of the memory of the machine, there's never a picture, never an image of the ballot. Only these cross sections. The scanner itself is reading 36 positions per line. That's all. The lines are separated one quarter inch from each other. In a sense, this is 36 pixels for the whole side of the ballot line. The scanner is using a laser diodes, which are orange red in color. On purpose, this voting machine is colorblind for color red on orange red. If you have a marking made with a red pen and mark your vote with a red pen, the voting machine will not read it. It will not understand it. The software itself is stored in two EEPROM chips, 128k and 64k chips. These are EEPROMs, not EEPROMs. EEPROM means that the chip can be erased only by using an intense ultraviolet light. The machine itself cannot erase the chip. Only way you can erase the chip and read program, you physically have to remove the chip from the voting machine, place it for 20 minutes under an intense ultraviolet light, and after it's erased, then you can use a universal programmer to write the new software into the chip. For this reason, the chips have a clear mirror for the ultraviolet light to reach the silicon chip, and always when the chips are programmed, a piece of paper or something has been put in top of the lens, preventing an accidental instability by a burst or ultraviolet light. Old camera flashes, for example, could cause an instability. This system doesn't have any traditional operating system. This is a special purpose computer. It's not PC compatible as such. The whole operating system is replaced with a microkernel, and the microkernel is living on the same chips as the applications of running the election. There is plenty of empty space on the chip, so the actual software is only 143 kilobytes, everything combined. This system doesn't have a hard drive or any other form, a flash memory, any kind of storage, which can be rewritten inside of the system to store stuff. Only place where you can store anything and where the election is living is an Epson-style J40 card edge connector, battery refresh static RAM card. Typically, these cards are manufactured in 32, 64 and 128K, but typically only the first 32K is in use on this card. The picture of that memory card is on the left. This is an extremely limited single-purpose computer, where an attacker's surface is very small. This is not because of the design of worrying about security. This is so old that at the time when this machine was designed, cyber warfare was bad science fiction. This is a hallmark of how old this computer is. This is even better than most of the hardened systems of today, just because how limited the capabilities of the system were at the time. Here's the picture of the voting machine's internals. There you see the two chips with a white stickers. Those are the RAM chips containing the software. The two chips, the left side of the chips with the white stickers, those are the RAM chips, 128K RAM chips 2, so 256. The square chip on the side, that is the V25 microcontroller. As you can see, this is from the era where a normal 74 series TL logic chips were used. This is a very, very old design. Also, I would like to point out that because of the limited horsepower, this machine cannot, even if you would introduce a wireless counter, this doesn't have the horsepower to drive Wi-Fi, this doesn't have a horsepower to drive Bluetooth. The only connections this machine was originally having was a RS232 serial port with a maximum speed of 9600 bits per second and a modem with a maximum speed of 2400 bits per second. But as a security measurement in New Hampshire, the modem has been physically removed, the sticker on the white sticker in top of the plastic, that's where the modem was before it was removed and also the cables for the serial port has been cut and removed. So in New Hampshire, this voting machine has no means of communicating with the outside world by any other means than the memory card. You can also notice the copper color inside of the plastic. This was the way how electromagnetic radiation shielding was done back those days. That is a Faraday cage, which has been done by spraying the inside of the plastic case with a copper. So this, from the electromagnetic emissions point of view, this is an extremely quiet and well-shielded machine. And of course, inside of the machine, there's additional shielding for a components, which might be radiating more. The compartment on the left is where the battery, so this machine is intended to be able to operate without an external power for limited time, so power cut wouldn't be stopping the election. I would also point out an interesting thing, which we will come back later, which is that when you look the center of the voting machine, you see a white line on the top of the metal casing. We will get back to that white line later. So this is the law, which was authorizing this audit, and as you can see, this is really a targeting, the Rockham district, number seven, which is Windham, and that a race only. But yes, a measurement, they are asking for recounting to other races just to make sure that whatever is the cause of the discrepancy wouldn't be affecting other races. So this was a security measurement to take into account a two wider races just in case it will reveal something. And the law called us to use all four machines in that county to count every single ballot, so it's a little bit different procedure than what is used in election day, where the ballots were divided between four machines, and then also take the custody of the ballots, understanding how the population of used, unused ballots, etc., is maintained, and then hand count everything, hand count those three races, the state representative governor and United States senator having a local race, statewide race, and federal race in hand count. This was the scope of the audit, and we were told and authorized if we find something, which expands a larger, we can follow the evidence wherever the evidence will lead us, because the intention was to find out what caused this discrepancy. Even when the discrepancy didn't change the outcome of the Rockham district seven race. The first order of business was that we machine counted every single ballot. There are a number of ways you can do the machine count. You could have a remove the cards from the voting machine, make an image of the voting machine with a third party device. We decided not to do that because this technology is arcane, so we would have needed to use a third party device, which nobody else have audited, so it would have raised a concerns potentially what happened when the cards were removed from the ceiling and placed to this device. We could have used the voting machine to make a copies of the memory cards, but since we were auditing the voting machines, it would be illogical to trust the voting machine to create a forensic evidence for us, when the voting machine is in doubt. Because New Hampshire had removed the cables, that was not an option, so the last thing which we chose to do was in order to have the best possible security for and also regaining the public trust. We only reset the counters of the previous election after printing on audit tapes, and we left the cards fully sealed into the machine with no physical access, reset the voting machine mode from post election back to pre-election and ran it as it is, and also setting up the clock of the voting machine back to the November election day. So when we started, when we process the paper ballots with the same memory cards, except that now we put all the ballots through every machine, we got the results which were a little bit different than what was the 2020 election. Remember at the time when we started this, we don't know which one is closer to true, is the hand counter problem, is the machine counter problem, are they both wrong? This is the starting point was we don't know what is the source of the discrepancy, was it the human counter or is it the machine counter? So first when we ran all the ballots through we found a plus minus two difference in 10,000 for the presidential race, we found in governor's race a 50 to 60 fewer planks between those four machines, because again all four machines had a little bit different results. The U.S. Senate, the Exit Council, very small changes, big issues in the state representative race, and then rest very small discrepancies. But these results were different than the 2020 election, so now these voting machines didn't even agree with themselves. So now we have to think of what has changed, with the memory card didn't change, what has changed, why we have a different kind of results. This was the first clue for us in our journey to follow the evidence. This is as a numbers, the hand counts between the 2020 hand count and 21 hand count and the machine counts. Now we see that the two machine, the two hand counts agree very closely to each other, which gives us a strong evidence that the hand count result was the right result, and the machine count results were the incorrect results. So this was the first critical piece, now we know that probably the main course is not hand count, probably the main course is the machine counting, wrong and the machine counting them different. The other part which we found immediately was that this, the voting machines, the four voting machines didn't produce exactly same results between each other, so we had a variation of the results between the machines. We can see that two of the machines are closely matching each other, one machine is a little bit off and one machine is way off from the others. Another interesting data point. When we looked at the other hand counts, the governor and us senator, we saw that in the governor's race there was a little bit more discrepancy and discrepancy is a non-votes being lower and more votes found. So about the same amount of votes were found for both of the governor candidates and the voting machine had clearly in this vote failed to recognize votes when there was vote and when we have a US senator race, very small differences. When I go back to the previous one, we see that the same thing was in the US state reference. If you have, there's a thousand less non-votes for anyone. So now we are approaching the question, what means non-vote? And the non-vote can be two different things. It can be that the voter didn't vote anyone or the voter voted too many because the other way non-vote can happen is called over vote condition. It means that when the voter was asked vote for four, the voter for example voted for five or six and that cancels all the votes. So if you have a race vote for four and you vote for five, the result is nobody gets any votes. So this is the condition we started looking and focusing. Maybe we are seeing what caused this problem. This is all 10,000 ballots superimposed over each other. This whole red blob in the left was that before we started, we made every single ballot to be unique by placing a unique number to every single ballot. We originally were planning to use a bait stamp machine but the bait stamps broke down and eventually that's why we just then did it with the handwriting. But this made every single ballot unique and identifiable. And after this serial number was done, we didn't use the scans of the ballots for recount purposes. These were reference scans to be used for analysis and we didn't know at the time when we did the reference scan how useful this will be. I want to make sure that everybody understands that if you scan ballots with a separate scanner, you cannot trust those images. There are a lot of features in the scanners which have been shown to alter the ballots. For example, the bunch hole removal from legal documents, bunch hole looks a lot like a filled oval. So when you have a filled oval, it can remove that. You also have to establish trust that your image population is a true representation of the physical ballots. So if you are creating ballot images for the purpose of, for example, making a recount, then you have to make extra steps to convince yourself that your image population is trustworthy. Just taking the images or hashing that, that is not nearly enough for this purpose. Anyway, when we superimposed the images, we saw two interesting things here. We see in Kirsten, Saint Laurent, I hope I pronounced her name correctly. We see a very high concentration of paper foldings going through her vote target. And we see on top of the ballot a way wider area of foldings, some of them being over the vote targets. I have to, the darkness of the folding is not representing how many foldings were in that place. This is purely maxing a type of superimposing just to find where they are. Other interesting thing is when you look the folding on the bottom through and going through the Kirsten's target, you don't see in a flip side of the ballot nearly a strong marking. So that was interesting. And the last but not least, we find that the folding machine seems to be a little bit, or the folder seems to be a little bit off-center. It doesn't go straight from the left to right. These ballots had a score line. Score line is a weaker than the perforation. It is a print shop created weakness on the ballot. And that weakness is created in a safe zone. So when you hand fold the paper, you automatically fold the paper always in a safe place and it doesn't ever hit the vote targets. So our question was, we know now that the folding, the correct folding line would be above the Julius Sotti's target. That's where the correct line is. The similar, the correct line is in above the candidates in the governor's race. So what caused a folding to appear in an incorrect place when the structure of weakness is forcing the hand folding to be always in a correct place. If you try to fold the paper incorrectly, hand folding incorrectly, you cannot absolutely do it because the weakness is where the paper wants to fold and you would need a tool like a ruler to make an incorrect folding. So the blood thickens. The other part of which we've started looking into the results, we found out that one machine, which was machine number two, processed 90 percent of the absentee ballots. And at the same time, that same machine had the highest error rate. So we had a perfect storm and coincidence where the absentee ballots which are folded are the ones which are going through a machine which have a highest error rate. Why we have a higher error rate in that one machine? The first discover we had was a paper folding machine. When the good people in Wynham started to have a high number of requests for absentee ballots, they had a hard time to cope with the demand and someone had remembered that they had rented a paper folding machine to be used in DMV, not in elections used in DMV. So in order for them to serve their voters better, they decided to use this paper folding machine and they use it as it is. They didn't change the settings of the folding machine. So it folded the paper exactly the same way as the DMV re-registration papers had been previously folded. And at that time nobody thought about the foldings. The Secretary of State had been conducting a few years earlier a testing of the foldings if they caused the voting machine to re-misregister those as a vote. And they hadn't found any problem at the time. So there was no knowledge that the folding can have a this kind of effect. But the folding machine is strong enough that it can fold the paper in the incorrect place. So now we found out what caused the folding to be in the correct place. Again, this was a machine which was rented by Wintham. We made a query to other jurisdictions to find out if they have been using folding machines and we couldn't find a single other township award which would have us the voting machine. So our query was not conclusive. We didn't ask all 3,320 but from the sample we did and asked the question we couldn't find another place with a fold. So that's a pointing that this folding machine is a unique occurrence. It's a black swan in this mix if you may. When we look at microscope what the folding machine looks, we found an unused ballot which had been folded on the election day. The folding on the right side is the folding which was done on the election day and the filing on the left was when we used the folding machine to recreate. And this is a bump and we discovered something else interesting. If the papers fold it up or down makes a huge difference. When it's up folded it creates a bump which creates a shadow and the shadow can be misinterpreted to be a marking on the ballot. While it's down folded it creates a bridge and it's almost invisible in the microscope that the folding was there. And this is consistent with what we saw in the superimposed image where the same strong folding which went through the Christie's oval on the front side. There was not a strong folding on the flip side and that was because the unknown factor was the direction it had been folded matters. And again we couldn't reconstruct from the Secretary of State office where they folded those up and down. They didn't know it makes a difference. Nobody knew it made a difference but now we found out what is the difference. This bump is permanent in the paper but it is not equal strength over time. So at this point of time we started looking into the bumps and we realized that why the machine count in 2021 and 2020 were different. The difference was that the ballots had been laying flat six months under the wake of the other ballots and it had straightened the ballots. So you still had the bump and it still created a false readings but it created less false readings because of the time the paper has been flattened under the wakes. Then we started looking something else and the question was why these voting machines have different results from each other. This scanner cannot be opened. There's no easy way to open it. So we used an endoscopic camera to look into the scanner to see what was inside of there and we saw a white powder. Well it looked like a dust so at the first time we didn't even know what we were looking. I used a highly compressed air cleaner and when we cleaned with the compressed air a massive amount of dust as you can see here came out. Only later we realized that this is not paper dust. This is actually an offset printing powder. In offset printing when the ballot is printed the ink is not fully dry when it comes out of the printing machine and in order to prevent the ballots to stick to each other in that kind of printing technology a very very fine powder is sprayed to the sheets to keep them a little bit separated and allowing them to naturally dry. So this was what we found out. It's out of the printing technology and that explained why the machines were a little bit different. We conducted 32 test elections with the 16 elections used by a flat paper and 16 elections with the folded paper. When the paper was flat all machines, all four machines, regardless whether they had the paper dust or not inside, were 100 percent accurate and then at the time when we process the folded paper the differences came out and these differences seemed to be indicating the level of paper dust contamination in offset powder dust contamination inside of the machine. So we found a contributing factor and this is just a we made a sanity check. We found how many ballots are in each way. We found about 300 600 ballots where the folding mark goes through a Kirstis target. We found a about four where the folding line went through Mr Sotti's target. We found a different combinations of the foldings in different intensities and this showed us that there's enough ballots and this also proves us that the foldings can both add votes and subtract votes. When a voter voted for less than four candidates then if the line went through Kirstis target it created a phantom vote for her and the hand count corrected that's why she lost 99 votes in the hand count. But if a voter voted for a straight party line republican ticket and the line went through Kirstis then it created in certain intensity a over vote which then tossed all the votes out and cancelled the votes. So this folding line both explained why the republicans gained 300 votes and it also explains why Kirsti lost 99 votes in the recall. Yet another example here we have the folding line going through the governors race and of course now in if that vote is if that folding line is misinterpreted to be a misinterpreted to be a vote then it cancels those votes and we saw that 90 figure in earlier what was the time but these there were not very many ballots and not all folding lines are always interpreters of ballots. When we made the sanity check we found out that the probability of the strong folding to be misinterpreted into a vote is 0.44 and it is the same with the two different ways of counting so this is bow time our our hypothesis to be sane with the regardless of which way you are analyzed. Again this is just a close-up picture to show that the external connections were really removed and destroyed and there was no way of communicating with the memory card from outside world. Just to wrap up the last things we did a lot of things we added in addition we we looked about poll book recalls we looked absentee applications we made sure then that all ballots are genuine ballot printed ballots handmarked not printed printed we measured the thickness we did a complete software analysis of the e prompts using gizara an nsa open source a reverse engineering tool we looked through e prompts to find verify that there's no a unexplained data there with because that would be the only place where malware would be so we complete all these analyzes and we found no evidence of malware we found no evidence of a any kind of irregularities in the digital media and we found no irregularities in the paper ballots everything was authentic printed and handmarked. We livestreamed the entire event and we published a real-time huge amount of documents all along based on a request of products and so we tried to be as transparent as possible i have spent more time to be an observer and a monitor so i wanted to run this the way i would want to be i would be wanting to be observer as observer so we want to be as friendly as possible to the observers make sure everything is is transparent when we were doing the hand telling the spreadsheet was a google doc spreadsheet was a real-time accessible from the world everybody could see all the data entry as they go and in the video every single ballot was a uh with overhead camera uh photo so every single ballot of the 10 2006 ballot was shown in a live stream in a full hd so that everybody can follow that process i think that's wrap it up please ask questions in the discord and i hope this was explaining why in this case in on the very difficult circumstances windham elections were windham election was run uh very well of course when you have a close scrutiny we have a recommendations how to improve it but this was just a coincidence a conspiracy of coincidences very many things needed to happen in order to create this anomaly and now the anomaly is explained and there was no malicious intent or malicious activity thank you very much