 So the topic of our presentation today is titled Backdormulatory, Another Security Tales From Gaming. So the overview of the talk will go through some introductions as to who we are so you can see the context. What has happened since 2011 and why that date is important to me is it's the last time I was here speaking at DEF CON about eye gaming security issues. Give a brief historical overview of some attacks and how the focus has switched from the physical to now more on the logical side and then probably what's most interesting and what Evan is going to spend a lot of time talking about is the Eddie Tipton case, how we discovered how he rigged the lottery, what Evan did to reverse engineer of the code, et cetera. If time allows, we'll talk a little bit about the recent Russian slot attacks that you know they were in Wired Magazine recently and the report is coming out with another story in a couple weeks and then we'll try to wrap everything up. So first one page, one slide about who my company is and say pay for us, Senate International, look us up, always looking to talk to people. Who I am, my name is Gus Fritchie, I'm a CTO of Senate International and about five years ago after my presentation on eye gaming, I transitioned a significant portion of our practice into the gaming sector because I was more interested in doing the government work that we had been focusing on and now I'm pretty proud of what we've been able to do and the client base we have going across from the lotteries to travel casinos, corporate casinos, daily fantasy sports, et cetera. Hey, so I'm Evan, I work for Gus. I work with Gus. So I live in DC, I have a Linux distro, it's a small one, I do reverse engineering and stuff and my free time I hike and climb and I live in a van too, so here's my van. So Evan limited me to only two memes, so this is one of them. So this is not going to be a super technical talk but then again, it's Sunday, the last day of DEF CON, I'm sure we saw, there's been a ton of great, highly technical talks. It's more of a story talk but Evan definitely does get into the details with the code which I think those of you who are interested in that subject should find pretty interesting. So what has happened since 2011? First I just buy a show of hands, how many people saw my last one on eye gaming security? I was just curious who attended, so a handful of people. So the good news for all of us online poker players is we got paid back our money from the sites that got shut down for the most part, so that's obviously a good thing. But what we have seen happen is not a lot of movement in eye gaming. The green illustrates the states where eye gaming is legal, so really we only have three, Nevada, Delaware and New Jersey and the yellow is pending legislation. Why I find this slide interesting is even though it's a small number, it shows that the attack footprint is expanding and I think if I don't have a slide but if I show the slide of where it shows where land based casinos are legal, you would also see that expanding. We have tons of states even five years ago that have casinos that didn't use to have casinos. So we see the potential attack footprint expanding. Is there a question? Sorry, eye gaming, online poker, online slots, gambling, basically online. Caesar's and other companies they like to call it gaming because gambling has a bad sound to it. So this sector has not been immune to security incidents, unlike just like any other sector that you read about in the news. I'm not going to talk about any of these in any detail. I just picked a random sampling of some of the breaches that have been disclosed and of course as you know there's plenty of other breaches that occur that never get disclosed that we don't hear about. So this is obviously an area that needs to be secured and I think when it comes to the public's trust and the integrity of gaming is extremely important because without that you don't really have a solid business model. With the Las Vegas Sands I do find this interesting. This happened in December of 2014. If you didn't read about it, I'm not going to spend a lot of time but this occurred to arguably the world's most profitable gaming organization. They make millions and millions of dollars and at this point in time where the breach occurred they had a very small IT security staff. I don't have not done any work with Las Vegas Sands so I can't speak to their size of their staff now but I understand it's increased a lot and they put a lot more money into securing those environments. But the way this breach occurred was just to show you how easy it was. It was just a development server that was stood up on the perimeter that they didn't know about or they knew about but it was used as a foothold and they pivoted inside and they just ran havoc destroying data and it could have been a lot worse if they were more sophisticated and did not look to be destructive. So history. Let's get into some of the early attacks against slot machines. Not going to spend too much time so I want Evan to get to his piece which I think is more interesting but obviously in the earlier days we had physical attacks with fake coins yoyoing with a coin on a string pulling it back out attacks against bank note validators when those started getting installed. And then we have Tommy Carmichael here who came up with a couple of inventions or devices I should say that allowed him to commit fraud with a monkey paw which he was able to use to jam up there and release the coin hopper and then of course with the light wand which manipulated the sensors and allowed the money to come out of the machine without being won. So we saw these early attacks were more physical in nature and another physical attack that we saw that happened in 1980 and my partner in the lottery sector Herb Delhante wrote about this in his book and it was also made into a movie. I don't think a very good one but it was there. And they waited the balls so they waited the sixes and fours and this obviously was able to happen because you had collusion between multiple parties. Obviously people realized pretty quickly that it was a fraud and no one got paid out even the illegal books knew that prior to the lottery even admitting it. So we had these physical attacks against slot machines and other forms of gaming. And then we see a transition to attacks on a logical side and I think when you start seeing these stories you'll start seeing how they tie together to what Eddie Tipton did in the muscle hot lotto RNG fraud case. So some of you may be aware of Ron Harris but he worked for the Nevada Gaming Control Board and his responsibility was basically to perform audits of the gaming software and platforms and one of his responsibilities was to audit the eProms in the slot machines and make sure that they were correct. But what he did was instead of auditing them he reprogrammed them so that when a certain sequence was pushed on the slot machines or a certain number of coins were entered it would pay out a winnings and I spoke to Rex Carlson who was the director of Nevada Gaming Control Board and he said that they really don't know how much money he actually made from this because he didn't get caught initially because of this. He got caught because he turned his attention to Kino and this is where it sort of parallels what Mr. Tipton did in the hot lotto case is since he had access of course to the source code he found a flaw in the pseudo random number generator and then he wrote a program that allowed him to predict what the winning Kino number was. So he went to ballies in Atlantic City and with an accomplice they probably got away with it but they just had such poor planning as far as what they were going to do after they won. So this of course raised suspicion and the authorities performed investigation and they went back and looked at the other work that he was performing. So what we see is we see individuals with trust who were trusted to perform these reviews and had access to these devices and this will parallel what happened with Eddie. And one last example from a technical perspective and I talked about this in my last presentation but this was from the absolute poker ultimate bet super user scandal where the owner of the site I guess I should say alleged because it was never actually 100% proven but he had a convinced one of the programmers that he needed a backdoor into the program so he could see players whole cards because he thought his whole idea was that there was cheating and only he could figure out if they were really cheating but he proceeded to use his backdoor in the poker software to illegally win about 20 million dollars from players and this graph shows the individual one of the accounts and you can see that he's way off there as far as the norm from a winning percentage. So once again we have examples from Ron Harris with the Kino being able to access the source code and find weaknesses. Here we have an example placing another backdoor in a system allowing the, allowing cheat and fraud to occur in the poker. So now I'm going to turn it over to Evan where he's going to get into the current events and what I think is the most interesting topic of this talk. Alright cool so this is my first talk by the way. So in case you guys aren't familiar with like how the lottery works, there's individual state lotteries and each lottery manages itself but there's something called muscle which is like an organization that oversees all the state lotteries, well most of them. So back in like 2005, what was it? 2004, it doesn't matter. So like a while back this guy named Eddie Tipton got a job at muscle to write an RNG. So some of the state lotteries use like computer RNGs to draw the numbers. Others use balls out of hats and stuff like that. Well not actually hats but yeah. So he got a job to write an RNG and pretty much immediately he rigged it because obviously he wanted to make money and they weren't paying it very much. So here's a couple of other faces involved in this. Tommy Tipton is like his brother, he was involved in it, he helped cash out tickets. Robert Rhodes also helped cash out tickets as like a friend of his. And then Rob Sand is the attorney general in charge of like the case. Alright so in 2010 Eddie went into a gas station in Iowa and purchased a ticket for a lottery game. He waited a whole year to claim the ticket which was pretty suspicious. And then also he used a mysterious company incorporated in like Belize and he went through an attorney to try to cash out the ticket. So that was obviously extremely suspicious and lottery security caught wind of that and decided to do an investigation into it. They also refused to give them the money until like they completed the investigation and because they refused to give their identities which is like possibly illegal. So they withdrew their claim to the prize. I'm assuming they did not want to get caught. Let's see here. And apparently they also received a tip from somebody that Tipton was the person in the video. I actually think I know who that was. But yeah. So they started a full investigation which involved the FBI and stuff too. And they determined that it was him that purchased the ticket. And because he worked for the lottery he was banned from playing the lottery. They kind of used circumstantial evidence to convict him but they convicted him before they knew how he actually break the lottery. Or that he did for sure. He was sentenced to ten years of prison. He was out on bond or bail pending a pill for a good like year. He actually just played guilty pretty recently. This should be 2017. But yeah. And here's a quick timeline of events. So he was hired in 2003 at Muscle. And the first known case of fraud was in 2005. That could have been earlier cases though. Because he was breaking the dozen four, right? And what's interesting about the timeline here if you look at where it says that the Colorado lottery fraud was constant, Kansas lottery fraud. Do you see a pattern there? They're all on the same date. And that's what Evan will talk about when it gets into how his code actually worked and how it allowed him to predict what the winning numbers were going to be. Yeah. And also this is just known cases of fraud. There were probably more. Especially since we found a third date in the code itself which there were no known cases of fraud for. So let's see here. No, I just wanted to also, a couple of gaps just in the story. As Evan mentioned he was really, he was convicted initially of a charge on really circumstantial evidence. I mean they had the video footage from the store which when you think about it he really made a big mistake because one in the other cases of fraud he actually never bought the tickets himself. He had his, accomplishes his brother or Robert Rhodes buying the ticket but in this case he actually went and bought the ticket himself and perhaps that's because the prize value was so large that he just didn't feel comfortable having someone else buy the winning tickets. And he actually went to a convenience store that had audio and also video surveillance. So there was other stores that he could have gone to that video or the audio and it's really the audio was what convicted him because his voice is very distinctive and they were able to prove it that way. Of course he felt and his lawyers felt it was circumstantial evidence and they did appeal it and during that appeal process that's when we were able to get a hold of the Wisconsin Lottery R&G which Evan will talk about and what he did but you know how do you go about rigging a lottery and this is my second and last meme so I was only allowed to but obviously you become a lottery developer, write code and have your friends buy the winning numbers. It's that easy and luckily for the first time we have video surveillance of Eddie Tipton actually performing the programming. Now Evan will tell you how he really did it. Yeah so I do kind of went over this part but so basically what he did he worked for muscle so not all the lotteries, all the state lotteries used the R&G some of them used third party R&Gs and others used like machines that just like throw balls up in the air and they come back down and stuff like that which is what lotteries should do by the way should not use computers to generate numbers but it's a business for us. So while he worked there so they have like a supervised build process so he got co-past that process which made the numbers predictable in three days with other conditions as well and the binaries and source code were certified by a major testing lab but the way he did it wouldn't have mattered unless he really went in depth and checked the binary against the source code and stuff like that. He also had access to the computer images too which he could have used like a root kit or something simpler to change the numbers as well so even if he wasn't the one writing the code he could have break the lottery as well. In 2016 we were contracted to perform imaging of one of the R&Gs and we were actually given permission to review the images at some point and I was asked to try to figure out how he did it because at this point nobody had any idea how he actually did it. He didn't seem smart enough to use a root kit so when he was convicted they assumed that he used a root kit to change the numbers because they didn't find anything in the code. Apparently they didn't find the binary to be malicious in any way and I find it questionable how they analyzed that but yeah. So I decided that since he was the one writing the R&G itself I would just look at the binaries and compare them to the code so I didn't actually started like just going through the binaries and just reverse engineering them so there were a few binaries like the main executables and libraries and stuff like that so the most interesting binary was the one that actually contained the R&G itself so it was actually the first binary that it looks at so I started looking at and this is pretty much what I just said so it was coming through all the functions in the binary and one of them caught my eye pretty much immediately and at this point I knew that all the winnings all the known cases of fraud run certain dates so this one I saw it had a bunch of date functions right at the top so I was like that's probably it like I saw another reason for them to use dates so I started obviously reverse engineering it and I saw pretty much immediately that it was referencing those specific dates the two dates that we knew in a third date as well so I reverse engineering it and I figured out exactly what it was doing and how it was seeding the R&G and everything like that so also it was at the end of the binary which is pretty suspicious because it looks like somebody tacked on that function at the very end of a file and imagine we got the source code for the R&G and we saw that there was no that function wasn't in there there were 25 functions in the source code and 26 in the binary as you can see but yep so yep that's about it that's how he did it he just I can go into details on that real quick so basically each time a number was drawn this function was called and it would reseal the R&G with predictable values on certain days of the year it also had to be like a Wednesday I think it's supposed to be Sunday Wednesday or Sunday and here's just like the code of his function so he actually seeded the R&G up a bunch of values from the computer I'm not really sure why he did that he could have used anything really he's like a computer name and he added it up threw that into the seed along with the values from the game and everything he kind of made it more complicated than necessary which made him have to buy more tickets than necessary in some cases he bought multiple tickets because he wasn't sure what the values would be yeah and so here's why certification did not work so it was certified by one of the major testing labs and their testing process was to run the output of the R&G through a bunch of statistical tests which is great to ensure the results are unbiased but it doesn't really catch anybody reading it they performed an audit of the source code but the source code he compiled was obviously different and he was able to slip that super bus bill process pretty easily so that's questionable and here's how he could have done it better too he worked it on only three days a year which made it pretty easy to identify the winnings if you work it on every single day of the year it would be extremely difficult to identify the winnings because what they did like what the investigators did was they knew that he was using like certain dates so they just went through all those dates and looked at all the winnings and they looked at the ones that were most suspicious and they followed those leads so obviously if you work the lottery definitely do every single day of the year and also he could have made the method of working it more discreet he could have used rootkits and changed the numbers in memory and that would have been much more discreet than having it in the binary because what we do now is we check updates to the R&Gs and spread them to the source code like Bindiff or if it's in Java I've got a custom tool for that so we can catch updates like if a vendor tries to like pass a like a backdoor into like the update we can catch that pretty easily so how can this be prevented source code should undergo end-up third-party reviews I think supervised builds are important too as an additional layer but for updates if you have a binary and you're not concerned about the system image it's pretty easy to check those like Bindiff and make sure they're not malicious after reviewing the source code because what we do is we compile the source code and we compare it to the actual binary we get but there's another issue with the system image too where these guys are building the system image and that's not supervised at all so they could obviously slip something in and it's pretty difficult to supervise that entire process and you can't check the image you can't be certain of finding anything in there yep, there you are that's about it thanks Evan, I think you covered that very well when you think about it he actually had a pretty good idea because once he got this qvrng.dll file certified by the testing labs and once again, the testing labs supposedly and I know they do now because we do review the source code line by line and they were supposed to watch them compile the codes because they end up hashing those binaries so in some way we're not completely sure he was able to get the testing lab to certify that this code was valid and then he had it on all these boxes and that never changed he would make modifications perhaps to the executable but he never ever had to modify the dll so actually what was interesting when we reviewed some of the more recent muscle rngs and you have to remember, he started this process in 2003 this fraud case didn't happen until 2011, he wasn't convicted until 2015 that's a lot of time going on to make modifications and as Evan mentioned there are other commercial rngs in the lottery sector but some of the state lotteries their whole mission is to give back to education or whatever whatever is written into the law so they're very frugal with their money and they didn't want to pay 200,000 whatever for rng so they would pay muscle at a much lower cost so you had all these rngs being used in other states and when we reviewed some of the other ones in other states at least the ones we could get the images to it was interesting because the executable was not calling the qvrng.dll correct Evan? yeah I'm a newer one so somewhere along the line I think he got scared or nervous perhaps after his jackpot win and switched to code so people wouldn't even look at that qvrng.dll and maybe his plan was once things die down he could just call it again in the future so I think he left the binary on there when he updated I think he just left it on there to be honest there's been some speculation that he left it on there to switch back in the future but I think he just left it on there like when he updated he never deleted it yeah there's different theories for why he did that but in the end I think it's a case of where we see breakdowns in a couple different areas and probably the most obvious and basic is that separation of duties here you had somebody who was the director of security of muscle who was a lead programmer he had physical access to the boxes he had everything and to give muscle credit now they have completely revamped their entire process and their operational management and technical controls are a hundred times better than what was committed so we can probably definitely go into more detail and talk later about the tipped-in case outside if you guys have specific questions for Evan like get into the weeds of it let's just check on time here we got plenty of time I felt we rushed because I was nervous about the video issues in the beginning so the Russian slot machine hacking and this sort of ties everything back to the present John Harris back in the mid-1990s we have the eye gaming the ultimate bet absolute poker we have this tipped-in case and here we have this Russian slot machine hacking which is a very good article which I'm sure most of you read because it was widely tweeted at least in my Twitter feed and I give a link to it there and also at the protection conference he was also quoted in that article and World Game Protection Conference is a conference here in Vegas in the December timeframe that pretty much focuses on physical security but he's broadened it and he has some inside information as far as how this fraud occurred as well but basically just to rehash the story and tie it back in in 2009 when Putin made gambling illegal in Russia there was this flood of slot machines on the black market so some of them or a lot of them were sold to other casinos but some found their ways into the hands of the Russian mafia and they ended up reverse engine I'll just get to that piece of what they did but in 2011 some casinos in Europe were noticing some suspicious payouts and then in 2014 in Missouri they noticed really unusual and high payouts on some particular slot machines so they started investigating this and this is where you have compensated controls and other pieces actually worked to end up detecting the fraud they had to go back to the physical surveillance cameras and tie this all together but these individuals they came back and they were later arrested in Missouri and were arrested in Singapore last year but what did they do? they reverse engineered similar to what Evan did with the code but they reverse engineered the software the binary on the slot machines and they found a weakness in the PRNG so then what the guys would do is they would take a video of a certain number of spins and that data was transmitted back to their comrades back in Russia where they were using very high powerful computers to process this data send it back to them they had an app on their phone which would then vibrate a couple seconds before they were supposed to hit the button and it wasn't always successful but it did result in a much higher payout so this is an example where there was a weakness in the computer programming in the RNG this wasn't built on purpose it was just a mistake that the developers made and this vulnerability impacted a particular vendor Aristocrat I believe on their older versions but the claim now is and I think not to steal the reporter's thunder but they claim to have working code for more modern slot machines and they're threatening to release this code to the general public they were trying to blackmail Aristocrat I believe but so far that hasn't worked but it will be interesting to read the story when that comes out so we talked about some issues here in a lot of different sectors what does an operator do to better protect themselves I think it's a lot no different than any other industry from healthcare to banking we get caught in this trap of compliance we gotta be compliant we gotta be compliant we waste a lot of money on paper doing that but we have to understand and I know I'm preaching to the choir that compliance is not equal security and when I first got into working with casinos and operators I went in there thinking wow these guys are going to be super secure you see all the movies Oceans 11 they got all the surveillance cameras and they are very good from a physical perspective for the most part but quite honestly they really are lacking in the technical security controls but that's really no different than any other sector now to improve themselves they're being proactive I know other firms are working with them so they really are taking it to heart to improve their security and protect their players data more money needs to be spent on information security and also the operators need to start asking the game manufacturers what how is your system secure what controls do you have in place and I'll give you an example we were working with a casino organization and we were doing a security assessment just a basic vulnerability assessment and realized that from the corporate network we could actually get to the slot machines and touch the interface card in the slot machines and when I brought that up to the director of IT's attention because I was curious I was like it didn't seem right he was like well let me fire off a note to I'm not going to say the game manufacturer's name and they came back oh no the only thing these need to talk to is a player tracking database it should be firewalled off so you have operators who are trusting their vendors to do their installations and sometimes not doing it correctly and then also questioning what other controls do you have in place in this process with this code obviously these huge organizations they're not like muscle they don't have one guy writing the code they have hundreds but how do we trust that code is secure especially when we're dealing with some a lot of the gaming operators do a large amount of offshore development what are the controls in place luckily we do have regulations with security components when New Jersey the second state to legalize eye gaming they came out with very very comprehensive security controls at least more comprehensive than what was in the unregulated industries if you recall at that time there was really no security controls in place and to date New Jersey is very proud of the fact that they haven't had a license or a breach or their controls are working effectively Maryland for example their gaming commission requires their land based operators to undergo an annual security assessment once again I'm not saying it's perfect but to step in the right direction to have these security compliance regulations in place and then of course we have a regular regulatory compliance standards PCI etc but as last bullet says there is no operator to determine the level of security that is implemented there's not strict guidelines even when it comes to New Jersey the testing is sort of left up to us working with the operator to determine how deep we dive and as you can guess probably a lot of that comes to funding and budget for those operators so conclusion while regulated eye gaming has added additional controls there's still room for improvement and this goes to brick and mortar casinos as well and also to the lottery sector you know in my opinion as I mentioned I think one of the key risks is in the code and that's what concerns me most you know and I think this applies to all forms of online gambling and also brick and mortar I mean it would be in my opinion pretty easy for something to be added to one of these eye gaming sites that could allow fraud to be committed it's happened in the past I think it's going to happen again and it's very important for the regulators and operators to work together and you know the last point is as you saw on the slide with the map you know while it's still small with eye gaming it's growing daily fantasy sports is here as you can't watch a football game without seeing draft kings or fan dual advertising you know it's become more widely accepted sports betting looks like it could become legal in other states besides Nevada no longer is gaming focused on Nevada and Atlantic City it's across the entire United States and as it becomes you know as it expands that tack footprint expands and the opportunity expands for crime and fraud to be committed so that's what we have I hope you enjoyed the presentation I'm sorry for some of the technical delays and we rushed a little bit in the beginning when we probably didn't have to but we would love to speak with any of you outside if you have any specific questions especially for Evan I know he went through those code slides pretty quickly in a little bit more detail as far as how that code worked the various functions etc etc but thank you for attending and enjoy the last day of DEF CON