 We are at the Defcon 30 CTF run by Nautilus and the teams have been doing crazy things with a few issues along the way, but it is, that's this year, because you have the teams and then you have this whole live aspect where teams come on over here. Can you just tell me, high level, how did you design this CTF? Like what are the teams actually playing? Well let's step back a second and say this is your first year doing the Defcon CTF. So many people on Nautilus have designed CTFs before or in this particular CTF. What is Nautilus doing this year for the CTF? That's either different or what are your takes on it this year? So what we're doing here is a relatively traditional CTF. The CTF has evolved over the years. The general structure has stayed the same. So there's N teams, this year there's 16 teams. Each team gets access to their own server. All the servers are identical. Those servers are running executables on them that provide services. Those services are vulnerable. There's stuff that we have created. None of the teams have seen these things before. And they reverse engineer those services, find vulnerabilities, write exploits and throw the exploits to other teams to get text tokens off of the game boxes, which are called flags, and then they submit them to our scoreboard. Sure. And we can get a little more technical or use some internal jargon. This is a typical attack and defense CTF. So the core of the CTF is a typical attack and defense. So they have servers that they're defending against other teams and servers that they're attacking that the other teams have. So this is where I like to get into it, which is the actual challenges, if you will, or the actual exploitables on the machines are new every year. And everyone has a different take on what should or should not go in. And you've already burned a bunch of challenges. Are there things that you're particularly proud of this year that you may have designed and got put on? So we had one challenge this year called Web 4 Factory from Itzin, one of the Nautilus members who's new to organizing CTFs or at least new to organizing DEF CON CTF. And I think it was deeply, deeply hilarious, as the Web 4 Factory implies. It's a Java applet, or a Java program, not a Java applet. It's a Java server that has deeply nested classes, like teams entered the applet, or the Java... I'm too old, I'm thinking Java applets all the time, but they went to institute.nautilus.web4factory.web4factory. All the way down this horrible Java chain of classes. That was one that we ran through yesterday. And I thought it was funny enough that I made a custom sticker for it in kind of secret without telling anybody about it. That's fantastic. I heard you like classes with your classes. Oh yeah, yeah. So we put a class factory in your class factory, seeking factory classes while you factory classes. Oh my God, you've been practicing that a little bit too much, haven't you? So we also heard that you guys... Well, I mean, we've been having some power issues and things have been going on and off throughout the conference. Have you guys got hit by any of the infrastructure issues? Honestly, DevCon has provided awesome power, awesome network. Unfortunately, most of the hardware issues that we've had were our own. We didn't have some of the servers kind of correctly configured before you brought them here. And so we did run into an issue yesterday where we had to retire a challenge early in order to basically keep the load on the boxes down. And this led to us spending all night actually reconfiguring all of the hardware so we could run today's challenges more smoothly, which did fortunately work. So would you say this year is going smoother or less smooth than a typical CTF? Because that sounds... I've run my own and that sounds like something that pops up in most cases. So both Vito and myself have actually run the DevCon CTF before. Vito's pretty old hat at it. I ran this last in 2017 with the legitimate business syndicate. And relative to the year that I was playing or that I was hosting, this is going a little bit more rough. It's our first year. Whereas I was helping the legitimate business syndicate in their last year. So they had had at that point four previous years to kind of perfect everything. So the first year is usually kind of rough. You're building everything from scratch. You're kind of getting all your tech together. This is the first time you've really had to have teams interact with it. But I don't know if your experience is... My experiences were broadly similar to that. It felt like as legit BS, legitimate business syndicate, we always had to do some kind of clean up after the first day. Something would unexpectedly break or scoring would break, checking for service uptime would break at some point. And there's always a little bit of fix ups. I think 2016 was a particularly rough year for us because we went in... That was the Cyber Grand Challenge year where we supported the winner from the DARPA competition playing our game as an autonomous computer. And that was a different enough game from our previous years that we had a ton of problems. That was, I think, the year we didn't start our finals on time. And there were just lots and lots of little issues that year. I mean, you keep talking about the problems. They are interesting. The thing which is fascinating to me is that you have legitimately the best hackers in the world in this room spend their entire year training kind of for this event. They pre-qualify, they get in here. And you have to design an event in such a weird way. These are not designing secure software. That's what's so weird about this. You're designing a maze, an insecure software, enough where players can solve it in a finite amount of time. I am curious, though. When you design a solution, I think of the movie Inception where he says, give me a maze that you have to solve and it takes me exactly one minute to solve. How has it been when you've designed challenges? Because I know you have this whole live aspect which throws a loop. You're designing a challenge where you go, this has to be solved. I expect this to be solved in one hour. How challenging, how difficult some of these challenges are when you're dealing with legitimately the best hackers arguably in the world for binary exploitation, things like that. So for us, I feel like there's a large aspect to that for people on our team is experience. And there's a lot of CTF playing experience on our team. So for some of the challenges, if somebody's not sure about how difficult it's going to be, we pass it to a legitimately great CTF player that's playing with us. Shellfish, for the last few years, has been Fish who has thankfully joined Dottles Institute. And Fish is one of the best reversers in the world. Fish is working hard every day in his normal job on better software for analyzing software. So he's an incredible play tester and he's not the only incredible play tester on our team. How do you balance? There's this whole key. I love playing CTF's play. The end-second, there's so many great ones out there. And when I've played them, there's this weird balance that happens between like security through obscurity which leads to mostly frustration on the team. Because it leads you down these rabbit holes. And the challenge starts becoming how do I overcome all these red herons to find the tone of truth? So full disclosure, we used to help run Defcon Darknet which is a more introductory CTF. It gets people into the con, gets them around the con, gets them towards these more advanced things. And the question for us is always that red herring question. Do you worry so much at this level about the red herring problem where someone picks up on the wrong thing or is that their fault? So in terms of our game, like the larger attack defense CTF, we don't really worry about that at all. The best teams are not going to get tripped up by those kind of things. They're not going to get lost down some sort of unexploitable rabbit hole. They've got enough experience to kind of be able to feel and hone in on the right stuff. For the live CTF event, that is a consideration because you want the challenge to be solvable within an hour and if both competitors are stuck on some, you know, rabbit chase down something that's not exploitable, that's not a good challenge design for that format. No, I was just telling him I might go a little closer. Yeah, well we had a short discussion with CypherTax about the live CTF and of course he's actually helping, he's one of the commentators actually running. How has that been going? One of these last rounds went a really long time. In part because you throw a problem at them and then it takes them an hour and then their brain gets fried and then you go to your backup challenge which is supposed to be really, really easy but by that point they're so frustrated that they can't even do that one. This is a thing that happens. It happens to seasoned professionals. Has that been happening a lot or has that just been popping up here and there? So specifically in the context of live CTF, I haven't seen it happen too much. I believe that's actually the only challenge of the day that went to like a sudden death. They couldn't solve the original challenge in 45 minutes so they were provided an easier one and then had to do that one. I was worried actually, so I had one of the challenges in live CTF this year. I was worried that my challenge we were going to have to switch to like a backup challenge to make sure the round went through and I gotta be honest, the person who solved it, what his exploit was when he was doing it, he was so fast on the keyboard like it was like watching a Starcraft player or something but with like code and I thought that we were going to have to give out one of these easier challenges kind of thing and then he got it. It was great. So this brings up the question which is what have been the pinnacle moments that have happened so far in the CTF? My favorite so far has been especially live CTF related but every time there's applause on the room or somebody jumping up or somebody cheering that is something I always really enjoy to see in a CTF. I haven't been super paying attention to see if that's been happening with first blood on any of the regular challenges and the regular part of the CTF but it's always cool to see when somebody gets that breakthrough and they get excited about it. I love it. So has that been happening primarily with the attack and defense in the main game because it's attack and defend and you have to have a sustained attack it's not like one little rush of yes I scored a point you're scoring points continuously there's not usually as much animation from teams it's kind of like yes okay we scored on that one now let's get another vulnerability and our challenges for the main game are designed to have more than one vulnerability inside of them so the reason we do that is to have more vulnerability but still be vulnerable in another way and that keeps them looking at this stuff for hours and hours and hours trying to figure out how do we figure this out. One of my favorite things from our game is two of our top teams earlier today were in kind of this interesting both teams were leading they both had exploits on a service they both had patched that service basically dossing each other's service like finding ways to make it do things that weren't intended and then we have an SLA checker that's like hey is your service up and responding to things correctly they were finding ways to basically get it to be in weird states it was interesting seeing them go back and forth making each other lose points and getting points off of other teams for a sustained period of time it was definitely these two teams going at it and that's really cool to see in our game. So I remember during the cyber grand challenge year that it was announced at the end that a number of the competitors had found a bunch of what would amount to zero days or just unintended bugs in the challenges that were discovered has that happened at this CTF yet where there was some kind of exploit that was just not expected? Yes that has happened there was a challenge we released earlier today it was called Mando there was a heat bug that we didn't intend to be in the challenge and a team found it and exploited it it was great that's exactly what we want Do you ever see escalation with the teams and what I mean by that is I imagine like myself in my day job even where I'm bashing my face across something over and over and I'm getting nowhere and then you start going all right jello against the wall and I try this and I try this and then they escalate almost to the point where the rules get a little wishy-washy in the fact where like they're doing stuff they're not really supposed to Has any of that happened? Yeah so we have had some team it's hard running a hacking competition right because there's it's really hard to enforce rules when the whole point of hacking is that you're getting around like rules basically in a program we have had a couple of teams that have so I mentioned the Dawson earlier if you're specifically targeting like a patch that another team has made that's kind of cool but if you're running like a fork bomb on a bunch of servers that's going to like take down the whole game that's not so cool so we kind of have some loose you know spirit of the game kind of rules that are kind of like hey make sure people can still play you know make sure that we're not like attacking our infrastructure we put a lot of time and effort and work into this it's really frustrating to have our stuff come crashing down just because you know somebody decided to run a fork bomb or something with hacking challenges you're designing them to be broken and then you have kind of the semblance of rules because I remember when you're like oh goodness as well back where the scoreboard gets hacked into and things like that with this challenge how are the teams really dealing with we'll see the size limitation so like with this you have open are you finding a mass difference in the sizes of teams we aren't actually completely sure about how big some of these teams are one of the things I was most worried about and I promise this is an actual answer to your question kind of in a roundabout way I was really really worried that the you know 16 tables in this room we're going to have one person holding a VPN open and responding to like events in the room for everybody else elsewhere instead what we've seen is you know most of the seats at these tables are full like all eight of them every time I look around it's it's a busy room I really really love to see that we don't necessarily know how many players are you know up in hotel rooms or back at home but I can't imagine that it would be wildly out of place to assume anywhere between you know eight people or you know 40 maybe one of the problems with bigger teams is you know kind of endemic to all software engineering or all engineering in general where the bigger your team the more communication that has to get done back and forth between groups and since we haven't had you know more than about half a dozen challenges open at any time there's dubious value to having you know 80 people working on you know one thing that brings up a question though of the teams that are doing well what do you think they're doing that the other teams aren't doing are they just getting luckier with the challenges I mean they're not luck but you get the idea where you're finding those problems a little quicker or do you think it's a communication thing what do you think separating the teams something that I find is really important when you're playing a CTF is to get into this mindset where you're failing very quickly so you try as many things as possible in a short a period of time and what I have found as a player and also watching as an organizer watching all of these players is that the best teams the best players tend to be people who are able to very quickly iterate through a number of you know thoughts or or you know like different scripts or things to try things out to see what they think is going to work and so I think that a lot of the teams who are doing doing better are just able to to make these iterations to adapt much more fast than the other teams as an ancillary that somebody you know in this conversation maybe was watching me write a sequel query to analyze some scoring event earlier today and I definitely was in the zone there for you know a minute or two like you know type a clause hit enter see if it works another clause hit enter and it felt really really good to just blast it out like that step by step getting to a destination so I want to get into a little bit of it is a game right and and I always like the game design aspect of it and and this is really hard because I'm I'm not a PvP gamer that's why I typically don't attend attack and defense CTFs he is a PvP game I'm no one's perfect but I like PvE so like I like a more Jeopardy style and I like a more learning CTF that's that's for me but how much thought do you give into how players are going to interact with each other when you're developing the challenges because I have to imagine that's one of the biggest as an attack and defense it's all about how teams interact with each other like what how do you go through the process of designing some of those challenges I know that's a big question but is there anything in particular that that you want to talk about yeah it it is super broad and I think the game design aspect is something that CTF organizers don't always consider and for the attack defense CTF which there really aren't very many not only is this the biggest and best one but there really aren't too many alternatives most CTFs are kind of a Jeopardy board of challenges and you just kind of pick one and do it and it's like kind of like a speed run you're just trying to do it as fast as you can faster than other teams that kind of thing so for our game there's like you said a lot of interactions between players so we have to for example model scoring very carefully so you know we don't want one team that's solving one challenge to like run away with the game because we make it too many points versus another challenge so we have to be very careful about how many points is a flag worth how many flags can be redeemed how does that look over time so modeling all of those things are really important and then also just kind of trying to encourage things with points so we have a king of the hill challenge and one of the design considerations in that is we want people to try to win in the little king of the hill rounds so an earlier iteration of that design we said well a win is worth two points and a tie is worth one and a loss is worth zero but then winning is not so much better than tying so it might be better for you to just try and like tie with everybody spend more of your time on other challenges things like that so we adjusted it so that a win was worth three points to give the winner a much bigger edge on things similarly I mentioned this like concept of SLA where you have to keep your services up the SLA penalty is much larger than the amount of points you score on somebody to ensure that teams keep their services up because we want them to be attacking each other and we want the main method of defense to be patching the service like fixing the bug rather than just taking the service down so that you don't lose any points by being scored on well so this was this was actually huge because in prior years the CGC year for example there was an after analysis of what was the most effective way for your thing to game the game if you will and if I remember correctly they came up with oh was it yours that's my blog post yeah yeah if I remember correctly it was like if you had just left your services up the whole time you would have finished third right and so that's that's fascinating but you don't necessarily know the rules of the game while you're playing it so it's really hard to game it and not just basically roll the dice in fact Alex has a story about how he won his black badge oh is this the one where I was preparing you're saying no no no yeah the pairing one oh it's because it's very similar in a different sense yeah so in DEF CON 22 that was the year I won my black badge is at the dark that there was tons of different points you could get for going all over CON building your badge and things like that and when I actually started looking at the math I realized that pairing with badges was it was weighted a lot it was weighted a lot more than I felt it probably should so I ended up sitting in the solder well it's a hardware hacking village because the soldering skills village didn't exist at the time but just sitting there and people assemble their badges and meeting them and I had to meet a ton of people which was great but then I would sit down and work on challenges the moment someone assembled their badge like hey let me go test your badge and then beep sync it on and test the badge it was actually really funny because the badge was only capable of holding 50 and what ended up happening was I didn't realize that so I got like 70 badges to realize we were starting to overwrite them so it was it was a lot of fun looking at the point engine so the reason that almost feels like a situation where it was working as intended so you know you got to meet a lot of people you got to help a lot of people with soldering at tour camp a few weeks ago there was a event called bureaucracy and I kind of realized early on it was like oh this is a way to get like people who are new at this camp a fun game to play where they have to go around the camp collect stamps from different groups of people learn how to like pull style a phone by wrapping on the hook switch and get them to meet new people for you know one of those like awful like soccer practice conical paper cups of you know Reneer Lager so the question I had though in this context is because this is a common problem once you game the game and you figure out what's the most effective way for me to get the most points in the most rapid way the speed run issue right do you guys do anything to combat that in the sense that one of the things dark net did was we didn't publish what things were worth and we didn't necessarily publish your scores until much later on we give you a timeframe where you could see that have you done anything to combat the gaming the game or is that just part of the game at this level I think my philosophy on it is if there is a way to game the game that is not in the spirit of the game then the game design is fundamentally broken and needs to be addressed and I know in past years like the CTF here has like hidden scores and things like that we're still toying with hiding some of the scores tomorrow to you know increase anticipation kind of thing but I think my philosophy is there's no sport there's no other competitive endeavor I can think of where oh we don't know who's winning in the fourth quarter because we turned the scores off or something right like you know what the score is that's knowing the score does not lessen the anticipation of you know seeing teams perform and and be scoring and and like you know win the game so let me ask you a wrap up question unless you know I was just going to say as I know just speaking personally I know that was a frustrating aspect to me and the scores go down because then just also encourages teams I mean it more than jeopardy style where encourages teams you're okay we're going to hold on to points where and here it's a little harder but I know of that can add some frustration I mean I'm maybe in boxing they they don't show you who wins so to wrap up this will not go live until tomorrow morning when is that when is the next round start the next round starts at 10am we're going to run to 2pm and then we have to end live CTF might go a little bit longer but we need to get our scores into the you know black badge thing like tell people yeah to dark tangent so we can do all the pomp and circumstance of you know awarding the winners and things like that so do you have any surprises for the teams on the final day or is it going to be more of what they've been seeing already I mean it wouldn't be a surprise would it I think that's the best answer they can get it's a good point thank you so much I am so excited to see some of the write-ups that come from this thank you for watching and as always hack on