 Hello, everybody, my name is John Hammond in this video. I'm really excited to share with you guys I had a conversation and chat with some of the developers and organizers for the all-army cyber stakes or ACI CTF that went on a few weeks ago And I think it had a lot of incredible and awesome insight that I wanted to share with you guys all about okay Not only playing capture the flag, but hosting a capture the flag putting on an event maintaining everything and all of that Goodness, so on the call with me. I have Tillerie one of the lead organizers over at Grimm who put on the all-army cyber stakes capture the flag competition and Alex from Storm CTF who also helped out in creating the event and making some challenges So without further ado, let's roll the call and conversation. So thank you guys Hope to see you at the end of the video and take care Well, cool. Thank you for doing this. I appreciate you got you being able to come in and hang out Yeah, absolutely. I Been looking forward to being able to chat with you about it. So glad to be here. I mean, I feel like it's totally casual We're just hanging out. It's it's I'm not trying to be anything formal I hate using the word interview because I don't feel like that's right But some good questions on everything about cyber stakes, I'll probably if you're cool with it have this recorded just kind of on my side and I'll go ahead and edit it afterwards Maybe add just a silly intro as to like who we are what we're talking about etc And if you're all comfortable with it, I can probably upload it tomorrow in place of what would have been the cyber stakes video for that day Okay, yeah, that works for me. Cool Should we just roll on a hedge? Should I just like badger you guys with questions? Are you all? I Again open to anyone whoever's willing or wants to chime in Can you give me a background on kind of what the all army cyber stakes is like your point of view? How do you define it? Sure. Yeah, I mean, I'll talk a little bit about that Obviously, it's name kind of describes it a little bit, but it's it's the premier kind of largest most engaged DoD CTF training kind of event that really exists There are a lot of these types of things some of them are attack defense Some of them are just kind of large CTF kind of concepts, but none of them really hit the level that the ACS does So I think of it as less of a CTF although it very much is and there's a leaderboard and more of an opportunity for training Which is why I'm so excited to work on it This is this is our second year at Graham working on this and doing development because ultimately this is less about Being a competition and more about getting all that information into people's heads and guiding them along as we go And obviously we work really hard to make it like fun and a competition But I run a training team because I want to do training and so that's that's why I built this out the way I do Awesome, that's awesome. How long did the competition go for it was only was it a week? Is that right? I? Runs for 10 days and the idea is that we want it to be able to go over at least two weekends So it starts on a Friday goes over one weekend keeps going for the entire next week and then over the last two days To give everyone a chance to compete whether they're doing this as their primary, you know deployment or not So a lot of people are in Development or training in some some way whether those are folks coming over West Point or you know elsewhere in the DoD or other Other branches and some of them can dedicate time just to doing this But most of the people that are doing it are going to be doing it in the afternoons in the evenings on the weekends in Between their actual deployment, so we want to be able to make give as much time as we can the people who can't just do this as their primary task Yeah, that's awesome. I Honestly got started on it on the weekend because I think it opened up on that Friday and on the weekend And then like once Monday and the rest of the week kicked in is like well I got to get back to my day job, so I couldn't submit as many times I wanted to is that normal for you guys is that always what you do I mean, I know other there are obviously tons of capture flag events and competitions out there Is that kind of normal for you to being able to have a competition that runs so long or is that always the goal? This I mean this it's definitely anomalous. This is by far the longest competition. I've ever been a part of I think it's really smart for For Army Cyber Institute to set it up that the way that they did given their audience But most of the CTFs that that grim does development for that I do develop for that I help run Run for a weekend three days maybe at the longest some of them as short as four hours So it's a huge struggle Actually chatting with major eggs don't some of the other folks at that are we somewhere to do during the events? You can you can definitely tell by the end of it that it's a long event and When you're giving, you know live help to competitors potentially eight ten hours a day plus doing infrastructure work It's tough. I don't I don't know how they put as much time as they did and organizing it doing all the infrastructure Me just trying to field questions from competitors for that amount of time during the live event was enough to kind of melt my brain a little bit but they also joked that By doing it for ten days straight their brand mitts melts enough for them to forget how much it is So by next year, they're really excited to do it again for the full ten days So it's definitely by far the longest. I know Alex and I have originally kind of connected Through work that he does running storm CTF and their events aren't nearly as long, right? What kind of are you looking at there Alex? So I know this year we're running one that's going to be two days long and I'm mentally preparing myself for that one as it is Yeah, so so many of you know ten days and especially with the turnout that we had this year, which was just ridiculous It's tough, but it's a lot of fun, too one of my I could Dovetail into a kind of a story But like one of my favorite things that I did during this ten-day experience was I was working with one of the competitors who was completely new to one of the categories binary Exploitation I did a lot of web work had done some crypto work, but didn't really know reversing or binary exploitation at all And it turns out that this year Army cyber and the DOD decided to open up the competition not just to folks in the DOD in the US But also Ministry of Defense folks from the UK So I had been chatting with this person trying to help them through the RE 101 Just a very basic reverse engineering challenge and I got to the point where we were looking at Looking at the the binary in obstump and at this point like yes, I could type out Here's what this column means and here's what this does But it was gonna be so much more work to type it out than to just have a conversation So luckily we were doing all of our work through Slack to actually like interact with competition in the event so just throw up a quick slack call and like over the course of the next 10 minutes We got to explain like here's how x86 assembly works Here's how pointers work like this is the obstump output and Managed to get him started and he was able to go through and solve the rest of that challenge from from having that so It's definitely definitely cool to see that and that wouldn't happen in a shorter competition as well or kind of frame. That's awesome No, yeah, absolutely. I mean both of you you will play capture the flag like you yourself, right Alex tiller Yeah, I see you I see grim all the time at conferences. I remember I'm still trying to get into some howdy neighbor over at DevCon And storm CTF over on elixide you're we're cool to I guess tiller. We mentioned. Hey storm CTF is one awesome thing that you do, right? Yeah, yeah, we also build some CTF's as well just We're more of a Deal with the non not the profits and things like that we deal with a lot of the local chapters and things So that's super cool I'm just excited to just kind of have the conversation with both of you guys and both of you all I mean not only as a CTF player, right? But I like to try and build some things myself when I can so what did each of you work on? Specifically for cyber stakes. What were you in charge of what were you trying to produce? What were you helping on in making? So I just have them first. Yes. Yeah, sure. So I was actually a developer this year Previous year. I was actually a tester. I did a little bit of testing, but mainly I was in charge of creating a few web challenges I'm not really the binary exploitation and or reversing guy just yet So web is kind of my specialty when it comes to building and so I ended up building two challenges this year and that was Blame it on the temp and the sequel always sucks. So I Know a lot of people struggle to blame it on the time, right? Yeah, those were two challenges that They really got a lot of depth of thinking from competitors I fielded more challenge more more challenge questions probably from From that sequel injection challenge than any other sequel challenge I've ever run and and that was because like we the way we designed it was not a simple sequel injection You win you actually had to take a couple of extra steps to get in there. So Those were those were both really good challenges I'm really really glad we had Alex to work on those because they definitely went beyond kind of that basics So I'll jump in now. I guess and I'll answer this question My my role was kind of split between a bunch of different things at its core I I was the technical director for the project on the Grimside So a lot of a lot of that was communicating with the technical folks over at army cyber Institute Working with you know, all the developers we had and the testers to make sure that everything was kind of going smoothly the day-to-day operations I had actually One of one of our really talented employees kind of handled and that was straight if you know her Was was handling most of that day-to-day and in the kind of operations there the bigger picture and the overall design of all the challenges It's kind of where I came in so My job was let's let's take What we want to teach and what we kind of have a speed back from last year since Grim also did the development on ACS 3 and kind of build out what what can we do that's going to be better this year So rather than starting with a bunch of challenge concepts like we did last year I kind of made a list of these are the learning objectives that we know we want to hit We want to make sure people learn these things and in these ways and then we took those learning objectives We took those and turns it you know my job then was let's make those into something that is a solid Concept for a bunch of challenges and then I got to make pun based names for all of those So I like to think my biggest contribution is that every single challenge had a pun for a name this year So that was my big thing, but um Yeah, the overall technical direction for the project was my goal Awesome, very cool How many people were working on the team like how many did you have overall? Can I ask is that okay? How many hands or yeah, how did you do the work? Yeah, let me let me take a look at the spreadsheet that we used to track our challenges and do a quick count. That's awesome Looks like we had I'm always just like a one-man show with a couple of friends We had a total of 14 developers 15 developers Across the challenges so everyone took on at least at least two challenges We developed 50 this year for cyber snakes So there were 73 I think challenges overall in the competition 50 of those were developed by us The others were developed as either the tutorial challenges or a little bit of introduction ramp into difficulty for each of the categories and those were developed over by our cyber institute to kind of fill out what we did and part of that is because Last year when we did this our challenges were just too hard Some of them were just ridiculous to the point that we couldn't even deploy them and then you know But the things that we called 500 point challenges last year on the scale that we use this year probably would have been 750 And and only a handful of them got solved so we pulled that back a little bit focused more on the learning objectives like I said and and Tried to have a smoother kind of learning ramp, which I think we did really well with absolutely. I think so too and One of one of the kind of the joint decision And our reserve is it made was you know the really low hanging stuff that every CTF needs But that is the same every time we weren't going to be doing the development on that, right? They were going to take that on kind of develop those which we get kind of that 23 intro ramp up challenges And then we would start coming in at like for example in binary exploitation. We have a We have a jump ESP smack stash, right? But it's not just like your basic jump ESP stack smash. There's a little bit more involved in there That was that your cup overflow with is that right? Yes. Nice. Yep So that was that was that was our version on our development side of re 101 And so anything earlier than that in difficulty was developed by army cyber and and again That was because they you know We had a conversation and made the conscious decision that if we're developing 50 challenges We want to put you know our expertise into the ones that are going to be more technical and more technically demanding And also try and hit those learning objectives in ways that are new I didn't want to make anything that was like an off-the-shelf boring challenge. So even the things that are You know, this is a snack smash. It's a challenge. You've seen a million times How can we turn that on its head just enough to keep you know? Not just the people who are new learning but also the people who aren't interested in and progressing cool Very cool. I definitely felt that like even with some of it So I I like web personally, so I spent some time on that but the stuff with Even the just the cross-site scripting one where okay This is normally a classic cookie catcher But there's so much other evasions and stuff I need to do to get past the filter and stuff with the the xx e the XML to be one. There was just so many cool It's like a classic vulnerability But then it felt like there were a lot of other gimmicks or gotchas where you have to be really clever and like your attack in your payload Yeah, that was exactly the goal, you know If you got the concepts if you actually understood the concepts you'd be able to make that pivot But if you were using off-the-shelf tools or you know We're just kind of trying to throw brute force at it You wouldn't necessarily get there and that was kind of that was kind of the goal The later the challenge is as well the more of that there is so you mentioned the xx e challenge That one was specifically designed so that you can't do this unless you actually understand what you're doing cool When did you guys get started on the whole process? When did you start development like how much time did you feel like you really had to pour into all this? Both of your developer side lead side, etc So the brainstorming for challenges for this year started when last year ended You know as soon as a CS3 was over. We were thinking. Okay. What do we want to do next year? How do we want to approach this? You know, we had a close-out meeting with with Then captain now major regsdale about like here's what here's what was good What was bad? How can we you know try to hit these goals a little bit better next year? So right after that I started brainstorming a bunch of folks on our team started brainstorming How are we going to do this next year as far as the actual development time goes? It was really accelerated and that was the case last year and this year Our goal was to have the initial like list of challenges done within two weeks of our start date and Then have six weeks total for development and that was it So we developed all 50 challenges in a total of eight weeks if you count some a little bit of development We did in the first kind of two-week period, but the entire thing is developed really rapidly. So That puts a lot of pressure especially on our testers I know that Alex had a lot of trouble last year and then this year actually made a full testing infrastructure So pick a CTF in the way that it deployed is kind of set up two different ways There's the you deploy it locally and it's Docker containers and you just kind of do it And then there's the full infrastructure, which is designed to be run on AWS with Terraform and Ansible scripts And Last year all of the testing was done in these kind of Docker container, you know test environments this year Alex and some of the other testers, but especially, you know, the ones that we're working directly with him set up a full local install of that, right Alex Yeah, so I ended up just building out an entire like local platform actually on this box right back here Just completely built it out locally using vagrant and Ansible Just like you were going to deploy to production so that any Dependencies or anything like that would immediately be caught Especially for the challenges that I was testing and that some of my guys were testing and the one especially the ones that I was developing as well I just didn't want to have that issue Awesome. I'm sorry go ahead. Okay. I was gonna say and that's especially important because last year We actually found a few bugs in the Pico CTF platform Such that if you're deploying it locally in the test environment the permissions like don't get set correctly on files So we had things that tested perfectly fine until you actually deployed it to a production server And then suddenly you can't get to the pieces you need And that's that's bad if for example your web challenge relies on you being able to pull a certain file down or you have to know What version of of libc is running to be able to exploit, you know Binary exploitation challenge and if you don't have permission set right in only file system. It turns out that's really hard to do Oh, yeah So that I mean those things became a lot more obvious once we started testing in kind of a production setup Rather than having to find them in the actual live event, which is what happened last year So I'm hearing Docker. I'm hearing vagrant. I'm hearing ansible. I heard a little bit of terraform, right? And I know Pico CTF was the framework and platform of choice So like I'm always just a poor man rolling with CTF D whenever I tried to tinker with Pico CTF I am kind of finding like oh man. This is kind of weird It's not working the way I would expect so what made you guys choose Pico CTF over any of the other frameworks or what are some of the pros and cons? Why you would want to go with that one versus another or was it just like hey, this is what's normal to us This is what we always use Yeah, so it was actually a really easy decision for us because ACI said hey, this is what we're using so we said great We're gonna use this There my first experience with it was a ACS 3 last year I hadn't I hadn't used Pico CTF before that and I actually fell in love with it There's a lot that it does really really well And and the biggest thing is that it's really good at scalability So if I have a whole lot of people all hitting the same CTF, it's very good It's spinning up spinning down rapidly without all of those people having to share flags That's the biggest thing I like about it is that it has this instance concept Yeah, so you can you can say when you when you deploy a particular challenge I would 10 instances of this challenge and it replicates that 10 times generating a new flag each time It also does a really good job of wrapping the typical types of challenges that you would see So if you want a binary exploitation challenge that normally acts on standard and standard out You could do that and it wraps all of that an X in that D to make it an actual Over the network communication, but it still feeds it into standard and standard out of the actual application So you could do things that don't require network But wrap it in network really easily because it handles all the X in that D deployment stuff So I like that a lot it has classes for PHP and flask for web challenges So if I want to develop a web challenge purely in flask I can do that and it will deploy that cleanly it'll scale that up or down as we want So a lot of the dynamic deployment options in it. I'm really fond of what I don't like is the complete lack of documentation There is there's really good documentation for the easy stuff If you want to create a new challenge like a new crack me and put that up There's a walkthrough for how to do that, but as soon as you dive deeper than that It's very clear that Pico CTF was made to run the Pico CTF CTF, right? right so the more complex things you can do with it you kind of have to dive a little bit into the code to figure out so That's a bit of a problem. That's the case with literally every open source project I've ever worked with so I can't really fault Pico for that It actually has better that documentation than a lot of things I've worked with But there are a lot of pieces that we kind of had to pull apart and then the other issue that it has which a lot of Frameworks have that aren't just scoreboards, right? CTFD is a scoreboard which you have to do all of the infrastructure yourself Whereas Pico is not just the scoreboard, but it's actually also all of the infrastructure including all the deployment And most things that do deployment can't handle anything that isn't off the shelf 64-bit windows Sorry on Linux, right everything is built on 64-bit Linux because that's what you're deploying in and that becomes a problem for For me because my background prior to my current role was in Windows exploitation. I'm a Windows VR Person my job is find vulnerabilities in Windows. So going to millions of CTFs and never seeing Windows challenges bothers me So we we managed to find a way around that and in some other kind of Scalability problems that we ran into with particular types of challenges by creating a new Problem type so everything in Pico CTF is organized into problems and we created a new Problem type called Docker and that just whenever you make a challenge in a Docker It's been some a Docker container for instance and handles it that way So we're able to do weird architectures and operating systems that are Linux by doing that So we use that pretty extensively in ACS 3 because we had a bunch of I Don't want to say non-standard, but maybe more esoteric hardware to selections. We had a MIPS challenge We had a power PC challenge all that last year We liked it not to do that this year because we wanted to focus more on learning objectives and things people are going to see more day-to-day But it also allowed us to do do you see what I see which was one of my favorite challenges this year Which is actually running. It's actually running a full command line only Windows install for each deployed instance It's running that inside of QMU and then QMU is sitting inside of Docker and then Docker is being deployed by Pico CTF Which is part of why that challenge was as slow as it was We also we were able to get that down to only actually requiring 256 megs of RAM Which is not something Windows wants to do but you can do it so The big thing the big game that it had for me I guess my short answer is I like frameworks that handle deployment for you. I hate doing deployment I'm not a sys admin. I'm terrible at that sort of thing So if I can make my framework do it for me great all I have to do is hack up So you know some Python code and I have a challenge But that's me as a developer and hacker if you look at someone who has more of a Architecture background or deployment background they want more power and more control and the ability to do things like create entire domains So they're gonna have more hands-on there and actually Alex and I were talking about this just earlier today about this exact question So I dive into some of your concerns with the platform if you want Alex. Yeah, yeah, so that kind of goes back into the infrastructure and having the The sys admin background because that's where I come from I come from being a Windows sys admin and systems engineer and some of the challenges that we're trying to Look into building in the future specifically myself for be it grim or my own personal reasons are Domain-based challenges challenges such as resource-based constrained delegation Which only exists in an active directory environment or Passing the hash attacks with domain-joined computers Exploiting vulnerabilities in group policy things like that those don't exist and and unfortunately some frameworks Such as Pico CTF and it's not to rip on that platform But they just don't support that specific type of problem just yet. I'm sure that there's easy ways to implement that with some plug-ins kind of like the Docker type and things to that nature but other platforms other CTF environments that don't have the deployment built-in Help me a little bit better in those kind of aspects Yeah Yeah, I think ultimately it's right What is our goal and does the platform we're using support that goal and in a lot of cases Pico CTF will and in a lot of cases It won't did you stick with kind of the flat set up that Pico CTF and vanilla gives you like AWS and Terraform deployment Is that right? Yeah, so so overall it's based pretty heavily on what Pico gives us All of the infrastructure was set up by ACI and they did a great job of managing that and keeping it all running So as close as I had to do as far as touching that was to have an account We added my SSH key to the deploy user so that if something came up I could fix it But for the most part, yeah, they were able to spin that up The things that are built into it are actually really good and then the documentation for it It's pretty usable for what I found although Alex you dove into that quite a bit more Um, yeah, I did run into a few issues with some of the the way that we were finagling some of the challenges I'll specifically remember Anything in particular till or did you have one that you remember that we ran in specifically during development? So I will say that we tried to do a lot of things that the platform doesn't really mean for itself to do So so we did have to kind of pack around around it for quite a few things The shared resource-based challenges are kind of the biggest one so blame it on the temp was tough because when you have you know We when you have as many competitors as we did or or even as we planned for which was significantly fewer than we had You you end up with you know ten or more people on one instance And if you have ten or more people on one instance and you have resource constraints Or you have to write to the file system in a particular way Then you you can definitely end up with things where it interferes with other users and that's hard to test for You know every to every challenge we developed has an automated solve script So when we're doing that we can try and throw that script around and see if anything breaks But if it relies on the correct process going the correct way every time we might not notice edge cases and corner cases So that's that's definitely something that that we ran into some in the manual testing And then we also saw during the event blame it on the temp in particular At least a couple of people Accidentally found the path to solve the challenge because of something someone else had done rather than the network Yeah, and that's not a bad thing necessarily especially for people on the learner side, but it's definitely also not the intention. I Think I remember seeing it even even while I was going through blame it on the temp It's like I can see some other leftover files or some entries that they're leaving in that in that Jinja template, etc But I mean yeah, I mean just kind of like you said the nature of the beast so You mentioned I don't think that it actually So I don't think that actually detracted from anyone solving that challenge or we're getting the most out of it If anything it kind of gave them just the prod that they needed to find their way down, which we're not too upset about overall I try to limit the amount that I give out to competitors beyond what hints were given for a couple of reasons one is fairness And the other is that you know a small prod and then you figuring it out Means you learn more than if I just walk you through the solution. So Ultimately those little artifacts being left around. I don't think that they're too much of an issue But definitely not the intention You mentioned yeah, it was a Docker type challenge. Sorry to Alex. I didn't mean to stomp on you No worries I was just gonna add to that those yeah, even even the we did actually put in I specifically remember putting in kind of some Deletion like automation that would delete those files every 60 seconds and then you can't always, you know Rely on that because if you're if you're competing in CTF and all of a sudden You know your files deleted every five seconds, you know, it doesn't really help you too much So yeah, it does come back down into how many instances of this can you spot them up? So Yeah, and mitigation can only go so far for for the SQL challenge for example We we wanted it to be Solved using an automated tool. We knew that and we wanted it to be a Step further than just running a tool. So we put we actually implemented that one in PHP And there are some just really basic checks for your user agent. I saw using it a fault user agent Yeah, it just drops you so you have to figure out how to change your user agent I'll just do it a little bit and and you know, we wanted to mitigate in part people just running those tools and walking away Even though part of the child part of the point of the challenge was yes, you need to use an automated tool We couldn't provide the resources for that to to have that available to everybody all the time But also we couldn't mitigate so much that people can't solve the challenge and we can't mitigate so much that You know, we can infinitely spin this up So it's a hard balance to catch especially for things like that So there were there were two major challenges now that the CTF is in learner learning mode that we couldn't deploy And one of them is that one just because we burned through more processing time on that challenge than everything else combined oh my god, and the other is the one that sends out emails and We actually got ourselves blacklisted from a couple of email systems because we were sending out so many emails to people trying to solve that challenge That's hilarious. I know you had mentioned the the docker instances and how you were that docker type that you added for the pico framework Is that the one what it will let you kind of start and stop a specific port that you can access kind of per player? Is that right? Yep, that's that's right Yeah, it kind of goes to an activate button that that spins up a new docker container You can communicate with that just for you and then when you're done it gets torn down or if you don't interact with it for a while So our goal was we want something that Works for challenges that have to be completely self-contained for one user We want something that works for odd architectures because that was a big goal that we had last year And then we want something that allows us to do things that running on top of just core Linux doesn't so a Lot of that requires containerization or emulation of some kind and the easiest way to do that was just wrap it all in docker All of all of that work was done by one of our really smart engineers that is also so Grimm has an IT department that is roughly three people and Our IT department is also on other work everywhere else So one of the folks that's in in our app sec team that worked really heavily on this project last year Ben Lipton also developed this this new class for pico because I Basically kept complaining. We couldn't do things and he wanted me to stop complaining So he developed this this docker Type so that we could create these challenges that we had ideas for And it's gotten improvements and iterations over the last year and it was better this year than it was last year more stable more capability So our plan now is now that it's it's safely in the hands of army cyber as its maintainers They're gonna push it back up to upstream pico and it'll be available for anyone that uses platform That's awesome. That's awesome. Alex. I know you had kind of been working with that or I thought right? Were you kind of on keyboard trying to wrestle with some of those? But I didn't see them on the sequel always sucks or blame it on the temp because those had the shared resources Was that a different mentality or well? Yeah, so I was I was a tester for a lot of the challenges that required docker So I didn't develop any of the challenges that were built into docker But I did I did have to interact with that docker platform quite a lot actually Probably a good five or six different challenges, whether I was testing them or Someone else was testing them off of my device. So yeah, it was it was definitely And I will say it's definitely night and day improvements from last year on that platform It was much easier to spin up after talking with one of the one of the grim guys on you know The actual process and the documentation down. It was a lot easier to do and I think Pico CTF is really going to Improve off of that very quickly Can I ask about your like specifically your challenges the kind of mentality and mindset and kind of what you wanted for? The sequel always sucks and blame it on the temp. What were you kind of going for or does it? Does it have the solution that you wanted it to would it have multiple solutions? Can you tell me a bit about those? Yeah, yeah, so I'll start with the sequel always sucks So that challenge like like Tiller he was saying it definitely Was supposed to be more of an entry-level sequel injection challenge, but we definitely just didn't want to run sequel map done You know, right definitely so the goal for this one was to implement some checks into the actual Into the actual sanitization but not use things like html special characters in PHP and things like that You definitely want to remove some of the restrictions while Making it accessible for a sequel injection That's not just straightforward. So simple things were put into place You could say the user agent filter There were a couple stringer places that were added things like that just to Make sure that you weren't just, you know Dropping, you know the command and then winning immediately. So That that was the main mentality there. There was a specific There was a specific Expectate or expected path for solving it And Tiller, I'm not sure if we can describe if we can disclose that is that cool Yeah, it's fine. Yeah, no that the competition's been around. That's fine. Okay. I just want to make sure so the the initial Actual solution was supposed to be more of a time-based Injection there were filters to kind of remove union based Injection so that was kind of the the thing there and one of the one of the other mitigations with swapping out Quotes and and double quotes for they were swapping each other So one double quote would go to two double or to two Excuse me one double quote would go to a single quote and one single quote would go to a double quote as well So things like that just to kind of shake up the automated Injection try attempts. So that was that was sequel. It sucks and and As as simple as it sounds as simple as I'm explaining it. It was actually a little bit tricky to kind of Figure those nuances out with how simple map, you know, does it's smart, you know enumeration of databases and things But yeah, so I'm sure there are multiple different ways to solve that But that was the intention Yeah, and so just just a little bit of insight into Kind of how that process goes When we generated our challenge list By the time that it got to Alice's hands as here developed this thing We had the general outline of here's the liver requirements, right? The documentation that we need back from you solve script that sort of thing and then a technical description of the challenge Which for this one was this challenge consists of a web app vulnerable to a sequel injection attack The vulnerability will require slightly more sophistication than the easiest injections, but it will still be approachable and that was it That's what Alex took and generated the challenge from so we definitely wanted to put Creativity into the hands of the developers as much as we could You know when we were coming up with this initial list and and all of the developers really really ran with it that way And we all worked, you know when all of us were developing challenges We all work together we had a slack space that we're all working in so we kind of bounced ideas off of each other as well But the the technical descriptions that I wrote for these challenges. We're we're short on purpose Awesome. Yeah, let the developers just kind of run with it I thought it was really cool to see sequel light for one thing because that Yeah, I guess I don't see that all the time Especially when it's supposed to have that bigger scale, but I guess hey, that's part of the problem with that challenge being so resource and extensive But the timing attack was really really cool I tried to avoid sequel map because it just kind of gave me so much trouble connecting back and forth But that was really really cool Yeah, one of the one of the things that a lot of people Overlook is that sequel light actually does have a way to do multi Multi-session writing really just happen. Yeah, you just have to set it up a certain way But I don't know if it's undocumented, but it is a pain to initially do if you're not looking for it So you can definitely do that But I can I can talk about blaming on the temp too So this one was actually a very interesting because I'm Kind of a flask fanboy. I really do like flask I like the flexibility of being able to go. Oh, this didn't quite format quite properly Well, it's Python so I can just make it do that nice and I really like that you can do that mix Python with a web application It's great But what was very interesting is that I had never really I know what a sst. I is never actually Built a challenge that implemented server side template injection before and So when I was assigned this project or this this challenge, it was one of those, okay Let me research this a little bit and figure out Exactly what's going to be needed for this. Obviously, we're going to flask because I have no idea how to write anything in node. So So it was quite a no-brainer to write it in Python The challenge the way you actually solved it in your your video that you did Was actually shorter than the proposed solution. Really? I yeah, I actually spent If I go back and and I look I think the solution that was written was probably 211 characters, I think Just for the payload itself the actual callback, but I did a full I think I did a full reverse shell and everything as well. So Yeah, I think you called one of the base classes a little sooner than I did That's what it looks like based off the video that you did though And it's great because I was learning how to jump through these in in Jinja and it was It was a lot of fun to be honest and then just throwing in that C-Surf check there I think drew people off a little bit as well having to learn how to automate doing a C-Surf Bypass I think that was kind of cool as well Yeah, I have a I guess kind of in my set and and some if anyone's watching you might have known from the verset con Another like a CTF that I'd put on recently I had I think it might even be in pico 2019 the the mask challenge that I have is a simple flat SSTI or server-side template injection, but it's just getting the config object So you can read the secret key and the secret key is the flag. Okay, nice and easy But I love that flask also for one thing as a development tool like rapid creation of websites micro framework is awesome But I mean in the CTF world it offers so much stuff between okay remote code execution through that And even like forging cookies or some of the other stuff you could do like once you get the secret key You can do a lot of damage and do some interesting things. So super cool awesome challenge Yeah, I definitely liked it could have been a single-step drop and you're done. I really enjoyed the the jumping through hoops aspect of it to kind of you have to do this and then do this and that and So again, it was it's the idea of let's take something that people should be learning or understanding anyway and turn it on It's had so it's still interesting to people who have done this a million times Did you guys run into any issues kind of before the event got started? I know you said hey between last year and ACS-3 it just Miles ahead of the production value, but were there any any hiccups or stumbling blocks kind of getting before up to the event and even during Or was it smooth? Development was definitely I mean there's always going to be stumbling blocks in development Especially when we're trying to do challenges that are a little bit outside of the norm So probably half the challenges had some kind of stumbling block in the process of developing it But but overall especially comparing last year to this year. It was it was super super smooth this year I'm really happy with everything turned out both during the process of development and during deployment and during the event itself the Single biggest headache that we hit was for do you see what I see really that was because We were we were trying I knew I wanted a binary exploitation challenge and see and we'd written We'd written some last year, but they didn't get deployed They ended up not being used because the overhead to do it was really high The plan originally was we have this Windows server We're gonna set that up but only for these challenges and Pico doesn't support that out of the box So we'd have to add additional stuff to it to make this one server work And so we just didn't deploy them last year and I didn't want that to happen again this year so we had to come up with a way to incorporate Windows as a binary an actual runnable thing inside of the Pico environment and Getting Windows to run inside of Q mu inside of Docker on Linux is not easy to begin with right Getting it to run small enough that its resource requirements aren't huge and we can actually deploy several of them also difficult So when you put all of that together, it's really really hard We also knew we wanted to deploy this thing from a single static image We could because having to duplicate the entire Windows hard drive a bunch of times Is a huge resource requirement on the disk side So we actually implemented it to use a bind mount so that it could just use the one bind mount for all the deployments It's been up really fast In the end when we actually deployed it in the competition We decided to take that back out and that did increase the requirements It made it so that every instance that we launched of do you see what I see took a minimum of 10 gigs of space on on disk Wow, just to launch the entire Windows system Which is why we had been going for a bind mount but that has its own restrictions and it limits other things that you can do So by taking that out we were able to deploy it more easily faster more of them, but it took a lot more disk space so we actually had to completely re-implement that challenge twice in the process of Developing it to get through all of the headaches and hoops that we had to jump through to make it work So that was probably the biggest takeout. Cool my roommate Caleb and he's much much more of a binary ninja and like reverse engineering and exploit kind of oriented than I am So I remember he rested with that challenge and I was I think that was the set comp filters Right and the kind of the buffers one that was working in one direction. The other was reverse. Am I thinking of a different one? That's Maybe who knows this one was a a mostly straightforward stack smash Although a lot of people ended up building Rob out of it Which is an interesting kind of interesting takeaway for me. We we built the binary such that it had a SLR enabled and depth enabled But we then disabled them on the system so that you actually could just jump straight to the stack and execute I saw that in a write-up. It looked like okay the binary had depth on but the actual server had it off Right and that's really a lesson learned for me There are lots of ways to figure out That it's off on this on the system and we intended for that to be kind of obvious But it turns out it wasn't nearly as obvious as we wanted it to be and that was a problem because we have this thing that that Like we designed to be a fairly straightforward, you know, binary exploitation challenge now turning into blind wrap and that's hard You know it becomes a really hard challenge Completely unintentionally, but that's also it's also my favorite challenge for two reasons one because I'm a Windows person But two it's actually based on a real vulnerability that I found in my first year as a vulnerability researcher on Windows I had a piece of software that was a single threaded application so it used selected a loop and if you went down one path it would read and it had a Read bug that lets you read more of the stack than you needed if you go down another path You can write and it had a right bug that lets you write too much of the stack We decided to modify that a little bit so it wasn't just a stacks match We actually had offsets in an array that we used for this challenge But I took this real-world vulnerability that I found back in 2010 back when I first started doing, you know VR on Windows and turned it into this kind of fun cool Slightly different CTF challenge. That's awesome. Very cool What did you guys use for in-game support for the players? I know you mentioned the slack channel and I saw that was that like hey The best means the only method like getting contact with organizers or facilitators How did that work for you? Why slack other than discord or I know people to use like cheesy email and that doesn't help Yeah, so there were email addresses that you could send things to particularly to like account lockouts that sort of thing but the primary method was in slack and There are a few reasons for that For one thing slack always feels slightly more professional than discord does fair. Yeah We also use the slack and the fact that we have we can do shared workspaces to do a lot of work during the event We had you know developers and management stuff in the slack prior to the event Excuse me. So it's really easy to just kind of open that up to everyone since we already had all the users there The other thing is that the private Channel support in slack is better than discord in my opinion So, you know, we had we had a bunch of private channels That only certain people were in without having to manage a whole bunch of server roles and make it really obvious Who's what and that kind of made it cleaner as well But that was definitely that was the primary method of contacting really anyone for support and especially the developers during the event. I Guess that's totally true kind of looking back and thinking I mean I know you have the different groups between okay The DOD side some of the DHS guys and different branches in between that's totally much easier to keep private Rooms and slack and it isn't discord cool. Yeah, very cool The the only way you can really do it in discord is to create group-based restrictions And then assign those groups to people and everyone there's no way to hide what groups of members of in so Very cool Yeah, I tend to use discord, but it's always like on all well and all that brawl when things get to it I mentioned that one and sorry I was gonna say Graham Graham has a public discord that we use for you know Communicating and interacting with the That we use for interacting with community at large and We don't use slack for that because if you want a whole bunch of people in there Discord is a great tool. Yeah, if you want a select group of people that have specific kind of roles responsibilities Maybe slack is gonna be a better tool. So Cool very cool. I Know you mentioned last year once you were finished with a CS 3 you immediately got started planning for The following year's cyber stakes. Are you guys already planning next year's cyber stakes? What does it look like? Can I can I ask can I peek behind the curtain? What are you thinking? Yeah, I mean we're definitely thinking about ideas and some of that's going to be taking the lessons learned we have from this one For example, I'm definitely not gonna do something that requires you to send an email to every user next year That was a big problem But as far as the challenges go I've tried to find think of new ways to do some of these same concepts But in a new way right because every every year is gonna have a snack a snack smash right every year is gonna have a sequel injection Right, how can we keep that interesting across yours? So that's kind of what I'm starting to think about now I also want to start incorporating more of the kind of important esoteric things that you don't see as much of so we saw that with Like over time paid like no one really thinks about OTP anymore Oh, yeah, it's still a really basic foundation of cryptography, right? So like what what elements can we take and we reuse and kind of do that at the same time? You know last year and this year we had very different focuses for our miscellaneous category because there's you know We always have the same categories and one of them is is our catch-all bucket for things that don't fit into the others So last year we had a set of challenges There were all having to do with radio waves and radio communication And you had to take a look at those and figure out what was sent or look at the the waveform pattern and the flag Is printed in the waveform pattern things like that so we did that for ACS 3 that was our theme And for 4 we wanted to try and put more of a kind of defensive mindset on it So we wanted more things that fit into forensics more things that fit into kind of defense and hiding of information So I think I want to lean into that a little bit more next year Start trying to think about how can I take a Jeopardy style CTF, but make that interesting to blue team players and I'm just starting to spin on that a little bit But that's I think that's what I'm gonna really try and do for our miscellaneous category the next year. That's awesome Yeah, I remember seeing the audit log challenge and I know you mentioned overtime overtime paid I was beating myself up on overtime paid because like oh, this is clearly a one-time pad Obviously between the challenge and title and everything I get seed in the source code But then it's like why can I figure this out? And it's like well the very first line is exactly what you need having that known value. I was like wow cool I spent way too much time we made sure that there We wanted to make sure that there were a few a few cases of that so there's actually three Lines that are the same for every file the intro a line full of spaces and then the outro And you can pick any of those three lines that I'll get you the key. Nice And then again that kind of goes back to our idea that we don't just want a challenge that you can throw a script out And win right we want something we have to actually understand what is the learning objective here? What is the thing we're actually looking at basically don't know how one-time pad works and don't know how XOR works You're not going to solve that challenge, right? But if you do if you actually understand what's happening in the challenge Then you'll start to see that path forward and be able to solve it And we definitely found in some cases that once we started doing play testing Some of those were difficult because the path forward wasn't necessarily Obvious without a lot of knowledge about it So in those cases we had to add to the hints because you know We want something that yes You need to understand it to solve but we want something that you need to learn it and understand that to solve That's something you have to have been doing this for five years to solve Cool with the exception I guess of some of the higher point challenges where it's okay to require some experience but the lower the point value the more The more hints we gave for one thing and the more kind of obtainable we wanted to make it well I think having kind of as you said the idea and the whole concept that it shouldn't just be a Throw a tool at it and immediately get a flag It's kind of being able to understand and know the concepts behind some of the source code that you provide and stuff like that So I think that is definitely a signature of a good and high quality CTF. So kudos to you guys Thank you What I think kind of to wrap it up which would be a cool thing for hopefully some others that might be listening in or watching What do you think is your best advice or what would you tell someone who wants to do This sort of thing themselves like hey, they want to host their own competition. They want to put on a capture the flag What should they know what what tools do they need to know what technology should they be smart on? What is your words of wisdom to someone that wants to do this too? Yeah, so I have a few things and I'm gonna split it up into sections What one of them is you have to host the CTF you have to host a CTF So knowing how hosting works is really really important and a lot of that comes down to infrastructure If you're not great at infrastructure development yourself You need a framework that handles it for you like Pico if you don't want to be limited by the things that your framework Let you do then you have to understand how to do that hosting and how to do all of that kind of management and spin-up And to do that you have to have a really good understanding of the scope that you're targeting We were not targeting 2047 solvers for this this competition. That's what we had We had 2047 people with at least one point and that is huge If we hadn't had something so scalable that wouldn't have worked So knowing what your target audience looks like is really important, but also being prepared for something a little bit beyond that We were really excited when we started seeing the numbers coming in and I think a large part of that is that you know This was a thing you could do remotely in a time when a lot of things can't be oh, yeah, but You know preparing for that kind of unexpected is really important as well And then the other half of it is to host a CTF you have to host a CTF Right. You have to have a concept of what you want to be doing So knowing what kinds of challenges you want or what your objective in doing this is is really important for us This is a training platform first in a competition second Let's not to say that making it a competition wasn't important And I think we did a good job of that but at its core Cyber stakes is about learning something and it's about everyone that's competing even the high-level people learning at least a thing That's not going to be the goal of every CTF Sometimes CTFs are to test your skills to see how good you already are I've seen companies use CTFs as Recruiting mechanisms and if that's your goal great There's absolutely nothing wrong with that, but you need to know that's your goal and really lean into it So know the types of challenges you want know the difficulties you want are you giving hints? Do you use something like CTFD that charges for hints? and if so how do you set that up right and So kind of knowing that's really important. So so my kind of two key takeaways are Understand your platform or what or what your platform requirements and have that ready to go and Understand what your actual goal is for the CTF and build around that a city that tries to do too much on Either side of that isn't going to be successful Cool. That's some good insight Alex, I know you do this all the time. Do you got anything to toss in? Yeah, no, just One of the major things that I've learned In doing this not just from a hosting site, but from an actual development side is You have to have a mechanism to keep yourself on pace and on track with development For example, some of the the CTFs that I've built in the past we started eight months before the events even supposed to start and we get done a week before the CTF just because of personal problems personal issues Work, you know this, you know might not be your primary job These things definitely get in the way. So having that buffer time planning for the unknown and and definitely using a tool I like two different ones whether it be Jira or Asana to just kind of keep yourself in pace with, you know What you need to accomplish and then having you know a project manager possible just kind of tell you to you know Hey, this is due I know you have other things going but this is due you should probably get on that Yeah, those are really good points I'm I'm blessed that I get to do this for a living You know building building this event was the thing I was doing while we were building this event And when I'm building other CTFs, that's the thing I'm doing what I'm building a new training program That's the thing I'm doing. So I'm definitely really lucky that I got to turn this passion into into the thing that I actually do for a living That's really exciting but a lot of people are doing this as a hobby or as a side job or whatever and it's it's definitely a good point that you Need something to keep you on track. We we set ourselves up in sprints So we had an initial planning sprint and then we had our development sprints Then we had an integration sprint and then the actual event was a sprint and all of that was accomplished in a total of 10 weeks not 10 calendar weeks because we ended up moving the CTF because of a couple of reasons, but it definitely if we hadn't kept on top of that and if if Straith hadn't done such a great job of keeping our developers and our testers in line and and on task and meeting our deadlines There's no way we could have delivered it. So Cool, those are really good insights I'm trying to like take it to my own heart because if I if I want to do some other Something like this to even just hey host a game on my own between some of the other events that are going There's so much remote conferences and virtual events right now I know a lot of people are asking. Hey, can you put on a game? Can you put on a capture the flag? And so those are that's cool. Those are really good words of wisdom. Thanks sweet All right, I don't have a whole lot of other questions I know we're wrapping up and getting really close to an hour point So unless there is anything else either of you would like to add in any other If not also totally cool Thank you so much for being able to be able to do this have this conversation Absolutely. Yeah, I don't really have anything else to add. I just really enjoyed the opportunity to talk about kind of some of the stuff that we're doing You know it Seeing a capture the flag a lot of people sometimes and I'm not saying everyone but sometimes people take them for granted That they're just available. They're just challenges that are readily available And they're not, you know, it takes a lot of work because people write them up You know, I know there are tons of write-ups for ACI already For some of these challenges those are burned. You can you can't use them again in a CTF. That's going to be You know for prizes and things like that. So these takes a lot of time. So if anyone, you know if anyone Is building a CTF know that your work is definitely worth something because it takes a lot of Effort to put on that stuff. So yeah Yeah, for sure just just to echo that I mean when we plan internally on my team We expect developing a single challenge should take between 40 and 80 hours of development time and you know when you're developing 50 70 challenges for a CTF that you're running once that's that's a lot of time, you know and and I Love doing it. So it's great that I get to spend that much time on it But it's not I mean, it's definitely a labor of love and it's definitely a labor to get these things out You know and sometimes We don't have write-ups and that makes it a little bit easier because we use some challenges A lot of the challenges that are in you mentioned howdy neighbor are you know have been there since we first made howdy neighbor Some of them we cycle out because a you know, maybe somebody did write up for it Or maybe this challenge has just been around too long and has been solved so many times and that's that's a huge consideration too so I Think I think it also, you know Alex kind of alluded to a lot of things that are the difference between a really good quality CTF that you want to play in and that you get a lot out of and a CTF that is just thrown together really quickly and you see a lot of complaints especially like I was reading a bunch of Threads on Twitter just today about how a CTF's are terrible because they always are solvable by the same tools or whatever And and that comes down to how you develop your CTF what your goals of your CTF were and if your goal If the goal of your CTF is a bunch of challenges people learn from and you don't care about automated tools great If your goal is make it really hard for competitors you have to test against those things We have some we have a list of automated tools automated solvers that people use on actual CTFs that we run against our challenges So we you know every crypto challenge that we write if it's an AES challenge We run AES crackers on them and say does well enough the shelf tool solve this and that's a big you know A big amount of time for that you should probably add katana to that now Yeah, yep. Yep. So it's definitely did my tool get used for testing That was I threw it together in a docker container like Probably a week ago and threw it against some of the older challenges that I've built and it was like yeah, no problem What else do you have? Yeah But you know that the advancing the edge of the art is really important too both in competitor tools and in CTFs And that's something that I really enjoyed doing and part of why cyber stakes is my favorite. I mean, I developed eight CTFs last year. This is my favorite. So Just the amount that we're able to do here and the goals of it really align with with where I want to be See what other parting thoughts do I have? I did have one more Yeah, if you are developing a CTF and you're going to be hosting it and all that good stuff If there's an option to have a larger server Go with that option. Especially if it's gonna be in More public setting definitely go with that option. It might cost a little bit more You're gonna have a better performance out of the actual event You're not gonna have people waiting on a challenge to to load or you're not gonna have time out issues That was one like I would say noob thing that I ran into one of the first times I ever ran a CTF was I needed a bigger server so Definitely put that into perspective Yeah, so the last thing I want to make sure that I hit is is that? events like this are really good for engaging with the community and as as someone who Spends most of my day thinking about how to train people and less of my time thinking about technical solutions to problems than I used to And especially as I move more into kind of management level things directing these projects instead of getting to develop the challenges That's really important for me Both understand where we are in the community But also to understand like how far behind I am so that I can catch up and that's really important to me But then also, you know building better challenges building better community in general is also really important. So I actually jumped at the chance. We had a much more active part this year in Kind of troubleshooting challenges as they went and interacting with the competitors Then we did last year and I jumped at the chance to do that so that I could be interacting with all these folks so that's that's really big and You know, I try to continue to do that after the event's end Which is why I was so excited when you reached out, you know To kind of having this conversation is amazing and being able to have conversations like this in slack as they continue to kind of ping me Or in discord in the grim discord is really great So I really hope that that continues both as an outcome of this CTF and all CTF's kind of as we go forward because ultimately You know competitions should be bringing us together rather than apart and it's a really good way to do that by kind of pulling the community together Absolutely. Yeah, I love it. That's a that's a great way to end it. So well, hey, both of you did a Absolutely superb job. The game was fantastic. I'm really looking forward towards next year's and I mean I have nothing but but great things to say and this has been a fantastic conversation I hope there's been a lot of awesome insight for others that are listening in and wanting to learn from it So, thank you. Thank you. Thank you. I can't say it enough. This is this has been fantastic Myself, I'm sure tell her he feels the same. So, yeah, absolutely. Yep. Lots of ways to go hold on me so fantastic and I Shout out and a special thank you to all the people that did compete because you know putting all the hours into developing challenges There's nothing if we don't have people actually solving them So I've been really really happy with the number of people that have not only tried and learned something from this But continue to engage with it and with with the other people that are still solving I think that putting the challenges back up in training mode You know in training mode is one of the best choices that army cybers do could have made because now You know that that conversation that's happening gets to keep happening So thank you to everybody that's been participating and thank you John for having a song