 Well, hello, thank you. We're really really grateful for the opportunity to come hang out with you guys This is kind of a passion for us and something that we really like to do for fun and also Thankfully for work. It's a good blend This is a presentation on security training at the harbor or running capture the flag with Docker I want to put a little bit of dev ops spin into it because a lot of this capture the flag Material is a lot more cyber security oriented, but obviously hosting a lot of the infrastructure and creating a lot of the Everything at scale right everything that we try to build we try to make it in a Stable way that is also automated and can be kind of encapsulated with code and and and dev ops, right? So what we wanted to bring is a conversation on our experience Hosting a capture the flag on a worldwide stage an international event and competition that we put on so We will dive right into it. We'll get through the obligatory introduction Kudos and thank you to Trevor for those wonderful words. He said it right my day job. I'm with the Department of Defense cyber training academy I'm a trained developer there and I used to teach with them I also do some kind of red team penetrations testing stuff on the side and in my free time I do like to play and create some of these capture the flag learning environments and training exercises I also have a pretty silly YouTube channel It showcases a lot of this kind of stuff with cyber security and capture the flag competitions and events and That's grown. It's a lot of fun. It's kind of a pet project But it's just my name if you have any interest you can go look me up and I'm sure you can track me down So John is also a government contractor. So he is a sub case you guys were curious So I'm actually active duty Coast Guard. I just graduated from the Coast Guard Academy this last year in 2019 So I am at Coast Guard cyber command as Trevor said I am a blue team lead. So most of my Day-to-day job is in like compliance and vulnerability assessments And I do at night. I do have free time. I play in cts With John sometimes by myself, I'm not as good as John is so I'll play in a lot of the beginner ones or me Intermediate John plays a lot more of the advanced, but I do also do some development for cts, which I did help with development for the ctf and I am I do some web development on the side. That's actually kind of like what my fun passion is so Yes, we are not six feet apart because John's camera cannot Cannot actually extend that far. So we are literally sitting right next to each other I don't know you might be on my side Okay, so Uh, this these are the topics we're going to run through this goes from like the background of the capture the flag itself that we ran as well as how we built it and also infrastructure That we implemented as well as infrastructure that we hoped to implement In the future because we are actually going to hopefully run another ctf in june so this talk and presentation is meant to be Both a story right us kind of hey explaining our experiences what we learned how we grew from it, etc And still kind of talk about some of the the tech behind it So we'll throw in a lot of terms between hey docker ns jail Some of those stuff that you saw in that little outline or agenda But bottom line is we're here for you There's not a whole lot of incentive. I guess We're not doing this for us. We're totally doing it for you We want to make this informal and casual and just kind of spend some time with you guys So please don't hesitate to ask any questions I know it's strange doing this virtually or digitally, but We're just happy to have the opportunity and spend some time with you disclaimer we We don't come to you with with gospel right some of the stuff may have worked for us and Worked again is still kind of an interesting word because maybe some stuff was successful. Maybe some wasn't Disclaimer if you had get any insight from this or you're interested in some of those concepts We can't promise you immediate success So your mileage may vary with some of this stuff, but we wanted to at least put that stamp on there just in case Why are we talking about this? So a lot of the stuff we did for the cts city This is the first large ctf that john had hosted and he had asked me to co-host with him So we made a lot. We put a lot of work into infrastructure and automation So doing that allows us to not rework or not We don't have to like make the same thing for every challenge You can just use one thing for all the challenges. You can use one thing to host everything Do you mind turning up your voice then your volume a little bit, please? Do you move to this mic? Is that okay? Yeah, I'm just going to change mics because apparently my mic is like causing issues. So it's fine. So We did have automated infrastructure for this and we For the most part it was stable throughout the entire time Also, we did learn some lessons with automation And that trusting players is not is not always a good thing to do with cts the whole point is to have fun and For everyone to learn and there are always going to be people that don't want that to happen So they're just kind of we're going to talk through some of the problems We had with specific instances or infrastructure that we had that maybe caused us to have some issues with certain challenges or wants us to Things that we can improve in the future things that we're going to implement That we're going to use in place of stuff that we used for this past ctf So a little bit of background to kind of set the stage for really the vessel of of the talk here A few months ago. I was asked to help host a virtual conference similar to Right what we're doing here the whole dev ops dc meetup and just doing some of this cool stuff online That was going to be hosted by ben Nomsack over at hacker one. They do a ton of bug bounty stuff And the cyber mentor or heath atoms and he's another kind of online personality for showcasing and training cyber security We wanted the event to be I guess wholesome right or for good, right? So There was no cost to attend. There weren't any ticket prices But if you did opt to buy a ticket all of the proceeds that went to the conference was going to be donated to charity Everything that came through either through sponsorship or through any ticket prices that people were willing to buy that would all go to the leukemia and lymphoma society So that was really cool for one thing because it set Just a really good tone for the conference and everything that we really wanted to do Even in I don't know the whole aspect of the capture the flag or the game or the competition that came with it So it was going to be a conference. So we had speaking tracks, right? And there were some time slots for speakers There were about 11 speakers that packed throughout a two day event This ran over a weekend and of course there was one Capture the flag and that was where we came in we were invited to go host that competition So, uh, as john said mileage may vary. So you want to kind of explain what a capture the flag is So capture the flag ctf same thing. Um, this is what we hosted for the conference itself So it's essentially a cyber security competition. It gamifies learning new tools new skills Or ways to think about how to solve a challenge Challenges can vary and they essentially will come in they'll come in a very a variety of forms You can have a service. That's an active thing that you connect to It can be a file that you download an image of pdf anything along those lines and It requires you to use either some hacking or cracking I guess or Possible defensive computer defensive computer systems to actually get a flag You want to get some form of it's usually an ascii text Flag usually we use tags for ours like jctf or verset con ctf was like our flag tag Some categories that we had in the ctf We had all of the ones that are on the slide with the exception of the oscent We didn't really have any need for oscent ones, but Some of the ones that had the active Services that you would connect to would have been we had web security So like connecting to a web server some of the Uh binary exploitation ones and I don't know if any of the other ones were most of the other most of the crypto We're going to be downloading a file or an image such as steganography or forensics But that dichotomy or the difference between kind of a static downloadable file Or a dynamic kind of an active service that you connect to Brings us to really the next topic and how we can kind of pour more into that dev ops sense The dynamic services and the challenges that the players would need to connect to and actively interact with Well They're going to have Vulnerabilities in them right because it's supposed to test cyber security The player might be doing some more offensive or kind of emulating an adversary and a lot of their techniques so If we create challenges or problems that have Vulnerable code or there's a real weakness or flaw in there Well, we don't want that running on our Actual computer right if i'm trying to host a website for someone to go hack into and run code on I don't want that on my legitimate laptop So we opted for one technology and maybe you guys have heard of docker Or you might already be super duper familiar with it, but docker is Incredible it is a technology that allows you to Essentially bundle up an application or kind of package it so that it can be shared and ran anywhere and it's encapsulated It's not so much a virtual machine, but it's a little bit smaller than that It's lighter and more compact than what you might expect in a virtual machine and for our purposes This is fantastic because we get to put them inside of a jailed Encapsulated environment where those vulnerabilities what the player is exploiting isn't going to break Our actual computer or our real infrastructure. It's all written in code, right? You can see some of the command line stuff there Creating a docker file or spinning up an instance or an image or a container It's all really easy to use and really easy to develop So we certainly use that for all of the challenges that we went ahead and worked with And when I say All of the challenges that means that we have a lot of them, right for every single challenge They all want to be in their own instance. They all want to be inside of their own container so not just one but 50 of them or more and they're all supposed to be Available and accessible to every player throughout the entire competition. So that means two days 48 hours All of these services need to be up and available The way that we did that is we utilized a docker compose Which again kind of a little sideline like carried with docker technology that allows us to Orchestrate or really kind of keep a symphony going of all of these different Instances and services. I can specifically say okay docker start All of them Let's bring up suddenly all 50 or the whole mass of services that I need to bring up or down and I can kind of declare and specify Hey, I want this network service to be available I want this port to be reachable by the player and accessible So docker and docker compose allowed us to do that and host the services On the back end really running that kind of the end of the competition event that the players would end up interacting with When they're working with challenges when they're working with the services So some of the issues that we did have with docker were Some privilege issues for users specifically configurations that we had that users were able to Change a flag edit Text different files add things delete things break things So that whole don't trust the user mentality We kind of had to actually take down one of our challenges because of that during the competition Which sucks, but eventually has we have to think of what is fair and fun for everyone because if a challenge just keeps breaking It's there's no point in keeping it up So this is when we kind of turn to nsjl as a possible improvement to that so John actually got invited to come to the google ctf finals in london due to his youtube channel he Talked to the people that created the google ctf And this is something that some of the developers over at google as an unofficial google product actually use for their challenges there For those services that are running or the web services that are running So nsjl is really cool. Um, it allows for process isolation and by default It's a read only file system So immediately the user is put into an area that they are not supposed to be able to change anything or do anything Other than what you intend for them to do So you can see there's a snippet there that kind of has Uh choices that you can give the user or what you're allowing as a default in there It does allow you to do customize configuration per connection, which is another really cool thing And nsjl, uh, is super easy to set up. It does have a github that I think we'll show at the end. Um I actually used this for a couple of new challenges that we made really cool infrastructure for just doing, um Like instant containers. So this is basically going to be a replacement for docker for us But it does allow us to integrate with docker containers that are pre-made So things that already have like a sequel database setup Or services setup that we want to use nsjl allows you to use those characteristics of that docker container But in a more secure fashion in nsjl's container outside of that So for the for the ctf, uh front end that we actually used, um, we use the framework ctfd So ctfd is one of the more common frameworks that you can use For a front end for hosting a ctf So pico ctf is a common one. You can also always just make your own But ctfd was a really easy one to do automation with uh for setting up And it allows you to have an admin panel that we could only look at and we could hide challenges We could have challenges ready to put out, uh, like at our periodic times So we could give the players more stuff to play with Uh, it also allowed for us to have stuff that you can have the dynamic or static challenges that we talked about earlier in a ctf As well as a lot of great visuals for players to see as well as for us to see So having a framework is really nice and an easy way to actually run the front end of a ctf Digital ocean is the cloud service the cloud infrastructure that we used to host all the back end and kind of the infrastructure for our ctf Um, i'm sure some of you are familiar with digital ocean digital ocean or aws has instances Digital ocean has droplets Uh google cloud and microsoft is there are two of the other ones that i people are probably familiar with but Primarily even for my personal web development. I used digital ocean droplets for stuff and That was john's choice for us to use for the ctf and it was really useful So for the front end, we had the ctfd service i talked about we had The docker services were the back end and we also had load balancers that john actually John actually set up all the load balancers I mostly helped with the back end servers and managing the ctfd and the discord that we had set up to Know when i needed to go check a challenge restart a challenge look at something so For we actually had seven of everything that way that when for example one challenge a user connected to the instance five A challenge went down We were able to have six other instances that if a another user connected the challenge wasn't down for everyone It was just down for some users So you can always have those backups and we were able to Isolate the one challenge that was down just have the other six be up and work on that challenge to try and fix it So then we could bring everything back up and everything would be operational So it was really helpful for us because we were the only two people running the ctf So a lot of times i would be working on one Or and or john be working on one But players could still play because the entire ctf wasn't down since there were other instances to connect to with those load balancers So something that we had to actually maintain the ctf were some scripts that we used That utilized ssh and ssh pass to actually connect to those back end Uh droplets i was talking about to like check challenges or to go in and actually Um edit challenges if we need to when someone did remove the flag or mess with the challenge We also had a bound script and that was really cool. So The idea of being able to like check a challenge or see if it was up or down Is really hard when you think about like well what how many people are going to join the ctf We had no idea when we had started developing the ctf How many people would actually sign up for the conference originally? I think it was we were thinking like oh just a couple hundred people But as we saw The registrations come in we were reaching like the thousands and we're like, okay Well us going individually to check seven instances of 40 different challenges is probably not ideal so John um started this writing this bounce script that allowed you to stop and start a service like on a dime So you would just run that if you there was a challenge people were saying oh, there's an issue I can't really get in so just to see if maybe stopping it and restarting it fixed it and it was just a weird Connection issue and that usually worked. Um, I would say for almost all the challenges the exception of two That was the fix nine out of ten times So another idea that we had To do that to kind of expound upon that stop and started service was a health check So we both worked on writing a script that actually would connect to those services To verify that it was running And then run that bounce script that we had talked about to actually See if it doesn't if it didn't respond to actually stop it and then restart it This was really helpful because As john said the challenges ran over 48 hours So we you know need to sleep eat do other things other than just like respond to discord messages and like check a cdf So having that health check actually Really helpful to know whether or not it was a user That was just having a weird issue if the challenge was legitimately down And we needed to go look at it more because if the bounce script Stopping it and then restarting it and then going immediately back down with the health check again Was a trigger to us like okay like stopping and starting it's not working still And there's got to be something else wrong with the challenge itself The issue that we had that we ran into Kind of this poor man's maintenance is what john galdick since it was just running scripts or Manually connecting to stuff a lot of the time or running a simple Bash script to do the bounce or the ssh logins was we had no ability to check Challenge solutions, so we weren't checking the flags on the challenges And we didn't have an automated way to like apply patches to a challenges or Like be able to see if it was one specific user that kept Dawson a challenge or that because we had one challenge in particular that we were pretty sure it was like one person Just hammering for no reason but was doing it and the challenge just kept breaking But we had no way to identify What was causing that so we eventually just had to take that challenge out Which is a bummer because it does cause issues at the end with teams that are like we solved everything but like They didn't end up at first because we took out a challenge that they had And all in all this was for fun So as organizers, we didn't really see it as that big of a deal There was no cash prize. This was just for a foundation But something in the future for us to look at and think about because if we are ever asked to do this for Something of monetary value like you have to keep that in mind competition and people being competitive with challenges so After the fact right After the weekend after 48 hours in a two-day competition where we had I guess you can see some statistics There are 67 challenges running All across seven front-end servers seven back-end servers balance from the load balancer Uh It was still a success It was actually really really well received. So we had just over I guess 2,500 players that registered Um, they asked hey, can you please keep the challenges online? This has been a lot of fun. We got a lot of training value out of this So I think even now I think that number has gone up to about 3000 participants, which is Just surreal and kind of crazy to me But when we asked for feedback a lot of it was positive everyone said hey We didn't experience any downtime whatsoever. The game was always available And probably the biggest staple or indicator that hey, we did a good job is that they invited us back So, uh, hopefully we'll be able to put on another event in competition like this Next month. I think june 13th is when we're planning another nom con and we'll have a whole new range of 50 challenges and 50 gamified uh Plays uh exercises to work with With that said, uh, we did have some success, but it's not all sunshine and roses, right? There were some things that we Got wrong or we could certainly do better with so We mentioned some of those dynamic services the ones that would allow the player to interact with them Some of them got vandalized, right? They could leave some files left over They could tear down some of the files that were there or remove the whole flag or the solution to begin with And that would just ruin it for other players Learning from that Well, we should make those docker containers read only and nsjl as an option Might make that a lot better and still kind of lock down some of the other processes Or memory or resources that might be necessary to hold that Another point While we had I guess over 2,500 participants a ton of teams that were registered and some people playing There were just about a few people I guess five or so participants that had solved all of the challenges And That's not a bad thing, right? That's good. It's it's awesome that they were able to roll through everything and But thankfully it wasn't super duper early on in the competition because that would probably bum out a few players like Well, I'm done with this game. I'm bored now But thankfully it was just before the end of the competition about a few hours left every uh, some of those teams were able to solve everything From the content creator and the developer standpoint I'm bummed at that because I want to keep providing That training value. I want to keep putting them into new exercises and having them learn as much as they can So I want to be able to make a game where no one solves everything No one should be able to 100 but that's hey some lesson to learn from next time We'll keep putting some more fire in there and as katlyn was saying One of the challenges was just too problematic and we had to end up pulling that one from the rotation So anyone that had solved that challenge, I guess the few people that had before it somehow got broken beyond belief They All the points that teams had gotten from that challenge were removed and it would it kind of shook up the score board Right because at the very end of the game those people that had solved everything They're really competitive and they want to say hey, I want that first place spot Because you remove that challenge it alters my standing in the scoreboard and that was a little bit of a storm but Okay, we'll know from next time if we release challenges throughout the competition or if we end up pulling anything We really need to be a little bit smarter on on that and how that experience is for the competitor and for the player So with a lot of that in mind How can we kind of do this better when we put on the same show next month? Truth be told a lot of the infrastructure that we created within digital ocean Some of that was by hand, right? We needed to have seven instances of the front end seven of the back end the load balancers and Of course, we have a database server and a redis caching server and some other kind of decorations off on the side We would create one instance Maybe of the front end and then clone it and duplicate it So we'd have seven of them to work alongside that load balancer Still we're manually creating that first instance or the database server or the caching server How can we remove that manual configuration? We want to automatically deploy all of that cloud infrastructure So i'm learning about and getting smart on terraform And that would totally replace all of that manual interaction We could automatically create that droplet or that instance in the digital ocean cloud infrastructure And then we could decorate it and provision it with the database installed with okay Configuration known to accept connections from whatever ip's those in the load balancer pool, etc, etc And again terraform is kind of like docker and that hey, this is a text-based thing You're gonna be writing some code or some configuration files and then it will Just speak it into existence and that's crazy cool It does have providers and it does have support for digital ocean So i'm really really happy about that you probably see it all the time with aws or i guess google cloud or anything But digital ocean isn't always as advertised So i did a little bit of digging and our research a little bit But it can go ahead and just spin up droplets for us so i can Rather than clicking around and spending however, maybe half hour or hours just building those machines I got a simple command that i can dot slash and hit enter on and suddenly everything exists And i can tear it all down whenever i need to That creates all the services that builds the whole game for me but How do i get away from that poor man's maintenance that i was doing with those dirty bash scripts or kind of Running around back and forth into one server and the next just trying to fight fires We had that problem where if someone were to Vandalize a challenge they'd destroy it for other people or maybe even There's an unintended solution and someone just was able to grab the flag really easily Well, i need to patch that i need to fix that challenge Problem is it exists across seven other back-end servers How do i do that as quickly as i can so it's not disrupting the game for these thousands of people that are playing So another thing i'm trying to get smart on and learning about is run deck So After i use terraform to deploy all of this infrastructure I want to be able to fan out all my commands and all my changes To each instance so i can script and run code on all of those front end or back end servers That would totally replace Those dirty bash scripts that i was doing in my poor man's maintenance Maybe we could update the challenge source code like to roll out a patch or to fix something or redeploy a challenge right? bounce it start and stop the service or This is another thought that we had because Not just even the dynamic active services and the challenges that players need to interact with but even those static downloadable files Those exist on the front end server rather than the back end server So again, we had another issue because if a player is trying to download a file And it exists on only one of the front end servers Well, then it's just a matter of luck if the person happens to be in the right rotation of their load balancer to get that correct file so Rundek would even help spreading all of those files across all of those front end servers and back end servers Rundek could allow us to automate Spreading out that infrastructure and provisioning it and working with it as we need to Yep, that's perfect So those are the technologies that i'm really looking forward to a little bit smarter on and How we could apply them into fixing some of the elements of the game that we fell short on There's still some stuff. I'm thinking about right We had a conversation as to we're getting a lot of traffic We're getting 2000 3000 people hitting these servers And maybe challenges go down or maybe something's getting Beaten up with a denial of service attack But i'm totally blind to that. I have no idea What ip address out there on the internet is trying to hammer these services if anyone might be doing any malicious activity I want to see okay, how much traffic am I get am I getting where is it coming from what ip addresses Do I need to be aware of or what firewall rules? Do I need a place to stop those people from just trying to flood this infrastructure? I needed network visibility and I still need that I'm still looking for what is a good solution to be able to get my eyes on all that traffic coming in Another thing that we discussed with our health check scripts and those bounce scripts that we were Starting and stopping a challenge whenever we need to maintain that That was good to test if the service was available and if it was live The problem was I didn't know if that service was still in that challenge was still working as intended Because maybe someone clobbered that flag file or it got vandalized and it's destroyed and it just broke In a different way that simply knocking on the door wouldn't find out So I need to automate solving that challenge. I need automated solution scripts That will go ahead and find that flag for me and verify. Hey, this challenge is not only still alive and well But it's also functional and working the way that we want it to That's another thing on our to-do list One last bullet point that we're thinking about is that All of these docker containers and these instances exist All throughout the game and they're accessible to every player, but they all exist Like in the whole scope accessible to every player and that's why we had some of those issues where Okay, someone could break that challenge for other players one thing that we could do Rather than using nsjl or really locking down and making that read only is we could Give each player their own dedicated specific Copy of the docker container so they can break it all day long. They're just ruining the fun for themselves and not everyone else Um, we're trying to figure that out a lot of other frameworks like captured the flag hosting frameworks in the front ends We mentioned ctfd Ctfd is what we're using But it doesn't have built-in support to be able to spawn these containers and instances per player Pico ctf another option does and so that's why you want to kind of survey the scenes when you're choosing your provider But also learning and also poking around It sounds like we might be able to use a plugin that will allow us to spawn up specific containers Individual specific to the player and that would be really really awesome Because then we don't need to worry about the read only file system trouble And if they want to break the whole instance They're only ruining for themselves and not for thousands of other people So, uh, I we can work with trevor or that to possibly give you guys the slides if you want If you like don't want to like copy and paste all these links But these were all the resources that we use kind of for when we were researching stuff either before We were doing the ctf during the ctf as well as after the ctf when we were working through stuff just kind of some general guidelines documentations good Um github references digital ocean references that kind of stuff So the github for nsjl is really awesome. I was in that a lot Uh, and digital ocean does have a lot of great tutorials for random Different tools that you want to integrate with it kind of similar to what john said it was a terraforma digital ocean Yeah, digital ocean has like really weird random if you search digital ocean with something There's a good chance. You'll find a tutorial on that which is really awesome And Now it's time for questions. I don't know if trevor has any or if there are any in the chat Immediately that we can answer but Yes, and those are both of john and i's contact info in there Yep, sorry about that katlyn. There are a list of questions. Um, I know there was a lot Like a smattering in new technology. Sorry that was a little overwhelming Well with the audience here what we typically do is we specialize in provisioning secure systems And you're sharing how to provision in secure systems But you're also doing it with a lot of the techniques and concepts that we utilize in enterprise architecture Which was mentioned early on which is providing the service to users quickly and timely load balancing High availability like that would that was really good examples to hear from But moving on with the questions, um, Peter asks if you have a demo for those that don't know what a ctf looks like Do you want them to watch your youtube channel? Oh boy The the easiest thing Yeah, to not I don't know how quickly or in the weeds. I would be able to spin something up Let me try it in another window here. But the easiest thing Yeah, go take a look at some of the stuff that I have out on on the interwebs on youtube We actually did do like a stream on john's youtube channel about Running the ctf and actually showing, uh tutorials on how we build stuff and solve stuff. So, um John's youtube does have actual like for this specific Conference and ctf like how to do that stuff. So yes, we do. I don't know if he'll be able to pull it up right now, but I can give you just a quick. Um, I will share out information If you're totally cool with the quick, uh, short demo here is a capture the flag pico ctf That is normally kind of targeted at more the younger age or just to train people and kind of get into the scene Thankfully, I was able to help out with that guy. You can see a stupid face over there, but Once you're interacting with the game and the competition You'll have a certain amount of those challenges to work with and they'll drop down as a hey Can you can you solve this specific task? Can you calculate this specific thing? Or determine one specific program with a lot of different categories, right? So pico ctf as you can see it's set up more for like an individual view Uh ctfd is more of like blocks and it does it's It doesn't look as more maybe personal as the screen for pico looks It's more of kind of just like here are all the challenges So pico is definitely the one that's john talked about just like having the individual instances That is kind of what pico is more geared towards Does that sort of answer your question? Sorry Yeah, that's um, great I see that can be used both for breaking things, but also just testing general skills. Um, that's cool. I Have to think about ways to use that Well, thank you. Uh, the next question The victoria asks if there's any paid product that you didn't have access to that would have helped you run the event None of the ones that we found were I mean digital. Okay, so digital illusion does cost money So we did we did pay for those Yes, um My thoughts it's funny. No one has ever asked that question. Like how could money help you? um I think about the network visibility as one Area and I think gigamon I think I think those are really really solid on kind of visualizing some of that stuff That might help Uh, digital ocean, obviously. Yes hosting all that infrastructure takes some coin I think I've actually I think I got a little bit of headway today actually and Getting a little partnership or maybe some sponsors sponsorships set up, which I'm looking forward to But so far I We're like a one-man shop or a two-person shop, right? So how do we do this with open source technologies? How can we do this without breaking the bank? So So far If I had to answer that question, I guess I'd look into gigamon that Yeah, I mean we did this We we just did this to do this. It wasn't like a cost the only cost we had was the digital illusion um cost but For the most part like other than our time, which I guess like, you know time is money But we enjoy doing this stuff. So this was all done Um using any everything else we used was not a cost and all the tools we talked about terraform in that Uh, they're all free that you can use and implement. So An nsgl stuff that we are going to use in the future. So yeah Thank you. Thank you. Oh, and then how much did the digital ocean actually end up costing you at the end of the event? Oh boy. Yeah, so Monthly it's like I can say from a monthly perspective if you're just using it like day-to-day stuff Um, it's usually somewhere like five six dollars because that's what mine is on a monthly basis But I don't know how much yours was. Yeah So I feel really bad about this because a lot of people ask in order to kind of encapsulate or get an idea of Uh, how much should this cost? Uh So I was stupid and I totally blame myself for this when I Hosted the game. I had some digital ocean credits. I had about $2,000 in digital ocean credits Uh, and then I was able to use it without yeah I got that as a prize for a recent game and We started with that amount in the pool But keep in mind we ran seven front end and seven back end and then three other servers So suddenly we add up to maybe 15 or 20 instances running in the cloud um For the weekend for a 48 hour event I should have done a better job of Capturing how much that really cost and then after the fact when the players asked Hey, can you please leave these challenges up online? We're like sure absolutely Only then did I go take a look back at digital ocean and see how much that had eaten up? I would think It would I would think the weekend itself maybe would have cost 500 or 600 dollars I don't know and that's totally my fault, but we're close. Maybe hopefully Okay, thank you, um Brandon asks two questions uh Did either of you get into any of the darker hardening processes the the user space Yeah username spaces set comp app armor se linux anything like that We tinkered a little bit with app armor In some set comp I truth be told in some of the challenges. I don't think we needed to go that far It wasn't that Extensive in some of the some of the things a lot of the ones that would require Some of those on the system command line defenses were showcasing and highlighting a really small trick Like hey, can you find this hidden file nested in us however many amount of of sim links or can you Use just this single command to escalate your privileges and now unlock the whole rest the file system or something something like that Where we intentionally had a vulnerability So putting in some of the defenses like app armor or some of the username spaces It wasn't something that we needed But maybe some more complex stuff. We certainly might have to but nsjl. I think looks really good for that Cool, thank you the the second question Brandon asks is what type of monitoring logging or metric solutions you used for the infrastructure Whether tools like prometheus or elastic elk Yeah Again, again, that's like a fault of like well, we didn't have all that visibility for for logs and Even cabana or elastic. We just didn't have that in place Yeah, so I I've used Cabana Oh here, let me stop this so you can do that. Yeah um So I've used uh cabana on that before and that was something I had mentioned like the day prior to the ctf But um, it just wasn't something that I think we thought of again We kind of We kind of just like just said. Oh, yeah, sure. Like we'll host a ctf um And did very a very bad job of uh prior like thoughts in terms of like monitoring and logging Let me try to defend myself. Okay, sorry background When I like to do these kind of events I like to bring them to local security conferences or local events like like a b-sides or like an o wasp conference So we're expecting a room of 20 or 30 people. That's what I'm saying Yeah, and and this was the first time They said let's put it on the internet Let's let's open it up to the whole world So it was a crazy learning experience that I think we did a decent job keeping it afloat and just really awesome to learn How can we make this better for the next time? All right, thank you um The last question is you a lot to ask how do you fix the bugs in real time or Did you consider that off limits for yourself? It do you keep it broken and you just move on to the next thing? So no, we we did not keep them broken There was one challenge like we said we had to take down but most of the time fixing the bugs was We would see Users saying like hey, I think this challenge might be down or like I was testing challenges periodically Most of them we did have solve scripts for or I had gone in or john had gone in and Solved them already when we made them Because I had made a couple challenges john made majority of the ones for the ctf we did I've made more for the next one just because I didn't have as many in my back pocket john had done a lot of development years prior but um We would go in real time and and fix them. We we would uh run our script and see if it would break Go back in kind of to how Like notes we had on when we actually made the challenge and kind of go back through those if we couldn't figure out Immediately what was wrong and we would fix it and for All of them except for one it was successful. Yeah, um as I guess the guy on keyboard Seeing all the messages and notifications that would Personally like set me on fire Caitlyn was probably Doing a good job tolerating me just kind of mentally completely scatterbrained because I have 2 000 people waiting for me to go push a fix And once I if I could either say Oh, I know how to do this. I can I can I can crank this out and I can I can get this done I would do that and then we need to go drag it over to Back end 1 back end 2 back end 3 back all the way up to 7 So again, that was a limitation and we're looking forward to having run deck or some of the other automation to put that in place Cool. Thank you very much. Yeah, I just want to say thank you very much for coming and speaking with us. Um You know, we use the same tools to build secure services to provide to users and you're using those very same tools to build insecure Services to provide to users. So it's really interesting to see that that uh, That's spin of the take there Thank you so much for having us. Thank you. Oh, absolutely Peter would you like to close out? Oh sure. Thanks. Um, I'm as one of the people helps run cloud.gov and It occurs to me be interesting to do a cft just general challenges like Team see if you can as a you know at a conference or something say Come in and see how long it takes you to spin up this particular kind of service So people could build their their skills not as a cyber security challenges as a general Skill leveling up challenge. So, um, thanks for So there's all kinds of different ways. I'm interested in taking this. Um Anyhow, thanks everyone for for showing and participation. Um, we'll try to keep you all in the loop as to, um, when our next Meetups gonna be and we will get the word out. Um on many of the one lovely resources that you share Um, thanks all. It's yeah, don't have turn on your cameras and wave. Goodbye If you're like doing that sort of thing and maybe I can capture a screenshot that put on the um Put on the page Okay, waving waving waving All right Bye. See you next month You