 Next figure up is Brandon working team. He's from security matters, and he's going to talk a bit more about CTFs that here in place and also in CTFs in general challenging the next generation of ICS security professionals, so Thank you Larry, thank you Larry Appreciate the opportunity to speak here. You'll notice there's actually two people on the Title slide that's because my colleague Harry Thomas Actually, it was instrumental in creating the CTFs that we're going to talk about His wife is about two weeks away from giving birth to their first child And so in the interest of not getting divorced. He did not come to DEF CON We are both ICS security engineers at security matters We're a company that focused on ICS security and we're happy to be a part of The ICS village this this year We're part of the CTF right next door So what I'm going to talk today about it's kind of three points. Why did we build the CTF? some problems and solutions that we ran into as we were doing this and Especially this weekend there were several and some tips and tricks for That last part is really focused on How you can use a CTF to be more educational rather than just a contest So the first question is why build the CTF? I Looked at the DEF CON Web page the contest page and there's 29 CTFs at DEF CON listed on their Contest page that's here. So why did we do another one? that's a lot well the main reason is because ICS is a very specialized niche and a lot of CTFs are focused on really that high level person. So for example the DARPA cyber grand challenge It was that DEF CON a couple years ago that was focused on artificial intelligence and you really needed to be a Expert level person in both AI and malware and all that in order to be successful at that or even to compete in it The DEF CON official CTF or whatever they Call it. That's the same thing. It's a contest to find the best at CTFs and Then there's niche ones like the social engineering capture the flag Where? ICS is not really talked about so we wanted to create something that was really focused on ICS For one one huge rate reason some other reasons But the huge reason is because there's this well-documented phenomenon that the talent shortage of ICS security professionals This little tweet thing is from a couple years ago now, but A.L. Peterson it runs ICS focus security conference and his guess was 5000 ICS security professionals in the u.s. With most of those hidden away where we never hear from them What this means is that it's a really small community and so I always like to throw this one if I'm talking at a place It is not necessarily ICS security people, but maybe General IT security people I like to point this out because it's a small community You can get involved you can meet people really easily and it's just a great a great way to get involved I personally have only been involved in the ICS stuff for about five six years And within a year I knew Most like a ton of people who gave talks at Def Conn and stuff on ICS security because it's such a small community It's a really great community to become a part of I've talked about ICS one of the problems when you're talking about ICS security is industrial control system security, so there's lots of acronyms If you come from an enterprise security background, I'm sure you deal with that in other ways But we talked about IT and OT so when I said IT security that means your traditional like sequel injection attacks or password cracking things that a Hacker a generic hacker might do that There's some things that we that a security professional Who's comes from the IT world? May not be able to get away with if they come to the OT world such as some devices if you scan them in the OT world They will fall down and cry into a little ball on the OT side It's just a different skill set because they have different requirements And there's other people here this week are talking more about that, but I just wanted to point that out There's some ICS focused Capture the flag stuff Reed Whiteman is right outside. He does one at the S4 conference every January There's some other ones, but there's not a whole lot of them. So we saw this We saw a kind of a opening for something like this So we built our capture the flag we call it the lights out hacking challenge Defconn it's a little different because we're a part of the ICS village capture the flag So we have flags and all that kind of stuff that you get points for in our other events where we do this Like this was from B sides Orlando and March maybe We call it the lights out hacking challenge where the winner is the first person to turn out the lights in Gotham City To turn out the lights in Gotham City, so that's Mike Mitchell was our first winner of the lights out hacking challenge We'll hear from him at the end of this presentation These are all people from the college students in Florida. So we were very happy that they Participated and won the first CTF we did Now I'm going to go into like the second area of my talk the problems with creating an ICS focused CTF Just in case you want to The first problem I'm going to talk about is the operating systems when I did CTF at B sides several years ago with the pros versus joes Everything was virtualized because you're attacking and defending Linux and Windows boxes that kind of thing In the ICS world we don't run many times. We don't run just your traditional Linux Windows Mac operating systems We run proprietary operating systems that do a certain thing Well, hopefully they do it. Well But they but it's it's a different operating system. You can't just go use the same the same That tools with them and so virtualizing them can be much more difficult So you kind of have to have some real-world stuff. It makes it up much better of a CTF And that gets into the next problem, which is money I See us devices can be super expensive. This was off the rack I just googled Siemens s7 device and took the first price. I saw $4,000 That's out of the range if of somebody who wants to just build a CTF for their local maker space or their local b-sides or something like that So that was the first problem that we had to overcome And so we did that with our business credit card because that's how can solve a lot of problems We Created that our lights out CTF. Here's the picture of it so we can look at something prettier than a credit card We created this and it has several different use cases. So one of them is CTFs at b-sides Here we use it for CTFs another use case is for trainings and b-sides Las Vegas just this week my boss gave a one-day training on network security monitoring focused on industrial control systems and This was a non-vendor specific. This is a non-vendor specific Using all open-source tools that kind of stuff just as a public service type thing It sounds kind of cheesy, but we we want to give back to the community in that way But then we also can take this to trade shows Which if you're which we take the same things we take it to a trade show and we do demos for Potential customers and so by making it a business thing that we use for business. We were able to build this Actually, I should say cyberacle Do we see cyberacle? Oh Don't quite see and there it is cyberacle built this they're an engineering firm based out of New Orleans who? We contracted with to build to build Gotham City for for us And then we use it to by using it to drive that business use case We're able to then also do things like those b-sides trainings and capture the flags and that kind of thing The next problem we get into is shipping This is a big problem sometimes like yesterday I'll get into that in a minute Shipping this can be kind of expensive costs a lot of money to ship a giant 150 pound box across the United States We did we put it in a pelican case. We have multiple pelican cases with custom-made Liners and all that kind of stuff to make The model fit the model sits in the liners and stuff and then we have a portable server rack that you can see outside Or on the other side of the wall you have the portable server rack that's in a pelican case To try to minimize any damage that comes from shipping Sometimes it's unsuccessful When FedEx loses a hundred and fifty pound giant black case. I don't know how they do that but They did and it ended up being a good thing because we were able to then make version 2.0 of our case and We were able to switch out We started with a Dell 1u server Which was fine, but it didn't handle the shipping very well And we were able to switch to a super micro Ruggedized box we were able to put a new KVM in it that was more ruggedized So we were able to kind of upgrade Learn some lessons and take the insurance payout from FedEx and build a new box So version 2.0 would be more More hearty as we shipped it I'm sorry How many four thousand dollars Siemens boxes were in that? our great question Come out and we'll show you we have a PL and Alan Bradley micrologic PLC SEL Schweitzer engineering r-tech and a Schweitzer relay Doing them. Yeah We ensure we ensure we ensure it every time so FedEx cut us a nice check They actually found the box But and after they cut the check and so they told us we could either send the check back or Or Get our thing back when we're like, well, we don't know what you did with it for the last two months So we better just who it might have been sitting outside or whatever. Who knows? Micrologics is a brand name for Alan Bradley. Yeah, Alan Bradley. Yeah So Shipping sometimes helps you but it can cause problems And then anytime you do something like this with technology You're going to have technical difficulties and you're going to have user errors and we've we've had both of those We went to when we went to our first CTF at besides Orlando. We got there and our Wi-Fi router that we use for participants to get on Was broken and so luckily my boss is always prepared and he had his travel router with him And so we were able to use that at the second at that first CTF I Told we actually have another person who works in our company who's Related to my boss and I told her about that that I was putting the Boy Scout slide up and she said, oh, no He's a Coast Guard person. They're always prepared. I never knew that Coast Guard Coast Guardian, I don't know what you whatever you call a Coast Guard person He's he was in the Coast Guard and so he's always prepared of course then we went to our next event and He was not there and we didn't bother to buy a router. So that one was definitely user error on that one So again, once again, we have the business card and we have Amazon to our delivery to the Denver Hotel So we were able to solve that problem by spending more of my CEO's money And then we have this week and I've talked to some of you about What happened yesterday and today we came in yesterday at one o'clock in the afternoon to set up our CTF like we planned to and That right there is where that super micro one you recognized computer is supposed to be This was after we started looking at it for a while. That is where it is That's where its RAM is and that's where its hard drive is It was shattered You can see the weld the metal welds that came apart from FedEx shipping our box to us That we found out yesterday at one o'clock in the afternoon that it was broke totally broken So if you're going to have that happen somewhere This is the place to have those kind of issues Because my boss did a training at B-Sides and he had somebody he trained there who he talked to a little bit And this guy was going to come test out our CTF for us run through all the flags make sure everything was working wonderful instead he showed up with three of his friends and this guy is a Hardware like genius the guy I think the guy in the blue shirt that you can kind of see in the back Is apparently Ubuntu genius and they With my boss they went to fries they bought Ram motherboard everything they built a server for us last night, which is why we were able to do the CTF this morning We've we got everything networked and working all the VMs on it at about 945 this morning for the 10 o'clock CTF After working late into the night last night And honestly it would not have happened without these random people from DEF CON who we never met this guy For eyes did not want to sell my boss the two 32 gigabyte Ramsticks that we needed for it Because they were the only two they had in the store and the only two that would work with that motherboard And so one limit one per limit so he put He put one on his own credit card For us so we could buy that and do it. I mean that this community is phenomenal, so So but everything's working now, so now we're into it. We actually have a CTF going so one of the problems you have with a CTF is Having to protect the devices So what we did was we enumerated all the attack paths that we might have We determined which ones provided too much access to the players And then we limited the risk from those attack paths and we did that in a couple ways The easiest way maybe not the most effective is to make it out of scope So like we said don't take out the host hypervisor We're not in the business of defending hypervisors So that's that's out of scope for the contest most people are pretty cool except for the guy who was Messing with everybody's DHCP server earlier this morning But most people are pretty cool in this kind of thing and so they respect that kind of thing The other thing we did is we made the attack vector too difficult to compromise. So for example We have admin panel that might have a 30 plus character path randomized password because in a two-day CTF realistically nobody's going to crack that in in a CTF so Even though we say nothing is unhackable Realistically for the CTF. That's a that's something that we don't have to worry about being compromised. I'm sorry I'm sorry Yeah, the length so 30 plus I said yeah, that's why I didn't say 27 or 36 All right, so With our lights out challenge of and the title is talk is creating a CTF that teaches With and that is near to me. I used to be a teacher. I taught middle school in high school for several years My boss does trainings all around the country like you said he likes to teach people like I said earlier He likes to teach people we want to do things that really Drove the educational side of this so that if we brought the CTF to somewhere like this people are getting introduced to the ICS devices So I'm going to throw a couple educational terms at you So the first one is scaffolding so just like you might imagine on when you're painting a building the scaffolding builds you up So you can reach what you're doing scaffolding and education is providing supports that allow the students to be successful So for instance My colleague Harry During the first CTF we did he put out the tweet about our friend crusty has taken up a new head hobby steganography Here's a tent a hint to somebody the order to the participants. Let them know. Hey If I find a picture, maybe I should look into some what the steganography thing is So we get we give hints we we work with people to build them up Another thing we we do is a guided questions. So we like to talk to the participants What are they doing talk to them about what they're doing talk to them about? Where they're having problems, especially like I said in Orlando when we have the college students That was like we were in Nirvana with that one We can ask them what worked So that they can kind of get reinforce if you're talk through what you've done then that helps you remember it better So they can that helps them Give them ideas for where to what to read up on when we close for the night Might talk to people about that. We've done that in the past. I kind of give the Those guided questions to kind of give people hints on where where they should be looking Another thing Another kind of tactic is reinforcement We can introduce skills early in the challenge and then we can Build on them or require more advanced use of those skills later in the challenge so for example if we have the password prison in our For something and no, this is not one of our passwords We might have that as a low at early in the CTF then later on We might change that password to something where they have to munch the password a little bit and we can talk to them About how they can automate the process of changing out those characters making very password variations on how to so that as they They use a skill then they come back learn a little bit more about that skill and build on it later on So that kind of reinforcing as you go on through the process Another thing we've done is collect traffic captures and Honestly, this is partially Selfish because we like to see the attacks are going on because we're in the security business but Participants can see what they've done and then this one's really big for for me when I've done I did a CTF a few years ago where That was actually focused on this this was the only thing they really focused on was seeing artifacts of what you did It's very helpful when you do an attack what network artifacts are Occurring on the in the P caps that the defender is then able to see So that if you're on the red team side you learn what you have you might be it might learn how to Avoid those artifacts or and if you're on the blue team side you can learn what kind of checks you might want to look for And it also allows us to improve the CTF over time We the more data we have about what worked and what didn't the better we can do so that hopefully things improve over time So I mentioned the guy from Orlando Michael Mitchell was was his name the winner of the first out He was the one getting that challenge coin He said one of only two CTFs that I have ever participated in that were designed to teach Participants rather than test them, which was awesome. That's what we were going for The little time I had to compete in that CTF taught me far more than I was ever expecting so That's what he put this out on his blog after after the fact It would that was the best feedback we could get that Honestly, if somebody's like Larry or Reed Whiteman next door if they win our CTF I think we might be doing something wrong because our goal is to introduce people into the ICS world not to Just prove who the best ICS hackers are So thank you Just about out of time My name is Brandon working team my invisible colleague Houdini infosec Houdini Like I said, I could not have done it without him for this weekend I could not have done it without the DEF CON community stepping up and helping us save that server that got destroyed by FedEx Please go check out the ICS village CTF or if you don't want to do the CTF check out the demos There's like an airplane cockpit or something that I saw in there. I mean, it's it's insane what they have here at DEF CON now And that are there any questions Yes, sir That we do not have a Siemens device in our portion of the CTF I do not know if any other groups the way the ICS village CTF is and there's like three or four or five Different people who all created parts of it. So the part that we did we have the Alan Bradley device for Alan Bradley micrologics PLC and Then we have an Schweitzer engineering Relay and Schweitzer engineering our attack that so the Schweitzer stuff is the blue box If you look at our demo rack the blue box boxes are the Schweitzer stuff. They control the electric city part The PLC that's sitting out on the table because it fell out of the server rack is it controls the ping-pong balls flipping up and up and down We can't afford hardware stuff like you guys have are there any software or virtual machine? Great question. He asked if there was any virtualized Integrated things that he can do for like a college cybersecurity club the first thing that pops into my head on something like that is called side body CYBATI That's run by a professor from I believe the Paul University named Matt Lou Allen, who's a great guy. He's been at the village before I don't know if he's here today as well It is the last time actually the last time I downloaded it was probably a year and a half ago And it and it was like 28 gigabytes or something like that So it's a hefty VM, but because he because he has that educational background as well He has a virtualized network with a bottle-filling thing so you can make It's it's a virtualized thing you can see the little bottle going across the screen he has Reference material but the reason that the VM is so big is because he has a lot of reference material built into that so that you it's all ready to go and That you you get it through a Google group So it's but Matt Matt Lou Allen CYBATI is probably the top of the line virtualized thing I've ever seen Yes again, yeah so the The for something like that in the virtualized environment I think the biggest benefit is it generates the traffic captures for you So you're able to then see what what is happening So you're you're able to see even though you don't have that real-world Experience of actually working with a real PLC you're able to see the network traffic Yes, yeah Coat so he mentioned that codis this CODESYS Has a Raspberry Pi version of their operating system. So you can get Cal Cali has a Modbus module I'm sorry Velasio. Yeah Yes, and the Yeah, any other questions or comments? Yes, like people who do that that you want to talk to It's a pretty small community, but I think it's a very accessible community. So There are people who do that and there's people who do red team and that kind of thing We're the top company Well, we are blue team. No, we're we're completely passive blue team. So Red seal is one Most of it. Yeah. Yeah, talk talk to people over there. Yeah. Yeah, talk to people on the other side of the wall Any other questions? All right. Thank you very much for your time. I hope you all have a good con