 Hi everybody, welcome to our talk Building an ICS Fire Range in our kitchen, sharing our journey lessons learned so you don't have to. We have roughly 30 minutes to introduce you to our little project and give you some of the details and most importantly also of course the challenges and difficulties we had along the way of building a firing range. Just wanted to take some time before to introduce myself and Moritz here next to me and also Enviso the company. I'm starting with myself. My name is Nico. I've been working in the IT security for 15 years and I'm the Dependentration Testing and Red Team Lead for Enviso in Germany. Hi, my name is Moritz Thomas. I've been with Enviso for over a year now and I've been mostly involved in this R&D project that we are going to present today, right? A few words about Enviso as a company. I was originally founded in Belgium in 2013. Then in 2018 we opened the offices in Germany in Frankfurt in Munich and currently we're counting over 100 specialized security experts from all areas of IT security really. And one point I wanted to highlight here is the development and research and development of Enviso. From the annual revenue of the company we invest roughly 10% into R&D projects, which is relevant for this project because it's been partly funded as well by this, of course. It helped us a lot to develop as well and try out some things and make sure that they work as we wanted them to be. And this is how it all started was what made us or what motivated us to build the ICS firing range as it is nowadays. It all started towards the end of last year where we did some external training on ICS security, OT security and we realized that there's an abundance of skills and knowledge that we need to acquire and also to share internally in the team for our pentester and red teamers. Most of them, of course, they come from the IT world so they didn't have any exposure to OT environments and ICS systems so far. And as you can imagine, there are differences between, for example, doing pentesting against IT systems and OT systems with different requirements in either case. So there was a really, really strong requirement for our team to get trained in that respect to develop that level of awareness for IT environments. So we thought about what's the best way to do it to maybe develop a lab that we can use internally where we do have a couple of OT components where people can try out different techniques and just see how they react in that environment. So that was one of the main reasons and, of course, having a lab once that is already built, we can use it for research and development where we evaluate specific protocols, for example, or components. But also we can use it for testing itself. So if we engage in ICS security testing, we can maybe isolate OT components and test them in our lab that would be then ready for that purpose. The first concept idea was building a water treatment plant that is comprised of a three-stage water filtration system with waters left with three stages until it is then cleaned. That, of course, is all driven by pumping stations for the various filtration system stages. And in the back we have a fully virtualized IT network to also simulate the enterprise part that you would find in a water treatment plant. This is how it first looked like. On the left-hand side you see the schema of how the filtration would work. So the water runs through all the three stages and then ends up in that quality assurance part on the right-hand side where we do have sensors to measure the quality of the water. And if we determined that the water isn't clean yet, then we would go send it back to the first filtration stage and then run through it again. We did it for the proper concepts. We did that with food coloring. So we had food-colored water, send it through the stages, and then in the end it would be clean or not. And then it would reiterate through the stages as well. On the right-hand side you see the basic setup of the OT components as well that I included in that model. So we do have a Siemens PLC that is specifically... But we also have a couple of Raspberry Pi's running codices to simulate the PLCs that we might have in addition to drive the pumps, for example, for the QA control and so on. And I'm showing you that first prototype of an ICS-5 range. I'm just going to show you this video very briefly. Once we were in the purpose of building that lab, I think we were pretty much halfway ready. You've seen it just in action now. We were contacted by one of our partners, that's CWD, in the Netherlands. And they got to talk to Riksbatterstad, which is the Ministry of Infrastructure and Water Management in the Netherlands. And they required a model or a lab environment for their forensics team to train their techniques and also develop some of the skills that they need in their field of expertise. One of the things they do in the Netherlands is operating basket bridges, for example. So this was a prime example for us to use as a model because it's something that I think we can get an idea of how it's working. It's maybe something that can be simulated and it's also very visible then as well. The strict requirements were having a mobile solution because they wanted to move it between sides. And then, of course, come with a workshop that is a scenario base. So the basket bridge that we were meant to build should be used in a way that is versatile enough to run different scenarios for different forensic teams, for example. And this is where the journey began, of building this bridge. And Morse is going to run you through the concepts and the details of how we came to the model that we currently have. So right, as Nico said, we now changed from the first context that we had in the first scenario of water treatment plant to the basket bridge. And it just turns out that this is, of course, part of critical infrastructure and it's not that easy to get good information about this online. So I would like to give a very short but brief and very nice shout out to the lovely folks over at the Florida Department of Transportation which were nice enough to make a bridge maintenance reference manual that is probably freely available. And this really contained a lot of useful information. So thanks to those guys. And here you can see our first 2D concept drawing that we came up with. So here you can see we basically have two sides of a bridge. They both have a leaf and a counterweight and a ground. And on top of the road, we also have some traffic indicators. We have a traffic light and we have also roads, barriers that then should actually block some cars from moving, right? So we had this idea and we then transferred it to a 3D concept. So how were we going to build this? Here you can see a very brief, a very, very short and simplified representation of our 3D concept. So here we got an aluminum frame on wheels, basically. It's divided into two parts. In the front you have the OT components mounted onto a steel plate. And in the back there are some black wooden panels now. But in the back there's space for a virtualization server that we can use to post different networks and different systems on. And on the top there we have our 3D model of our scenario, those bridges that we transferred from the 2D concept drawing. Well, we had this, but we wanted to build it, right? So we did this using the lovely magic of 3D printing and due to COVID we couldn't do this in the office and we wanted to do quick iterations though. So we had to do this at home and in my kitchen actually, much to the dismay of my wife, to be honest. It was noisy, it took up a lot of space, but it was worth it to be honest, it was really cool. So what did we do with that? Well, we started working on the mechanical challenges. Here we can see a very simple axle holder, right? Here we've got a bearing that is mounted between a clamp or more or less and that's fixed with some screws. Here are actually two iterations of the same part and this actually presented some challenges to us. One of those being, okay, if you want to use screws, how big should those holes be? Because you don't want the screw to just fall out again because it's not tight enough, but you don't want to have the plastic break because it's too tight, right? Also, you need to find out, okay, how can we hold other parts with 3D printer parts? So we had to figure this out. But once we did figure this out and we're satisfied with the results, we can then just proceed with other challenges. So how would we use two of those axles and do, so how would we print two of those axles and connect those to something in between that can actually make use of this axle? So what you can see here is more or less the center part of a bridge leaf and then we also want to control this, right? So we have to attach some kind of stepper motor to it and then you have to engineer some gears that you attach to it. And let me tell you this took quite some iterations, but we kind of got it really, really nice. So, but when you 3D print, you will run into problems and so did we, of course. One of our problems was just the infamous spaghetti print. This is just a result of very poor bed adhesion, as we've learned, and we just solved this by using a mirror as a print bed and turns out that this greatly increases bed adhesion. Another problem was warping. So here a 3D printer part just cooled down mid-print and then it came loose and went places. Turns out if you make sure that you maintain a steady and constant temperature in the environment, this will probably not happen. So what we did was just get a nice and big enclosure for this 3D printer. And we also did some tests with ABS and PLA and found out that if you, especially if you mix between and change between those filaments, that you will run into problems with your nozzles. So just changing them may help mitigating the problem of irregular extrusion of filament. And this was really a problem. But yeah, eventually we got to a point where we could easily and steadily print new parts. And here you can see our CAD model of our vision of what the bridge should look like and actually some printed parts of it. So those were very nice wins once we had those and once we could assemble those to a full model. And this actually is our full model. But what do we do when you have this full model? Of course you have to put it all together, right? And that's what we did. So we got this full model actually we built to and we printed to which took hours and hours and hours. But then of course we had to assemble it into a finished and then complete and then also usable in mobile ICS lab. So what you can see here is our aluminum frame structure with the metal plate for the OCS components boots onto it. And then of course we installed the OG components onto it with an awful lot of awful wiring to be honest. Then we just installed the 3D models on top of it using some spaces that we printed and designed and bolting those to the wooden plank there. And just for fancy points we added those panels also that you cannot reach into it during operation of course but it looks fancy in my opinion. And of course you need some lighting, right? Because if it's not very, very, very bright inside then you also want to have some lighting in there so you can actually take a look inside. Or else it's really gonna get really dark in there. And then lastly we did a lot of debugging. A lot of software debugging and hardware debugging just aid up a lot of time. So really, so if you want to build something similar something like this just plan ahead of time that you will use a lot of time for debugging. Stuff will break and this will take a lot of time to fix. But let's go into detail about the components that are in there, right? So first of all maybe most importantly we got the HMI for the bridge control. This actually represents all the movable parts of the bridge model on top, right? So for everything like the lights and for the movable parts like the bridge leaf and the various QCOTS individual controls that actually display their current status, color coded, text. So that's nice to have. And actually we pre-programmed it so that you can tap onto them and control the whole setup using just the HMI. Of course this needs to be powered, right? We got power supplies for five volts, 12 volts and 24 volts. These powered then our PLCs. Here we got a big S7 1500 PLC that coordinates the other PLCs. We got the S7 1200 PLCs that drive all of the motors and the S7 300 PLC, the ledger C PLC that controls the lights. And of course an awful lot of the wiring was due to the motor drivers that leaves in barriers. And if you could see the insides of those cable tunnels you would be shocked to see how much cable fits into such a small place. Last but not least, we also got some raspberry pies for the CCTVs that are mounted on top of the bridge model. Now having the 3D model printed and all the wiring done already with the OT components the next step in that process would be the entire IT infrastructure in the back. As you remember, so the idea of this step is also to have it for workshops, specifically for forensic workshops. So we do need that simulation of an enterprise network for example and a scalar network to realistically work on a text that might have happened against a basket bridge to then investigate what was going on in that case. So in that model there is space for a visualization server and what we did is deploy this infrastructure on it for that IT infrastructure part. So comprised of the enterprise network where we do have a domain controller. We also have office workstations and a visualization server which then gets its data from a historian that is running in the scalar network in the top of this slide here. In the scalar network as well we have the operator workstation where we can observe and supervise the bridge and make the status of the PLCs for example but also control it remotely via the HMI. We get access to the camera feeds of the CCTVs. Next to the operator workstation we do have the engineering workstation where we have tier portal deployed the tool to then deploy the ladder logic to the PLCs but also maintain the OT components as such. From there on we are connected to the lower levels of the bridge infrastructure so this is not a virtualized anymore that is where we actually have connected the PLCs with the centralized PLC in the area supervision control level including the HMI that you did see on the screens before for the manual but also automatic control of the bridge components. We do have the CCTVs that are essentially just piecams connected to the Raspberry Pis of that model and then on the lower levels we have the three different PLCs to control the barriers but also the leaf motors to change the lights from green to red and so on. This entire IT and OT infrastructure then serves as the basis for a scenario that we carry out in preparation for a for instance workshop. The idea was really that we go through all the levels of the purchase model to simulate an attack as it would most likely be in a real life scenario so you can assume that for example a workstation in the enterprise zone is compromised through an email where then the attackers deploy beacons that communicate to a C2 server somewhere outside then at some point the attackers won't say they got that initial photo they escalate the local privileges and do some credential harvesting to finally then get access to the Scala network where we do find the operators workstation but also the engineer workstation with the tier portal on it. Assuming that maybe it's not fully patched that could be one of the scenarios then we can exploit some vulnerabilities against the engineering workstation gain access to it gain also access to the leather logic we can then deploy any arbitrary a quote really on the PLC set or some disruption to the bridge operation but also maybe severe harm to actually the people using the bridge so just imagine that if cars are driving over the bridge you slowly lift the leaves so there might be accidents happening then in that case that's the scenario idea of our ICS firing labs really that's an attack that always took place and then in the next step the 4N16 would get their hands on the lab and would need to investigate what happened there and so much talking now we're going to show you the lab now in the demo so the relab that we actually build and just give you a few ideas on what's going on there this is the complete 3D model now with all the OTE components connected you do see once it's powering on you see the HMI turning on you see the PLCs already blinking they're also booting up they're all connected by Ethernet together also with the server behind the display that you see there now we should see the HMI UI also being displayed in a second there it is and this is now where the bridge automatically starts into its cycle so that is just for demo purposes right now we're just cycling through the entire process every 30 seconds starting with the leaves lifting and then the lights going on for the ships and boats to pass through the bridge you can see in the HMI on the left-hand side that this is now green so that means the leaves are open the traffic lights are green so the ships can pass and soon the lights will go red again the leaves lower and once they have done that and in a second we'll see the barriers go up just any second and now the barriers going up you can't see the lights for the cars unfortunately but you can see it in HMI that they switch green as well and now the cars can drive this is the automatic mode you can at any time go into the manual mode and then operate any of those components manually really so you can just lift one of the leaves or the other one and then the barriers whatever sequence you like so let's get to the lessons learned of course there were quite a few lessons to be learned to be honest well let's start with the ICS lab setup to be honest designing this stuff yourself you have to account for the assembly of how you are going to reach parts of how you are going to connect them of how to put them into place we didn't really take this into account and then had some problems some issues with really complex assembly it just took a lot of time maintenance was hell don't do this think of this ahead of time this really pays this is really good during research of course we had to find out then which hardware has which dependencies and which compatibilities this really required a lot of digging through data sheets you have to acquire those you have to read those you have to understand those you know if you're not an expert and we have no means experts in this field this will take your time to take this into account too and of course if you want to use industrial hardware you will have to pay for licenses especially the IDEs they come with heavy licenses and a practical issue just was that we managed to kill two stepper motors by overheating we supplied too many amps and did not really get rid of all the heat didn't manage to dissipate at all and they just died on us this was really, really bad because then you have to order new ones and you have to install those again you know as I said the assembly was not really straightforward so this just took a lot of time it was unnecessary but hey this happens regarding the printing as I pointed out earlier there will be mechanical design that will be challenging if you're not a mechanical engineer by no means we are mechanical engineers what you saw in our setup was not very, very, very complex I mean it was just an Excel and two gears that drove this Excel you have no idea how long it took to just build this and to test this out until it just worked so if you're not very familiar with it again think of it ahead of time also if you want to iterate quickly you have to take into account that printing is really time consuming overall just one side of those bridges took multiple hundreds of hours to print like they were about 200 hours to print just one side so the whole setup was about 400 hours of printing time so this is insane for quick iterations well print small parts and test them right also if you're not familiar with CAD yet well I suggest you will because this is really the preferred choice you can parameterize everything and you can make quick but very consistent changes to your designs and you can just iterate better with this in my opinion it's better suited than let's say creative software like Blender for 3D modeling because this will give you other problems right and to add some numbers to our project well we started in January 2020 and we invested more than 1,000 hours of manual work into this and more than 900 hours of printing time into this well I just said we it took about 400 hours of printing all the components that make up our bridge scenario well of course we had to do a lot of prototyping I had a huge box of failed prints or prototyping prints still in my kitchen laying around there taking up space again but anyways we also processed about 15 pounds of filament so take this into account as I said the socket licenses are kind of pricey well they are we used the TI Porto first for the Siemens POCs which cost us in total 3,500 US dollars all the hardware combined cost roughly well 14K US dollars and you know for all the coffee that we just consumed during this project it was roughly about 570 US dollars I know it's only specific but it's just a rough estimate right and then of course we broke stuff as I said we broke two stepper motors we actually managed to break one POC and today just until today I'm not quite sure how I managed to do this but this happened after I replaced it everything was okay so I think that's nice one motor driver died in the process due to wrong wiring but hey that happens and of course at several occasions we just lost our sanity on the line right so if you're going to go into such projects and work on those and you are not that familiar with all the technical aspects well you will have problems and you will pay with your sanity I hope that you got a good idea of what we had to go through in the past 6 or 7 months and I hope that you found this interesting and maybe there's something for you as well to take away from it in any way if you are interested in this project specifically or if you are working on a similar project and just do some follow-up questions feel free to reach out to other moors of yourself anytime also I wanted to point you at the invisible region to ICS securities we do have that ICS page there please have a look at it maybe there's something also for you there and again reach out to us if there's more you want to know about and last but not least we do have a couple of blog posts coming out for this ICS hiring range so we want to go more into detail of how the development went show some of the close-ups maybe some of the schematics and how the wiring works more specifically how the attacks in order that we envisioned comes together in that lab and also with some more detail describe the actual difficulties we had just before I worked here to see and share with the community as well thanks very much for joining and have a great day