 Hi, my name is Dieter Sarrasen and I'm going to tell you a bit more about fortifying ICS environments by applying hardening and by performing security testing on those environments on those hardened environments. First little bit things about me. I'm active within information security since 1998 and within ICS security since almost 2008. So, mainly I'm doing or challenging ICS vendor solutions to verify cyber security settings and I'm also assisting in performing security and start testing for those vendors and customers. I'm also assisting and customers by applying hardening to those environments or applying hardening to the vendor solutions they bought or they have. Now, a little bit more about fortifying ICS systems. So, it's also known as hardening, of course. It's actually the process in which the balance between usability and security of a system is researched and enforced. So, why would you think of doing hardening on ICS systems? First of all, well, the more the systems are interconnected, the more there are potentially attack factors of those systems. So, hardening is the process in which we can try to reduce the attack factors of those systems. We can try to reinforce the exposed systems if there are any. And also, securing remote access or limiting user access or verifying or making operated jail protections more foolproof so that data cannot escape from the operated jails in question. When would you do hardening? Well, during projects, of course, when there's new systems applied or when there's refurbished systems into the environment. Or when you're upgrading or applying or installing a new software component or hardware component to the environment. Most of the times, if you cannot do it during projects, you can try to do it at the next maintenance stop. But then you'll probably have to have done some hardening verification before that. As most of the times the next maintenance stops are sometimes too limited to perform the proper hardening and testing of the hard environment. Another reason why would you or when you do hardening is to close identified security weaknesses. So, if you perform security testing within your environment, you often, almost always, you'll often find security weaknesses. To close those identified security weaknesses, sometimes patching can be insufficient or not possible. That's the time when you would apply hardening to those systems as well. How would you do hardening? Well, performing hardening can be done in three big different ways. So, manually, first applying every single setting by manually changing register keys, removing software, applying other settings to your system. Or you can create scripts to do so, to assist you in doing the hardening. Or you can use group policies. That can be either local group policies or central control group policies, if you have, for example, next directory. So that's the main difference between the main join systems and not the main join systems. But in both occasions you can use group policies either locally or centrally. And a big question now is how not to perform hardening. Well, actually, hardening is not a click and play configuration. So, you cannot just take a script, run it and hope that everything still will work. That's something that some of the vendors I've been working with during hardening processes are doing it wrong. They just take the scripts that were published for a certain hardening guideline. But they didn't look at any setting within the hardening guideline, so they failed to understand what settings were applied or going to be applied by using the script. So, an example here is that a hardening guideline I created for one of my customers was for a completely isolated system. And the vendor was applied to that hardening guideline and accompanying script. But that was for a connected system. So he applied the completely isolated hardening script to the connected system, resulting in a loss of connection over that system. So that's why I mentioned hardening is certainly not a click and play configuration type approach. That sometimes works, but in most occasions you run into problems. Before we continue further, you should ask yourself your questions. So what is your actual hardening strategy? So what systems or environments will you harm? And what systems or environments will you harm first? So those are questions that need to be answered before you dig into the further hardening process. There's also a list of potential problems that you can run into during hardening. A lot of things can go wrong. For example, the whole list here is no more access to user interfaces, no more access to file shares, printers. So no more access to file shares is actually one of the most occurring issues that run into during hardening. Because, well, they want to have file shares, of course. HMI applications can stop working or your system application can crash. Even sometimes you can run into domain-wide problems. So the main can get down or something like that. Another thing that you keep in mind is that time differences between systems need to be sorted out properly as well. Certainly, if you're using ICS applications or HMI applications, the time needs to be almost completely the same actually to prevent information and data loss. You can lose touch screen functionality if you disable the touch screen functions in the operating system, of course, and operators can lose access. So the whole list of things that can go wrong is the key is here that you should solve that particular isolated problem or you should revert to a known good state before you start applying the hardening. Sometimes you have to live with some settings that are not as strong as they should be for a particular environment. So sometimes you just can't do anything else by accepting those settings. So what's less likely to be hardened? Well, more and more black box systems because those are often used as guarantee buy-ins or lock-ins. So the vendor lock-in here can cause guarantee issues if you try to touch or even change settings on a black box system. So that's something to keep in mind. So the best thing to do then is try to mitigate the potential issues of that black box systems in another way, but we'll come to that later on. Other systems that are less likely to be harmed as office clients because they need to be able to do anything by using legacy systems. It's often the case that people are really reluctant in touching legacy systems because they do not know the system. They cannot predict what's going to happen if they come through something on a legacy system. Here's that little flow chart that I created for potential problems during hardening. So what if you cannot perform the hardening? Well, you have at that stage an insecure system. You cannot do the hardening. So then the first thing to check is can you do an upgrade on the system? If you can do, well, fine. You'll end up with a secure control system. If you cannot upgrade, you might want to look at if you can replace it or not. If you cannot replace it, well, then you have to investigate if you can isolate the system. But if you cannot do anything of those, so no hardening, no upgrading, no replacing, no isolation. Well, the thing is then what could go wrong? Well, a lot of things can go wrong, of course. So what things would you harm? You have identified that you want to do hardening. You have identified the systems that you want to do hardening, when you will do the hardening and why you will do the hardening. But now the things that you want to do hardening for, so what you have to define those as well. So actually, all things that could pose a risk to a system are candidates for potential hardening. Meaning software, all installed software packages, not only the operating system software, but also additional packages, services. So it's not only enough to stop the service, but it's also key to investigate when or if you can disable services so that they cannot start up again after you have launched something else. So account policies, user rights and permission onto the operating system, network interface settings that is very often forgotten about. That a lot of network interfaces keep their default bindings, that's more on Windows systems, of course, their default interface bindings enabled. Even if they do other style hardening, those bindings are left to left present. Remote administration solutions, if there's not a repeat that's used, often you have erotmen, you have TeamViewer or other solutions out there. TeamViewer might not have been the best example I gave, but let's take erotmen. For example, it's often the case that default erotmen installation, it might change already, but default erotmen configuration is not always the best configuration for your environment. Also not to forget as peripheral port access, so USB ports, serial ports, whatever ports there are there that are not used should be disabled, of course. You don't need, for example, USB data storage, you should disable USB data storage. If you do not need to install additional USB drivers for keyboard mouse or Ethernet dongles or whatever, disable that possibility. Just to prevent people from attaching other USB devices that get recognized on the system that you can get access to the system by using those. Operating system access by using operator gels. I gave a talk on that a few years ago in Stockholm, but operator gel escaping is almost always possible. But it depends on the service account or user account the application is running with that will determine your access to the further operating system below. So that's why applications first of all never run as administrator access. For example, we used to need administrator level access, so that was a problem already. But by further protecting the operator gel, so by limiting what the operator can do onto the operating system will further limit possible attack paths into the system. Now, the learning process. First of all, you begin with choosing the right operating system for job. Yeah, normally you would have been able to select a good operating system based on what the functionality needs for the system. But very often you are forced into certain operating systems based on what the vendor solution requires. So today this is unfortunately very often Windows based operating systems, although the last versions are getting more secure, which is good, of course. Using custom operating systems. Well, that's problem when those systems have a lot of functionality embedded, which are not always disabled by default. That's why we do the hard right. And of course, sometimes you have to sometimes you have to live with using obsolete operating systems and software. So sometimes you cannot do anything else besides just putting the system onto another environment and making sure that nothing else can happen to that system. One thing that you should not forget before doing the hardening is taking backups and, of course, testing those backups to make sure that if you have done the hardening and something was wrong during hardening, you can revert to a known good state, as I showed a few slides before. After you chose your operating system, it's a good thing to analyze the necessary functionality and the current security of the system. If you have an up-to-date operating system, the current security of the system without the application installed should be pretty much okay, but still might need some additional hardening, some additional security settings depending on the location and the criticality of the system. But as you will introduce potential applications, HMI applications, DCS applications, whatever application is required for functionality of the system, you will have to recheck the functionality and recheck the current security of the system. Recheck the operating system applications, what data flows are required, what sources are needed, and then, of course, you can try to determine what can or should be hardened. So this, of course, depends on the identified security issues, if there are any, identify on what's needed for the system. If there's software that is installed, you should check if that software needs to be removed or upgraded before you do any hardening to make sure that you have on software aspects and the pre-system aspects the last updated environment for your software set. Then, of course, do the hardening. It's often in a trial and error style approach because in most occasions or in a lot of occasions, you will not be able to determine what things or the vendor might not be able to determine exactly what's needed for the hardening or for the environment and whatnot. I've been in that situation when I helped hardening for a solution vendor. It was done through GPOs and we had to change GPOs, I think, five or six times because the vendor even didn't know his environment properly. After you do the hardening, after everything is running, first thing is, of course, verifying the functionality of the system, making sure that everything is running properly, is working properly. But then it's also key to verify residual risks of the systems after you have applied the hardening to making sure that after hardening, the system is good, is secure, or as secure as possible, and try to identify what you can do with those remaining risks after the hardening. A big thing to understand is that whenever there's major system changes to the hardened system, you have to verify if you do not have to reapply hardening after that. Because in a lot of occasions, installing patches, installing operating system patches or application upgrades, some settings might have to be reapplied or might have to be added to the hardening set to make sure that the known secure state is as secure as possible. Sometimes, of course, it's necessary to revert changes. If the HMI or the DCS or the hardened systems run into problems or something doesn't work and you cannot solve it by just changing one single setting in the hardening, you might have to restore or revert changes. So there's a few options there. First, restoring the image. Well, that's a full recovery, so there's no more hardening present on the system. The second choice is restoring a snapshot. This will only do a full or partial recovery depending on the applications that are installed. And it also depends on when the snapshot was taken. Of course, for the image, restoring the image, it also depends on when the image was taken. And you can, as a third option, choose to restore only some safe settings. That's only a partial recovery and a partial hardening remains, of course. In this case, if you need to restore or revert changes, it's important to verify or re-verify the analysis phase to make sure that you haven't missed anything to be able to restart the hardening if you're allowed to from your customer or vendor position. This slide shows you some indicators in the IAC 62443 in where hardening is mentioned. First of all, in the dash 2-1. Of course, in the dash 2-4, because that talks about requirements for IACS suppliers. And of course, during system and component, there's also mentioning of hardening for having a secured installation. Now, this was the hardening part. We can also fortify IACS systems by performing security testing, by knowing your weaknesses. And then, of course, depending that we can do hardening after knowing those weaknesses or not. Fortifying by testing is, first of all, verification of what needs to be hardened before doing your hardening. Verification of what has been hardened after hardening, of course. And identification of any other security issues and residual risks. So you can do that by doing regular cybersecurity testing if you're allowed within your environment, of course. Or by doing cybersecurity, fat, and sub-testing in projects. Actually, cybersecurity fat testing is one of the few times in the IACS environment where you can go all the way with your security testing. Because it's in an isolated environment and then you can do full-blown cybersecurity testing at will. So, cybersecurity fat-sat testing is the cybersecurity validation of new upgraded systems, new or upgraded systems and solutions within your environment. Why would you ever do cybersecurity testing within IACS environments? First of all, as I said already, verifying cybersecurity. Second one is verifying compliance to cybersecurity or project or user requirements. But you should not stop there. Because by doing testing, you also, at the same time, create awareness with your vendors, operators, with INC people. And you are making sure that most cybersecurity issues are known and are candidates to be solved before you go into commissioning. So, cybersecurity testing together with hardening, together with fat and sub-sat cybersecurity testing will gradually increase your security within your environment. When would you do cybersecurity testing? Well, for every new system or solution, for every updated system or solution. Of course, during project fat-sat testing cycles, in which you have certainly new systems or upgraded systems at certainly. Or during regular cybersecurity evaluations, if you're allowed within your environment. So, this is a timeline in which the requirements for testing and residual risk determination is shown. Hopefully within projects at your organization or your customers, they set cybersecurity requirements at front during procurement. Together with the doing a risk assessment to identify potential risks to your environment before running into cybersecurity fat and sub-testing. I put cybersecurity fat and sub-testing on the construction phase because actually that's at the vendor side during construction, during test environment. You can choose to do that before, during or after functional fat testing. And the same with the sub-testing, cybersecurity sub-testing that's during commissioning. And you can choose to do so before, during or after functional sub-testing. It is sometimes very interesting to do functional testing and cybersecurity testing during fat or sat at the same time. Because then operators, I see people, even a vendor will see how the system reacts on the cybersecurity testing events during a real scenario. What should you test and verify? Well, first of all, everything on the network, everything on the system, everything on the physical side and everything on the application side. That's in a nutshell. But on the network, you have the ports, network vulnerabilities, traffic encryption is firewalls used on the system. You have system vulnerabilities, configuration issues, not only in the operating system, but also for additionally installed applications. You also have system physical security and so on and so on. So you have the list, complete list there. The main thing is that hardening baselines or hardening guidelines, if there are any, should be used to verify against, should be used to test against. But you should not stop there. You should verify if those hardening guidelines and security testing checklist should not be updated with specific test requirements, specific project information or vendor specifications. So your standard checklist should be updated regularly. And you should not forget that you should verify logical, physical and human aspects of the to be tested environment. Do note that testing of hardened systems might require temporarily reverting certain settings. If you, for example, do hardening on window systems, some things that you will certainly do is disabling the remote registry service. If you disable the bindings on the interfaces and you enable the firewall and so on and so on. Those are all things, typical things that you might have to temporarily disable or deactivate before being able to do some authenticated vulnerability scanning on systems. Depending on the vulnerability scanner that is used, of course. Another thing that's important next to the hardening strategy is creating a cybersecurity testing strategy. So cybersecurity testing should be embedded within quality assurance processes. Cyber security testing should also be embedded within project processes. So regular testing, FATSA testing, those are all things that are as important to keep in mind. But also having some own testing equipment or adversary emulation solutions at hand to be sure that you are able to test everything. If you don't have that capability, you can always outsource that capability to trusted parties, of course. But you should always include logical, human and physical elements within cybersecurity testing and play the what if game. So what happens if I plug in my USB to Ethernet download, for example? What happens when I click on this button when you do operator geo protection checks and all that kind of what if questions can pop into your mind when doing testing or when trying to define hardings and testing strategies? And last but not least is a cybersecurity FATSA reporting. So security assessments is nothing without a decent cybersecurity test result report. But here in this story, in the fortifying ICS story, you should use the cybersecurity test results to adapt your hardening strategy and in the end also your testing strategy to make sure that you are able to cover everything. So thank you for your interest and should you have other questions on this topic, you can always reach me out, reach out to me on the mentioned information on this slide. Thank you. Talk to you later.