 Hi, this is Austin Scott, and this is five quick wins for improving your ICS cybersecurity posture. I'm a principal industrial penetration tester as part of the Dragos professional services team. And I really see the world from the perspective as a pen tester, just like a hammer sees the world as a nail. I really see everything from an adversarial perspective, and that's the point of view we'll be taking in this presentation today. Really, we're looking for ways to make it more difficult for an adversary to move through an ICS environment. Now my recommendations in and of themselves won't give you a bulletproof cybersecurity program or industrial cybersecurity program, but they will raise the bar for your security posture. There's a lot of other things involved in running an industrial cybersecurity program that are outside of the scope of this presentation. But in my presentation today, we're going to talk about a lot of the common things that as an industrial penetration tester as an industrial cybersecurity assessment practitioner. These are the things that we see over and over again, and a lot of them can be addressed fairly easily without a lot of capital or operational expenditure. We'll talk about how to address some of these risks and what these these common issues that we identify are. So a lot of this is based on the 2019 Dragos year in review report. We see a lot of these common issues as we do assessment after assessment. It's common to see things like limited visibility or credentials laying around or routable connections from your corporate environment into the ICS environment and even ICS components directly accessible from the internet these days. So a lot of these common findings are highlighted in this year in review report and we'll be talking about the top five and how to really address those or identify them in your own environment and remediate them. So we think it's important to take a threat based approach when you're when you're looking at risk in your industrial cybersecurity environment. Now within Dragos we track a number of activity groups that target these environments. And this will be really done from the perspective of our own internal internal activity group called Kyberide, where we develop our ICS tactics and techniques really based on a lot of the other activity groups that we see. So we try to do some threat emulation as we're working through these networks and use a lot of the common, a lot of the common techniques that we see being used against ICS environments. And of course, tactics, techniques and procedures are part of the equation, but we find these really change depending on the environment that you have. And it's important to, of course, start with understanding what you have, whether you're running Windows Active Directory or Windows Work Group, what kind of security controls you have, how the internet access works, how industrial vendors get in, what your firewall rules look like. Once you understand these basic environmental factors and understand how they change the tactics and techniques of an activity group, that can really help you reduce your risk and understand where you need to invest your time and energy to mitigate some of these risks. Of course, if you study activity groups like Electrum and Xenotime, there are publicly known techniques that they have used against environments. And I certainly believe those techniques would change depending on what was available. As a pen tester, as an adversary, we're always very dynamic with our approach. We need to roll with the punches and adjust our techniques to align with the environment we're faced with. So that's why it's not one size fits all. You can't take all these TTPs that are identified and directly apply them to your environment necessarily. Often the environment dictates what the TTPs are going to be used, which TTPs are going to be used. So during this presentation, we'll be talking about our top five ICS assessment findings for 2019. These are related to firewall rules, access management, system hardening, logging and network visibility. So we'll be talking about some of the tools that we use as penetration testers and ICS network assessment professionals. Some of the tools that can be used safely in ICS environments without introducing operational risk or at the very least minimal operational risk when they are being leveraged in these environments. How to identify these risks using these tools and then also how to mitigate some of these common findings. So at the end of the day, if you're able to take ownership of your industrial cybersecurity posture, do some self assessment work. It's not difficult to do once you understand what tools are available and how to approach some of these problems and highlight some of these problems. If you're able to mitigate a lot of these common issues, even prior to doing another assessment, it allows that red team or that penetration testing team or that assessment team to really focus on the more interesting problems. So you're getting rid of all the low hanging fruit. So the adversary group has to dig a lot deeper and focus on some of the more challenging problems. And you're also raising the bar like when you when you address a lot of these low hanging fruit, a lot of these low level issues. It really reduces the playbook that the adversary can can use. They're not able to fall back on their normal, normal operating plans, they need to think outside the box a little more they need to experiment more, do more reconnaissance. And whenever they do that when they have to work harder, you have more opportunities to detect them you have more opportunities to stop them. And, and also you make their lives a lot more difficult, which is, you know, something you want to do you want these, these adversaries who are targeting critical infrastructure who are trying to turn the lights off in your in your town or, or impact these important industrial processes, you want them to have to lay in bed at night, questioning their life choices that brought them to that position of why they're targeting civilian infrastructure. Starting off with firewall rules, what we see when we do these assessments, often will ask for the the rules to be shared with us. Usually like a white box approach is best. When we're doing ICS assessments, we really need to be transparent with the operating operators at the sites and the site personnel turns out industrial asset owners and industrial operators don't like surprises. So, we found that it's very important to be open and clear with what we're doing whenever we're doing it within these ICS assessments. The more information we can share, the more we can work closely with the operations team and start to build that trust and that build that bridge between the cybersecurity team and the OT or the ICS operations team which is which is so important. To be successful in this environment. So when we ask for these firewall rules will usually use a tool to make sense of them. There's dozens of different firewalls out there and they all provide their rules in different formats. Fortunately, there are some commercial tools that you can use and even some free tools, you can use to make sense of these rules and work through them in more of an Excel spreadsheet to kind of format. One free tool we like to use is the SolarWinds free firewall browser. So what you can do is export your firewall rules and then import them using this SolarWinds free firewall browser and then just go kind of line by line. What you're really looking for is interactive protocols that allow remote access between your different trust zones. So typically between your corporate network and your ICS network or even between different trust zones within the ICS network. You're looking for SSH, Talmud, remote desktop, VNC, even things like WMI or remote management, RPC, SMB, even protocols like OPC like OPCDA, the older OPC protocols can allow remote access and often do. So we often find there are temporary firewall rules still in these configurations from the time of commissioning. We find there's vendor dictated rules and vendor access rules that have never really been evaluated or questioned. So we go line by line and really question why do we have this? Like what purpose does it serve? Who's using this? And to knock off as many of those temporary or vendor dictated rules as you can. Of course you want to communicate with your vendor. If that vendor is using that for remote access, you want to identify those and make sure they are still able to access or maybe propose a more secure method if the method that you're using is introducing risk in your environment. So this is what the free firewall browser looks like. It just breaks down the rules line by line in sort of an Excel spreadsheet format. And you can see we've got a couple interesting interactive rules here on 3389, which is the remote desktop protocol. So things to watch out for when you're going through these assessments. There are commercial tools you can use as well like Nipper that can help you identify issues. Often we still find any any rules or what's equivalent to any any rules. Sometimes these firewall rules can get quite complicated. And you put enough rules overlapping on top of each other and it can basically equate to an any any rule. So it's important to study these firewall rules closely and identify opportunities to pivot through these different trust zones. Now, another common issue that we running to is access management. And what we really find is it's not necessary. Most of the time to use any exploits or have have to dig too deep in these ICS networks for to pivot or escalate privilege, because the access management is so poor in these environments, often will run into shared active directory environments between corporate and the ICS. So, once we once we take that corporate active directory environment we're able to easily pivot and take full control over the ICS environment. Other situations where there is a dedicated active directory environment in the ICS, it's poorly maintained and hasn't really been configured properly so usually we see almost everybody's a domain admin or there's lots of service accounts that are running in these environments, lots of these common issues in active directory that we run into. So what what you do how do you identify these issues. Well what we typically do is run a tool called bloodhound. It's the open source tool, and it's been used. It's used by pen testing teams and red teams to unravel the yarn ball that is active directory for years active directory kind of experienced this security through obscurity, where it's so complicated that even the adversaries couldn't really figure out all the groups and groups and users and groups and how all those unwind to different permission levels but with the introduction of bloodhound this this tool uses graph theory to truly map out what the implications of all these permissions are what how to unravel all these groups to determine who's really a domain admin and and how you can pivot from the average user to domain admin fairly easily so it shows you all these different paths and over privileged accounts that you can identify and potentially locked down. And it's fairly safe to there's a very low operational risk, because it only communicates with the active directory server in the network it's not going to scan your network or hit all the PLC isn't in your network or anything like that. It's only going to communicate with the active directory environment. And it's only going to send LDAP requests which are fairly normal. It's just the same kind of network activity that you would introduce if you were logging into a machine or remote desktoping into a machine it's, it's nothing unusual for that environment, there's certainly a big spike of LDAP when you, when you use that tool but it's nothing that would create an operational risk in that environment. So what does, what does that look like. So here's an example of bloodhound unraveling a path to the domain admin from this RTAM user we can see he's a member of this group that's an administrator on this machine, and because they're an admin on this machine. They're able to gain access to potentially the password or the hash of this user who is also logged into this machine, and then use that user's privilege to gain domain administrator access in that network. So this can really make sense of Active Directory and help identify some of those common issues that and common misconfigurations. So access management part two. So what if you don't have Active Directory what if, what if it's a Windows workgroup environment which is also fairly common to see in ICS. And even if, even if you do have Active Directory there's other passwords that exist outside of Active Directory within these ICS networks. So I'm talking about things like VNC and SSH credentials into switch gear network gear. Stuff like that. There's usually passwords just laying around the network and Excel spreadsheets or notepad files or default credentials for a lot of devices. Often we find credentials are stored in things like Chrome or stored in things like putty or WinSEP or batch patch or other other tools like that. And when you when you click that option to save your username and password in these tools, these tools don't always securely manage those credentials so it can be quite trivial for an attacker to pull out those stored credentials. So what you can do is leverage some of these client side tools, like session gopher that are free session gopher from FireEye, or even performing things like an LSAS dump, or using tools like MemeCats, MemeKittens, and some of these Nersoft password utils. Now MemeCats is it's something that's used by almost every activity group. Once these activity groups get a foothold in an environment, one of the first things they do is dump their post exploitation tools. A lot of those are just trying to find passwords to escalate privilege and move laterally in that environment. So you want to try to understand how you're storing your passwords, what passwords are stored on different endpoints. And you can automate that process just like the activity groups typically do. So what can you do once you've identified the issues or the misstorage of passwords, you can implement some kind of password storage mechanism like a privilege access management system or even just like a vault like or a last pass or some password vault solution that does a better job protecting these important credentials than WinSCP or putty or other tools typically do. So here's a quick example of running MemeCats and it's something I'd recommend like it's something we always try to do on a safe environment, running MemeCats just to see if it's detected, if it triggers any kind of alerts, if we're able to do like what we're doing here dump the LSAS memory to a file. So if we dump the LSAS memory using the task manager, just right click on the local security authority process and go create dump file. We can copy that dump file off of the machine and run it through MemeCats to see what kind of passwords we pull out of memory. If your Windows endpoints are not hardened. You'll usually be able to pull out the hashes and often clear text passwords from any account that has recently logged into that machine. So it can be quite eye opening to to see that and see these passwords coming through in clear text using MemeCats. So it's something we we recommend that our customers do in a safe and safe manner, and even just copying the MemeCats executable into one of your ICS assets just to see if it gets caught by Windows Defender or Norton AV and and to see if that alert makes it someplace to see if you know the your monitoring is set up properly. It should create alerts it should set off all kinds of alarm bells so it's a great way of testing your your monitoring for malicious files in your ICS network. Here's another example of another tool the the fire I session go for it looks for passwords and other tools like WinSCP and putty and you can see like your stored passwords in your remote desktop client. So it'll pull out clear passwords like we see in this example below, and this can be very valuable to an adversary who's looking to move laterally or escalate their privilege in the network and we almost always always find credentials. When we're doing these assessments, one way or another there's there's always almost always poor storage of these credentials. So that can help address that credential storage issue is some basic Windows system hardening. It's a very common issue that we see where a lot of these ICS Windows assets, they, they haven't performed any hardening often we find things like the firewall is completely turned off on these windows endpoints. Just because these ICS networks are so sensitive at times, once, once the operators or the system integrators get things working they're afraid to change anything or lock anything down in case it breaks something so usually, once they get things working they just kind of leave it as is, and it's rare to find any system hardening really performed and and without without some basic Windows system hardening it's, it's so easy to cut through those networks as an adversary. The default Windows installation, especially Windows 7 and older Windows versions. There's just so many, so many backwards compatibility features that are turned on that make it so easy to pull passwords escalate privilege to system that once you're in that network you can own it within a matter of minutes without some system hardening happening. Looks like my internet connection had a bit of a blip there. I'll start that over. Now system hardening does have the potential to create an operational or introduce operational risk. You'll need to work closely with your vendor to ensure that any hardening you're doing won't impact your operating process. Often the major vendors will have system hardening guides that you can follow and the recommended hardening that that they have tested and approved. So it's important when you're building a new Greenfield system when you're building an ICS system from the ground up to ask them to implement these system hardening features to turn security on because what we find is if you don't ask the vendors or the system integrators to do these things it just doesn't. It doesn't happen. If you don't set that standard or or or make that request, they're just they're not going to do it. So it's very important to be clear that you want the systems hardened. They need to be part of their commissioning plan. Part of their site acceptance testing or factory acceptance testing checklist that these hardening features are turned on and the recommended best practices for system hardening for that vendor have been implemented. And if the vendor does not if they're not very mature and they don't have a hardening guide. You can, you can use some of the tools like some of the ones I've listed here like chaps the configuration hardening assessment power shell script from cutaway security to identify some of the common hardening issues. And you can raise those up with the vendor to get them approved or ensure that they're on board with making these changes. But there's other tools Microsoft has a great tool called the security compliance toolkit, which has a very thorough analysis. It does require you to install some software in your ISIS environments that could cause some issues with your vendors same with the CIS tools and the state tools they do require software to be installed in order to make them work. But that's why I love the chaps tool it's just a power shell script you can run. It doesn't require any software to be installed it just will do some data collection on that endpoint and highlight some of these common hardening issues. And you have to work closely with the vendor when you're running these things but here's an example of the chaps hardening demo in action. You can just run it as a power shell command like I've done above, and then it gives you sort of a pass fail view, things like W digest DNS client. This, if you implement these hardening recommendations, it will prevent an accessory from being able to pull clear text passwords of a memory, clear hashes out of memory. It'll prevent them from downgrading to PowerShell to and bypassing a lot of the PowerShell security features. So it. It also helps reduce the chances of man in the middle attacks and things like that in your ICS network so just a little bit of hardening and lockdown can have a huge impact. What we're seeing from a logging perspective is usually a complete lack of logging or no centralized logging. Sometimes logging is turned on on the windows endpoints but it's just not going anywhere it's just being stored locally. And if it is turned on and being centrally managed, they're not always logging the right stuff in these ICS environments, they're not logging PowerShell commands or, or new processes, you know, using things like sysmon to really get the the details you need to do in the proper forensic analysis and incident response in these environments. And it's not hard to do it's it's not difficult to turn these, these things on. And again that CHAP tool can help you identify some of the common logging issues that you may encounter and some of the things you'll want to turn on in your ICS environments. And actually just having that centrally managed logging environment can be such a huge win. If you ever are doing like an incident response, you'll be so grateful to have that centrally managed logging environment, and it's all built in the windows you don't need to have Splunk or anything like that you can just use windows event forwarding to centrally manage those those windows without having to spend any extra money, just having that those events all in one place can really facilitate things like threat hunting and can speed up incident response and give you better visibility into your ICS network as well. So what we recommend is understanding what your windows event logging capabilities are today, what's being logged what's not being logged where it's going and using again that CHAP tool can help identify these issues. There's really a pretty low operational risk. It may produce a little more traffic on your network, but for most modern networks this shouldn't be a huge issue. This is the output of that CHAP tool again that PowerShell script, and it can show you some of the issues if you have the PowerShell task scheduler win RM, WMI activity, all these different log files are important to have turned on and have a larger log size to ensure that they're being forwarded to a central location so the CHAP tool can help identify a lot of these common issues. So if you turn on the recommended logging from CHAP it will make a big difference and reduce your risk quite a bit. Now on to network visibility. Another common issue that we see is as a pen test or a red team, we're able to operate within these ICS networks undetected. Once we're in them, there's usually very little or no visibility so we can move about, move laterally escalate privilege, take over the domain without any alarm bells going off and maintain perpetual access as well. So what you can do is if you don't have network visibility today, you can start to lay that foundation and start to see if you get the value out of it in a low cost sort of introductory kind of method. First of all, if you just identify the points in your network that you should be monitoring what switches you should be attaching or configuring span ports to, or better yet, purchase some network taps and install them. And just having those points in your network that you can tap into to collect data and collect PCAPs, it's extremely valuable in an incident response or it's extremely valuable in a threat hunting exercise. So that can really enable your security operations team to do a lot more just knowing where to plug in. That's the first step in getting that network visibility. And once you have that, you can start to collect PCAPs and do some analysis to better understand what's going on, what kind of traffic you see in your network, what's normal. And then you can start to use some, even some free tools that are available or commercial tools to perform analysis on those PCAPs, or even install some hardware and software to do continuous monitoring of that network traffic. And again, it's a pretty low operational risk. You're connecting to span ports, or taps. Now ideally, we always recommend you use dedicated taps, network taps rather than span ports, span ports, when they're configured on a switch, there is a risk that that switch can get kind of overloaded, especially if it's an older piece of equipment. You should be monitoring the CPU usage of those switches. Once you enable your span port, if they're in the 80 or 90% utilization kind of threshold, you may want to consider an alternate option, because that could just put it over the edge and create a network outage. So you need to be a little careful with that, but it shouldn't, it typically doesn't cause too many issues to set up those span ports. But in some of the edge cases, it could create some or introduce some operational risk. So something to be aware of, something to watch. And of course, having visibility in your network can improve your threat detection and threat monitoring capability with the right tools and techniques and procedures. So there are two free products that Dregos provides. One of the free products is our old CyberLens product, which is, it's well suited for PCAP analysis. It was really designed to just take a PCAP and help visualize what's inside, specific to the ICS protocols and ICS content. It's a great way to help you understand what assets are in your network. That's a common challenge we see in the ICS space. What do I have? What are my, what's on my network at any given time? So CyberLens can help you identify those. And Sophia is designed for more of the continuous monitoring. Sophia was the next defense product that was commercially sold to customers all around the world, and now it's available for free from the Dregos website. So if you want just continuous asset identification monitoring just to know what's on your network and that that updated perpetually, Sophia is a great tool for doing that. And of course, there's commercial tools you can use like Network Miner to do analysis of PCAPs. That's a very handy tool for digging into PCAP data. And of course the Dregos platform is our commercial product that does a threat based monitoring of ICS networks with playbooks and feeds into our Intel with the latest activity groups that are targeting these environments. And if you want, if you're ready, if you're seeing the value in your network monitoring, then maybe a commercial product is the next step for you. But it's always nice to kind of learn to walk before you run, kind of identify those span ports and kind of ease your way into it to make sure you really see the value of and take full advantage of that value of OT network monitoring. And of course, this is something these self checks these common self assessments can be done regularly. And there's there's huge benefits to taking ownership of these just doing like some mini assessments, once a year once every six months just to see what's on your network, look at how passwords are stored, understand who the big accounts in your active directory environment are who the domain admins are and who has access to that. All these things should be done regularly. Once you really take ownership of your industrial cybersecurity of your industrial cyber risk, you can, you can make a big difference in that risk reduction. You can start to address that. And it's something that should be done on a regular basis at a set interval once a year, once every six months because ICS environments are quite dynamic they do change. They're constantly being modified and updated and maintained so it's good to do this on a regular basis, and it can be augmented once you once you get into this self check and you're you're covering off a lot of the low hanging fruit. So that's, that's when you can bring in a professional team to do an assessment, and then they'll get to really dig into the interesting stuff, the stuff that is uncommon, and would require an adversary to dig a little deeper, do more research, and have to sweat a little to move through your network. And that's the end of my presentation. Thank you so much for attending.