 Hi, thank you for attending our talk. I wish we were able to see you in person and hope that in the next year, the things will get back to normal. As you can see from the title, the topic is very important because the internet, the biggest storage of all your funny pictures and memes runs on electricity. And we are here to talk about internals of power generation automations, about vulnerabilities that we are able to find, and about what you can do about that. We are security consultants who have been working with different industrial solutions for many years. Actually, we have been doing it for so long that we have a long list of content information of different system integrators and industrial vendors. So when an asset owner, for instance, a power plant comes to us and asks for service, we don't just give them information about vulnerabilities that we found. We closely work with all involved entities, including the vendor to find together safe and reliable fixes and workarounds. We work at Gaspersky and this research is a result of the teamwork, not just of the three of us, but also of Lebedritsa, Sergey Andreev and Sergey Sidorov. Everything that we are going to discuss today was reported to the respective vendor. I'm talking about Siemens a long time ago, but actually this is not one vendor problem. You can find similar issues in systems of other vendors as well. We are going to talk about real vulnerabilities in real power plants out there and at the first glance, it may seem irresponsible. But if you think about it, for good guys to do such research, it's a challenge. You will need to have relevant experience. You will need to have a lot of time and, of course, you will need access to the industrial environment. So there is no working welcome sign on power plant stores, right? So for good guys, for penetration testers, for auditors, for power plant separators, it's challenging to get access to all of the things together. On the other hand, for bad guys, for those bad guys who would be willing and able to attack a power plant, they do a job to do such research. They have significant investments, they have a lot of time, but they keep their vulnerabilities to themselves for their malicious purposes. So we assume that those guys already have all this information and from our side, we would like to share this information with good guys with you so you would be able to act upon this. Power plants is the main source of electricity on our planet and thanks to carbon monitoring, anyone with internet access can get information about where different power plants located on the world map, what fuel they use and which is their capacity. The heart of every power plant is a turbine. We don't have a picture of a turbine here, but if you ever saw a modern airplane, I believe you also saw a turbine. This is a giant rotating thing generating electricity. So, and on power plants, they look and work quite similarly. And turbine manufacturers are generous in us to share information where the turbines are used. Here, for example, a screenshot from Siemens website. Turbines are used not only on power plants, they are used in many different areas like chemical or oil and gas and many others. If you correlate this picture with the previous one, you will be able to understand a solution from which vendor is used in a particular power plant. In addition to that, online, you can find a lot of information about what solutions are used in certain power plants in different pressure leases and marketing materials. You can find a lot of interesting information about software, hardware version, generations of different systems. Sometimes you can even find building plants. Here, just a small example. Okay, Google, show me some power plants of California. Here, just a few power plants of California, and they have one more thing in common. They use the same plant control system, SPPAT 3000 from Siemens. This is exactly the system that we are going to discuss today. But before we move on to power plant generation, automation, let's talk a little bit about power generation process in general, what we are going to automate. Here, and throughout the presentation, we will be intentionally oversimplifying a lot of things, partially to make them more suitable for the audience and partially because to be honest, we don't fully understand them ourselves. So let's go from right to left. We will need some fuel. Here is coal, for example. You put fuel into a combustion chamber, put it on fire, and it generates pressure to rotate a turbine. The turbine is connected to an electricity generator through a shaft, so when it's rotating, the generator starts generating electricity. It's important to mention that electricity doesn't go straight to your house or city. At first it goes to a special place called power grid. Power grid knows information about market demands, it receives energy from different power plants, and this is power grid who distributes electricity to consumers. So the burning process in the combustion chamber, you can have a lot of excessive heat, so costs called waste heat, and there are different ways how to approach it, you can release it to the air through a condensing tower, or you can reuse it during recuperation. For example, warm water and send the steam to a turbine to generate more electricity. Now that I used to automate this process called distributed control systems or DCS. They are designed to make life of power plant operators much, much easier. They allow you to conveniently start and stop the power generation process. They allow to control the amount of electricity you want to generate, you just enter the amount in megawatts that you want to have in the output. You can monitor everything. These systems are literally very powerful. They are connected through PLCs, they're connected to many parts of the plant including the turbine, and they control such interesting things as the amount of fuel, the temperature of the combustion in the combustion chamber, the speed of the turbine, and even control the turbine not to go into different dangerous modes. Obviously it's not a small piece of software that you just install on the server and it's magically worse. It's a very sophisticated system consisting of different hardware, software, PLCs, input output modules, the turbine itself, and a lot, a lot of things. And often starting to create such a system starts even from building constructors so someone comes to a vendor and says we have an empty field, please build us a power plant. And there are many vendors who do this, but today we will be talking about Siemens. The DCS from Siemens that we analyzed is called SPPAT 3000. Just like other DCSs, it consists of many different industrial components, PLCs, OPC, different servers, HMI, a lot of things, and it may have very different architecture depending on the site. So there will be two components that are unique for SPPAT 3000, they are called application server and automation server. And this is how we will structure our talk. First we will talk about application server, then we will move on to automation server, and then we will move on to conclusions. In different manuals and documentation from Siemens, you will see how the system should be built in a perfect way. For example, they will say that the application network where application server is located should never be connected to any external networks and how everything should be designed. However, also, we again here worry over simplifying everything. In reality, in real power plants, you will meet a lot of interconnections to other systems. For example, you will need a sensor network to monitor different vibrations inside the turbine. You will meet a demilitarized zone because you will need to get some kind of remote support from the vendor, you will need to get updates for your operating systems, for antivirus, and so on. You will need to push out OPC traffic to your corporate network or to a regulator because this area is very strictly regulated. So in practice, you will meet a lot of interconnections. The life of our vulnerabilities started almost two years ago. Back in November 2018, we reported a bunch of vulnerabilities to Siemens. And about a year later, Siemens published an advisory containing information about how to approach these vulnerabilities, and also a set of other vulnerabilities that were reported by other vendors. And that's great that this area receives a lot of attention from security researchers. But also about a year past, it doesn't mean that during this year Siemens didn't do anything. The thing is SPPTA 3000 is an exclusively supported system. It means that a system integrator for these Siemens themselves. So soon after we reported these vulnerabilities, Siemens started to roll out patches and working directly with their clients to fix everything. So December is just when this information became public. We don't have a lot of time to discuss in detail what kind of attacker, what can do with the system. I will just highlight that CVSS scores given here obviously do not represent how critical these vulnerabilities for real power generation process. They're just individual scores for vulnerabilities in the vacuum. To better understand what kind of attacker can do what you can take a look at our threat model that is published in our white paper. Let's start with application server application server is a logical core of the entire system. Everything has connections to application servers in the logical sense. If anything needs to connect to the network, it will end up in application server. Other servers start they work from the loading software from application server and launching it. So this is the heart of the system. And what can possibly go wrong if you open over 40 ports there. This is for an attacker. This is a huge attack surface. If they are lucky to get into their respective network, they can choose what they want to attack. Do they want to attack Windows operating system because this is simply a Windows server. Do they want to attack third party components or they want to attack on SPP 83,000 services, which are based on Java. Also, we hope that these vulnerabilities are already fixed. In industrial environment, usually you don't update your systems very often. Usually it's about a half of a year or several months between update major updates of operating system. So it's always possible to find a time window to with remotely exploitable abilities in Windows operating system. Configurations also could be better. You can find some statistics for security benchmarks on the right-hand side. But one of the biggest problems is actually passwords. The life of passwords has three stages. At first passwords were the same for all power plants. They were default and the same everywhere. And you can find them online. Obviously this screenshot went out from information published from Siemens. Some power plant separators decided that it would be a good idea to share passwords and a lot of other technical information with the internet. But they will be online and we also have a world list in our white paper. Around 2014-2015 Siemens started to generate different passwords for different locations. However, the process of changing them was challenging. You would have to be familiar with the process to change the passwords by yourself. And only in the last year in 2019, the process became easier and now you can do it yourself much easier. Now please welcome Radu, who will tell you about vulnerabilities in Java. Hello everyone. Let's look how SPPS software works on application server side. Operator can communicate with system through thin or fed client. In case of thin client, it use Java blade of internet explorer and communicates with server over HTTPS. So it can be outside of application network and its communication can be constrained by firewall. In opposite, in case of fed client, operator should belong to application network and its end fed client directly communicates with RMI registry to found RMI services and after that directly communicates with these services. Illustration of SPPS architecture was kindly provided by system through public URL. So not to be messed, let's divide it into spaces. In red zone there are items that process HTTPS request and in green zone there are RMI services. RMI service looks like network services which assert on dynamic TCP ports. SPPS consists of containers. All types of containers are represented on these illustrations and containers have self-explanatory names. So before we go deep inside internals of SPPS, let me introduce some tools which used in this research. First of all, all just files of SPPS are up to skated with commercial product. But the security measure can simply bypassed by public available tool. Secondly, sometimes it is useful to see how related software communicates with server. It helps to understand architecture and to see client workflow. In case of SPPS, RMI detector was written. It represents raw TCP streams and in human readable format. Inside it use read object method of STDK. It is known that this method is suffer from insect or desalination vulnerability. So be sure not to be exploited through remote pickup. The first pillar of SPPS is Apache web server. According to its configuration folder, Orion software config can be accessed by unauthorized user. But in fact, this folder contains some critical information about application and about items of application and automation network. Also, configuration of Orion web application in Tomcat also can be accessed. In Tomcat, there are three web applications registered in Tomcat. There are remote diagnostic service, manager and Orion. According to configurations of Apache and Tomcat, all these applications can be accessed by unauthorized user. Inside Orion web applications, there are surlets which can be accessed. All of them are listed in configuration web.xml and the list is huge. Some of surlets has attractive names for attacker. For example, Browse surlet. It allows unauthorized user to perform directory listing on arbitrary folder. But in case of exploitation, another surlet is more useful. File upload. Surlet allows unauthorized user to perform file upload. And folder is fully controlled by parameter-based dir and target name of HTTP request. This vulnerability can simply bypass to remote code execution. For example, attacker can corrupt some startup scripts of SPPA or simply inject JSPS shell in Tomcat web application. And get remote code execution with system rights. Also, the surlet which has service factory in its name, this surlet directs HTTP request to remote services. Inside it uses parameter service url of HTTP to identify RMI service which will be called and serialized object in data section of HTTP request contains information about method and arguments which will be called. So the situations when a theme client and friend client can communicate with RMI services. But in case of FED client, it can communicate with RMI registry. So if application server miss some important security updates of Java, then server contains in sector deserialization vulnerability and public available to vice or serial can allow exploited and perform remote code execution with system rights on server. So the next steps will be to identify all input vectors for attacking. And for this task, we will try to list all RMI services in system. At first step, we will use a class locate registry office.com and get a big list of RMI services. All but one are jmx RMI services. And I assume that they used to control and manage containers of SPPA. For this investigation, we will choose lockup service. In fact, it looks like a collection of next level RMI services and using its public methods. We can get the reference to this next level RMI services. RMI says should implement interface service factory. So it also looks like some collections of another next level RMI services and using public methods get service and parameters such as client ID and name of the service. The instance of RMI of next level RMI service will be created and reference to it will be returned to client. And this next level RMI service is a server which perform real job of SPPA. But in fact, it contains a lot of public methods which can be accessed by unauthorized user. So input vector of SPPA is very huge. The next question is to understand how authentication perform on a system. For this task, let's look how client perform request to security service. To do this, first of all client tried to get reference on security service with some client ID. It needs to PC service factory use this client ID to get valid session in session manager. If session manager will fail, then exception will thrown and client request will be rejected. But in case of success, valid session ID will be returned to PC service factory. And in its turn PC service factory create instance of security service where session ID will start in login ID. And reference to this instance will be returned to client. Further client can call some public methods of this service. And in its turn this methods can perform some previous checks of client. So we have two security measures of system. But there's a question how client can perform login operation on system if he doesn't have any valid client ID. For this task at startup of SPPA session manager create anonymous session with client ID 0. And client used this client ID and perform login operation. But attacker can also use this and simply bypass this security measure. So to sum up we have, so there are a lot of RMI services which has a lot of public methods and permissions checks are fully delegated to this method. So it's really difficult to perform security management of this system. So we understand all input vectors and security measures. So it's time to find vulnerabilities. In the list of RMI services there is admin service. It can be accessed by an authorised user and its public method transcript doesn't have any privileges checks. In fact inside it first step create instance of class loader using bytes from arguments. In fact this step will load arbitrary Java class. This class should implement interface admin script and defined method execute. This method will be called by run script of RMI services. For this case we create Java class which simply run OS common. And this OS common will be run with system rights. In fact there is more powerful post exploitation because we inject arbitrary Java class around SPPA software. So we can use some Java reflection and patch some private variables of SPPA. And as a result we can corrupt some technological process of SPPA. To bypass privileges checks in methods as we can use second vulnerability. It's using RMI session service and its public method get login sessions. Attacker can get information about all login users on the system. This information contains user names, IP and client ID of login users. Attacker can reuse this and if in this information there is a user with admin rights then attacker can simply reuse his client ID and get the reference to security service with more privileged session. And after that attacker can call public method, get all users of security service and as a result get all private information about all users on system. All password hashes also contains in this information. Both of these vulnerabilities can be accessed from either external or application network. All communications between RMI services are encrypted so user name and password hashes are transferred in plain text. Moreover system doesn't have any session protection mechanism. This fact is more critical in case of fed clients because of attacker can perform mid attack on the user of SPPA and get valid username and password hashes from the traffic of user. And after that simply reuse this username and password hashes and perform login operation on the system. So attacker can also change the password of the user. I talk a lot about password hashes and users so it's time to understand how these items are organized on the system. So Alex, you're welcome. Hello everyone. Let's continue our discussion about application server. In this slide you could see how remote authentication works. Now I'm going to tell you about how it's organized locally. After the system gets started, it begins to read the content of files users 1.xml and pdata1.exe to get a user list and the password hashes. pdata1 is a simple xml while pdata1 has a slightly more complex structure. It's a jzip archive encoded in base 64. There is a javascript object in the jzip archive containing a specific xml. The field of this xml from pdata1 file are presented on the slide. They are used to calculate password hashes and check it during user authentication. In this slide you can see a password check algorithm in sudo code. The cryptographic scheme is a typical crypt hashing scheme like in your Unix and Linux machines. It has sold to different number of iterations and the only one thing which was added is hardcoded sold, the same for all users, which is concatenated to the password. The tool to extract password hashes and their parameters from pdata1 file has been developed. Its output is presented on the slide. The tool can be used during password auditing to check a week or dictionary passwords and their hash calculation parameters. The tool is available in our github repository. Draw the line under application server analysis. As you have seen, attack to face is really huge and include java remote services, tomcat applications and a lot of other. Secondly, it's about remote connections. Whether SPPA has or has not remote connections according to when the system integrator or someone else, you should check it. The good starting point for this is application server because OPC maintains remote operator. All of this thing will end up on application server. Thirdly, despite the fact that there is automation network between application server and field devices, an attacker can affect generation process from application server. This actions includes to start stop generation, change power output or just get information how power plant works. Of course, such ventilation will always be visible for operator always monitoring technological process. So real attacker will also be required to change or modify data on operators HMIs, but it's also possible with revealed security issues. That's all about application server. Now let's move on to another main component of SPPA infrastructure automation server. Automation server. The main goal of the automation server is to execute real time automation functions and tasks and tasks for power plant control. Depending on power plant project architecture and features, the role of the automation server can be different. We have distinguished three roles and the first one is automation. There may be slight confusion because the term automation is used both for server and its role, but analyzing automation server configuration and publicly available information about it. We have found that whatever the role is almost the same hardware and part of software are used. So we have decided to use this kind of roll classification, which seems less confusing to us but at the same time it's slightly different from the vendors one. In other words, having an automation role means that the server is responsible for interaction with input output modules, which control and monitor power plant equipment such as turbines or electric generator. The second role is communication. This role is used to communicate with certain parties systems. This is just a protocol converter supporting such protocols as Modbus, IEC-1001-104 and some others. And the last role is migration. In this role, the server is used to connect previous version of SPPA such as SPT-2000 or PERM-ME. In automation role, automation server can be run both on Simatica 7 PLCs or industrial PCs. In case of other roles, it can be run only on industrial PC. Let's talk a little more about each role and let's start with automation role based on PLC. PLCs are what directly control full devices and access to them is more for any security discussion. Any configuration changes and updates for PLC are required to stop technological process. Therefore, these devices usually have security misconfiguration and firmware without security updates. Another common security issue for PLCs is using unsecured industrial protocols. In case of SPPA, they are Simatica 7 protocol and PLC data protocol. There is quite a lot of information about SPPA, but not so much about PLC data protocol. So we had to deal with it and analyze it ourselves. It's not a special protocol for SPPA. When you program your PLCs and you need to exchange some data between them in real time, you use this protocol. It's pretty simple and maybe its description is available somewhere in the internet, but we couldn't find it, so just in case, show its structure here. It doesn't have any security mechanism and the only obstacle while do manage the middle attack to SPPA data is the sequence number, which we can get from a packet and just fuzz implementation. For protocol analysis, the Wireshark D-Sector has been developed and also available in our repository. During PLC security assessment, one of the main things which we check is unauthorized access to reading and writing the PLC memory. Availability of unauthorized access is determined by the position of mode selector of Simatic PLCs and some other configuration parameters. The metrics on the slide shows unsecured states for Simatic PLCs when unauthorized access is possible. The tool for gathering information from PLCs over the network and for its analysis has been developed by one of our colleagues and also available in our repository. Now, let's talk about automation server based on industrial PC. The workflow of all roles in this case is quite similar to each other. When automation server is based on industrial PC, it's just a Linux box. Now that we're in the start, it tries to download some additional files from application server. These files include JARs, which represent SPPA runtime containers, bar script, some configuration, protocol configuration files, and other. To execute downloaded JARs, PTC Perk virtual machine is used. It's a real-time Java virtual machine widely spread in industrial and military areas. Now that running JARs open NMI services or some of the extension, in case of migration server or your own RPC services, which are extension of classic Java NMI services are used and you can see the listing on the slide. Now, automation server based on industrial PC has following security issues. Firstly, there is a possibility to spoof downloaded from application server files. These files are downloaded over HTTP and there are no security mechanism like authentication or integrity check during this process. Secondly, it's about using default credentials. You can use the same admin with password same. Thirdly, it's RMI services running on automation server. The research has found two vulnerabilities in our own RPC services, which allow to perform sensitive data exposure and RC. And the last group is vulnerabilities found in the software used to fulfill a migration role for connection previous version of SPPA, SPPA T2000, also known as TXP. These vulnerabilities found in migration server software are related to different kinds of overflow, stack, heap, integer and other. Actually, there are so many overflows here that the stock would be overflowed by that if we started to describe all these vulnerabilities in details. If you was about RPC vulnerabilities, these vulnerabilities have been found in runtime engineering service. This service has a method requested on time container container where the first argument defines an action to be executed using the action read file. It's possible to read any file from the system using write config file action. It's possible to write any file to any folder in the system. And for example, it can be jar file which executes shell command from the command line. And then using the same request runtime container method, it's possible to execute to this jar later. Now let's talk about automation server to sum up automation server can be run on PLC or industrial PC. In case of PLC, it's a usual PLC with well known security issues. The industrial PC is a Linux box, which tries to download jars from application server and then execute them with PC or virtual machine and attack again spoof downloaded files or just exploit revealed vulnerabilities in network services. I haven't mentioned any network equipment used in distributed control systems. During the research we saw a wide variety of network devices and network infrastructure. They include different kinds of switches routers firewalls and more rare devices such as data died for example. In order to summarize all this information and got common as we pay network topology scheme presented on the slide. We have shown in purple usual places for network devices but it also also should be mentioned that the same devices in the same places can can be found in other distributed control systems from other vendors. In industrial network usually have a lot of security issues. The reason is these devices don't require any configuration and can be run out of the box. So for the things like we community strings in SNMP we credentials in HTTP, HTTP, telnet and other services from where this publicly available exploits and just security misconfiguration, all these things is common and typical for industrial network devices. Additionally, some industrial protocols, for example, such as a profanet dcp can allow to get a set network configuration of these devices. In case of profanet dcp. In case of profanet dcp it also can be very useful for device enumeration. And the last thing about industrial network devices is always remember that these devices with years of uptime can act unpredictably for completely legal and normal activities, such as browsing web page with statistic or login into telnet. Thank you. Thank you, Alex. So the topic of power plants is huge and the topic of DCS is huge. So we just tried to poke a couple of things here and there and show that to affect the power generation process you don't have to go deep to field level to PLCs, but you can focus on higher logical levels like application server and automation server. And what we do not talk about today is havoc or damage that you can cause by such attacks. Actually, that's because it's not that bad. If you already imagine a hacker and a hoodie who writes a couple of lines and after that the entire city goes into the darkness. It's not like that, because it's power grid who is responsible for power distribution is it's just not power plant. So here we mostly talk about more local damage like wouldn't be possible to damage a turbine. Since a turbine is connected to DCS PLCs. And the turbine is a giant mechanical monster. And it's self degrades by working probably by putting it into different uncomfortable modes or quickly starting can stop in it it would be possible to make it degrade even faster or even break it. Unfortunately, or actually this fortunately we were not able to find a spare turbine on eBay. But we are making an educated guess that the damages are there based on the system architecture. To make life of power plants a little bit easier and we prepared a small do your own assessment guide that you can use to check your SPPT results and system for vulnerabilities that we will discuss today. You just connect your laptop to a couple of places in the network. Go through a simple checklist. You don't have to hire expensive security consultants for that. And after that you can fix some parts yourself or you can call your system integrators to discuss what to do further. Just any other industrial system DCS is not so good in terms of security, it could be better. And we recommend you to take a look at I say I see 62 443 set of standards to know what to do to improve security in terms of communicating with different entities because in the industrial area is usually a challenge when you, for example, when a vendor is not interesting to fix in something. So it describes in particular what kind of relationships you can build with regulators with vendors and other parties involved in industrial area. Of course update your systems change your passwords improve your configurations according to the vendor security guidelines. We also recommend you to set up monitoring since most servers are based on standard windows and Linux boxes. You will not able will not be able to detect different Java attacks but at least you will be able to detect some parts of attacks. And again, this is more about DCS and industrial security in general, rather than, than by one system by a single vendor. We released a lot of information there was the white paper different tools that we mentioned today. I recommend you to take a look at our GitHub page to find all of that. And we would like to thank Siemens product search who made all the communications very effective they were very responsive. They allowed us to contact Siemens product team. And of course they released the patch. Take a look at the advisory and Siemens always tries to raise awareness of their users, how to better build such systems, what are better configurations, but they also highlight that it's not the vendor who is solely responsible for security of an industrial environment. There are a lot of things that power plant separators can and must do themselves. So please follow the guidelines for that. Thank you for our talk. Thank you very much for your attention and we will be glad to answer any questions.