 It is a system which controls industrial processes. You may have also heard them referred to as SCADA systems, and you can find them in power plants, water treatment centres, all sorts of critical infrastructure. You would think that these would be super secure, locked down. Are they? Are they shit? There's so many vulnerabilities, there's so many hacks, you've probably heard about them in the news. These two folks here are two people who are well aware of the insecurities of ICS. They both come to us from CompaTest. They in fact compete in competitions to hack ICS systems, and they in fact won this year in Miami, I believe. So I think we can all be impressed by that. Please give a good warm MCH welcome to Dase and Dan. Thank you. Thank you. So ICS security is becoming more and more important. Now that factories are connecting more ICS devices to either the internal IT network or in some situations even directly to the internet. So this talk is the results of some research we did in software that's commonly used in ICS networks and the vulnerabilities we found. So my name is Dan Koeper. I'm here together with my colleague, Thais Alkamade. We work for CompaTest, a full-server security provider in the Netherlands. We do everything from penetration testing to incident response, but we too are the research facility, meaning we get to research cool stuff and talk at conferences about it. So if you have any follow-up questions after this presentation, we have a big tent full of arcade games on our retro square, so feel free to drop by and we can have a talk and a beer. So last April, we won Ponto On Miami, which was a Ponto On edition specifically about ICS security. So this was the result of a couple of weeks of research we did prior to the competition and during this talk, we would like to share as much as we are allowed to about the vulnerabilities we found during this competition and we demonstrated successfully during Ponto On. However, before we go into the details, we noticed that when the news was published about our victory in Ponto On, that not everybody is familiar with the competition and how it works, because the main comments were about the fact if we indeed use Mac OS as our main operating system or if we run Kali Linux on it for our security research. So before we dive into all the details, I'm going to head over to Thais and he is going to tell you a little bit more about the Ponto On competition and what the rules are and how you can compete. Yes. So Ponto On was started 15 years ago and the idea is that you need to demonstrate new zero-day vulnerabilities against the target in a live environment to win a money prize and to also win the laptop that you hack. Now there currently are three different editions for this. There's the Vancouver edition, there's the Miami edition which we participated in for ICS and there's the Tokyo edition or it moves around a bit for IoT devices like phones, TVs, stuff like that. So the idea behind Ponto On is that around three months before the actual on-site demonstration they announced the targets and then for each target there's a cash prize and they select these targets based on what they think are interesting applications in that environment. So in this case there's a control server category with two applications and you need to demonstrate a remote code execution vulnerability here either over the network or you need to let somebody open a file on that machine for that application. And there can also be different kind of targets. So in Ponto On Miami this time there was the OPC UA server category. There were three different levels of payload. If you have a denial of service payload then you get five points of $5,000 because even just being able to crash a ICS application can be very serious if it's something that's for example controls the electrical grid. Then the next level was remote code execution so taking over the machine but there was even a higher level here which was the bypassing the trusted application check. And the reason this is higher is because remote code execution can be after authentication so they really wanted to make sure that the authentication was working correctly which is why this got more points and a bigger reward if you could demonstrate that. And I'll talk more about that later. So the idea behind this competition is that they announced the targets and everybody who wants can just start researching it, finding stuff, inviting exploits for it and then you need to enter into the competition, you need to submit your application and then nowadays you can also participate online but you can also go there and then you need to demonstrate your exploit against a live target. And to do that you get 20 minutes and three attempts and each attempt can only take at most five minutes. So you really need to have an efficient exploit that runs well and you need to make sure that you don't need to update it in between but you can prepare quite a long time in advance. So normally it's three months but in this case because of COVID they had to move the conference so also the competition moved so we actually had about six months of preparation in this case. And the vulnerabilities need to be zero days which means that the defender must not know about them. So how they do that is immediately after you demonstrate it and it succeeds you go into this disclosure room where they either there might be somebody on site from the vendor or they call somebody from the vendor and then they fully disclose all of the details of the vulnerability so they can go and fix it. But they also look into their bug tracker to see did they already know about this, did they already have a record of this bug and if they did you get a reduced price because then it's a collision so it's no longer zero day. So it's also a goal of this competition to make sure that these vulnerabilities get fixed but also yeah you really need to work out the complete exploits. You cannot just point to a vulnerability you have to actually demonstrate it live on stage there. Now for this competition we had five different entries. There was a denial of service, three remote code execution and one of the trusted application bypass. In the OPC foundation OPC UA.net standard they're really better naming these things and we're going to talk about the last two because for those we have permission to talk about them because the other three are not yet completely fixed and the vendors are still working on fixes but luckily these two were quick enough to update that we can now talk about it which is great otherwise we would have 15 minutes to talk about very little here. Now as mentioned we also participated before in PontoOwn and this was a year ago we participated in Zoom and that exploit was a memory corruption vulnerability which was quite a lot harder than anything we did here and memory corruption means that you have to spend a lot of time getting the memory in the right shape. So it was very tricky getting this to work and as you can see it succeeded with only 20 seconds left on the five minute timer. So what we decided this year because it wasn't really a main focus project for us like Zoom was we wanted no memory corruption at all. Everything we wanted was had to be logic bugs or something simple that we can just if you find a vulnerability then we take another day and we have to exploit for it. So we wanted simple stuff low-hanging fruits no memory corruption or stuff like that. If you want to know more about this one we're going to talk about this one tomorrow at five at this stage. So here we have an example of one of those exploits. This one was streamed live and now I press enter and then about seven seconds later it actually succeeded and we had already demonstrated it. So this is much better for your heart than having to wait those five minutes and then getting really stressed about those last 20 seconds if it's going to work. So the first one we're going to talk about is the ignition one and I'm going to hand over to down for that. So the first vulnerability we're allowed to talk about is a remote code execution vulnerability in Ignition. Ignition is a SCADA software solution. It means that with Ignition you can intercept multiple channels of data from your ICS network and you can write certain control statuses for them. So basically it's like a central component in your ICS network to monitor and control everything on the every appliance on your ICS network. So before we start big kudos for how the vendor inductive automation handled this year edition of Pone to Own because they were very quick with releasing their patches but they also gave full disclosure about every vulnerability that was mentioned during this competition which is really something that well not every vendor handles vulnerability disclosures in this way. So I just described what Ignition actually is and with certain confidence but this was no way the case when we actually entered the play to go to Miami because Ignition was one of the software packages that we had a working exploit for long before we even knew what the application was supposed to be doing but well it's software so we can hack it that we don't actually need to know what it's what it's used for. So but we learned a lot during the conferences even where Ignition was supposedly to be used for. So the target for Ignition was getting remote code execution so the first step was we looked for vulnerabilities that would actually allow us to run code and in the case for Ignition this was rather easy because Ignition has a script editor. You as an operator you're allowed to write some script to based on certain data inputs you can write scripts that's well change the machine operating status or etc etc. So it has a full Python interpreter built in the application. So that was quite a neat feature. This means that if we could get access to the application it would be game over. We could just use the internal script editor to upload a reverse shell and then we have code execution. So getting code execution was easy but the only trouble was that this editor was only available for logged in users of course you needed to be authenticated. So that meant that we could shift our focus rather than finding a remote code execution for vulnerability we could find an authentication bypass. So since this was a Java application doing code auditing is relatively easy. You can just take the jar file for decompilers to readable source code and they can start auditing all the authentication classes. So Ignition has a couple of authentication methods. You can just authenticate with the username and password and it will verify this against an internal database. But of course we all love single sign-on because yeah then you only need one account which you can use to log in to all your application. So Ignition also has a single sign-on implementation and it can authenticate to Active Directory for example because if you're on an IT network that is what you want. You already have an Active Directory that is where all your user information lives so why don't you want to have Ignition to authenticate against Active Directory. Well seemed like an excellent idea at the time but then we looked at how they actually implemented this SSO implementation. So this is the whole function. It's not quite a lot of code but I'm going to shift the focus to only these four lines. This is the whole authentication method. So basically what you need is you need to have an Active Directory name and you need to find a user that is active within the domain. But there's an important step missing here like verifying some tickets or passwords or anything that is a secret. As long as you have a username you have an authenticated session. So that was quite nifty, quite handy for us. Finding a username in Active Directory is not particularly hard. You can just ask the Directory service or you can just take administrator for example because that one will always exist in an Active Directory system. So combining those two using this application bypass to get a valid session and then using the script editor to spawn a remote shell we got remote code execution. We are going to demonstrate it here. On the left side you have a Windows VM running the latest version of Ignition with the process explorer so you can see the reverse shell spawning. And on the right hand side you have our exploit. So we're just going to start the exploit. It's going to try to log in, get a user cookie that works and we get a remote shell. So the cool thing about this is the whole Ignition Gateway runs as system because of course you want to have Python code execution as a system user. So we automatically have the highest privileges as well. So that was the first vulnerability. Tys is going to talk about the second vulnerability which was a little bit more complex. Tys. Yes, I'm going to talk about the application bypass in OPC foundation OPC UA.NET standard. Now OPC UA, you've probably never heard of it but it is a very important protocol within ICS. It's created by the OPC foundation which means that it's not a vendor specific protocol. So in an ICS network you often have a device connecting to software that is written by the same vendor and you speak a vendor specific language, a protocol that nobody else has implemented to communicate. So there's all of these bubbles of systems within ICS networks that communicate over one vendor specific protocol but that's often not enough. You have multiple of these things and you want to combine them into some larger network. So OPC UA is often used as the glue between different vendors. So if you have one vendor that has some stack of devices and software and another vendor then you can just choose OPC UA which is often implemented by everything to communicate between them. That's also why it's created by foundation because that's yeah there's multiple members multiple vendors are a member of that foundation and part of the OPC foundation's work is also to create a reference implementation. They have a reference implementation in .NET which is what this is about but it's more than just a reference implementation it's also used as a library in many products. So many people don't actually write their own UPC UA stack from scratch but they just grab the reference implementation. So we looked at a lot of these applications and often they have either this implementation or they have a C version of the reference implementation. There's just only a couple of OPC UA implementations out there. You have a demonstration of the OPC UA reference client so that's part of the yeah the same coded OPC foundation develops and we can demonstrate it. On the left is the server on the right is the client that connects to that server and as part of the reference server there's some randomly generated data there. There's a boiler with a temperature or something that we can read from. It's just fake data that you can now read from the server. Now because we didn't want any memory corruption we thought those just application bypasses were quite interesting because that means that you essentially only have to audit one function. You have to audit it completely and make sure that you understand every part of it because it's such a constrained target it's just in one single place means you have very little other code to look at. So we spent quite a bit of time looking at the different certificate validation functions in the the four applications that were in this category and eventually we found one vulnerability. What's actually quite interesting is that this vulnerability was quite similar to something else we found last year. There were some vulnerabilities we found in the corona check app used here in the Netherlands that basically in the iOS version of the app completely disabled TLS validation except for one single check implicit check at the end that still worked. So they wanted to add certificate pinning but they completely messed it up so there was no certificate validation at all and this was actually quite similar to what we found here. If you want to read more about this you can find it on the website. So the process of how they want to validate the certificate in order to validate the certificate first you need to build a certificate chain. Quite often the other party only sends one certificate and then you need to make sure that you know the full chain to verify the cryptographic signatures on each step of that chain. But for some reason the developers of this reference implementation decided that they don't like the trust sore for certificates in Windows. They don't want to use it because it's tricky to configure. So what they do is that they they use their own logic to construct a potential certificate chain, a candidate. They construct this based on just the names in the certificate so it's not yet cryptographically validated and then they pass this chain to the X5 or 9 chain API to get it to validate it to check the signatures but because they don't really use it in a way that API was designed to be used they may encounter errors like an interested root because they don't configure the the trust store because they don't want to use the Windows build interest store. So then they have some codes to ignore these errors that they expect like the certificate root is the root is interested. So this was the idea behind this dysfunction but it's not actually how the code that they had written works because why they wanted to verify that chain what this API was actually doing is it was building a new chain using whatever methods it had for finding those certificates and then verifying that new chain. So if you could find some way to make those chains be different then you could make those errors disappear by making them coincide with the types of errors that would be ignored. So I'm going to try to show this to you within the code instead of actually trying to do it on the slides. So this is readable it could make it a bit larger I think. The main logic for verifying those certificates is in the internal validate function. This function gets that list of certificates that the other party sent and then it wants to check if it is trusted. So the first important function here to look at is the okay I'm this function that I accidentally deleted. We get issues no exception on get issuer and what this function does is that it tries to construct that chain and it has a this list it will at the return of this function contain that candidate chain that's not yet validated but that probably is going to be the chain of certificates. And this function works in a loop and every loop it tries to find the issuer of the current certificate so it has one certificate and now it wants to find what is the issuer of this certificate so it wants to go one step up into the chain. To do that it calls the function get issuer no exception and it has a couple of different certificate places it can look for that certificate and this function calls into match and this function loops through the list of trusted certificates or the interested certificates that are still installed or the list of certificates that the client sent and tries to find a match between those and then that function calls compare distinguished name which is the place where really this vulnerability originated. So a certificate has a name but name means a bit more than what you might think of as a name so a name has components like for example you can have a country and you can have a state and you can have a common name and that entire thing is considered the name and it has distinguished names components and normally in OpenSSL or many other TLS implementations to compare two names you just compare them bit by bit if it's just binary equal then they are equal but this function is way more complicated than that it actually takes the name apart into the different components and then sorts them and then compares them ignoring the case so uppercase lowercase are considered equal and that this is not how any other x5 or 9 implementation works as far as I know so this is something we can use to create two diverging chains so if you now go backwards to that internal validate function here it starts that x5 or 9 chain so here it goes into the actual api that's it hands it the certificate and then tries to build a chain with that and here we have the function which is going to ignore the errors that they expect to get here so one of the errors that they expect is an interested root and what this check here basically means this check if if it's the last one in the chain so if it's in the place where they think their trusted root is then it is okay so that that's the part that we can then use to hide the errors so to make this a bit more visual suppose the server has two trusted certificates let's encrypt and one named root and then it needs to validate the certificate and that's the certificate at the top the certificate says that it's issuer's root but with the case is swapped so lowercase are uppercase oot but the root is actually named root but uppercase r so then opcua this code considered as a potential candidate for um yeah the trusted root certificate and then it goes into that x519 chain to make this to do the cryptographic checking of that now the x519 chain api it doesn't really consider this one even a potential issuer because the name doesn't match so it's not going to say this cryptographic signature is wrong because it's not it's not even an option to check it so it doesn't really have a root certificate but very helpfully there's this extension it's called authority information access uh ca issuer's url something like that basically what you can do is you can put a url into a certificate which says the issuer of this certificate can be downloaded from this link so even though the server didn't have this certificate the the malicious certificate we can tell the server to download it from us so at this point this api will download that certificate from us and then because we have generated this one and that one the signature is just correct but the certificate is interested on the server now comes that function that ignores those expected errors so it sees well there's one interested certificate at the root well that was expected so we just accept the certificate so in this way we can by only knowing about the root certificate on the server we can forge a new root certificate and then create a new and certificate that is accepted by the server even though it was not signed by one of the actual trusted roots on the server and therefore we have then bypassed what's known as the application check so some more background on opc it has an option for encryption it can also be used without encryption and that encryption uses those certificates and both parties need to have show a certificate to the other party so in this case you could bypass both of those authentications if both use the same implementation and then intercept the connection and then we could manipulate whatever systems are trying to communicate here which is really hard to say in general but yeah it might be some ics systems at both end of the connection now we have also have a demonstration of this one again on the left that reference implementation and on the right our exploit connect to the server we see that it has a certificate issued by some root certificate we copy that root certificate and then generate a new certificate and connect to the server and yes you can see we have a incoming connection there and also here we can read that random boiler temperature values from the server now i'm going to hand over to Dan again to talk a little bit more about ics in general and our thoughts about it yeah so the word of two vulnerabilities we are allowed to talk about at this event the last three are still to be fixed so what we learned about the ics world is that in the ics world everything is about availability and we already noticed this when we discussed ics pentesting with our clients yep they all think that's a good idea and they all want to do pentesting on the ics network however at some point we need to say okay but we cannot guarantee that all systems will stay online because we are going to send well malicious data to every device and they might well go offline or a crash or whatever so as soon as we mentioned this the pentest is off the table and you're back to tabletop exercises which are also useful but they're all theoretical so yeah nothing you do in the ics world should jeopardize the availability of components having downtime means that the whole facility grinds to a halt because for most component most important machinery there is no backup so that can of course raise all kinds of issues can either be liability issues but it can also loss of revenue you can imagine if a factory grinds to a halt you cannot produce anymore or in some situations maybe even safety issues if the component is used to control bridges or whatever so that also means that things we consider common practice in it networks are not that common in ics systems one meaningful example is the installation of security updates that is not done in the ics world because well you have to take the devices offline because they need to reboot but even if that only takes 30 seconds there is still a chance and it might be a minor chance but nonetheless there is a chance that the security update also changes something else which makes the device unavailable so typically security updates are not installed because that could jeopardize the availability secondly is that the devices in the ics world have a considerable longer lifespan than in ics world and you have components that might have been online for the last 30 years which also means you need to defend or protect devices that use 30 year old technology and they are not built to withstand attacks from adversaries via the internet for example we spoke with multiple operators at the conference with with pwn2o miami was and they all had windows 95 or windows 98 devices still in service and they needed to be connected to the internal network because they needed to be remotely managed or they needed to share data to other components about for example the pressure in the boiler or whatever data they needed to share so thinking about security in the ics world really differs from security in the it world so in the it world we're getting to a point where the network no longer matters so even if the network is compromised devices are able to protect themselves this comes for example due to secure defaults that devices nowadays use and things like transport security that the network no longer matters that much this is also the reason why we can build zero trust networks where devices authenticate each other rather than trusting on the network ideally we also want the ics world to move in this direction however we see with the vulnerabilities we found in only a couple of week of research that most components in the ics network are unable to defend themselves all the vulnerabilities we found were what we consider low-hanging fruits there are no complex vulnerabilities there are no complex exploits i think the shortest time we went from installing the application into a running exploit was 15 minutes so we're still a long way from having a zero trust network in the ics world and if you speak with people that operate or are responsible for securing ics networks they have one defense strategy and that is network segmentation making sure that the attacker can never reach the ics network which if you have only two or three bridges to either the internet or the internal it network looks like a solid plan there are only three bridges to the internal network that is something you can monitor and defense and you can reason about all the different vulnerabilities that could arise in that small setup however speaking to that same same people most ics networks no longer have three bridges to the internet or the internal network they have a tenfold of that because every component wants to do remote management and most likely the management is done by different parties you have people responsible for machinery at site a or maybe a specific device at site b and not only for remote management but also for monitoring and sharing data the world talks about having doing big data analysis by sending all your ics information up to the cloud where you could do big data analysis and you could send that data back to the network so you could make monitoring decisions or operating decisions so network segmentation is a good strategy maybe if you have only a couple of bridges but in the current ics world we don't see this as a winning strategy so we think if we want to prevent the critical infrastructure from becoming hits we need to start making security devices and of course this is not something that is going to solve this problem immediately but this is of course also a hard problem to solve this is not something we're going to solve in the next year or so but i think that is the way forward secondly we don't think that having somebody responsible for ics security and somebody responsible for it security is a long-term strategy we think if you want to really make your ics network more secure you should consider it a single network with all kinds of devices if that either are laptops phones etc and you need to make sure that every device is secure and not making the distinction so hard about the type of device so if you want to read the full write-ups you can find them at our blogs we will also publish the three write-ups we haven't been able to talk about today as soon as the vendor has fixed them those vulnerabilities are much more like the first one in ignition we showed you like a really simple vulnerability to exploit and the trusted application bypass where thys talked about that was by far the most difficult vulnerability we demonstrated during this year's pound to one so if you are responsible for ics security we would love to have a talk with you so please come to our tent and we can discuss this further and if you like memory corruption for abilities we will talk about zoom tomorrow at this place at five which should also be a fun presentation thank you all for your attention and if you have any questions we'll be happy to answer them cool and slightly scary thank you very much folks we have plenty of time for questions so if you have a question for Tess and Dan there are microphones in the central one the back one at the front please line up talk nice and closely to the microphone but this is probably too close thank you thank you for a very interesting talk can you give us a little inside for you into how you research in the applications to find those vulnerabilities sure so one of the first things you do if you want to look for a vulnerability is that you look at what everybody else has already done so one of the things we did is there was another early edition of punts on miami where also there were some vulnerabilities in inductive automation ignition and we read those write-ups we tried to understand what was wrong there was some java deserialization vulnerability in the same api that we used so our starting point was to look at that same api can we still do the same thing but the conclusion was no they check for those deserialization vulnerabilities for an authenticated api calls if you are authenticated then the checks were much more lenient so that really led us to yeah this idea of can we somehow bypass that authentication and that's how we got to looking for that active directory bypass so it's often just trying to find what everybody else has already done we're trying to find easy places to start especially in these applications where you have no idea what they are doing it's very helpful to have somebody who points to a specific place in the code or a specific feature that can be useful to exploit typically you want to have a focus point because otherwise the attack service is too broad so we typically focus on things we consider difficult to implement correctly or easy to miss certain security checks and we really focus on those areas and we just do a deep dive and we try to find a vulnerability and if not we try to find a different focus point but you need to have a very specific focus point which for the trusted application bypass was easy because yeah the only answer to one function the certificate check function and all the other stuff because we didn't care about the remote code execution vulnerabilities etc we just ignored them so that made it easier about the sorry about the certification bypass what was the reason that defender decided to ignore an invalid root certificate so that's because as they described it to us they don't want to use the window certificate store because it's difficult to manage for people in practice according to them so what they wanted was to just have one directory where you can play some .pam files and that's your trusted certificate root so that's why they didn't configure the trusted root certificates of the that API that they were using so yeah that leads to certain errors that may happen because they didn't configure it in a way that is intended to which can yeah they then have to ignore because yeah it's actually trusted because they checked that i think that it should be but we haven't looked at the API ourselves but i think it should be possible to just use that API and give it some trusted root certificates say okay check this chain these are the rest root certificates we trust don't use the windows trust store use this one but they didn't do that so they built their own implementation and then verified that with the API yeah and then you get two different change and all kinds of complexity yeah we are not net developers we are not sure if it's really not possible to do this and they also need to keep in mind that they need to it needs to run on windows macOS analytics so they there's some differences between the implementations there as well i see thank you very much of course hey so you call these vulnerabilities long-hanging fruit um do you know how many others looked at the same software and if the how many collisions there were so there were quite a few collisions we even had one collision one of the remote code execution vulnerabilities was a collision there was also the vulnerability we found in 15 minutes so we already figured that there is quite a reasonable assumption that somebody else has found the same vulnerability yeah so the war is on collisions but uh yeah most applications have very wide attack surface so there is still more vulnerabilities to be found uh in uh more obscure places good things new okay if there's no more questions uh i think all that remains is to once again give tess and dana a very big round of applause thank you very much