 Okay. Hello everyone. My name is Dori Ardeny. I'm the head of incident response and hunting at Otorio. I've previously served as a search team leader in the IDF, the Israel Defense Forces. I managed threat hunting operations and also dealt with targeted cyber threats on big organizations. I also bring hacking knowledge from my experience as a red team researcher. So I hope you will enjoy this session. The mutual effort together with OSI Soft and Mike was really cool. Thanks Dori. And this is Mike Lemley. I work at OSI Soft. We were in response to the wonderful research that Dori presented to us on the vulnerability discovered in our product. I've been at OSI Soft since 1999. I focused primarily in software development. And then about seven years, I started focusing on cybersecurity and now work as part of our central architecture team. I'm hoping to improve the cybersecurity practices of our development at OSI Soft. Thanks Mike. So let's start with a quick story. And at the end of it, we'll get to my point. So two weeks ago, a shift leader at a chemical plant received an alert from the data management platform. Platform like the one of OSI Soft. The alert reported about a boiler that exceeded from its threshold temperature. The shift leader was really nervous. And in addition to that, databases of all historical plant data and their backups were deleted, which means they had no access to industry 4.0 analytics, like people like to say. The plant was shut down due to this abnormal activity. The concern was physical damage and human loss. The shutdown caused a loss of significant production time, which means a lot of money. So why am I telling you this story? I'm telling you this because it's not a nightmare. It's not a fairy tale story. When you base your decisions on the data management platform, especially what has been manipulated, you are technically blind. You are blind from your operational environment. So we have two core goals that we want to achieve from this session. The first one is to raise awareness about the implications of industrial-oriented vulnerability, where each vulnerability is a serious threat to our sensitive environment. And it's not only ICS vulnerability. It could be also IT vulnerabilities that somehow affect directly on our production flow. That the second goal is to showcase how the discovery of one vulnerability can help other vendors. Mike will talk about the way things proceeded after we identified the vulnerability in their product. I want to talk a little bit about what the pie system is from OSISoft. We can start off with our core customers. So these six areas of business are where most of our customers' industry. And the company OSISoft has been around for about 40 years. We have customers in 140 different countries. And this world map gives you an indication of where we have support offices to help. This reference architecture explains how a lot of our customers. Now what we want is we want to reduce the amount of people that have access within the critical system. And the OSISoft pie system infrastructure allows them to do that by pulling the data that is needed for analysis and business decisions out of the critical system. So that those people don't need to have access there so they can in order to do their job. This additional data reference architecture also represents how the pie system is used. Now the pie server inside of the control center is used to extract the data from the control center using hundreds of different protocols based on the different types of equipment there. Data is then sent over to the corporate access or corporate network zone where it is then used by those employees in order to do their analysis. Now the target for this particular incident is within the vision server. And that is where the API the API server exists that is being used and attacked by this particular incident which is cross-sites. Okay so before I dive into the details of the vulnerability suppose you ask yourself why we even started looking at pie server. And the answer is that we had to put some effort into product tasks. We had to integrate our OSISoft platform the pie server into our security orchestration and response platform. But as a curious researcher I decided to test the asset framework of the pie server on XSS. It was very late at night so realistic as it sound. I thought again the web API parsing mechanism to in order to inject JavaScript code into client browsers and in the end I succeeded and we are here. So here is a screenshot from the asset framework. On the left we have a simple plant tolio plant with boiler and tank. Each element has attributes for example the boiler has attribute of temperature and the tank has an attribute of volume. And element one is the child element only for my tests. On the right you can see the description field, the vulnerable field with encoded base 64 JavaScript. In order to bypass the web API parsing mechanism you have to insert some magic characters at the beginning of the beginning of it. Once an attacker manages to do so he or she can run JavaScript code on client browser for phishing, key login, etc. So here is a possible scenario for this attack. First an attacker is stealing low privileges credentials inside the network. After that the attacker is injecting the malicious JavaScript to the vulnerable field that we saw in the previous slide. Then a victim for example an innocent engineer is accessing the web API infected page. And then a fake login form is prompting for username and password and also in parallel changing the notification rule template on behalf of the engineer high privileges credentials. At the end of it the attacker is receiving the victim credentials and also causing temperature force positive alert. Okay so let's see it in action. On the right is the server of the attacker. This is on port 8080 for the stolen credentials. On the left the victim browser. Once the victim passes the cursor over the infected field the description field the fake login form is prompting for username and password. The victim is inserting the pyadmin password. Once the victim submits credentials the attacker gets the stolen credentials on the right. Pyadmin and the strong password. So from here it's pretty much game over. And in the same method in the same technique I could also add a JavaScript code to change the notification rule template in order to change the rule and to cause positive alert of high temperature. Okay so this screenshot was taken from US side. The vulnerability has got rank of 7.7. OSI soft also added us in their all of thanks list here in the bottom. And I suppose you ask yourself how to avoid these kind of attacks. So the answer is patching. OSI soft released a patch for the vulnerability very very quickly but we have to do a lot more. A lot more proactive actions such as right hunting operations on our sensitive environment to out and fire rules and also to get visibility into our assets and their version in order to in order to perform vulnerability management etc. But that's that's for another talk. Now I'll talk about the lessons that OSI soft learned from this incident and how we responded. Now to start off with I'm going to talk a little bit about our security development lifecycle process at OSI soft and that'll define how we change what we responded. So we at OSI soft establish several different security best practices and then we do as we measure the adoption by our 40 around 40 development teams to apply those security best practices in their development. Now why do we measure? I constantly say I mean what good are best practices if you don't measure to see how well you are adopting them. But in addition to that it really helps teams plan. When you go through the process of identifying measurements for the best practices it forces you to identify milestones and those milestones are very useful for the teams to to plan against and plan their development activities. And other thing that it does is it forces you to organize and prioritize all these best practices and that also helps the teams identify how to best move forward. Now looking at our SDL process focused entirely on cross-site scripting that allowed us to see where we could improve our process. So we have education of defensive programming. We also have various tools that we use and the vendors which we use for these various tools are listed on the right side along with some of the security researchers we engage with. So as I said we have various security tools we use. There's static analysis, software component analysis which actually helps us find cross-site scripting vulnerabilities in components that we consume in open source products. We have various security headers that we encourage use of as well as the external reviews and incident response which was very key in how we handle this issue that came to us from Otario. Now in looking at those processes we defined there were some areas we thought we could improve or add on to. So the first one is an internal security tech talk. We do tech talks weekly at OSI Soft. Every third week they are focused on security and we did a tech talk specifically on this incident so we could share the lessons learned from this with the other of the 40 teams at OSI Soft. We also looked at how we use the static application security testing tool and identified some process improvements for our use of that tool. And the last is we decided to get much more aggressive on the adoption of content security policy at OSI Soft. Now I did say that we do measure our best practices and I just wanted to give you a sense of what some of the measurements are that we use associated with just our cross-site scripting defenses at OSI Soft in the best practice areas I find. Now we didn't do any process improvements on our DAS tool and I'm going to talk briefly about that. Now the DAS tool that we use is from QALIS and for those of you who don't remember what a DAS tool is it's basically it's run as a your application is up and running and then the the DAS tool will send dynamic tests at your web server typically OSI Top 10 attacks and see whether or not your web server has the appropriate implementations in place to protect against it. Now in this particular cross-site scripting issue as Dor explained it was a stored cross-site scripting attack but in looking at the tools it's more of a DOM cross-site scripting vulnerability with use of insecure use of NRH GML. Now our DAS tool QALIS is very good when the source of the NRH GML parsing the bad package is from a document URL cookies or refer header but it doesn't really handle well when the source is from an HTML document so we decided not to focus there. Now our static application security testing tool is from synopsis and if you look at the bottom right as a reminder for what SAS tools do they analyze your code so they're looking directly at your code look at call stacks how functions are reached and identify insecure call paths or insecure calling sequences. So we felt that was the best tool it should identify where NRH GML is used and then it should be able to flag insecure uses of it. So what went wrong with this? We identified that there was a tool compatibility problem with the with the NuGet package that we used it's a razor engine and our tool synopsis did not work well with it. It does work very well with the razor analysis built into ASP.NET. So consequent to this problem our scanning for this particular project missed all JavaScript and CSH GML. So the process improvements we identified is we need to be able to do some central evaluation of the SAS configurations and we're working with synopsis on how to best move that idea forward. The other thing we're going to do is for this one particular development team we're going to move them to using the ASP.NET MPC razor engine instead and then we're also considering increasing the sensitivity to completely disallow the use of NRH GML. We're not sure exactly how this is going to play out but NRH GML is definitely not something we want to use so we're either going to be heavily scrutinizing the use of NRH GML or completely disallow it. So we're still working that one out. Next slide please. Now I'm going to switch context and talk a little bit about content security policy and just to give you a why behind to that I'm focusing on jQuery. Now over the last couple of years jQuery has released several different vulnerabilities. These all in this chart are the cross-site scripting vulnerability and as you can see with the overlay that was just added they all fall in the high range for CVSS scoring which means you have to take these two vulnerabilities very seriously. Cross-site scripting in general does tend to fall into the high vulnerability area and dealing with all of these different patch updates from this open source does present challenges to vendors which consume open source and components under the scrutiny of security research. Next slide. Now content security policy just to explain how it works. The web server will send down HTML and JavaScript content to your browser and now your browser is theoretically susceptible to some sort of a JavaScript injection via a cross-site scripting payload which has been architected by the attacker. Now by adding content security policy from the same web server you can now inform the browser of the allowed list where JavaScript is allowed to come from. Now the browser has the ability to identify that hey this cross-site scripting payload is not on my allowed list I'm not going to allow it to execute. I also added the CDN content delivery network so just to let you know that the content security policies allow list does incorporate more than just the web server that it came from so there's lots of flexibility there. Next slide please. Now we strongly believe that content security policy has reached the point where it is extremely mature. The major browsers all have implementations for it so we think that the industry is in a good place where we can start demanding the use of content security policy in the products that we consume. Now a question I often get i.e. Internet Explorer. The Internet Explorer I just want to talk about it briefly is Internet Explorer. I get this question a lot you know it doesn't support a content security policy is that a problem and this will not stop Internet Explorer from working so you don't have to tell customers they can't use Internet Explorer anymore but what we are taking the position of this is that we want to inform our customers that you should be using modern browsers for their security benefit. One issue and one protection is content security policy. Now I talked about having measurements for our best practices and this is the way one of the ways that we do it we've identified poor, better, and best practices for our development teams. I came up with ways for them to measure their adoption against that. So the poor practice would be using unsafe inline and unsafe eval. The next better practice would be if you do have inline JavaScript to provide nonsense or hashes for it. But the best practice is to remove all those inline JavaScript and eval statements and report any failure. I give the browser an endpoint to report any any occurrences of defenses or blocks that it had to execute to a central reporting location so that those issues can be investigated and fixed. And just to give you a general sense of how we do this measurement within OSIsoft, all the measurements we've identified from the best practices have response options. They're all given scores. So you get a score or a scale from 0 to 10 for all of them. And they can all be combined and you can look at collectively how teams are doing it in response to adoption of these best practices. But in addition to this we decided to not only revamp the responses in order to encourage better adoption, but we decided also to take the step of making this required for releases going out by the end of this year. Excellent. Now focusing on the key takeaways, the things that I covered. The first one was security tools. Using security tools is not enough. You really have to evaluate how you're using them in our case for better code coverage. Next one is content security policy. We believe the industry is mature enough that content security policy is in a place that this should become an ICS standard practice, we believe. It gives you a strong second level of defense. Like with a lot of open source components out there, I mean most of us consume open source. And when the vulnerabilities are discovered in those open source, the public disclosure happens to the public at the same time it happens to us. We know about the vulnerability at the same time everyone else does. So there is limited time that we have to patch and get the protections in place. But by having that second level of defense content security policy there, it gives us more time to get the patches in place correctly. And the last thing, engaging with companies like Atario and Doork is very, very successful in moving security forward for our products and it's been a great experience working with them for us. Thank you. Thank you. Thank you very much Mike. The mutual effort was really cool and we very enjoyed this research.