 All right. So this is hacker machine interface state of the union for SCADA HMI vulnerabilities. Um it's not uh you know I don't want to the title slide here to indicate that this is about stuffy shirts and racks. I mean this is about hardcore exploitation in this talk. Um we're gonna cover an in-depth analysis of a corpus of 200 plus uh confirmed HMI vulnerabilities that have come through the zero-day initiative program. We're gonna detail out the popular vulnerability types that have been discovered in these HMI solutions uh and we're gonna talk about how they're developed and and how the weaknesses actually manifest in the underlying code. We're gonna talk about some of the biggest SCADA vendors that exist on the planet including Schneider Electric, General Electric, and Advantec. But the uh the all of the vulnerabilities that we're gonna be talking about can be applied to pretty much every SCADA vendor that's out there today. This talk will also cover uh and compare a uh time to patch performance for the various SCADA vendors in the industry. And it will also compare the SCADA industry itself against the other software, other entities in the software industry. And finally we're gonna use the data that we presented uh to provide you additional guidance on what SCADA researchers should be looking for in HMI solutions and what we can expect in future attacks against SCADA HMI solutions. But first let me let's uh introduce ourselves so you know who we are. I'll let Fred Fritz introduce himself. Ah good day. My name's Fritz Sands. Uh my Twitter handles Fritz Sands because I'm occasionally boring. I was a long time developer at Microsoft 25 years in uh the Windows operating system and then I joined the trustworthy uh computing and secure Windows initiative in 2001 when the big security push happened to Microsoft. I uh left Microsoft in 2014 and joined the zero day initiative where I have been investigating software in the real world which has given me a deep appreciation for the code quality at Microsoft which I did not have when I was there. Uh so I'm the senior manager uh vulnerability research in Trend Micro's tipping point organization. Uh my primary responsibility in this job is to actually run and manage the zero day initiative program which represents the world's largest vendor agnostic bug bounty program. We've been in operation for 10 years. We spent over 13 million dollars on vulnerabilities over those years. Um we do a lot of root cause analysis working with researchers around the world to buy bugs uh define how they actually fire and help the vendors get them fixed in a proper way. I'm also the organizer of the ever popular PON to own hacking competition where I spend probably half a million dollars this year just on exploits uh against the hardest attack surfaces in the world. So before we get started we want to give you kind of an overview and level set everybody in the audience about what we're talking about here, what the SCADA industry is, what HMI is, and who are the heavy hitters in this industry. A lot of the marketplace if you look at it is really focused on developing hardware and selling control systems uh and not so much focused on the selling of the HMI solution itself. In fact many of them are freely downloadable which makes them good targets for auditing. So in this case they kind of focus on hardware hardware uh software um or software that runs on hardware and less on Windows applications and that really shows in the type of vulnerabilities that we're seeing come through the zero day initiative program. It is a highly regionalized market. So there are vendors in China that specifically develop SCADA software and hardware for Chinese uh for Chinese implementation. There's also ones in Germany uh and we've even seen code developed by uh by Italian developers. Um so it's it's a very active market and if you if you are focusing on SCADA uh products in one region it'll be completely different in another region. As you can see on the slide we've got a bunch of big names on there. That Wecon brand is the one that is the is actually for China. We found, I personally found a dozen bugs in their in their HMI solutions uh and submitted them and and had them fixed. Um Siemens is also a major brand uh and NGE Electric or General Electric Adantek which we'll talk heavily about in in this presentation. Now there is also a lot of mergers and acquisitions in this space. Uh it's very much like the rest of the software market, lots of buying, lots of selling. Um but and the one interesting thing that we find is when we buy a vulnerability in some of the mom and pop uh SCADA developed uh HMI shops which there are a lot of them. We'll see that by the time the patch comes out they've actually been acquired by one of the bigger companies like Schneider Electric or Siemens. So there is a lot of merger and acquisition that goes on which makes the disclosure process uh process a little bit more complicated. If we look at the human machine interface what exactly is it? Well its primary job is to provide status of the critical infrastructure. Things like alarms, notifications, they also provide highly advanced and customizable visualizations that give operators insight into what is going on in their critical infrastructure. Um and a lot of these you know there you can kind of develop these and customize these uh these visualizations for different components in your actual uh infrastructure. Now there's supposed to be air gap and run on isolated and trusted networks but this is really not always the case and we'll talk about attacks here in a couple of minutes where they took advantage of HMI's that were not on uh isolated networks. Now even isolation uh is is not guaranteeing security if you ask the Iranians back when Stuxnet came out the air gap network didn't really provide them much value uh when they when they were being exploited using USB uh link vulnerabilities that existed. Um so if the developers are actually spending the time and and and thinking that the fact that there are HMI solutions are going to be used in air gap networks and not putting security in that's what we're seeing in the code that's what it feels like. They're not actually spending a lot of time implementing best practices of the industry. So why would you target an HMI solution as an attacker? Well it's because it controls the infrastructure you can actually see and and get configuration information about the devices on the network and it can actually be used by itself without a vulnerability to shut down a network to shut down critical infrastructure. This is this is the case in the Ukrainian attack that happened at the end of last year. Ukrainian uh the attackers who were going after the Ukrainian power companies just used the HMI solution by itself to trip breakers and shut down the power and they were not actually exploiting HMI vulnerabilities but they were using the HMI system to actually take uh take the systems down. Now it can also be used you can actually attack these to deceive uh and disable alarm systems in the in the control system itself. And this is the case in Stuxnet where they actually deceive the operators um and and about the state of the centrifuges that that they were controlling and actually send send uh commands to trigger uh self uh self uh uh destruction um conditions in the actual control systems themselves. So there are active attacks in in um HMI solutions. If we look at uh you know Stuxnet is obviously the most popular uh one that we we talk about. Everybody knows about this one but it did leverage vulnerabilities in in HMI solutions including uh Siemens Semantic Step 7 DLL hijacking vulnerability uh along with a SQL server authentication bug. Alright so these are really simple bugs very common bugs in HMI solutions and it leverages those to deceive the operators about the state of the centrifuge. Now black energy is an ongoing sophisticated malware campaign against ICS environments um and it actually does target HMI vulnerabilities. The GE simplicity path traversal vulnerability it's used it's it's uh we believe it to have used the vulnerabilities in um Siemens 1CC and advantage of remote uh web access. So quite famously in the ZDI program uh the GE simplicity vulnerability is actually one that we purchased from an anonymous researcher and disclosed the ICS cert and it turned out that that was actively being used by black energy. So it's kind of interesting to see that happening uh in the wild. Now another big player in the industry is ICS cert and as a researcher you need to understand who this organization is and where they sit in the government. So I'll rattle off the title and their location in the government. They're the industrial control system cyber emergency response team which operates within the national cybersecurity and integration center a division of department of homeland securities office of cybersecurity and communication. I mean that's a long name. I'm almost getting paid by letter there. So but in reality they are a very important organization and people um and who are researching in HMI uh need to know who they are and and and learn how to work with them. We work with them every day uh in our jobs because we're purchasing a lot of vulnerabilities uh in HMI and they they do a lot of things they actually release a report every year about all the stuff that they're doing and according to the 2015 report they actually responded to 295 incidents and handled a 486 of vulnerability disclosures and that's significant. That's uh that's a lot of vulnerabilities passing through that organization every year. Because it's so regional uh it's it's often hard to get a hold of these mom and pop organizations when you find a a vulnerability in their solution and at this point you come to programs like the zero day initiative or go to ICS cert to help you disclose those vulnerabilities and get them fixed. So let's talk about a tax that leveraged HMI features or vulnerabilities uh uh in uh in their active attacks. Um if you read the Verizon data breach report they talk about a uh uh an incident where their team went in and actually it was called in to analyze um the security of a water utility. Now they don't give the name of the water utility but they do talk about their findings and in this case they found that there was an internet facing AS-400 system responsible for HMI like capabilities uh like manipulating uh PLCs. But the system also did network routing and managed customer data. I mean how ridiculous is that that all of that information is sitting on one system connected to the internet. Both critical infrastructure and billing systems. Um this is kind of a an example prime example that there's no focus on the separation of responsibilities when they're architecting these critical networks. Now what they learned it is that what they discovered was there are four separate connections to this AS-400 over a 60 day period where the IPs were tied to hacktivist uh activities and they actually altered the the water flow and the chemicals in in that system. Now according to the report they they say that the attackers really didn't understand what they were what they were working against and they didn't couldn't really didn't do a lot of damage but they could have done a lot of damage by that by accessing that system. The most recent example of a really high profile skate attack was the it was in the Ukraine where there's several your uh Ukrainian companies that experienced unscheduled power outers which is which affect almost a quarter million people. These were caused by malicious actors and there's actually a really great report on the ICS website um ICS cert website that describes all the details about that attack that are unclassified. Um so and they talk about how the attack was coordinated and they all attacked within 30 minutes and in this case they didn't use HMI vulnerabilities but they leveraged that HMI solution because they were because there's no isolation they were able to VPN into this into the network and get access to the HMI solutions and use remote administration tools which dost the operators from making any changes and they actually just tweaked the the the knobs in the HMI solution to turn off and trip breakers and as a result the power went out. They also uh put kill disc malware on the Windows based HMI systems which which basically brought them to their knees and really hurt their restoration efforts. Now this is obviously used to destabilize the Ukraine a little bit. I don't think they've actually attributed the attack but there is a lot of political stuff going on in that region so you can imagine. Now there's also some interesting report uh some interesting research that came out of a sister organization here inside of our company uh where they actually looked at the malware that was used in the Ukrainian attack and actually found links to malware in other companies in the Ukraine including a rail company and a mining company around the same time. Now black energy was supposedly not used in the attacks uh against the Ukrainian power company but it did exist in that network. So you can imagine that the attackers who were going after those are probably the same attackers who had access to some rail and mining companies in the Ukraine as well. How did they how did how did our the our sister organization know this? They looked at the infrastructure of the malware and the naming conventions that were used and they released a white paper on it and it's actually really really interesting and it's worth a read on the it's on the Trend Micro blog. So let's let's talk about the prevalent vulnerability types that exist in HMI solutions and what the current state really is. So the reality of the situation is the HMI solutions have not seen any benefit of the evolution of secure software development life cycles over the last ten years. We have looked at a lot of the code you know dozens and dozens of code bases uh we've analyzed and looked at vulnerabilities and confirmed zero days and that's what we've learned. There really is no security built into that software. They haven't seen any any benefits of the secure development life cycle that Microsoft Apple all these other companies have and this is actually a really scary thing. In fact most of the solutions that we're vetting bugs in do not have ASLR uh safe safe safe SEH or stack cookies enabled which is really really scary and we actually urge skater vendors to turn on all of these mitigations including things like building 64 bit apps and and to to make ASLR better uh and actually reducing the reliability of heat sprays and also turning on just the the basic mitigations that are available by flipping a toggle in the compiler. It's all it's actually really embarrassing. It's also there's also a lack of understanding of how these are really these uh solutions are actually run. They they seem to think that they're going to be running that isolated environment and they're using that as a way to not implement security mitigations but they are continually being integrated and you've seen attacks in the wild that leverage interconnected HMI solutions to take down critical infrastructure. So this is probably the only pie chart that you're going to see a DEF CON and I'm actually really proud of that because I did a lot of work to generate this pie chart but in uh what we ended up doing was we pulled all the 2016 and 2015 ICS cert advisories uh and identified all of the HMI solutions that had bugs fixed over the last two years. We cross referenced that with our 250 plus zero day vulnerabilities that we've purchased in HMI solutions to come up with what the most popular, most common vulnerability types are in HMI solutions. Um we also cataloged the CWEs to kind of get an idea of what vulnerabilities existed and and they're here they are listed on the slide. So uh the number one is memory corruption uh followed by credential management usually hard coded passwords, insecure defaults, authentication and authorization and code injection issues. Now what about cross site scripting? What about cross site request for delivery? Well most of these are windows based applications. There are some web based applications but most of them are windows and as a result you're not going to see a lot of that cross site scripting stuff but there are some in that gray area on the slide. So what we're going to do is let now let's let's get down dirty with this. Let's let's look at every single one of those categories and we're going to give you case studies of what these look like so you can understand how terrible this code base really is and what you need to understand to actually go find these bugs and to protect yourself against these bugs and that's the most important part. So first we're going to talk about code injection vulnerabilities. This makes up about 9% of the common vulnerability types that exist in these products and that you know it's the classic stuff, SQL injection, code injection, OS command injection but there's other domain specific languages that exist in this software and that's what we're going to talk about today. We didn't want to cover like stuff you guys already know. This is we're going to talk about gamma code injection right so this is a domain specific language that that's used in this industry. Specifically we're going to talk about cogent data hub and we're going to talk about CVE 2015 3789. Now this allows this vulnerability actually allows an attacker to turn on an insecure processing mode in the web server which allows the attacker to send arbitrary scripts to the server and execute arbitrary code. This was discovered by anonymous researcher and purchased by us disclosed to ICS cert and fixed and we do offer the ability for people to submit bugs to us in an anonymous fashion and we get a lot of that actually through our program. So what is cogent data hub? Well that's what you see on the screen here that's one of those visualizations that I was talking about. Cogent data hub is a real-time middleware solution that is deployed across several sectors including chemical, commercial, critical manufacturing, energy, financial et cetera and it's used around the world. It offers the end user the ability to create those really intense advanced visualizations that what you see on the slide here customize those so they can monitor their underlying network. So what is GammaScript? Well GammaScript is a domain specific language specifically designed for the use within data hub. It's a dynamically typed interpreted programming language specifically designed for rapid application development. It looks like C and I'll show you some here in a second and it has a range of built-in features that's got libraries and everything. It actually has a fully documented API that you can read on the internet and it's actually pretty full featured for those application developers. Now the attack itself is a flaw in a Val expression method and it allows an attacker to execute arbitrary code on the system. It actually sits and is accessible through an Ajax facility on port 80 and you simply supply a well-formatted GammaScript which allows underlying code execution. Now the interesting thing about this is domain specific so there's a lot of functionality in Gamma that's specifically used for developing that stuff but unfortunately it did have in that script the ability to execute system commands. So what is the vulnerable code? It's right here on the screen it's very very simple. A Val expression basically takes an expression and checks one flag. Am I allowed to execute this expression and if it does then it executes the expression. And this is whatever you want to send to the system. Now the question is how do you actually get that to load up and how do you change that value? Well it also allows you to do that as well. And the exploitation steps are you send a request, an HTTP request to the port 80 which will load the GammaScript libraries then you go you call Ajax support to allow expressions and which will set allow any expression to true and then you call a Val expression with whatever script you want and you execute code. So let's demo that exploit. So what you see here is the installation of data hub. You can see, let me kind of zoom in for the audience here. I'll forget that. So what we're going to do is right here on the screen what's highlighted is cogent data hub version 7 and it's running and sitting on port 80. And what the first thing we're going to do is we're going to run a proof of concept here at the bottom that is a just basically a Python script that sends the three commands that we need and will actually disclose information on the server it actually is disclosing autoexec.bat on that box. So then we're going to send another script which will actually execute the evil evil calculator. And you'll see here it's actually a very very reliable bug and a very reliable exploit. You can just kind of send it over and over and over again and there's those evil calculators you know. So that's a pretty fun bug. Really simple bug and they did actually do a really good job fixing it. And you can see here all the calculators being spawned by the process. So now how do they patch the bug? This is kind of one of the interesting things for the ZDI program because when bugs get patched researchers will also submit bugs POCs that actually break the patches which is kind of interesting but here it's going to be kind of difficult. So on your left is the old code and on your right is the new code and you can see up here that they actually removed allow expressions so you cannot access that at all. So you can no longer toggle that flag in the system. They also removed eval expression entirely and actually gave it a really great comment which is actually a best practice. This method is dangerous it could allow somebody to execute arbitrary code by an HTTP call. If you absolutely need it create a script and define it and then make sure that your web server is on a trusted network. So that code is buried in the application itself so it's highly unlikely that developer is going to go look at that but they're just going to call the APIs. But it is good that the fact that they actually documented that so they won't regress that bug at some point. So that's how that bug actually works. So I want to turn it over to Fritz and he's going to cover the rest of the prevalent vulnerability types and then we'll talk about some some disclosure statistics. Hello again. So the next section we're going to look at is authentication and authorization problems and authentication bypass improper access control improper privilege management bad authentication and what we're going to focus on is an advantage case and you're going to hear advantage a lot and this is actually a pretty fun one as information disclosure. And this is a CBE 2016 5810 and I see a search says a properly authenticated minister can view passwords for other administrators but the terminology is a little unfortunate here because this is not a system administrator. This is an administrator of a given SCADA solution a given project and so that is sort of akin to unprivileged users of the system. So this is an essence saying one user can extract the password of another user and this was discovered by zoo you and disclosed by the zero day initiative and it's sort of fun and basically they have a script and ASP script that allows you to change your username your password your description and this is great but it can be abused and the way you do it is you log in to the account you have so this is not anonymous you have to have an account on the system but then you can change the URL to give any other name and then pass that in and it will bring you back the password of the second account I can't see the password because it's got asterisks in front of it. Yeah. So here's the demo showing it. So first log in as the admin and by the way you can also get the full system administrator account this way and you can see that there's a test one and a test two user. So now you log in as test one and put in your password for test one and that's all great. Now if you try to change you change to test two using the UI it will quite properly give you an error saying you can't do that but if you change to a user name a test two in the URL it will pop it back but it's got those asterisks so we got to fix that. So you view the source and there's the password and then you can use that password of course and log in as anyone else and including of course the complete system administrator of all of the solutions. There you got it and you're logged in and what okay here it does. So that one was sort of fun and there's also a lot of insecure defaults in this space. Bad transmission of information, missing encryption, unsafe act of X controls, yes we're backed to act of X controls. So the one we're going to focus on here is the Schneider electric DS and VS and this is a bad act of X control with memory corruption. Now even though it's memory corruption we put it in here because this act of X control well first it was set as safer scripting from untrusted source and but what's also interesting is it was never meant as a control to be used in Internet Explorer in a web page. So it should have been configured as automatically kill-bitted. So it's really bad configuration so it was wide open to Internet Explorer when it should not have been. And this is CVE 20150982 and the Schneider electric PELCO here is an HMI for digital sentry video surveillance systems. So it's really great. You can use this to you know get information on video surveillance systems which is always fun. Well I wanted to do show this for people who are going and auditing looking for act of X controls it might be vulnerable. This shows an interesting second step you often need to take. There are two ways to tell the system that an act of X control is not safe for scripting. The standard way, the fast way is statically in the registry to mark it is not safe for scripting. But if you know it turned it on to make it safer scripting is to flag it as safe for scripting. But if it's not marked in the registry as safe for scripting it can use the interface I object safety and in dynamic run time assert that it is safe for scripting. So even though it's in the registry as not marked as safe for scripting it still is potentially vulnerable. So you've got to look at the dynamic situation as well as the static situation so you can't just do one. And here's just the demo of how the memory corruption works. Which is you use internet explorer and you go to an attacking web page which invokes the control in IE and it does a stack buffer overflow and builds everything with your classic 41s that we all know and love. Let's talk about some credential management problems. This actually I was really shocked when I ran into this because it's like you're kidding right. This happens a lot that they hard code credentials in the code hard coded passwords. You know I thought we'd gotten rid of that 15 years ago with IIS. But well it's we're in SCADA space you're hacking like it's 1999. It's it's awesome. It's awesome. We're back then. So this is the one we're going to look at is GE MDS pulse net and it's got a hidden support account and this is really fun. So this is used to monitor devices and industrial communication networks and it's deployed in energy water and wastewater sectors and it's used worldwide. This is CBE 2015 6456. So if you take a look at the user management panel using the UI you see that there are exactly two accounts in the system. There's an admin and an operator. Well that lies. If you go in and use the I use tidy SQL but if you use anything that extracts information from the database you see that there are not two accounts there are three. There's a hidden account called GE support. Now now it's really super subtle because it only stashes the MD5 hash of the password not the password itself. You certainly no one here could crack an MD5 hash right. It turns out that the password is actually pulse net but they made it leak by changing the L201. And here's the demo. You can see on the right the two users that are officially there. And on the left we will log in as the user that isn't there. And what I think is really cool is that even after you log in as the user that isn't there. As you're logged in as the user that isn't there it tells you that that user is still not there. Which we just sort of slick I think. There's also a lot of other misconfigurations. One of the other ones that we see a lot is where companies decide to roll their own. Ackles and they decide they don't want to put things under program files as Microsoft intended. And so they create their own top level directory with their company name. Under the C drive. And they often put you know world has full access. And then they put their service binaries in there. So any local user can drop new binaries in and they will run as a system service. So this is very standard. Now we get to the joy of memory corruption. Stack based buffer overflows, heat based buffer overflows out of bound read right. Just the classic ones. And the advantage is our whipping boy here because they did an awesome job here. We got a hundred bugs in one day. From an anonymous researcher. This was like this this data dump from heaven. And we analyzed them and passed them on. And they were all buffer overflows. And it was quite impressive. And I'll go drill into one in particular. This is CVE twenty sixteen oh eight five six. And it was an anonymous researcher and disclosed by us. And this is their web page and what's really interesting about web access is that the SCADA solution. But they also advertise as you can see in here. That this is for Internet of Things. So this is widely deployable. And it's awesomely vulnerable. Launches a service web. The RPC in the context of local administrator. And listens on forty nine fifty two and the web the service calls are configured to look like Microsoft IO access control calls. So they've got an I octal value and they do jump tables off of that to perform. Hundreds and hundreds of types of services. For this particular one. The parameter that's passed is a window name. Which is then copied using S. print out to a stack buffer. That is hex eighty characters. And as you can see in this packet. The link is hex eight C. So it copies hex eight C bytes into a hex eight zero by buffer on the stack. With predictable consequences. And so inside. You've got this and the flaw is the stack based buffer overflow. Here's the classic S. print F. call. You know nothing of a surprise there. Here is the stack layout and this is sort of fun because you can tell the window's name is at minus eighty. And then zero is your return address. No stack cookie why no stack cookie they didn't flip the bit in the compiler and linker. Probably because they first built this twenty years ago. And they never changed their configuration. To handle to add. A. S. L. R. to handle safe S. E. H. to handle. Stack cookies. So all you have to do is overwrite the return address. Point it to the first of your rock. You can handle the rock chain well because there's no S. L. R. life is good. Life is really good so you can see us jumping to. The address and here I will pop the glorious calc. And this was fun to do. Bingo. And that's running at high privilege. Life is good. Let's talk about the patch analysis. But S. print F. Microsoft published the band API list. A decade ago. And there's a reason why Microsoft published the band API list. And so what they did when we reported this. Is they changed S. print F. into S. N. print F. Now S. N. print F. is also in the band API list. Now it's a better band API. Because it won't buffer overflow. But if you give it too many characters it also will not null terminate. So if the stack is not pre-cleaned out and it isn't. It is possible for you to then use stirring manipulations on this window name. Where you think it's hex 80 characters long where it may be longer because it didn't null terminate with the copy. So there still may be problems. As I said 100 bugs came in. 100 bugs. Advantec fixed 75 of them. We have disclosed the other 25 as not fixed. You guys can enjoy. There are also when they did fix they did not do any kind of global replace they did specific point fixes of the ones that they fixed. There are thousands of string copies S. print F. et cetera in the code base. And I would not bet 10 cents. That none of those can be reached by attackers supply data. So have fun guys. Have lots of fun. Ah yes researcher guidance so. What do you people want to do? Well the first things to do is fuzz. Right. These things are easy to fuzz. They don't have CRCs. Most of these file formats are wide open. Just do bit flipping. Remember to turn page chief on on the process that's being attacked. That's a great way to find memory corruption because then it breaks at the corruption point not later on when it's being used. Use your tools that you've got for fuzzing. Use your tools for analysis SQL map is great for finding SQL injection possibilities. One of my favorite tools is attack surface analyzer by Microsoft. One of the reasons one of my favorite tools is I helped write it. Microsoft released a public version in 2012. It creates a snapshot before and after the installation of your target software. And then it will highlight security problems in the configuration and it will highlight increases in attack surface. So it'll tell you your new com objects, your new active X controls, your new RPC endpoints. Here's an example. It shows you for example on the advanced tech software of the new RPC. Information here is attack surface analyzer telling you that the web root directory which is you know where files are going to go that are being executed in the high privileged web server context that this entire directory has right access by the world. What could possibly go wrong? So ASSA is a great, great tool to use. Now if anybody in here has pull at Microsoft, we need a new version drop of ASSA because it doesn't work on Windows 10. And if Microsoft wants to be really cool they could release the source of ASSA because I know what needs to do to fix it so it works on Windows 10 and it would take me about an hour if Microsoft would please release the source. Also audit for banned APIs, go look for the S printouts, go look for the stir copies, use IDA to trace the tainted data back and see if you can get to the source of these unsafe copy APIs if you can get to those from attackers and it's wide open people. And now back to Brian for more of the corporate things. So we wanted to give you an understanding of when you find a vulnerability how long it will actually take to fix. Kind of talk about the vulnerability exposure window. So what we did is we actually took all of the HMI vulnerabilities that we've received in the zero day initiative program. Again over 250 now and looked at how long they actually took and if you look at the last four years it's not exactly trending down. It's pretty consistently about 140 days from the time that we disclose a bug to when the patch comes out. And the thing about the SCADA industry is that when they're applying those patches if the patch is bad or there's an issue it will actually denial of service the critical infrastructure as well which is not good. But that means that patching actually takes a really long time. It's almost you could imagine almost twice as long so that leaves you know almost probably around 300 days when the patches are not being applied. So you know that's how long the bug are existing in the software even after you find them. So we wanted to do is actually call out a couple vendors who we've disclosed vulnerabilities to because that's what we like to do. And so what you see here is all of the vendors over those years and Cogent Data Hub I want to call out as being one of the better SCADA vendors for doing patching. I mean in fact one of the first bugs we disclosed to Cogent Data Hub their CEO actually emailed us and worked with us on the fix and they fixed it in like six days. It was amazing. And they've continued that trend you know we've still purchasing bugs in Cogent Data Hub and they're fixing them relatively fast. But if you look at the big vendors out there you see you know ABB, GE, you know Indusoft those over 200 days to release a fix for a zero day vulnerability that we've purchased that is known. So it's kind of interesting you know it averages out around 150 days for bug fixes. A lot of these are going through ICS cert and so just to kind of call that out. If you look at the SCADA industry and how it compares to other industries you know in the highly deployed software we consider that Microsoft, Apple, Oracle, the big name vendors they're doing a decent job. It's about 120 days for them to fix a bug when it's disclosed. And SCADA and security products are kind of battling out for second and third with SCADA coming in third. And kind of worse of all of them is business software, things like HP you know and other big name businesses like IBM takes them a long time to fix vulnerabilities. So you know we're almost 200 days for those types of vulnerabilities. So just you know as you find bugs and you work with ZDI to get them fixed or disclose them directly to the vendor it does take a significant amount of time but in certain cases it does take more than just 180 days. So kind of wrap things up. We present at these conferences and provide this level of detail because we want you to go find bugs. We want you to work with the vendors to get them fixed. We want you to work with bug bounty programs like the Zero Day Initiative to get compensated for your research. And so we are definitely interested in buying vulnerabilities and that's why we provide this detail. There is ICS you know focused malware that is actively exploiting HMI vulnerabilities. These code bases are plagued with vulnerabilities. And you can use those simple techniques to actually find them. It does take a long time for them to fix but they do end up fixing them. And so we're going to be continuing this research and we're actually going to be releasing a white paper in a couple of months. We're going to release some proof of concepts and all of our disclosure data is publicly available on our website zero day initiative dot com for you to analyze yourself and draw your own conclusions. Again we are the Zero Day Initiative. We buy bugs if you find zero days. We are a white hat bug bounty program. We've been doing this for ten years. We like to watch researchers grow and provide feedback to make sure that they are finding better bugs and getting higher payouts. And so if you are interested you know come up and talk to us we've got basically the whole team here in the front row. And we do a lot of research and look forward to working with you. Thanks for coming and spending the time with us.