 Welcome everyone to leveraging S-Bombs software build materials for ICS security. Looking forward to sharing some of this material and content with you guys. I appreciate everyone taking some time out of their day during DEF CON and this virtualized experience that we're all going through together. And I just really appreciate you guys attending this. So kind of jump into things. First off, just by way of introduction, my name is Tom Pace. I'm the co-founder and CEO of Netrise. We are a startup that is focused on providing visibility to a class of devices that has historically not had very good visibility within them. So things like IoT, ICS, networking devices, embedded systems, medical devices, things of that nature. So prior to that, I was the Global Vice President of Enterprise Solutions at SILANCE where I traveled the world, basically helping clients solve problems related to product services, direct research, all manner of things. During that time also I handled a ton of incident response engagements, specifically on many regarding ransomware. So that garnered me a bunch of attention and I ended up getting interviewed on 60 Minutes for some of the work that I did there, which was a really cool and fun experience. Prior to that, I worked at the Department of Energy doing industrial control system security and other security engineering things. And then before that I worked at a bank and then prior to that I was in the Marine Corps where I spent most of my time in the Middle East doing not cyber security things, essentially. So I've also done a bunch of things at RSA, Black Hat, other conferences of the like. So speaking of my time in Iraq actually, it's always fun to share a picture of me and Chuck Norris over there in 2007. So it wasn't also bad. It's a much, much younger and youthful looking version of myself. So from an agenda perspective, what is a software bill of materials, then we'll go into market trends that are driving software bill of materials like why now, why isn't important now. What what things have happened that are driving this adoption in need device risks associated with not having a software bill of materials for risks that could be reduced by having one attacks that that have occurred and attack types. And then we'll move into identifying and discussing the value proposition to use cases that software bill of materials can can provide. How do we generate the in consumer S bomb. The challenges associated with such and then what do I do once I have one. So, jump into it. The software bill materials is basically a complete formally structured list of components libraries modules that are required to build, which essentially would mean compile and link a given piece of software in the various supply chain relationships between them. And so in the bottom left to you can see the relationships of the ACME application, where you basically have you know bingo buffer or Bob's browser, Carol's compression engine things like that this is an example. This is used by the NTIA, which is a part of the Department of Commerce that has really been spearheading the software bill of materials working group of which I've been a member for roughly eight months or so. They're doing amazing work, and it's a really exciting thing that they that they've done there and what they've done for the community in general. So, on the right hand side here you see basically a an adaptation of a ingredients label for for food that that could could be modeled for software. And so, you know, an example here would be. Imagine a scenario where you have a peanut allergy as an example, and you go into a bakery and basically ask the baker hey, I have this severe peanut allergy. I need to know if any of this cake that I want to buy or cookies or whatever have been made within the vicinity of peanuts or something like that she had like a very intense allergy, and then basically not being able to tell you. So you have you now have to basically make the decision on whether or not you're going to accept the risk of purchasing this this baked good and putting your your health at risk. Well, that's essentially the state of affairs that we are in from a software perspective. Now, all companies aren't don't have that problem to that extent, but generally speaking, software bill materials are not available for or distributed to the great for the great majority of software that is that is created. So, and this provides a bunch of problems which will kind of dive into. So, these are things that we do have a bill of materials for food, obviously, nutrition labels ingredients. So, you know, the various sizes and dosages of different things hardware. So this these are the this is the bill of materials for a red wagon. So it has all the components parts required. And then also chemicals now this is this is less of a specific list of ingredients for chemicals but it is a, hey, here are the things that you need to be aware of. But if this happens to you from a prevention reporting storage disposal perspective, it has here are the hazards associated with this. And so you, these would basically, you could think about these as potentially like from a comparison perspective vulnerabilities associated with the software component from the chemicals perspective. So these are all examples of things that we have a list of ingredients, a list of risks a list of harms, potentially associated with different things in our lives. But what we don't have this for in many cases are our software and even less so for industrial control systems, operational technology, things like that. So the things that are driving and providing critical functionality for critical infrastructure and oil and gas and power and utilities and things like that are are especially deficient in this realm. So this one is more important, having a bill of materials for a red wagon, or having a bill of materials for a programmable logic controller. It's obviously a, you know, facetious question, but as you can see, we do have a bill of materials for a red wagon and we don't for a programmable logic controller in many cases, and that's a problem. So, diving into the market drivers and trends, why do s bombs matter and why now. The executive order that recently came out, which was a huge step in the right direction direction from software bill of materials perspective. Has the has the phrase s bomb mentioned 11 times operational technology is specifically called out. It also highlights various use cases of a software bill of materials vulnerability and license analysis risk management use cases centralized storage and distribution of software bill of materials, so that you can do things like a higher level analysis and have all this data in one place. 5G deployment and adoption is a huge driver here. We're talking about a significant increase in overall telecommunications infrastructure that we're going to need visibility into. And that we traditionally don't have visibility into. So smart cities are are going to be heavily reliant on this and we know that there is a enormous push for that kind of technology in cities. The infrastructure alone here is is enormous to to to even just rolled out 5G from a like cell phone perspective, let alone all of the other things that people want to do from a from a smart cities perspective. So, the other problem here is the adoption from 5G is going to be much faster than than 4G because there's a higher desire to gain the additional technical capabilities that 5G can provide. So, that's going to increase a huge attack surface that then currently exists. I already kind of mentioned smart cities, but the idea here is right. This is a, the problem you have here is there's a adoption and time to market are really are almost certainly going to supersede security requirements. And that becomes less of an issue if we have software bill materials and we have all this data available to us that can provide us insight into things that are going on. So, if we are going to get things to market really quick and ignore security, at least at the beginning, at least we have visibility in these things. Digitization of manufacturing like we are here, digital manufacturing exists, obviously the this combination of cyber physical systems is a common trend that that's being discussed very often. But once again, security is kind of viewed as an afterthought here, even though we all know how important it is. So, security is hard, baking these things into the, into the build pipeline is hard. This isn't a discussion necessarily that all of these things are like the easiest things in the world, but it is important to highlight that these things are incredibly important. The manufacturing environment impacts that that can potentially occur are huge here. I mean you have something like the colonial gas pipeline which was not the OT infrastructure was not compromised. But the OT infrastructure was made on available as a result of it compromised and that dovetails into the last bullet there about it and OT convergence. So, even though the OT environment was not particularly, we're not specifically compromised in that attack, it was still made on available, which clearly violates the CIA try add specifically availability. So, these kind of issues are becoming more and more common, and we can't really, you know, allow these kind of things to keep happening, otherwise, not having the East Coast be able to get gasoline and other oil based products is obviously a pretty big problem. Legal regulatory and compliance issues so there's a huge push here from a legal compliance perspective. You have the IOT cybersecurity improvement act that got signed into law in December of 2020. You have the FDA using the underwriters laboratory standard 2900-1, which is a standard for medical devices which can also be used for other device types well. You have a handful of NIST special publications that that approach the like either IOT or firmware specifically or other kinds of device types and in specific environments in which this is a problem. And then you have international standards standards as well like ISO, IEC and Enisa as well. So, from a device risk perspective, how are the market trends driving the risk of maintaining our current approach to buy security now we talked about this a bit already in terms of like more devices and things like that but let's dive into a couple specific examples. So, a real lack of visibility here now if we're talking about firmware specifically which is, I think fair to say the overwhelming set of software that needs that we need visibility and a software bill materials for especially related to ICS and operational technology. So, the firmware and the devices that are being powered by firmware that are running these devices are often treated as a black box. So, does a device exist and where does it exist or common use cases. That's important obviously that's why that's the critical security control number one. This ignores the risk within the device itself. So, it's great if we know where devices and what it is and all of that, but we also need to understand what's on the inside. So we've been approaching this problem from an outside in we now we also need to approach this problem from an inside out also fully updated firmware that's perfectly patched does not always indicate it is without risk. Whereas if you look at something like windows. If you do have a fully patch windows operating system that does provide you a reasonable like modicum of security like you can say, hey, all of the vulnerabilities that are at least known have been patched on my windows operating system now that doesn't prevent you from having to run for days and all of those other things of course, but for for these kinds of devices they're not even given that luxury, meaning there could be a litany of vulnerabilities that are very old, you know, five 10 years old known as N days. And those are not patched and maintained in the same way as like a windows operating system and there's lots of reasons for that, but not having visibility into that is a big problem. So we have to deal with materials, passing the risk to the Smith to the manufacturer. I have, I talked to a lot of people who mentioned to me hey we're, we're, we're passing the risk off to the manufacturer. Just to be totally frank, no, no you're not. If you buy a device in or using that device if you buy anything, anything in the world, you have just chosen to accept that risk to go back to the, the example around going into the bakery. If you buy that cupcake and I have a peanut allergy and there's peanuts in it or whatever. And I eat it, it like the bakery doesn't get sick. I get sick. I chose to accept the risk. So this idea that we're passing the risk off to the manufacturer is just totally false. It's just, it's not what's happening. And we need to just stop saying that it is. And this to compare again to Microsoft. And no one says, Hey, we're just relying on Microsoft to make sure that nothing bad happens here. We spend millions of dollars installing agent after agent after agent on these operating systems to provide visibility, provide telemetry, and give us an idea of the overall risk posture of a given computer. But we don't do the same thing for these kinds of devices now you can't install agents on all these devices and things like that, but there are things that you can do. But that's, that's, that's the important piece. So this just kind of highlights what I kind of just mentioned around traditional endpoints versus I CSM points this is not an exhaustive list and either column, but the point being, you have a lot more options to provide a lot more visibility on a traditional endpoint than you do an industrial control system endpoint. So, like I mentioned, most of the analysis is done from a network layer perspective of outside in. You have limited options and technical capabilities for endpoint telemetry because you, you, a lot of these devices just can't even support an agent infrequent updates patch cycles whereas on traditional endpoints you have a million opportunities to gain visibility as, as everybody, you know, as everybody obviously knows. So, here's another example that something I experienced, while I was working at Department of Energy. And one of the things I was tasked with was determining the impact of vulnerabilities such as shell shock heart bleed others. We had no visibility into into our own devices, no proactive announcements about what was happening. I had to reach out to, you know, all of the vendors that were that were running various device types with within the environment. And it took a long time to even get answers that were somewhat confident, not even perfectly confident, which was another problem. I have, I have the opportunity to talk with a handful of people who were on the front lines of this from a manufacturer perspective, and one anecdote I received was from a very large OTCS manufacturer that they had to email 635 people internally to ascertain, you know, where specific versions of, you know, an open SSL binary was being utilized. So, needless to say, that is a, that is a not scalable approach, and makes makes dealing with this problem, even, even more challenging. So, an inability to answer are we affected which piggybacks on that. So, if, if we determine a vulnerability exists a set of credentials exist an expired certificate is there whatever it is, we don't have like a repeatable consistent process to determine if you're affected. Answering questions can be hard, right, companies acquired don't have an inventory. So this is something that we can help solve manufacturers have the same problem as the end user. They're not. Most manufacturers don't own their entire supply chain, they are relying on other third party vendors for components and hardware and software and open source things just like other people. So, they actually have the same problem as the end user in many cases, which is an important thing to notate. The organizations have 30 plus vendors, they're probably conservative. So, being able to manage all this is time consuming, not comprehensive and not done in a way that that really scales. So there's no continuous monitoring or analysis, which, which is, I suppose fairly obvious, if we don't have visibility into things. We have no way to continuously monitor and analyze them. So, what you'll see is a lot of organizations will do this via a consulting engagement, which is not continuous not comprehensive and only focus on really what's outlined in a statement of work. So, aspects that need to be continuously monitored here in no particular order vulnerability identification compliance risk tracking the ability to just ask questions of this data. So, here's a particular example with related to a Siemens vulnerability that came out in April of 2020 that highlights that this continuous monitoring and analysis is not occur. So, the device notice date is April of 2020. The vulnerability release date is from August of 2018. This is what I mentioned as an end date like this is not a zero day this is not a new vulnerability. This is an old vulnerability that is just identified. And it was related to the kernel version. So this is 609 days that at risk devices were running critical functions. So, this was not a difficult piece of analysis that needed to be conducted. You, you identify the kernel version, and you look up that kernel version against the national vulnerability database I mean it couldn't be simpler. And something as simple as that just is not occurring. So you can see how this provides a giant problem. And just a, you know, as a result of these risks that exists, it's like, okay, so we have risk. Great. What are, are there actually attacks happening as a result. The answer is obviously yes. So pathway one is you have a targeted supply chain malware attacker or not necessarily even malware but putting in like a backdoor, which isn't necessarily a malicious binary but could just be like hard coded credentials in a configuration file or something like that. And this is a great example of this recently, obviously, a targeted device vulnerability attack. This is the most common I think, based on what we've seen. You have like the Mariah botnet you have Ripple 20, you have all of these various device vulnerabilities from from code that is running on these different kinds of devices. And then you have the infiltration of it networks across the OTR gap. So you had, you had the municipal water system where they were able to, you know, compromise some things and, and get access to like the chemicals and whatnot. And then you also have, as I mentioned previously, the colonial gas pipeline issue. So how can software build materials help. So shining a light on this device risk is really what we're what we're after. From a value proposition, you know, we can reduce unplanned and unscheduled work, reduce code bloat adequately understand dependencies within broader projects, comply with license obligations, monitor components for vulnerabilities, obviously, end of life awareness, make code easier to review, maintain a blacklist of band components, which is super important, especially for things like high assurance environments, provide software build materials to customers for that transparency. For example, if you have an identification, you can build this software build materials into your third party risk. So now instead of relying on like a paper based questionnaire that's always like questionable and the amount of value that's providing. We can now actually have a data driven approach to to doing third party risk management for the software that we're procuring from these vendors. And then compliance here. And so I love this. So, like, as bomb flower on the right hand side that highlights different use cases around supply chain software development, software vulnerabilities compliance, all these different pieces apart so there's a lot of, a lot of good use cases here for this data once we have it. So it's the it's kind of an age old tail. Once we get the visibility once we get the data, we can, we can extrapolate that and solve a ton of problems downstream. So as bomb generation and consumption. There's a lot of software development tools that can create as bombs for you easily. As you're building code, going back and creating a small for something that's already been done is just a much more challenging proposition most of the time. So, you know, the standards that exist for as bomb, you know, adherence Cyclone DX swid and SPDX are all well documented you can go find them online. And if you go look at like Cyclone DX as an example, they have an entire page dedicated to tools that can be used to generate software build materials that can be used to generate them from Cyclone DX to SPDX format. There are tons of opportunities here to make this much easier on yourself on the right hand side you see this is a set of the use cases that can be that are basically a part of the Cyclone DX standard. So, and whatever ones you can fill in great whatever ones you can't, you know, prioritize them based on the needs of what your customers are asking for. This is similar to like the food label piece I talked about earlier, where it's like here's the contents but not necessarily like the recipe recipe or proprietary blend, which is would be a problem. So from a challenges perspective here, you know, a software build materials is not is not a panacea. We're not solving all the problems. It's another set of data that we can leverage to help identify risk, provide visibility and reduce the likelihood of a problem happening and allows us to be more proactive in terms of identifying risk and mitigating or eliminating it as best we can. So, this is not a preventative security control, right. All security controls don't need to be preventative. So, if you don't have visibility into something, you definitely aren't going to be able to prevent it. So, it's important to get this visibility first, and then hopefully something from the data that we can gather here can be integrated with or tied into something else that can help prevent, you know, some malicious activity from occurring. I mean, it's important to realize right, you know, security logs, the logs in general aren't preventative vulnerability scans are not preventative data lakes are not preventative Sims are not preventative, but we all have them, we all spend a ton of money on them. And so, all we're trying. All that is really mentioning here is getting the data for these kind of devices is probably more important in most cases than some of the other devices that we're getting data for so let's do it. So, the equipment is a challenge right, you might only have compiled code. So, access to source might not be possible for a million different reasons. And so component identification and things like that can be really challenging when it comes to generating a software bill of materials here, but it doesn't need to be perfect right perfect is the enemy of good. So, it's important to make steps, or take steps, iterate, iterate, iterate, make things better over time. And if something can't be perfect that doesn't mean we shouldn't do it. You know, imagine if you're a person who has, has been blind or is color blind right we've all seen those amazing videos of somebody being given a pair of those glasses they can finally see colors for the first time and then we all cry. That's what we want to do here is basically give visibility give people a set of glasses that allows them to see what's going on within these devices serving critical functions in their environment. And so it's possible, we can do it. So some misconceptions about a software bill of materials I think are worth highlighting here. Am I providing a roadmap to the attacker. Operating under the assumption that attackers can't acquire this data on their own is just, just, I mean, just totally inaccurate. In reality what we're doing here is not providing a roadmap for the defender. So we're making that we're putting roadblocks in front of the defender, whereas the attacker can go get this data. So it's actually the opposite of this. So let's level this playing field some step. Is this intellectual property disclosure the answer this is no, we're not providing proprietary source code or anything like that. These software components are just a piece of the puzzle that that allow, you know, organizations to identify risk and and understand what they need to defend and what they have. So this awareness allows manufacturers to address unknown licensing commitments. It allows them to address these issues in a proactive manner not reactive gives them the visibility to make good decisions ahead of time through fees or more favorable terms. It also helps the end users and clients and show that they have compliant things. So it's kind of a win win win for everybody from that perspective. So first off from billing materials now what's so a bunch of things we can do here as I mentioned once we have this data, the use cases for it are pretty infinite enrichment of the data with other sources, third party sources, vulnerability data threat intelligence data, things like showed and like what you name it operationalize the data, we can send this data to a sim. For triaging we can we can make it part of our vulnerability management process, we can integrate this data with a sore platform, we can integrate this data if we're the manufacturer into like our CI CD pipeline. You know, if a critical vulnerability is identified then open JIRA issue, whatever I mean the logic we can think of and build out is, you know, you know, a lot. But make decisions based on visibility that was previously impossible or incredibly difficult to ascertain is, is, is the biggest one. So, adding additional monitoring to this device do a vulnerability, push back on manufacturer for lack of license compliance, third party risk management, data driven insights for lifecycle management, segment a device or a class of devices, you now have data to drive better decision making regarding these kinds of devices, and that's what we're really after. So, this is what you can do once this is done. So I appreciate everybody taking some time to to attend this talk, I hope it was helpful and informative. I will be on the discord server answering questions as this goes live. So please reach out if you have any questions. I guess I should have put my contact information here, but you can reach out to me at thomas dot pace th o m a s dot p a c e at net rise dot io net r i s e dot io. Once again, thank you very much. I appreciate it and I look forward to speaking with you during the discord conversation. Thanks.