 Hello, good evening everybody. Welcome to this last talk of the last day of the conference. I know everyone is eager to get home and I'll try to make your next few minutes worthwhile and we're going to go through this quickly, but hopefully you're able to take away some good tips from this talk. Here is a quick agenda, a quick intro about me, and then we're going to discuss software supply chain security challenges. We're going to discuss about multi-bomb as a winning strategy and your role in adopting and taking forward this solution. So here's a quick intro. My name is Pallavi Kalpatapu. Can you? I did the slide. Oh, sorry. Can you see? Okay, sorry about that. So yeah, a quick intro about me. My name is Pallavi Kalpatapu. Those are my social handles. I'm an engineering director at Cisco Systems. I've held leadership positions in engineering, product strategy, and innovation. Mostly I've dealt with cloud security, AIML, and some done some open source advocacy and that's how I'm here to talk about an interesting open source project and how multi-bombs can help you with that. And at personal level, I don't have very many interesting hobbies. I'm like a very simple person who loves friends and family. After work, I want to get back to home with family and gardening. Don't have very many interesting hobbies to share here. So let's start with a small trivia. What's trending in technology world today? You think everyone knows the answer, but let's let's make a little fun trivia here. Okay. Yes. Yes. It got to be AI. There is no talk, not one conversation goes without talking about AI. And very recently, just off the grill, we have this new executive order for AI security and everyone is trying to kind of even understand what AI security means. But don't worry. This talk is not about AI as promised. It's going to about software supply and chain security and S bombs. This is another executive order which came in 2021 couple years back, but still the industry is scrambling to catch up to it. This is no longer an option. Anybody who consumes or produces software is bound to understand what S bombs are, especially if you are producing software. This is an executive order that we publish S bombs. And that's why I'm sure most of you are here to understand the pain points of S bombs and what are some of the solutions you can adopt. So looking at it, why did executive order has to come up on software supply chain security, cyber security and S bombs? What's the deal with it? It's because as you all know, the software supply chains are breeding grounds for cyber crimes. There's source code all the way from source code, your CI CD pipelines. Once you go through your build artifacts, then your licensing, your conflicts, your infrastructure as code deployments, monitoring, logging compliance, all of this stack all the way from start to your deployment. Any point can be a attack vector. So that's what makes software supply chains because there's huge attack vector and we see a lot of attacks happening in this space, which brings down the entire software and actually companies and organizations, liabilities, all of it. So there are very many ever growing attacks log for J. I'm sure each one of you is bitten by that log for J and we all had to scramble to apply the vulnerability patches. And there are many others like SolarWinds, Equifax, some of it we are going to see in the next slide as case studies. We're not going to go over the entire list, but these were some of the recent supply chain attacks that were popular, that were noteworthy, that caused enough damage. And all of them had different targets, different impact, what they caused, the scope of the attack, the exploit through which they got in, all were different, very many diverse attack patterns, which is what makes even devising a solution or even anticipating what adversaries can do here very challenging. And as we said, let's look at few case studies like SolarWinds. If you take SolarWinds, I don't know how many of you followed this incident, but there were 18,000 customer accounts and businesses were compromised due to this malicious software update. You'll be surprised how many of these are through software packages, update, patch management, as you see Equifax is another 147 million records here. And then CC Cleaner also, it's also a classic supply chain attack. Again, it is due to a weak code signing processes and verifications, so the vulnerabilities creep in. And then there was another Apple XGhost one, which was secure development tools. And all the time we're thinking our software is secure, so I'm good, I have done my part, but what about the third party components that we are integrating? And that's what makes this whole software supply chain securing it. That's what complex and they have put together some recipes for that, that SBOM is one of the standards that lets us take a peek into all of the components, which we need to go ahead and secure. Especially if you're dealing with cloud native Kubernetes workloads, it's even more important because the workloads are very ephemeral. As you know, the containers come in and out of scope. We need to be able to know nonetheless, even if they run for a short time, what are they comprised of? What images are you downloading? And that's why to gain insight on the building blocks of software from code to production, we need that SBOM, Software Bill of Materials. And then we want to be able to detect the vulnerable building blocks of the software, not just during build time, even post deployment during runtime. As you download dynamic images, new vulnerabilities can come in. So you need to have mechanisms to be able to detect them post deployment and during runtime. Again, once you have these vulnerabilities, like mostly none of the applications usually stand and operate in isolation, they are calling into another application. There's a daisy chain of applications going on, one API is calling into the other. So we want to be able to correlate these vulnerabilities across applications to understand the blast radius of where is this vulnerability going into and how does it impact me? Most importantly, vulnerability finding is just a chore, a task. What we are ultimately interested in is finding the remediation, like the vulnerabilities you found. That's the crux of the whole exercise, that once you identify the vulnerability, you want to be able to go ahead and remediate it. So all of this becomes very important in a cloud native context. And there are some mechanisms to, like again, SBOM is supposed to be the standard for everyone to come together and publish what are the components that are involved in your software. And bear with me if you're, you know, SBOM experts here just to level set everyone, just wanted to share this SBOM 101. There's lots of myths around it. What SBOM is, is really like a nutritional label for your software when you're when you're eating, you're all looking at what is it that we're eating. It's the same thing when you're consuming the software, you are, you can see what are the components, but obviously it's not a linear list. It's very complex because they're a bunch of built in dependencies. So giving the transparency into your software is what SBOM does. And just because you have an SBOM, it doesn't automatically solve all your vulnerabilities. So people do mistake it for your vulnerability scanner. I got an SBOM, so I'm covered. No, you're not covered. You have SBOM that only gives you a means to understand what your vulnerabilities could be. So it's very important to know that it is not a vulnerability scanner. And also why is it important? I already mentioned, this is no longer an option. First, we need it for executive order, even if we go from top down. And then to find out incidents, response remediation, first, you can't protect anything if you do not know first what vulnerabilities exist, what are the loopholes, and that's what exactly SBOM gives you. Now, okay, there is SBOM. So what do we do about it, right? Like there are a couple of gold standard, SPDX and Cyclone DX, there are several differences between these formats, I'm not going to be able to exhaustively cover here, but SPDX is more for detail, it's more heavy weight, it's more like a spreadsheet format while Cyclone DX is more lightweight and it's more like your XML kind of hierarchical format. SPDX has been around for a long time. It's for non code assets as well, if you want to declare your documentation, the format of it allows for you to do it and mostly the legal and regulatory firms tend to stick to the SPDX format, only because it has a lot of mature ecosystem and it's supported by Linux Foundation. The Cyclone DX is a lightweight, it focuses on specific use cases more like your container workloads because it's hierarchical. So any modern software that is containerized, you tend to use Cyclone DX. Again, it is no, I mean, there's ton of information research we have to go through. If you really want to pick an SBOM that suits you and in few minutes, you will see that there is no one such SBOM actually that suits you. I don't want to give away the thing early on, but there are differences. The point is to note and understand what are the differences. For example, the dependencies in SPDX are represented as part of components, but in Cyclone DX, they are first class objects. So it really depends on how you want to take this data and manipulate it. So all right, there is SBOM, there is framework, there are standards. So once there is, is it very hard to generate this SBOM? Yes, it is because again, of the variances in the programming languages and package managers we deal with, these are some of the examples, Maven, MPM, PiPi, they all have their own ways of representing packages and package management information. And then there are various OS distributions package, different dependency information is again usually stripped off at build time. That means you have loss of information, you're only working with partial information because some of your library information has been stripped down during build time. So you have now like a gray area over there that you do not have visibility into it. So it depends again, each of the SBOM format, some let you give you much more detail than the others. Okay, once you have generated this SBOM and the next task, like I said, was earlier was to pipe it to a vulnerability scanner. Now, just like SBOM, you can imagine there is not one as vulnerability scanner, it expects different kinds of input formats, the idiosyncrasies of these vulnerability scanners are again very different. Some are better with specific programming language while the others do well with the more like distributor Linux kind of softwares. So on the right hand side here, you see like bunch of like open source solutions, vulnerability scanners, but again, the point is not one of them really will solve all all your problems. So it is very important to be on lookout understand the feature set of what it offers and mostly end up with a combination of these. So there are some commercial tools and solutions. For example, the very first one here is Cisco panoptica, like you know, I'm from Cisco, I'm just a little plug here, but rest of all the popular names that you've seen anchors, scribe all, I mean, I'm not going to go through the list, but those are the commercial offerings if you're interested to check out. And then there are open source solutions in this family as well. Cube clarity VM clarity again started by Cisco, and we contributed it to the open source. And it's everyone is welcome to check out these projects, trail by some of the other open source projects tree we sift and grab these are also heavily used open source projects. So we're going to talk about mostly cube clarity in the rest of the slides. But one thing I wanted to call out is what we are doing here with cube clarity open clarity is not try to reinvent the wheel but leverage and integrate already available existing open source solutions but take best of both worlds of many of them and try to stick together for a better together solution. And then if you are tired of this commercial open source and you think none of it is solving your problem, you're welcome to go by standards and choose, choose one try to build your own solution. But obviously, that's a tall order if you want to do that option is available. So rest of the talk, like I said, we're going to focus more on cube clarity. That's the project that I wanted to bring it to your attention today about the container security. It's the Kubernetes container security workloads. If you wanted to secure them and generate a software bill of materials, one software bill of materials is generated. You wanted to be able to track your comprehensive package and OS vulnerability information. You have an ability to scan clusters, your Kubernetes cluster spards, files, images, directories, root file system packages. And you are also able to run these vulnerability scans during CI CD and both during runtime as well. Like I mentioned, runtime scans are very important because you're dynamically loading some images at runtime and we want to be able to have that coverage. So top level features again for this is it offers you two faced scanning. We're going to talk about it a little bit in the next slide and then multi S bomb integration, multi scanner integration, multiple inputs, runtime scans and also CIS benchmarks because a lot of it you want to be able to run against CIS benchmarks and see how you're fairing with industry standard. Again, I want to draw a little attention on that single S bomb generator shortcomings, right? Like I mentioned trivia and SIF happened to be these are open source again, not any commercial solutions. Anyone could go and use, but they are targeted for very different purposes. You have to go research up and understand which one is suitable for me. For example, SIFT is very heavily geared towards Python workloads, all your anything that is very Python centric, fares well with SIFT and then trivia is more generic, but then again it is by itself not able to show you all all the idiosyncrasies that you're looking for. So here is something for you to take a look. I'm not going to exhaustively literally go over this table right here, but just to highlight the differences between these providers. So it's important that we apply like a multi S bomb strategy, that's because if your inputs are partial, your outputs are going to be partial as well, then your vulnerability coverage will not be full. So for that purpose we need to have that full coverage inputs for, so, and we already covered, we already saw there are variances and none of it is going to be your solution by itself. So the best strategy to apply is to integrate multiple S bombs with multiple format, be an SPDX, be it Cyclone DX and try to stitch together a more unified view for your, for your vulnerabilities, right? So you want to have a unified view, you want to be able to perform multi-stage CI CD, you know, either you call through API or you automate, but you want to be able to run it at multiple phases and be able to compute it. And most importantly, you want to preserve the lineage of where these, which S bomb analyzer has reported this. So if you take all of this merge and throw away the lineage, later when you want to backtrack and audit or any purposes, you want to do it, then it'll get exponentially hard. So what we do upfront is we realize that we preserve the lineage through metadata tagging. That way, you have the best of both worlds, you have a unified view. However, it is, it's going to show you like where it actually came from, the genesis of it. Also, a lot of other solutions that I've shown you, whether open source or commercial, their S bomb generation is usual because S bomb by itself, yes, executive order is recent. We are supposed to publish it, but most other cases, S bomb was not generated for the sake of fancy, here is my S bomb document. Most likely the next step was to run it through a scanner and find the vulnerability. So what happens is a lot of solutions try to combine these two steps as S bomb and vulnerability scanner as one black box. And then your flexibility to do anything with it is very limited. So what we do here is like we really split it into two phase approach. In that approach, we are trying to make your S bomb generation very pluggable with your downstream scanners. That way you can really do a plug and play and generate needed visibility into all your software components. Again, the recipe, right, how do we do it? I explained that this is what we do, but how do we actually do it? Because we are unifying multiple S bombs into a single S bomb, we need to be able to come up with a standard and we pick the Cyclone DX as our unified format to normalize all these incoming S bombs into a unified format, that's step one. And then we iterate through all components. If you remember when I showed you the table for Cyclone DX, it is the one that requires this package URL and version name as opposed to the SPDX format. So we kind of normalize and create that package URL, PURL, what we call it. And then if we find the matches, we merge them, but we retain the metadata. So you are always able to go and say, this library, this one was found by SIFT, this one was found by Trivi, this one was found by Godmad, but you will have a unified view so that you don't have to tap into multiple S bombs. You just have one source of truth and you get all the information from there. That said, I don't want to oversell on this point, there are new analyzers coming into play. So our Q clarity's goal is to be able to consume as they come because advancements are coming, everyone is innovating and trying to find better ways. So what we are trying to do is build a pluggable model that takes any incoming new analyzer and you're able to plug it and integrate. So because all the three or four that I mentioned, they are not going to solve the world hunger, right? They do give a lot of coverage, but we need to be able to let the new S bomb analyzers come in as well. So that's exactly what the pluggable model is. If you want to take away, this is one image I want you to remember, walk away with, is that this is what comprehensive S bomb and vulnerability management will look like. You really want to break it down into phases. The first phase being you're completely dedicated to your S bomb analysis with multiple analyzers. Like I said here, I showed you three as an example. It's completely pluggable and you're able to really plug in another one if you want. And then it outputs a unified S bomb. You then pipe it to downstream vulnerability scanners. Again, this is a form of vulnerability scanners. It isn't one scanner because each scanner requires different format. Like maybe one of the scanners doesn't like or is not compatible with like an SPDS S bomb analyzer. It only probably consumes Cyclone DX. So all this kind of formatting differences is what we abstract out and we completely separate these two phases. Ultimately what you produce is this golden mine is that you want to be able to have like a DAG where you are able to really navigate from a given application. What are my application resources? What packages? What libraries exist? And each of these libraries and package, what vulnerability is it pointing me to? And given a vulnerability, let's say that you say that there is a vulnerability CVE XYZ. Given that vulnerability, okay, you know that you read it somewhere that there is a CVE. Now, how do you know how is this CV impacting any of your applications? So you have, when you do something like this, you're able to take that vulnerability, run it through, and you're able to reverse traverse and go from vulnerability to which package, which application resource, it's an image, and then which application it belongs to. That's how you know your blast radius. That's how you'll understand, okay, this is the impact and that's how your remediation actions can start. So this kind of visibility view is very important for us to derive. And Q-Clarity tries to solve this for you. Again, just to give you a visualization of how these analyzers are piped, it, today it uses SIFT, Cyclone DX, Godmod, and 3V. These are the three ones that integrates with and it is completely pluggable, like I just said a few minutes ago. So any new S-Bomb, you already know of, welcome to try it out. But whether you can point an image, source directory or something to these input analyzers, or you have an option right here, like you see that if you already have like a partial S-Bomb from some of the third party application that integrates with you, let's say you have already pre-computed that S-Bomb, somebody gave you, you don't want to waste cycles doing it. So you are able to ingest an already available S-Bomb and combine it with your pipeline here to bring a format. Again, all of it has to be formatted to unified format if you remember, because ultimately you want to work with one S-Bomb. So there are options to pipe these as well as ingest an existing S-Bomb, ultimately to create a single unified S-Bomb view. Again, if you want to see how your pipeline, your CI-CD pipeline could have multiple touch points and S-Bombs can be like generated out at multiple points either at build application time, your building image time, or when you're pushing, it depends on what phase you want to start tapping into your S-Bomb. So this is how you will change your CI-CD pipeline. Now, enough of S-Bombs. Once S-Bomb is generated magically with all the secret sauce that I've shown you, you want to be able to run through this vulnerability scanning. Again, the goal is to be, why do we want to do it, is to be able to go and bring these CVSS scores and try to really find what are my most critical vulnerabilities, what should I be able to act on. You want to be able to seamlessly integrate with vulnerability scanners. So here is the integration. Just like the S-Bomb, you have a pipeline of scanners. It is again, very pluggable. Tomorrow, a new scanner is out there. You are able to also bring it in into your pipeline. The ones that we currently integrate with our GRIP, Godmad, Intrivi, again, you have an ability to give raw, point it to raw data, give it to directory and config, or your pre-computed S-Bombs, you can pipe them through the scanner pipeline and then you would get a formatted unified vulnerability report. Okay, I'm sure that I've talked about it. You would want to see some of it in action, how the vulnerabilities would look like. So let me show you a bit of a live demo here. So this is the, are you able to see? Yeah. This is the Kube Clarity UI dashboard. What we have done is like I'll just, this is the dashboard, we'll come back to the dashboard view. But here in the runtime scans, I just want to show you that this, there is a cluster via, I have pre-deployed this in my Kubernetes cluster, a few apps. One of the app is just a dummy web application called Sockshop. Here it is, just to show you there is a Sockshop app. Yeah, I wish there were no holes in that socks. That's why we need a Sockshop. But anyway, so I have pre-scanded just so that, it runs fast and I wasn't sure how the network connectivity is here. So I just pre-ran this and I'm going to step through some of these vulnerabilities here. So if you go back to the dashboard, once you scan your cluster, like what I have done here is Sockshop. But literally you can scan your cube system or any other application, any namespace that you point to, you are able to scan and find the vulnerabilities in it. Again, vulnerabilities how? They did not come automatically. It is all constructed through your SBOM analysis and your vulnerability scanners. Let's look at some of the information here. I need my glasses for this. Yeah, so these are all first class objects. The application resources, it shows you what package, package exists deep down. You can go and what pod these things exist. I can click and show you one of the things. For instance, if you see then application, you will see the application ID, what pod and some of the licenses it's associated with and number of packages, all of that information. And this is packages given, for example, anything, right? Like you can see the version of it, all the lineage, the history of it you are able to see through here. Application resources, let's say how many are involved in this application. Remember if the blue graph that I have shown you, from an application you are able to go drill down to package and from a package and a vulnerability or drill down back into application. So that navigation is available here. And then what would be of interest to you most is like this. You see that the scanners here, it tells you the lineage like okay, gripe is the one that reported it and it was detected at runtime and this is the CV score given a particular version like the package version, fix version. And what is most important is you will have ton of vulnerabilities. Usually like it's a lot of noise. You'll have like here what? How many 1,800 vulnerabilities? A lot of times you want to know out of this how many are actually fixable for how many are the fixes even available and what can I do about it, right? So you can see all the trends and drill down on your package counts but here are fixable vulnerabilities because you really want to get to the ones that you're able to actually act on. So you are able to see that view here in the grand scheme of things. See like what is a vulnerability information which scanner found this and then one more thing. Yeah, this is probably the most interesting and you know that this wasn't black magic. This is true, I kind of showed you like a lot of other competitive solutions that you will find. You will not see this as bomb analyzers, this multi as bomb analysis here. Usually it's one, right? And you deal with whatever SIFT has reported for you. In this case, what it means is this package right here, this one, the resource name was only reported by SIFT. It was not found by Trivi. I did this deployment right here, configured both but this library if you can see they are all detected mostly by SIFT and here there are some that are detected by both and you want to be able to at any point get back an instrument. Let's say you had a better patch and you know that there were some of improvements available for Trivi. You are able to run your pipeline again just for that segment. You don't have to run the whole thing and able to recompute. And if you go to the scanning again, here affected elements like applications. Again, you are able to literally there are very many options to go drill down and there are ways to also run I don't want to run the scan now. We'll get distracted, but you are able to actually also run the CIS. Well, it already started. So right now we just have to wait until it finishes. So that pretty much concludes what I wanted to show given that S-bombs, multi-S-bombs, how do you really plug it in into a UI and be able to navigate all the packages, software versions that you see here. For example, you see that this is a Zlib library. The license is Zlib version is this much. Like let's say you found a vulnerability in this version. You know that which application is actually impacted by it. So you are able to go and just patch this one, for example, because you know which application. See, given this package, know which applications in your deployment are impacted by that. So that's the kind of information you want to know to be able to correlate. And yeah, that pretty much concludes what I wanted to show you as part of the demo here. Again, you will have detailed information from, again, this is a lot is coming from S-bomb on what package names exist, what version it is at, language, all this vulnerability. Given this package, what's the vulnerability? From there you are able to go correlate it to a particular CDE and then be able to actually find and fix it. Again, let me, so really that concludes the demo and I'll probably just point you to the project URLs here again. This is open clarity on GitHub. You can go find the project. There are a ton of blogs written on this to get you started, how to deploy, whether it's EC2 or like your kind clusters, lots of information, lots of deep technical blogs are available on sysco.techblog.com on this project name if you want to search for. There is lots of documentation available as well for you to check, oops, sorry, yeah. And ultimately, we want you to check this out, use and contribute. We would love to have you as a contributor to this project, be it like bringing in another additional scanner, analyzer, or maybe any improvements that you find docs, maybe a new use case that you want to suggest to our, everything is welcome and this really a community-based project and would really love for you to try it out and give your inputs. Just to bring home, again, this is all under open clarity umbrella of projects started by sysco but donated to the open source community like it's open to one and all. There are multiple projects under this umbrella. Cube Clarity is the one that I touched upon today for container workloads. There is another one called VM Clarity. If your workloads are very VM-based, it also offers very similar vulnerability scanning and in fact it does even more with secret and root file system scans. And all of it, if you want to try open source, give it, if you want to try like a commercial solution, you could try that to sysco has like a commercial offering for that. But either way, my call to action for you is whether you are using open clarity or any other solution, please look, ask yourself that is my SBOM strategy strong enough? Like am I like fooling myself thinking that I have an SBOM and I'm covered? Or are you really covered? And then look for strategies whether it's automated one or like you build your own. But I hope I convinced you enough today that multi SBOM is very important. One is not going to solve your problems and you want to be able to ultimately when crisis happens, you want your team to act on it. You do not want downtime of your software. So for that, you need to be prepared and have that visibility. One for yourself, one for like I said, this is executive order. Like if you want to partner or ship your software, guess what, folks are not going to, they're going to first ask you can you have your SBOM disclosure? So either ways it's very important for us to get to this point. And that pretty much concludes my talk. I'm sure you appreciate a few minutes going back early today, last day of it. But yeah, I'm here to take any questions you have on this. I hope this was useful and helpful. Thank you. The third party scanners. Yeah, we didn't want any vendor lock-in right. Literally take the open source ones and make it available. Just package it so that it's convenient for you. Thanks for the talk. This was really awesome. Thank you. I was wondering, so we actually scans at a few places in the SDLC. So like run Tervi in FS mode to get info from package managers and stuff. Then we'll run it in root FS mode on images. Can Q-Plarity consume both of those? Yes, absolutely. And then be able to tell you where the vulnerability occurred or whether it occurred. Yes. Yeah, definitely give it a try actually and yeah. Cool, thanks. Hi there, just a clarifying question. Are there multiple scanners being used on that unified SPOM or is it one scanner? Sorry, I have multiple scanners being used. Yeah, are there multiple scanners being used to scan to analyze the SPOM for vulnerabilities or is it a single? No, there are multiple, at least three that Tervi, God mod that I mentioned. Yes, there are multiple scanners. It also allows you for plug-in, bring your own scanner if you are using, like it doesn't have to be an open source one. You can bring it in and integrate with it. And so what is the format chosen for the output vulnerability report? Cyclone DX. Cyclone DX. Yeah. Thank you. I guess that we are done and if you have a moment, please leave me a feedback here.