 There have been a couple of interesting setbacks, I would say, for this panel because we had four people that were supposed to do the panel. One of them couldn't make it, but he was kind enough to record a video and send it over to one of the questions. And the other person fell sick just last night, but she was kind enough to get up this morning and make some videos. So they're not going to be in person, but we've gotten the talking track from them that will play. So let's see how this goes, because we were planning for a conversational kind of panel. Now I'm going to have to go back and forth between me setting some background, asking the question, and then playing that video. So imagine it's a conversation that's happening on the screen. So this panel is about all of the things that you've been hearing in this conference. The primary topic of this conference has been around supply chain security. And S-bombs, especially for cloud native applications, is something that is emerging as an extremely important and urgent thing to do, thanks to Joe Biden's executive order, a lot of the organizations that are building cloud native applications and other applications are rushing to create S-bombs, at least as a first step towards their journey of achieving proper supply chain security. And this has to happen, especially if you're a regulated organization selling into federal government, et cetera, by June 11th of this year, because that's I think the date for complying with the mandate, the presidential mandate. So our goal in doing this panel was to bring some people that are experts in the operational aspects of application security, developer security, supply chain security, et cetera, to make sure that we talk about the practical aspects of how do you go about achieving this mandate? How do you go about creating S-bombs? Because of course there's a lot of talk about future and things that you can do to make the perfect supply chain security solution over the next year, two years, et cetera. But what do you do now from an operational perspective? So we're going to touch upon some of those things first. We're also going to touch upon what can you do after you've created your original S-bombs and so on and so forth. So let me introduce the panelists. I'm going to start with the people that are not in person present here today. Rose Judge, she is a senior open source engineer at VMware. She's one of the maintainers of the TURN project. The TURN project is responsible for creating. It's a tool just like Trivi and others that allows you to create S-bombs. And Rose has been an integral member of that team. Alan Friedman, he works for the Cyber Security and Infrastructure Security Agency in the U.S. And he is responsible for running the S-bombs initiative for the U.S. government. And I will go with Andy next. Hello, I'm Andy Martin, CEO and founder of Control Plane. I have an interest in supply chain security from back in the original in TOTO days and specifically around if we're signing artifacts, then what are their composition and how did they work themselves into the state that we are currently dealing with? Awesome. And my name is Kiran Kamiti. I'm the founder and CEO of Deep Factor. We have a booth here. So check us out if you're interested. Deep Factor is a modern developer security solution for docker Kubernetes type applications that will help you address some of your supply chain security problems, among other things. So let's get started. The first question that we wanted to talk about, we actually wanted to break this into very practical aspects of S-bombs. What is an S-bomb? What is a VEX file? What is a Cyclone DX and SBDX file? What do you do with it? Why is it important? Number two, we wanted to talk about how do people go about creating it? Number three, as you go through from left to right in your pipeline, talk about where do you store it? Why is it important to store it in such a way that you can actually search through it and make sense out of it? And then we're going to talk about how do people consume it? And lastly, we're going to talk about is this the beginning of your supply chain security or is this the end all of your supply chain security? Obviously, you guys know the answer, but we'll double click on the details. So let's get started with this question. What is an S-bomb and why should people care about it? We asked this question to Alan Friedman and he did record a video for us. So imagine that he's actually having a conversation here with us since he's not available in person. Of what is an S-bomb? Why is an S-bomb? Can we do this today? And what's some of the broader security context? So I'm assuming most of you are understanding no one an S-bomb is, but essentially it's a dependency tree. I use this software, the software uses this software and so on. Transparency in the software supply chain is going to be a key part of really moving forward and making progress on how we think about software. And in fact, it's a bit shocking that perhaps the vast majority of software today isn't built with greater transparency. So S-bomb can help developers, can help people selecting software, and it can help users to make sure they have a key point to react. There are real risks out there as we think about software supply chain, a lot of them are vulnerable vulnerabilities, right? Many of us spent a lot of time hunched over laptops in guest rooms at the end of 2021 trying to understand where our risk was for the log per J crisis. That's not a one-off. I would not bet heavily on saying, well, there will never be another log per J. But there's also a broader question around risks, right? How much of my infrastructure depends on projects that have no active means or how many of my projects use components that have been end-of-life or have no support. And S-bomb won't directly tell you that, but good luck managing that kind of risk without really being able to look into the entire dependency graph of your software. It's also going to be coming from your friends, the governments around the world. The 2021 executive order said, ultimately, everything US government buys is going to have to get an S-bomb. The FDA, which regulates medical devices and said, hey, you're not going to be able to put a new medical device on the market without telling the regulator what the dependencies are for a full S-bomb. And of course, our friends across the Atlantic in the EU have proposed the Cyber Resilience Act, which says that everything sold in the European Union must have an S-bomb. So can we do this today? And the short answer is yes. The slightly more complex answer is we're working on making sure that it's interrompable, right? I think everyone, whether you make software or use software, can start to produce S-bombs and ask for S-bombs. And then over time, we're going to really push for better interoperability and more full integration. There are plenty of tools out there today, both proprietary and open source. And there are two great S-bomb data formats, SPDX and Cyclone DX. They're both open source projects. They continue to evolve. And one of the things we're working on is to make sure that they're fully interrompable. At the end of the day, it's just data. So what about the broader security landscape, right? The vision isn't just to create this brand new thing. It's to build this into our software security architecture and how we think about risk for our organizations. So we're busy deciding what are the requirements to make sure this happens and then linking this data to other things that we care about, such as making sure we can map it to vulnerability databases, making sure we can map it to other sources of intelligence. There is an idea that says, hey, it's not just the risk that it's whether or not it's vulnerable, but if there's whether or not something can be exploitable. And so we're going to be publishing very shortly a document called the minimum requirements of VEX, the Vulnerability Exploitability Exchange, which will allow an organization to say, yes, there is a vulnerability in here, but it doesn't actually affect the security of a software. And being able to communicate that at scale will help us prioritize and make sure we're really focusing on the vulnerabilities that matter. And increasingly, we're interested in the security of an SBOM. So some of it is pretty basic. Hey, you should sign your SBOM. How should you sign your SBOM? Well, if you have the ability to sign your software, use that key. And if you don't have the ability to use it to sign your software, well, then we've got bigger problems. So let's focus on sort of overall modernization. And the last step is going to be, how do we verify that the data is true? How do we actually show that there is a path that comes from my build process so that it's not just trust me, but it's actually we can have some artifacts that have a full chain of trust. And we're very excited to see that continue to evolve. I think that's now in early days, but like all security processes, we need to start with a crawl, then walk, then run. Hope we have a very productive day and I'm looking forward to having conversation. If anyone would like to know more about CISA's work, email us at SBOM at cisa.dhs.gov. Thanks so much. Awesome. So to summarize, there are two formats for SBOM that, again, boiling this down to what can you practically do between now and June if you're mandated to comply with Biden's executive order. Two formats, SBDX, Cyclone DX, pick one. And even if you don't know which one to pick, I would say pick either one. It's not a bad choice to pick either one. There are tools available to create both. There are also tools available or being built to make sure that you can interoperate between the two and convert from one to another. The second thing that you need to do is understand whether you need to generate a VEX file. I would say if you're hard pressed for time, think of it like a phase two project, like do your SBOMs first and then add VEX later on because VEX basically helps you identify where which of the vulnerabilities that you have is exploitable, et cetera, and you may need to do some additional work to get that data in place. So that's step number two. And then of course things like Salsa, et cetera, and other things that will help you establish provenance, signatures, those are all things that I think you can phase down the road. Either it could be phase two, phase three, and beyond. So if you were to ask me, I would say create your SBOM first. That's step one. Step two would be think about signing. Step three would be add VEX to it as an addition. Would you agree, Ellen? Yeah, I do. Andy, sorry. That's how. The mystery of VEX is the CVE that is in the dependency either one, two, three levels transitively down the call graph, actually reachable. So for example, worst case scenario is a remote code execution. Can somebody do maybe a shell shock looking dump in specific bad string into my web-facing server into my socket and end up triggering, miss what I'm looking for, an out of context execution. So untrusted code being executed and then potentially firing a reverse shell. That's the worst case scenario for any web application. Gives an attacker a foothold that they can then begin to enumerate what's within that, let's say a container at this point. So VEX aims to say, okay, we have got shell shock five layers deep to pick any example. But actually, we are going to remove bash from this container image. So there is no mechanism for that, that particular function signature in bash to be executed. So we know that even if this is exploited, even if that malicious payload is written into our web server, there's no way this can be triggered. Another way you can do this is static analysis of the call reachability graph. So you can think about that kind of like decomposing an application into an abstract syntax tree and then walking it. And if the piece of code, for example, you might have a vulnerable library with three function signatures and you know that one of those function signatures just doesn't get called in the code at all. So you can either walk that AST with the reachability analysis tool, code QL, semgrap, sneak, they'll all have these capabilities. The factor as well. The factor are all able to analyze and understand whether that code's reachable. You can also with some languages do something that's a bit like tree shaking. So just removing the code, if it's an interpreted language where the source is fed to an interpreter at runtime, we could just remove the source code. But the complexity of doing this is really quite a heavy burden on a user. So as you say, the ordering of these things, publishing of Vex for a piece of software that you're publishing has to come with a deep level of engendered trust because if the Vex is wrong and people then run vulnerable code in production, there's a question potentially of liability when deeply uncharted waters at this point. So Vex is not something that we've seen particularly widely adopted as yet. There is recently a filleted version of the specification called Open Vex, which I'm yet to dig into. It will be interesting to see. I believe they are compatible. It's a subset from what I understand. Yeah. I do not know if they are compatible or not. Yes, so it is, it's definitely early days for these things. If S-bombs are nascent, then Vex is just a twinkle in the proverbial milkman's eye. And then finally we get to actually signing these things. Alan paints a perfect picture. If we have the capability to sign our software, then it makes sense for anything that we want to give that same level of trust to because of course the signature just says the person in control of this cryptographic key at this point in time liked this thing enough to put their name on it. It's not really anything else. It doesn't mean that the artifacts sanctity is secure. It doesn't mean that this composition is necessarily secure. It just means the person making a point in time judgment call wanted to put a stamp on it. So the ability to then do that signature signing process for additional metadata that you publish about your software, awesome. Yeah, definitely worth doing. But really the question is, is the thing that you're signing, do you have a guarantee that you're actually giving with that artifact? If you're just signing it for the sake of saying it's passed across my desk and I'm just gonna put a rubber stamp on it. That's not the level of trust and security that we want because there is nothing fundamentally there. And then the final point in that process is if we are publishing that VEX document, we have to be correct. So we have to be running not only perhaps the manual review or the manual exploitability check to make sure that we're not triggering that remote code execution, but also some automation around it, test harness, this is where the automated tooling comes in, reachability analysis, because the temptation will be to essentially allow that document to atrophy. It's done once, yes, we're happy, software moves quickly, it evolves, this is what we want. So if we then make a second release without revalidating the sanctity or the safety of that VEX document, we risk undermining the trust that is absolutely necessary for ultimately the internet, I'd say, but for the exchange of cryptographic information and trust and signing. So yes, small soakbox on VEX, I'm super excited for it, but it has to be done with a whole bunch of caveats. Yeah, yeah. The next question, I think there's a good question for you, Andy, a quick question that comes up. Whose job is it? Is it engineering team's job to generate these things at build time? Is it the DevOps team? Is it the security team? So how would you, these questions come up in organizations all the time? So how would you answer that question? Yes, it is a great question. So we can consider the composition of an SBOM to be similar to CVE introspection. So we already scan package manifests for dependencies and we take those dependency names. So for a package URL, MPM, left pad version 3.4, whatever, 0.3, that is a common practice in organizations because we use that information to go to our CVE feeds that are published by, in that case, the Node Security Project and then embellished by software security vendors to say, well, actually that version is vulnerable and it's got a CVE rating of X and you can then make a decision whether or not to ship that to production. Generating an SBOM is really a very similar experience. It generally involves going to a package manifest, pulling out the versions and placing those into another metadata file, which is your SPDX or Cyclone DX packaging format. So organizations have the capability and the patterns and practices to do that already. What is that doing? That is asking a static manifest for its information and potentially not including things like, did I just W get something from GitHub? Did I do a Git clone for a bit of open source code that I'm gonna package up into my final artifact? So there is a nuance there immediately already as to what is the final artifact that's being shipped? There's also a consideration here. Are we talking about generating an SBOM for a container image, which is all the runtime dependencies in the file system that we need to run an application or are we talking about a library that is then composed and consumed as part of a larger piece? So putting that in the problem space rather than necessarily answering it. At build time, we have the most information to build a complete SBOM. But if I give you an SBOM, you should probably revalidate it against the contents of the file system because plenty of SBOMs are wrong. It's perhaps even the question for the package producer, do you consider that actually part of the software that your software you've produced consumes? And then finally, we've got so build, packaging, ingestion. The difference there, build is obviously in the build system. Packaging is when it's being pushed onto the NPM registry, for example. But what about ingestion? Well, if we're a regulated organization, we probably pull code in third party code into our organization across some form of perimeter. Zero trust is a wonderful and necessary part of modern security, but it doesn't mean that we don't still establish a perimeter. We don't run all of our Kubernetes nodes publicly because defense and debt means that we don't throw the baby out with the bathwater. So when we're ingesting third party code into our organizations, we still care about its composition. We should still, at the top level of risk and compliance, we should still analyze those packages. We should still run static analysis on the code itself. We can still run those code reachability questions against those packages. And ultimately, at some point in the future, we probably want to do contributor analysis as well. The OpenSSF tried to do this and kind of had to back out because there's problems of collusion. You can't actually unmask all the Linux kernel contributors because they may be under authoritarian regimes. We don't want people being pressured to put backdoors into the software that we know and love. So there are questions of trust throughout this whole process and ultimately what S-bonds do is give us a better visibility into where we might have a vulnerability, but they do not solve the software supply chain security issue. They just enhance our view and give us more data to introspect. Got it. Okay, so just to kind of answer, you bring it back to what do you operationally do and how do you go about, which team actually creates your S-bonds? The recommendation here is S-bonds need to be created at build time and therefore the DevOps person does need to be involved, but I think the strategy of what S-bonds need to be created needs to come from the security experts. So security experts, AppSec, ProdSec, et cetera, teams are the ones that basically determine what do you need to do for the S-bonds, work with the engineering teams and the DevOps teams to go ahead and implement that as part of the CI pipeline to create it, right? Yes, I did get slightly sidetracked on the actual answers to the question. I think the DevOps engineers need to generate these things, but that question of exactly how many artifacts go in is really something the security team should be signing off on. Awesome. The next question is, okay, we've generated S-bonds. We've decided what formats to create it in and we've generated S-bonds during the build step. Where do you store them? Is there even a standard at this time? And after you store them, how do you share them? So let's talk about that. And we asked this question to Rose and this is her answer in a recorded video. The storage and distribution of S-bonds is something that up until this point, most people have had to figure out for themselves. There's no standardized best practices as the focus has mostly been on simply generating S-bonds. When we think about where S-bonds should be stored, I think the considerations around the storage itself is really not much different than any other type of cloud native artifact. The same considerations are still at play. Life cycle management, caching, versioning, access control. When we think about where we should store S-bonds, those qualities around where you're actually gonna put it are less specific to S-bonds themselves. But what is unique to S-bonds when we think about storage is their association with an artifact that they are describing and really making sure that there is some type of meaningful relationship between the S-bomb and the artifact, right? Ideally, if you're receiving a piece of software from a vendor, you would have access to the relevant parts of the S-bomb that describe that software in the same place or very near to where you are pulling the software from. As of right now, how S-bonds should be stored is less clear. There are lots of discussions taking place that are trying to answer this very question of how. So while there's no good answer, I do think that there's a couple of things to consider as we work to try to answer this. And the first is access control and the parties involved in sharing. If there's no reason to restrict access to an S-bomb, then sharing is a lot easier, right? You can post it on GitHub on your website, like any public download location. But when there's a need to restrict access for certain parties, things start to get a little more complicated and there's more to consider. Next is the actual data to be shared. S-bonds are after all just a collection of metadata and some of that metadata may not be appropriate to share without some sort of ULA or NDA in place. We need to consider the mechanisms for sharing. So likely having some type of network with the ability to push and pull documents or parts of documents. Life cycle management of S-bomb metadata and protocol around when certain metadata might expire or no longer be shareable. And this is just kind of the tip of the iceberg when we think about the best way to distribute S-bombs, which I think illustrates why there is no canonical way to do this yet, right? There's a lot to consider. So again, there is no clear and canonical answer for this, but a few considerations beyond what I've already kind of mentioned for how and where an S-bomb should be distributed needs to first and foremost consider what is the software and who is the consumer. Obviously, if you're downloading an app from the App Store, you're not expecting an S-bomb to come with it, right? But if you're receiving source code from a vendor to build into your product, you probably have more interest in the contents of that code. Similarly, if you have proprietary software whose S-bomb contains privileged information, you're not just gonna give it to anybody who asks for a copy of your S-bomb. So understanding the exchange can parties is important. If we look through it from the lens of open source, I think the answer is easier because there's less access control considerations in the open source context. Ideally, S-bombs would be distributed with the artifacts that they're describing. So most likely in some sort of registry with an easy to use API or client tool to pull the necessary information. Awesome. So to boil us down to, again, actionable and results, we've seen at Deep Factor as we work with various customers that customers choose either registries like JFrog Artifactory, or they're also sometimes used tools like ServiceNow even or just FileRepost to store their S-bombs at this time. The most important thing, like Rose mentioned, is not just where you store it but also how you give access to it and who's supposed to access what S-bombs. So thinking around that part and making sure that you work with your IT people to put the right checks and balances from an authorization perspective on your Okta or your SSO provider to restrict access to certain users, to certain types of S-bombs, that's the operational challenge. So thinking through that is the painstaking work part. The next question would be, should organizations be asking their vendors for S-bombs? Let's say you do have vendors providing you software. You know, if these vendors are selling into federal government, of course, they're mandated by the Biden executive order to provide their S-bombs. But if you guys are not a federal agency, are you still going to benefit by asking for S-bombs? Let's see what Rose has to say for that. So like most answers, it depends. Mostly I'm of the opinion that yes, you should be asking your vendors and cloud providers for S-bombs. If the software you're consuming from them could be exploited, which is most software, not all but most, if nothing else, as a way to establish trust with those vendors. So if you receive an S-bomb from a vendor that you're consuming, you can kind of do a sanity check. You can verify that the binary they give you matches the S-bomb that they give you. And this kind of will give you an idea if the vendor you're working with has an idea about what's in their code. And if the S-bomb only has two components and it's a huge binary with lots of dependencies, then maybe that's a red flag. Maybe you have to kind of reevaluate if you should be using that vendor. The S-bomb from the vendor might also provide additional information that you wouldn't be able to glean from just the binary. So maybe there's build time information in there that's relevant to the way you use your software. And three, the S-bomb from the vendor can help you understand your exposure and risk when CBEs come out affecting components that you might be consuming. And I think the caveat to all of this is that if you're asking for S-bombs from your vendors, then you need to have a way to make sense of that S-bomb. So just asking and then letting it sit there isn't gonna do you much good, right? You have to actually be able to parse the information and put it into use. Okay, so this is an interesting question. Let's say you are the vendor, you create an S-bomb. You know your S-bomb has some serious vulnerabilities in it. Are you still going to go share that S-bomb with customers or are you gonna shy away and not share it until that vulnerability is fixed? Like what's the ethical thing to do here? And also what's the practical thing to do here? So assuming that you regularly share S-bombs with your customers, I would argue that not only is it okay, but it's something you absolutely should be doing, right? If nothing else, if for nothing else, then to really establish trust intentionally, not disclosing an S-bomb when you otherwise would because it has a critical CBE seems dishonest to me. As a vendor, I would think that you would want to position your customer to know and mitigate the risk of CBEs. Excuse me, as soon as possible and they can't mitigate what they don't know. And, you know, notifying your customer of a high severity CBE that affects them is something that can 100% be done without an S-bomb, right? Like this is kind of typically how it's done today. You get an email or like a notification in a portal from a SaaS provider like, oh, you should update this product because it has a high severity CBE. But eventually we'd like to get to a point where S-bomb generation happens automatically, it's delivered automatically and the consumer has the tools and processes to consume the information automatically when a CBE for a package in an S-bomb exists. And really this communication around vulnerable components and CBEs is what Vex was created for. Alan mentioned Vex in his intro, but essentially it's a framework to assert the status of a specific vulnerability in a particular product. Awesome. I'm gonna ask you a question, Andy. Now that we've created the S-bombs, how do people consume S-bombs? Because, you know, there's no point just creating a bunch of S-bombs and just letting it sit there. What are the things, you know, proactively, operationally and reactively that one could do from a consumption and making sense of the S-bomb perspective? In the interest of time, you know, I'll give you two minutes. Thank you. Yeah, there's some interesting points there. One on the actual generation of S-bombs by cloud providers and whether they're disclosed or not. Ultimately, the metrics we can gather from these are meantime to remediation of the providers of our software. If somebody has a CBE and then they ship a patch the next day, well, we've got like a 24-hour remediation window, that is a risk window for our production software, but that's quite reasonable, you know? In large organizations, it might take two or three days just to get that process moving sometimes. So what should we do? Well, as there is no distribution mechanism for S-bombs at the moment, it is incumbent upon us as consumers to pull these from when we're actually doing an update of whatever the software is, to pull them from the vendor. Then we might do something like actually scan the package manifest with a container vulnerability tool because still those package URLs are the same information that we use for CV scanning. But ultimately, really to close this loop and to do so in good faith, as Rose says about the sort of dishonest lens of not disclosing a vulnerability or an S-bomb with vulnerability in, vendors should be pushing out updates and saying to us, we've determined that this zero day has been announced. It is in this package and we are actively working on it. The VEX document might then be updated for that. So the S-bomb still exists, but the new VEX would say, well, there's actually a exploitable vulnerability in here. Recommendation is we're working on it, which is one of the subfields of the VEX. And then they push the next S-bomb with the next package version at the point for mediation. So it is a complex dance at the moment and there is no solution for it. In my mind, that is the cleanest way to do it. There's still questions, who does that email go to? I mean, we're just gonna lose that in the security at inboxes. So yes, for the automation required. Right, a good resource for you guys to look at. As you think about how do I make sense of these S-bombs is the guac document from Google, GUAC project. Basically, I like the way it breaks it down into three parts. Proactively thinking about consuming S-bombs, operationally thinking about admission control, et cetera. So the proactive part would be, does my S-bomb have any vulnerabilities? Like if so, is anything critical? Do I need to tell my developers to go fix something? Essentially the SCA aspect, software composition analysis aspect of it, both from an OS packages installed in your container images perspective, as well as the dependencies that have vulnerabilities, et cetera. That's the proactive part. Then there's the operational part, which is I'm gonna start my container, and my Kubernetes is gonna start my container. And at that time, from an admission control perspective, do I even launch certain containers that have a certain set of vulnerabilities or a certain set of signature mismatches and things like that? So admission control is the second operational part. And the third part is reactive. There's a new CV that just came out. Where all in my Kubernetes clusters is a vulnerable component running that has that CV. I should be able to go and search for it and then look at it from my S-bomb from a reactive perspective. So all of these three aspects are very important. So don't just think about creating S-bombs and storing them somewhere. Actually think about this from a strategic perspective. How do I make sense of it from a consumption point of view? Otherwise S-bombs is just a bunch of files that are sitting in your folder. We are out of time, isn't it? Yeah, okay. So because we had to do this recording and as opposed to the conversation, I think we overshot on time a little bit. But if you do have any questions, by all means, please reach out to Annie and I. We're gonna be here for a few more minutes to have a chat with and answer any other questions or just geek out with you guys. So thank you.