 We're going to be talking, this panel is going to be on not just S-bombs, so it's going to be a talk about supply chain security focused on all of the other great things that we're doing to either complement S-bombs in addition to S-bombs as opposed to S-bombs for various use cases. I'm going to introduce myself and then I'll let the panel introduce themselves as well. Unfortunately, Dana from OpenSSF had to run out, so she won't be joining us today. So I'm Mike Lieberman. I am a CTO and co-founder of Kusari, a software supply chain security company. I'm also a CNCF TAG security lead. I'm also a member of the OpenSSF technical advisory council and a member of the OpenSSF governing board. And I'll hand over to you. Oh, is the mic down? I'm John Schell, director of open source at TestifySec, contribute to many of the same things as everyone else up here on TAG security and CNCF. This also projects the security tool belt in OpenSSF among many other things. Hi, I'm Marina. I'm a PhD student at NYU doing research on software supply chain security, especially secure software distribution and updating. I'm also a co-chair of TAG security, the CNCF technical advisory group for security, as well as a maintainer of TUF in Toto and a couple of other security related projects. Hi, I'm Arnaud Lors from IBM. I'm part of the OpenTechnology group. I'm actually not originally a security expert. I'm an open source, open standards expert, but I've been focusing on OpenSSF for over two years plus. I serve different roles there. I'm currently the vice chair of the technical advisory committee, and I'm also a maintainer of the Salsa specification and participated in a bunch of different groups. Cool. So let's reintroduce the problem yet again. So this is from the Salsa documentation, which I think is a pretty good introduction into the problem. So really, at the highest level, at least in our opinion, software supply chain security is really security that is about the SDLC, and by that we mean security that impacts the production of software, the storage and distribution of software, and then the consumption of software. So not how do I prevent a SQL injection attack, it's more of how do I prevent myself from using software that has those vulnerabilities in them, stuff like that. And so quickly here, you have lots of different threats that can happen at all these different stages throughout the SDLC. A developer might make a bad commit, or you might have somebody impersonating a developer tries to commit bad source code. Your source code repository could get compromised in some way. When pulling down and pulling in source code into the build, all sorts of things can go wrong. Think of the SolarWinds attack where the build system itself had gotten compromised, and then the build system had access to all the keys, and so everything looked good, even though actually everything had been compromised. You can pull in any number of bad dependencies into your build, whether it's purposefully or inadvertently. And this includes things like purposefully malicious dependencies like the ones we saw in some of the MPM attacks that had compromised, that had been looking for folks crypto wallet keys. And then also when you go to package everything, obviously somebody can inadvertently or purposefully push out a malicious package or the incorrect package, and we've seen that sort of thing as well, where dev builds had gotten published out as production-ready builds. And then the consumer can inadvertently download the wrong software through stuff like typo squatting at sorts of attacks, where they think they're installing one package and they're in fact installing another. So that's really at a very high level what the software supply chain problem is. And then what's an S-bomb again? So a S-bomb or software build materials is an ecosystem agnostic data interchange format for software composition metadata. So you think of your requirements.txt, you think of your packages.json, those sorts of things. But what if you had just had one way of reporting that to others, so you can actually include all sorts of stuff that exists outside of an individual ecosystem inside of an S-bomb? So oh, I'm using Node, but I also am using OpenSSL, that's a dependency. Now S-bombs also include all sorts of other stuff like AI, service bombs, hardware, you know, so on. And so what are some of the problems with this? S-bombs are only a piece of the puzzle. They're very much right now a reporting tool. Some folks sort of view this as mostly a compliance tool. A lot of the tooling hasn't kept up to date with all the various features and things that are coming out of there. And then there is a lack of focus on S-bomb consumption use cases. Currently, there's a lot of focus on producing S-bombs, not so much focus yet on how do we consume S-bombs, how do we actually do stuff with that data. So what else is there? So here's just an example of some of the tools, frameworks, and projects that are in the supply chain sort of security space. And of which the panel will be probably talking a bit more in detail about some of them. But just at a very, very high level, you have stuff that is focused on production side of software. So things like Salsa, which is focused on secure builds. Things like Scorecard, which is focused around securing, you know, repositories as well as All-Star. Witness, which is also focused on securing builds. In the middle, you have sort of stuff that is focused on things like, around storage and distribution of software and software metadata, the supply chain security metadata. So this is stuff like Guac, which I'm a maintainer of underneath the OpenSSF. Projects like Archivista, which John is a maintainer of and, you know, so on. And at the bottom is stuff that's focused on consumption side. So how do I now consume all that software securely? And yeah. And so then on the left-hand side is just essentially the rules by which you would then apply all of that. So how do I know, you know, who should I trust? What are the policies, that sort of thing? And that's where things like in-todo, six-door, tough, et cetera, come in. And so now I'm gonna start asking some questions to the panel here, right? And so I guess to start off with, what are your thoughts on, you know, since a lot of folks have been hit over the head with, hey, supply chain security is a super important problem. What areas do you feel like today are not getting enough focus, right? We keep hearing about S-bombs. But what do you hear today that you don't see getting enough focus? All right, so I'll stop. I mean, so first of all, I would say I think overall, the way I see it is, the industry has become better at bolting down systems that are in production. The problem is we kind of forgot or lost sight of the fact that basically, I mean, the good news is open source is everywhere. There's basically no software today that doesn't use some kind of some component of open source. So, you know, for me personally, having dedicated my life to open source, I find that very great, you know, I'm very grateful for this development. The problem is it's become kind of the Achilles heel of the software industry because we have not been very careful at making sure that those dependencies are also secure. And so essentially, S-bombs, what do they do? They tell us what's in the software. And it's especially helpful in governments, you know, I've been taking action saying, okay, industry, you cannot just keep doing these things. You're doing where you're distributing products with no liability saying, hey, use it as is at your own risk. So governments are starting to put regulations in place that impose, you know, liabilities, responsibilities to the producers. And so S-bombs is the first answer to the problem of, well, what's in the product to start with? Because we have all sorts of systems as an industry have developed, like, you know, ways to disclose vulnerabilities. But people don't even know if it concerns them or not, right? So S-bombs are supposed to help you better understand what's even relevant to you because you can tell what's in the product you're using. But it's only a very small part of the answer. The next thing is, well, what are we doing as an industry to make sure that all these dependencies we have are equally secure? And quite frankly, I think one of the big problem we have is people have taken it for granted that, well, there is a community out there that's maintaining those open source projects for me, so I don't have to worry about it. And clearly, this is not working. We've learned that, well, you cannot depend. And what we are teaching people in IBM and a big problem we are facing, still as an industry, is to educate everybody on the fact that they need to take responsibility for all these open source dependencies they're taking on. Of course, if you use Kubernetes, something really big with a big organization like CNCF behind it, you can kind of assume that there's a big community around it that's going to take care of vulnerabilities when they show up. But there's tons of other little projects that don't necessarily have this kind of backing behind it. So we need people to be educated about the fact that they need to take responsibility for all the dependencies they have, they introduce in their products. That's the number one. But then the problem is, I think this example of this slide shows the challenge. It's not an easy fix. There isn't like a one tool, one setting kind of thing you can turn on and then you're good to go. I mean, I don't know if some of you have attended Marina's presentation the other day. She went a little bit more in the detail of all these different technologies that are at play. And I think the big challenge is for people to get their head around the complexity of the problem. There are so many attack vectors all along the supply chain. It's not an easy fix. And it will take a long time to get to fix it all. I'll stop there for now and let them talk. Yeah, so I agree with all of that. I think the one thing I would add as a place that we could use for more focus is on this idea of a trusted route or a trusted identity and where that comes from. I think there's a lot of discussion that's been happening not just around S-bombs, but also around signing software, signing S-bombs. And I think signatures are great, but they're only as good as if you know who's supposed to be signing it. And I think that idea of figuring out who's supposed to be doing what in your supply chain and be able to enforce that policy, I think that's another big area that could use some more focus. Yeah, adding on to what Arno said about the complexity. In the security hub yesterday, we were having a long conversation about vulnerability management. And how about I show our hands, how many people have one security scanning tool, at least one security scanning tool they use? All right, how many have two? Three, four, five, six, 10, 15, all right, so how do your developers triage hundreds or thousands of vulnerabilities that are found? Maybe false positives, maybe they really exist, maybe they're in a code path that's executed, maybe they're not, maybe there's a configuration change that solves the problem and then your customer goes and turns it on anyways, even though you told them not to, what is a vulnerability, what's not? You're not gonna solve this problem through a brute force method. You're gonna need to take a fundamentally different approach and that's where you're gonna need to go back to the beginning. You're gonna need to turn this software into an actual supply chain. You can't have a signature at the very end of it or a signature of the artifact or a signature of a get commit in the beginning. You need to really follow everything from the beginning to the end, make sure that the output of one step is the input to the next step and that's where projects like Intodo really come in here, that specification giving you that ability to communicate across these things and set up policies and be able to enforce the compliance and really ensure that what you think is happening is what's really going on. Yeah, so I guess, yeah, to follow up with that, since you described that the problem is quite complicated, I think a lot of folks, their first question is where do I get started or what's the simple path for me? And we'll throw it back over there, which I know is, that's the hard problem right now, that's the hard question. So it's interesting because there are many ways you can approach it, that's the flip side of it. There are so many angles, do stop doing anything, that's better than doing nothing. And as you go, you'll get, you have to expect this is a long road. And by the way, there's two sides to this problem, right? Right now we're trying to get the producers of software to improve the security posture of their supply chain so that when they build artifacts, these are much more secure and they come with all sorts of information, provenance and everything. But the other side is going to make sure that the consumer of those artifacts also check that what they get is what they expect it to be. And my analogy in this is that we're still very early on. It's a bit like in the automotive industry when we said, oh, all cars should have a seatbelt, right? So we basically, there were regulations put in place to ensure that every car that gets produced gets a seatbelt. But quickly realized this was not good enough, right? Because if people don't fasten those seatbelts, they don't do you any good. And we have a little bit the same problem here where we're going to have artifacts that get produced with a lot more security built into it, a lot of more like metadata associated with signatures for all the different pieces that went into it. But if people don't check that what they're actually using has all the right checks in place, well, it doesn't really do much good, right? So I think that's going to be the next big challenge. And so depending on where you are in the spectrum, if you're more, and often we are both, right? Because we use components, we as part of building the other solution that maybe we either pass on to somebody else. And so we have to play both sides of this problem. Yeah, I think step one, everyone here is doing it already. You're learning more about the problem. Everyone who's here, you're like, you're not at level zero anymore. You've gone up a step. And how do you continue that incremental adoption? Because you can't boil the ocean. And so we've seen this throughout the industry with things like Project Sigstore enabling signing to become much easier with the introduction of things like keyless signing, not having to solve that problem of how you manage keys. But as Marina pointed out, that signature is an assumption of trust about all the things that came before that. And now we're seeing progress from that with things like Salsa Project having build provenance. Now you know, when you see that signature on a signed provenance document, you can say more information about what went into that build or the source of the build and how you're consuming it in your organization. But once again, that's also not enough, but it's the starting point you need to get to and we need to keep progressing after that. The Salsa Project is gonna continue building out more tracks, a source code track, a dependency track. And at the same time, there's things that will never be in the Salsa specification. The results of static analyzers or dynamic analyzers, the custom and proprietary things that your products may be building into themselves. And that's where the general nature of Intodo allows you to customize that, but still communicate that information across the different steps in your supply chain. And so every extra step that you create an attestation for that can communicate some piece of information reduces the scope of the trust boundary that you have. And so you gotta start somewhere, make it as big as possible and slowly over time make it smaller and smaller. Yeah, I think I agree with everything that's been said and I'd like to add that. Yeah, I think that as you build these attestations over time, you can also build up the security of the attestations at each step. You can start by looking at your supply chain and just attesting what you're doing today. And then at the very least, you're generating these attestations and you know what's going on in your supply chain. And then over time, you can require more things. You can maybe lock down your build server and require that there's a specific ID from that build server that's signing these attestations or so on, right? You can build up those different pieces after you have kind of a visibility into what's happening today using these same building blocks that can then let you build that over time. Cool, yeah, and I think one thing to actually, I would add on there, right, to tie it back a little bit into S-bombs is actually, hey, start attesting your S-bombs, start signing your S-bombs, so that folks, even if the S-bombs is potentially inaccurate, you know where it came from, you know who generated it, right? Whereas I know a lot of folks keep saying, oh, I got an S-bomb, it's here somewhere. Well, where did it come from? I don't know. So yeah, I think that that's also another piece there. So I guess to tie it back a little bit into sort of the S-bomb story, how do you see S-bombs fitting into this bigger picture of supply chain security, not just from a check the box mentality, not just from a compliance perspective, but where do you think that S-bomb fits into that picture? Yeah, I think an S-bomb is a great place to start with that visibility step. You can see what dependencies you're taking in and what's happening in them, and then you can use that to then dig into each of those dependencies, maybe check on the health of those projects or learn other things about those projects, or use that as like an opening to then start signing things, start by signing that S-bomb and then use that to see how the process works for you, how you can then use that to sign other steps in your process that you're doing as well. So yeah, I think it provides that visibility and it provides kind of an entry into looking at your broader supply chain security, what's going into your software and how you can look at that more broadly. Yeah, and I think it's all about the use cases for that information, right? If you're doing vulnerability scans of your container images, how long does it take to scan? I'm sure no one has a container image larger than a gigabyte, right, or 10 gigabytes, but the bigger it is, the longer it will take to scan. If you have this well-formed S-bomb document listing out everything that's in it, do you really need to scan that container image or can you look for a tool that can actually process that S-bomb in a way to tell you if there's newly discovered vulnerabilities? And so I think there's a lot of exciting progress in the area like that. How do you discover information like that that's right there at your fingertips? If you store it the right way, you can query it, you can index it, do all those things to enable that analysis. Yeah, so practically speaking, if you're a producer of artifacts, you should definitely start producing S-bombs along with other metadata, like we've mentioned several of them in total attestation and so on. And a lot of the offerings that are out there, I mean, IBM as an offering to build in your products, your software in the cloud, and we have this ability now to produce S-bombs along with salsa provenance and a lot of other cloud providers have similar offerings. So that's step number one is turn those things on and start producing S-bombs. If your consumer start demanding that your providers give you that information as well. And I can share with you like, you know, an anecdote from a customer in the automotive industry, a big manufacturer, they started demanding from all their providers to get their S-bomb and they started building a system internally where they ingest all the S-bombs, they could some kind of database, and that way they can be on lookout for CVS. Of course, you know, and I think this is the reality today is that S-bombs often have a lot of crap in them. There are many different formats, they are not even following the schemas. I mean, one of my colleagues actually wrote a validator and he was horrified to see that most of the tools that are very generally used, you know, broadly used, don't generate valid S-bombs. And so, you know, it's a bit of a rough road right now because we're still early on and you know, you have to be able to pass them and then once you pass them you have to be able to make sense of them. And then it was pointed out later on, you know, earlier is that later on when you get all these CVS, you have to figure out what it relates to and whether it impacts you or not. Knowing that many cases, CVS might be pertinent to some of the components that you're using but not actually in the code path that you actually use in your product, right? So there's different levels of false positives that actually make you, you know, making use of this not so easy. So we shouldn't try to, you know, brush the perfect world on this. It's by no means a perfect solution but we're all into this. We know we have to make progress on that road and we just, you know, hopefully as we progress we'll have better tools that will help us filter the noise and all that stuff, but that's where we are. Yep. And I'll add in a quick point before opening up to questions from the audience. Yeah, so I very much agree with everything the panel said and I think one thing I would add on there is, you know, anything we could be doing as software producers also for the security engineers in the audience to make this simpler for both producers and consumers to actually use would be super useful because, you know, we wanna be able to take as we mentioned, you know, all that great salsa in total attestations, open VEX, S-bombs, et cetera. We wanna be able to ingest that information into, you know, tools like Archivista and Guac, right? That can then be queried by various tools to then, you know, answer questions like, hey, if I do have a reported vulnerability, where does that live? Do I have assigned attestation so that I know did this actually come from a trusted party and all that, so that, and then, but at the end of the day, right, we wanna make sure that as a consumer it should be relatively transparent to you, you know, it should be baked into the tools that you're using in on a daily basis as opposed to, oh, as a developer you need to learn everything that's up there. No, no, no, no, let's try not to do that. So yeah, with the last 10 minutes, wanted to open it up to the audience to see what questions folks have about supply chain security, about S-bombs, about any of that. Don't be shy. Oh, sorry, there's mics on the side of the room. Feel free to line up. I've worked with artifacts before, I know the struggle of getting a developer to understand that an artifact exists, it is not just a config file, it makes stuff happen. What do you think, I can see some of the same problem here, that when I try to explain the existence of this issue, people have a hard time even comprehending the space. What do you think can be done to try and overcome this and make it less technical to accepting systems of this issue? I think there's two reasons to that. I think you can look at the attacks that have happened as examples of why we need to look at the space and really secure the different steps of the software supply chain. I think that can help illustrate why people should care. And I think the other thing is just making the tooling easier. I think not every developer, not every person at a company should need to be an expert in all of this. Like somebody at the company should, but then there should be tools that exist so that people can just add small steps to their existing workflows. So yeah, I think over time as we improve that tooling and it becomes easier to do secure supply chain practices, it becomes less of a question of why do I care, but why not, because it's easy to do. It does improve the security, it prevents these attacks that are out there. Yeah, I think you look at solar winds and it costs the company literally billions of dollars. You could have spent hundreds of millions of dollars securing your supply chain and you would still come out ahead. So we talk to people all the time who struggle with this idea of how to get more funding for the work and things like that. And I don't have a simple answer for that problem, but it seems straightforward as far as justification. And we'll always work to try and make it simpler. And I think the emergence of these new tools, like I said earlier, there's no brute force way to solve this. We can't solve it with what exists today, which is why we're building new tooling to solve this in a different way. And it will take time to adopt it and make things better and easier for everyone. But I think we're going in the right direction. Thank you. Yeah, on that front, I want to add, I mean, this is why I was mentioning there are regulations coming out a bit all over the world, right? Because it's actually costing a fortune to the economy in general. It also has a public safety aspect to it now, because there is definitely personal information being leaked and so on. So I think the justification indeed is easy to make now and to find. And by the way, it's not anywhere near getting better because if you look at the numbers, there's an explosion of attacks and now they're all focusing on the supply chain, right? The vast majority of attacks that are happening now, they come from being infiltrated at the open source level, which eventually make their way into the products. So this is a real problem and this is why a lot of the industry players have started Open SSF and we are active in this OWASP and so on to try to tackle this problem because it's bigger than any single enterprise can tackle. It's really an industry-wide problem. And to add on, I think, two pieces. One is, I think, and actually one of Arno's colleagues, Jeff Borek, brought up this metric, something like two-thirds of all security teams don't share their data outside of their security team. And that sort of thing, I think, has to change where we can't have the situation where, and I've lived that at a large enterprise where often you just get a report or you get an email saying you have a critical vulnerability in some software of your own and the report was sent to your manager's manager's manager and you get tapped on the shoulder right before a go live. We need to be more proactive about this. We also need to get the data in the hands of the people who need it, in the way they need it, right? So as a developer, I don't want to be looking at a vulnerability dashboard. I want to be looking at a, I want to be looking in my code and, hey, if I get the little red squiggly in my code saying, hey, you're using a vulnerable dependency, tell me so I can update it. So I'm hearing that we're gonna give people a lot more information about what's in their software. In the past, we've relied on distributors like, if your vendor is Red Hat, for example, to kind of parse all of that and you could say, hey, I can trust this because Red Hat said it was trustable. Have we thought about ways to help people figure out what data they can trust? Like if I have a six store signature, that doesn't mean I can trust it, it just means some person signed it with six store. So what do you recommend on the, like how to figure out who I can trust front? Oh, Marina. Yeah, I think that part of it is internal policy that you have to kind of just figure out and the other part of it is figuring out like what are those roots of trust? Like especially for figuring out like, exactly what you said, like signature tells you that somebody signed something, but you can say that, for example, somebody else trusted that somebody to sign that thing and you can build up a series of those statements and so then if you have an organization and you have a number of developers at that organization, say you're just following the actual like manager structure, it's like this manager says that this developer should be signing this package. At least then you can then just trust that manager and so on and kind of build this chain of what in tough we call delegations, right? That tell you who should be signing what, for what purpose? And I think those individual decisions about why I trust this developer to sign this package, I think just comes down to either the structure of the organization or the open source community and how they decide to kind of delineate that trust. But fundamentally it comes down to the root of trust, and it's like, well, there's always gonna be that question that at some point you have to trust somebody and there is no universal answer as to who you should trust. It's a personal choice kind of thing, but you're right, that vendors do, I mean, it's at the same time, it's a business opportunity, right? There are vendors that will provide indemnification and so on in exchange of money and so that's one possible answer. Next question. My question is more on the proactive actions you've mentioned. So one of the fundamental steps I feel of prerequisites to some of these challenges is threat modeling, threat modeling of any system or application or the DevOps cycle or supply chain cycle. How do you see that, what's your perspective on that is what I want to understand. Mainly from threat modeling basically gives you a sense of where things could go wrong and in what way, whether it's buffing and how any vulnerability could be exploited. Oh, I was just gonna say, yeah, so my opinion, yes, that is very like threat modeling, your SDLC and your supply chain is probably step one, especially at a large enterprise, right? Where you want to be able to understand what the biggest risks are, like are they existing in the build? Are they existing more in the packaging area? Because that will help you out. I also, I don't have any left, but I have written a book where we go into sort of threat modeling this. Yeah, I was just gonna say that, yeah, that threat modeling step can help you determine where to start when securing the supply chain. So we're gonna help you see what are your weaknesses today, where should you start as you build that up. So I think it's definitely part of the process. Thank you. Okay, one last question. I wanted to ask what the, I guess, ecosystem is like for people that are consuming S-bombs today. Yep. Or like for people that are asking for them, what are they doing with them? Are they rolling their own things to like comb through them or their services or what actually happens once you provide those? I don't know that anything does. They get printed out and stored in a filing cabinet. I mean, unfortunately there's, I don't think there's many people using it. It's hard to use them. So I think that's part of the problem. Yeah, I mean, I think there's some projects like for example, open SSF project guac does ingest S-bombs. There are a few other ones like dependency track that do ingest S-bombs, but and there's various vendor tools that do, but I would say by and large, it's kind of currently still a compliance function, more than a security function, that at least from my experience, but I think the shift is happening where we are seeing more folks actually ask the question of I have an S-bomb, I wanna do something with this. What do I do with it? And I think that leads to more folks building out tools that can then consume those S-bombs, analyze them, do all sorts of stuff with them. Well, as I said earlier, I mean, there's a, some people definitely try to ingest those, build the database with it, and be more like proactive about CVs that get reported. Yeah, so it looks like we're at time. So thanks everybody for coming and... We are happy to hang out a little bit outside. If people have questions, we can entertain some additional questions outside. Thank you.