 The plan is, I would love it if this was a very interactive session, so we're going to have a microphone. If you have a question, raise your hand at any time. I have one we'll start with, but then the rest of you, whatever you can think of, whatever you want to ask these folks, like it's fair game. So I think the more audience questions we have, the better, because I have plenty of questions, and my questions are probably boring. So we'll kick it off on the very end. John? Hey. Hey, everybody. My name is John Yeo. I'm head of research over at the Cloud Security Alliance. And what that means is we manage a portfolio, not just on the Cloud Security, but securing technologies and implementations and adoption methods. And we have a think tank of around 9,500 different contributors and experts that are part of that group. So it's pretty cool to be a part of, because I feel like I'm centralized on what providers are doing, what customers are doing, what auditors are doing, what implementers are doing. So we get to hear the best of all sides. And I get to be a pretty big role there. So it's kind of fun. That's me. Awesome. Thank you, John. Tracy. Yeah. Hi, everybody. I'm Tracy Miranda. And I've been working with open source and open source communities for over 20 years. So communities like Eclipse and Jenkins will help them scale. And currently I work at Chain Guard, which is a startup whose mission it is to secure software supply chains by default. And as part of that role, I oversee a research team which are looking at all the data that we can gather from open source around software supply chain security. Amazing. Wonderful. All right. I'm Eric Tice. I work for, I feel like my already loud voice is making this worse, but I work for Wipro Technologies. Wipro has a group called Lab45. We're all about innovative strategic new technologies. We work in all of the emerging areas like AI, sustainability, 5G, and everything in between that you can think of. I have been working in the industry for about 25 years and started open source back when Java was a command line language and everything was done with Pearl and CDI and all those fun technologies. I'm a member of FinOS, OpenSSF and a number of other foundations and just happy to be here. Fantastic. And I'll finish up. My name is Josh Pressers. I'm the vice president of security at ANCOR. And my kind of other claim to fame is I do the open source security podcast, which is a load of fun. I do a hacker history podcast, which is a lot of fun. And I'm just kind of an all around supply chain and security nerd. So this is right at my alley. Thank you all for being here. I appreciate it. All right. I'm going to start us out. The questions raise your hand. We'll get you a microphone, but I just want to start with a silly question, which I've written down is like, what is your favorite source of data when you're kind of thinking about the supply chain? And Tracy, I'll start with you because you specifically mentioned data in your intro. Yeah. No, I think I'm going to the source itself and just looking, there's a wealth of data around open source itself. And like a couple of great examples I can give when we like I'm really interested in ways people look at securing open source. And you can look at things like ruby jams and pi pi and they have like mechanisms for signing the software. So that's kind of built in, but then when you look at the data, you'll see in practice, it's only like 5% of the community will actually use it. So that's sort of more telling than the capabilities. And the other interesting one, we recently did some research at chain guard where we were looking at base images. So you know, you can go to GitHub and you can look across all the Docker files. So we looked at about 3 million Docker files to see what are the most popular base images folks are using. And then we ran them through various scanners to see like how many vulnerabilities are in each of these images that people are using. And it gave all sorts of interesting insights, you know, some were pretty scary like node image had over 800 vulnerabilities, but also other interesting things like some of those vulnerabilities had been there from 2003 and 2004. So sometimes it's really sticky and it's really hard to overcome in practice. And even though people might say, okay, it's low vulnerabilities or medium severity. We think it's still important to address these. So but yeah, starting with that grounding off, what is the data telling us? I think it's important for a baseline. More context of what you're looking at, right? Yes, absolutely. I like that too. That's cool. Right, John. Oh, I volunteered myself next by speaking to that. Whoops. What's your favorite data source? Man, I had trouble like thinking about this too, just because there's so many cool things that, you know, an open source world and supply chain world. But there's something new called the global security database. It's called GSD for short. Big surprise, a new acronym, but what I like about it is that it encompasses everything in the vulnerability space right now. So everybody knows CVEs and then NIST does a good job with, you know, NVDs, which kind of spins off of that. GSD, the intention is to encompass everything so that if we think of what a CVE is, it's a software vulnerability that's customer owned. And especially playing in the cloud space, man, we don't own, you know, most of the applications and builds that we use in our businesses now. And so this expands that model to include vulnerabilities outside of your control so that downstream you can see what the configs should be and how that impacts what we're doing when we're playing in the cloud space. So I think GSD is really cool about that. The other aspect of it is it accelerates how we do vulnerability identifiers. So instead of submitting something, as a researcher, instead of submitting something and waiting for the application owner or in this case, MITRE to issue a CVE, this goes direct to a platform that's community driven. It's open source driven. And so I think we can do things faster and we can also have a larger collection of information, which is valuable to everybody who's, as far as I know, all of our environments are very different in how we operate, what data is precious to us, what applications we use, what source code we use. And so being able to do this based through a community, I think the benefits are just huge and I hope it just continues to grow and that we get more people contributing to GSD, more researchers, and more people using the information. So GSD, I picked one. Thanks, John. All right, Eric. Yeah, I don't think I have a specific, you know, I'll take a little of the comment there. While community is obviously very important, there are a large number of foundations like OpenSSF and OpenChain and SPDX and, you know, there's SBOM working groups and SIGs and a whole bunch of others that provide a lot of meaningful data, as well as, you know, companies like, you know, not to give Josh too much credit, but companies like Anchor and others who put out the state of documents every year. And all of these are very useful, but realistically the type of industry that I'm in, working for a GSI, we also have the insight of what our customers are going through, be they, you know, high, heavy proprietary tool users, which most of those tools are built off of open source, or, you know, an open source-first type of company. So we're able to pull a lot of metrics through different domains in the industry, not just specifically from some of the core sources, but we see how people are using them across the board, be it, you know, cloud, completely cloud-native, and what the potential issues everyone has, and how quickly we have problems. You mentioned one of the key components of just looking across the board as companies take and fork open source, how many of them haven't updated that fork in five years, and how many of those transitive dependencies are now problematic. So you gain a lot of metrics there as well as just through the community. So that holistic viewpoint is really where I try and pull most of my metrics from. Awesome. All right, so let me, let me, oh, sorry. Question? Perfect. Thank you. Is that on? Yeah, so I've done a lot of work with looking at the uses of software dependencies right in this space, and you talked a bit about how, at companies, you have kind of a different view of what packages are actually used in practice and how they're used throughout the thing, the ecosystem. What can we do as, like, an ecosystem to make data available to a broader community of researchers and other people about these internal insights of various different companies so we can kind of collate that in a way to really see what people are using and how we can focus the efforts to secure packages that are used more often? Does that make sense? I think it makes perfect sense. I don't even have to summarize that. That's wonderful. All right, would one of you like to take it, or shall I assign it? I can start, if you don't mind. So one of the key things in OpenSSF is one of many potential options to gather this type of information. It's broken off into a number of working groups, and beyond that, there's an education SIG that's happening right now, which is all about solving that problem. How do we educate people, provide the appropriate data, and not only training materials, but how can we take all of the information that exists and disseminate that to a wider group of people across the industry? That's actually broken into three subgroups now, if you haven't been generally, if you haven't been to OpenSSF and seen that it's all in GitHub. But technically, the key right now is it's very, very new. All of these working groups are working to catalog all of the existing problems. And then beyond that, there's a cert SIG that's actually working to how we can help the industry remediate these problems. So that's one avenue that you can look to, to start solving those problems. I would say that, you know, from what I heard is like, what kind of visibility do we have when we're building anything, right? And I know, you know, S-bombs, the software build materials, they've been around for a while, right? But I think finally, now that there's an executive order from the U.S. government on, hey, we need to make sure we're declaring S-bombs on the things that we create, I think that's really important to have is the build materials of what's in our software builds, but also a standard approach on that. Because if I just have a list of a bunch of, you know, you know, code builds and scripts that, you know, I don't know what version they're on. It's not really signed properly by whoever wrote that S-bomb. I think that's where things get really confusing that doesn't make our lives or our jobs any easier. So having a clear understanding of what's in the build materials. And now as we talk about supply chain, it's not just our software builds. It's, you know, cloud services, it's cloud operating systems. How is this going to inter-operate with, you know, what I'm creating and into, you know, what our customers are using? So I think there's that huge chain of build materials that we need to start having a standard approach on. And I think we have a standard way to do that with the S-bombs now. And I think now that as we go to cloud and down to other operating systems, it's going to be really important to do that. Yeah, and just building off that, I do think it's a great idea, but it's super hard, like even just looking in open source, where we're starting to kind of drill down into things, it's hard to do so. You add in, you know, companies and closer software and proprietary things, it makes it really challenging. I do agree S-bombs should play a key role in that and certainly standardization will help. I do feel we're not there yet, like there's still a lot of gaps and we're just starting to understand more about S-bombs and how they don't necessarily see everything and they don't see the right level of depth of dependency. So just pushing to have more clarity around what should this look like, how can we measure the quality of those S-bombs and then how can we start sharing them and just building up this wealth of data which we can tap into. I think that would be a great direction for the industry. And there is something of a movement in a lot of different publications that have come out that say roughly by 2024 or 2025, that even all of the proprietary technology will be expected to produce. 90% of those should be looking to produce S-bombs for everything they provide or they won't be compliant. And again, that's speculation, but definitely starting to go in that direction. Awesome. And I saw a hand up, oh, two hands. I think the young lady in the back had her hand up. Or did they? Am I wrong? Yeah, that's right. Oh my goodness, look at all these hands. It's a contest. This is great. Do we have an history play? Maybe we need a line. Hi, thank you. Rewind a bit and go back to basics for a second. I think you just touched on that in the last question, but I would love to hear your distinguished opinion about what the definition is for software supply chain security because I've been hearing a lot of conflicting things in the industry and even here in this conference. Is it only the stream of things that are coming from outside of the organization or are we focused on the entire, like the real software supply chain and everything, all of the dependencies and proprietary things and builds and all of that? Thank you. You're gonna pay for it, because there's a scanner. Well, you know, an attacker doesn't care about a definition of secure software supply chain. So I think we do need to address it from all aspects of that, right? Where the build, the vendor partnerships that we have, the developers themselves too and the entire software supply chain process or SDLC. So I think there's what we do internally, but it's that build versus buy. And today it's so easy to buy things. So even though we're gonna build things in the house, we're always going to do that. We're buying so much like we at the CSA, and we buy so much more than we build because, well, one, we have a small dev team and then we have tons of other partnerships there too. So yeah, when you talk about supply chain, it's what we do in house with the SDLC, but it's also having that visibility into what our partners are doing. Do we have security policies that scale with SaaS services, with the platform service that we're building? Are we able to do that all within a standard that meets either compliance metrics or things that we're doing internally? So yeah, we have to attack it all aside. So I don't think we can just define it within one small scope for everybody. Every team is gonna have to play a role in securing the software supply chain. Awesome. Anything to add? Tracy? I can add to that as well. I'm kind of building off of that. One of the key things that I, and again, it's very much a developer centric. A lot of what I talk to people about with software supply chain security is the traditional kind of ivory tower of CISO that drives all of security within an organization or just a security portion of an organization is very production centric. And you've deployed something, we need to make sure that it can't be penetrated, it's not vulnerable. And with the shift left kind of mentality of what's happening in the industry around cybersecurity and really making it more of an education for developers and starting at the inception of a project and not just open source projects. He mentioned buy versus build. One of the key components when making that decision is what have they done to actually secure their project during their own STLC process? So as you think about from when the developer takes it from the initial check-in of code all the way to I'm going to deploy this, how am I securing that code as much as possible and not just the code. If I'm using cloud native, it may be my containers, it may be an environment, am I signing my code repository? All of these types of things. And the salsa framework is one of many that have come out recently that have discussed what are your levels of provenance and are you tackling this at each phase in the STLC. So for me, it's that constant conversation of regardless of whether or not you're building a proprietary or an open source technology, how are you solving your own security so that your downstream user can be reasonably assured that you've given them something that is as secure as possible. Thank you. All right, we got a lot of hands. So I'm going to move this along with God. Can I ask one question to everybody though? Like how many people, like security is part of your job? Nice. I'm hoping everybody's raising their hands. So, okay, cool. Hi, can you hear me? So my feeling a bit with S-Bombs is I know the cost, but I don't know the value. So from a consumer perspective, I know my dependencies, but I don't know what they're worth. I need to do two diligence on each and every dependency to know whether or not I'm at risk or not. So maybe we need some kind of classification system for libraries like a class A that's well maintained, yada, yada, yada to class F last update more than two years ago. And basically, you can automate your software deployment by saying, well, I don't want more than 10% of the class Lord and C in my deployment system. All right, I'm going to summarize this one. I think I capture what you're talking about is criticality of your dependencies, right? Obviously some dependencies are going to be more critical to your environment than others, perhaps. And Tracy, this one sounds like right in your wheelhouse. Yeah, so like, I think having scales would be nice, but when the fundamental problem is nobody even knows what they're running. So that's the problem we have to solve in the first place. And this was highlighted when log for shell happened and this was a library that had been around for a long, long time. It had been adopted everywhere, but when the vulnerability was announced and people had to figure out whether they had it or not, it was a case of, well, actually we don't know and we don't know how to figure it out. And I think that kind of sparked a lot of the conversations and a lot of the direction. I'd love to get to a place where we could talk about grading open source or grading the dependencies. But I think first we have to tackle that problem of what are we even running in the first place that solves our problem. And then let's build on that to say, okay, now how do we work out which bits to focus on first and expand our effort accordingly? Agreed. Next question. So a lot of talks now with QCon have been on the production side of like S forms and stuff like that. What kind of discussions are you having or seeing around the consumption of them? Like especially between products? Because right now it's mostly people figuring out, as you're saying, what they have in their own stash. But there's very little in regards to like, okay, if it's up on record, I'm assuming that I'm processing it to be the good stuff that's up there. But like, what am I supposed to do with it? Okay, so the question is about S-bomb consumption, which one of you would like to take that one? This is a big answer, always. Any of you want to answer it? Yeah, as before, it helps again that we now have, at least in the US, an executive order and education now from CISA, part of the Department of Homeland Security that is emphasizing S-bombs and trying to develop standard criteria for S-bombs so it can be more consumable. And it's a start. I think S-bombs are really just a start to getting us in the right direction on knowing how to fix things. So when a log for shell pops up that we know where we're using log for J, we know how it impacts us. Because one, it's nice to know what's in a software build materials. It's nothing to know what's vulnerable and how that impacts me or what is that doing. I know that's what Tracy mentioned before too, where we need to know where things are before we can even begin to fix things. But yeah, S-bombs, I mean, we're doing them at CSA. I know I've talked to a lot of people who have kind of bought into what they're doing. I don't have records saying that like, hey, 90% of our application builds have an S-bomb. Does anybody else have statistics like that? Yeah, I don't think people are consuming that. We haven't solved that part. And even the regulation is gonna make people generate them, but we haven't got that second part of the story. And I hate to be the person going, it's all terrible and we need to do better. But we are starting to see a few tools out there which do tackle consumption side, but we're building some ourselves and we're starting to see some in open source, which at the very minimum, we're encouraging people to have policies that say, do you have an S-bomb on your software? And then get that in place and then build on it. Like, can your S-bomb actually tell you anything useful? Does it have enough information to tell you that you're using Log4J somewhere in there? And it's coming, but we still have a lot to do as an industry. And there was a small group here yesterday, actually, who got together and had a conversation about where do we take attestation and start to automate some of these, what software should be leveraged to start thinking about this. Conversations around some key open source components and of course OpenSSF, I keep bringing that up, but has an S-bomb everywhere, say, as well. So there's a number of people working on the problem and the conversation about how do you really make any use of it, provide some level of provenance and attestation for your S-bombs beyond just, okay, here's my catalog or repository of them. Now what do they do next to your point? So there are, the conversations are happening, but it is still very early. I'm gonna add one more thing and move this on. Anyone interested, the OpenSSF's S-bomb everywhere effort is I'm one of the co-leads on that. If you want to join, anyone can join, you don't have to be an OpenSSF member. Just hit me up on online or after or whatever and we can get that ball rolling. But yeah, like everyone's welcome and we love all opinions. All right, next question. The keynote on Wednesday highlighted container images and a large number of vulnerabilities. Secure base images, eventually people load their applications on that, Python and whatever and then you start bringing in hundreds. So the question here is what interesting approaches that you all thought of about actual exploitability of those vulnerabilities in the containers? Because scanners give you lots of information but many of the vulnerabilities identified by them aren't exploitable because they're never called from the entry point of the container image or aren't in the execution path of the application. Interesting information. Yeah, I'd like to tackle that one and I think it's simpler. Like I think the approach we've taken is like why even have that conversation? Let's just get rid of the vulnerabilities. Let's build software that by default. You're starting with zero CVEs. Like today if you build something on the node image you've got 800 CVEs to figure out how exploitable are that? You know, how is someone supposed to go through 800 things to work out what the exploitability is? When the alternative is and like give a shout out to a project called Wolfie which builds base images with zero CVEs and updates them constantly. And we're looking to take that approach off in the same way once upon a time people would ignore warnings around code and now it's like best practice to have zero warnings. Let's go in with a mentality of zero CVEs with your base images. And the Wolfie project is one open source project looking to do that and build base images from scratch with zero CVEs. Can I add too so that global security database the GSD that I mentioned before that's kind of the pathway for that. So if we look at CVEs and how they're issued and what it tells you it doesn't always give you a lot of that of what you're mentioning too. And so having a community driven effort where we can ingest CVEs and people can talk about how that impacts them and proper mitigations and counter measures. And I think that's the way to that's at least the path to get there, right? It's having more people that are working with these CVEs who are fixing them and sharing how that worked for them. And I mean, that's what we're here for, right? Open source communities help solve problems like that. And we just need a path for that. And I think that's what I like about the GSD program. It just piggybacks off a CVE and gives you so much more robust information about fixes and mitigations together. Anything to add, Eric? Yeah, I think just above and beyond that it becomes an internal governance conversation, right? So if you don't use Kubernetes and you're just building an application in a VM or on EC2 or some other platform, as I mentioned earlier in the talk a lot of organizations don't bother to do scans as much as they should now. This is increasing rapidly but they may have things I haven't updated in two years, right? So now it's the I need to constantly be aware of this and once it's in a container, a lot of people think less of it because it's now it's in a golden image. It's something that doesn't change where the reality is. If you're not constantly doing governance on what is in my environment and again if you have S-bombs that tell you all of these containers have this in it, I can go back because they should never update a Kubernetes instance once created. And this is probably a whole another conversation we could argue about. But if you have that image of a Docker image or whatever that lives in Kubernetes, that in and of itself should never change. But again, Kubernetes, if you're gonna auto-update it or do those processes, you need to be very cautious and have a strong governance platform to assure that nothing is changing and what you do have is constantly up to date. And so it's process, not just technology that you have to worry about. I will give a shout out as well. Actually, maybe think of the VEX tool if you haven't come across that. It's a vulnerability exchange format and that's a way of cataloging does this vulnerability actually affect the software? And there's increasingly more tools like VEX CTL, can see the ortho fit in the front row. So shout out to that as a way of using it to say what does this actually affect the software if this vulnerability is there? Does it affect it? Has it not been investigated yet? Or is it unaffected? And yeah, I'll check that out. All right, we're gonna get our five minute alarm here. So we'll go to some closing, but I have one more thing to add. There's a project that's part of FIRST called EPSS. I don't know if you ever heard of it, but the intent of EPSS, it stands for I had to look this up, exploit prediction scoring system, and I don't remember what it stood for. But the idea is can we look at metadata around vulnerabilities to predict the likelihood they will be exploited? Which I think it's a really cool project. It's very early days, but again, it's an open source thing. If anyone wants to get involved, like go hunting down and get involved, it's a cool thing. All right, five minute warning. So closing statements. We'll go in reverse order of our opening statements. So Eric. Yeah, I think a lot of what's happening in the industry is continuously pretty new. It's one of the key components is up until a year or so ago, about 95% of released software proprietary or otherwise was open source. That's dropped to about 90% in the last year. And that's happening primarily because of security issues. People are worried about identifying and mitigating potential security in open source and how it's leveraged. I think in the industry with this move for software supply chain security is more of a subset of this, the importance of S-bombs. But we all look to tooling to solve this problem, but we need to understand that it's very much a process oriented thing as well. It's not just simplistically, what's the right tool? I need a tool that solves my issue. So open SSF with all of the SIGs that I mentioned are looking to solve those problems. And I highly recommend any of you that aren't involved to at least go listen to the recordings for the meetings and read the minutes, they're all on GitHub. And if you can, always looking for more people to contribute. Tracy. Yeah, software supply chain is pretty challenging because you have to solve a lot of different problems at the same time to get it right. And that can be pretty overwhelming. But yeah, there's a lot of work happening in this space now and we encourage folks to just pick one area, get some wins with that, whether maybe it's software signing, achieve some wins and then keep tackling it. And yeah, I just echo the sentiment of there's a lot happening in the open source communities, open SSF and other places. So do get involved because this is like priority number one. Everybody needs to be tackling right now. And if we all do it together, we can all go further, much faster. And John. You guys took my answers. No, no, I'm kidding. But those were great answers actually too. But I think visibility is so important, right? And we need to push for things like S-Bombs. We need to push our partners to have visibility on the platforms that they're using because we're, like I said, we're building, we're buying, we're connecting and we're integrating with all these other tools. So visibility is so important, but community I think is maybe even more important with our current jobs, because that's what we have control over. So coming to the KubeCon, talking to colleagues about things like this, scraping Twitter and Slack channels to find information and to share so that we understand, hey, am I doing this right? Are you having the same problems that I'm having? No, there's no central source yet, but having these channels with colleagues in this room and people on the stage and people facing the same thing that you're doing every day, that's the way that we get answers and tackle these things together. So I think those two things, visibility and community are pretty important. I love it. All right, I wanna thank the panelists. This was an absolute treat. I wanna thank the audience for the lovely questions. You made my job so much easier. That's fantastic. So thank you everyone. This has been awesome.