 All right, everyone, thanks for watching this panel today. We have a bunch of experts on the topic of Sbombs or software bill of materials. Let's start with everyone introducing themselves. Alan, do you want to go first? I was going to say we were going to jump on all at once and then try to squeeze the doorway. I'm Alan Friedman. I'm sometimes referred to as the guy who doesn't shut up about Sbombs. I failed academic who got talked into joining government about six years ago and helped coordinate the international cross-sector effort at NTIA. That's created a shared vision of Sbombs, recently moved over to CISA to help scale and operationalize that while still making sure that Sbombs remains a community-led effort. Awesome. Nisha? Yeah, I also failed. I failed long time ago at manufacturing. I used to be an RF engineer, but now I do DevOps-y stuff concerned with the container images at the open-source technology center at VMware. My focus is, well, it started off in the compliance space and you will, if you squint a little bit, you'll find that a lot of concerns about compliance and security overlap. One of the places that they do overlap is the Sbombs. I originated and now I'm a co-maintenor of TURN, which is a tool that generates Sbombs for container images. What I am aiming to do is try and integrate Sbombs in the container build and distribution ecosystem and I have talks accepted around that space that's at KubeCon later. Hello, I'm Frederick. I focus on quite a few areas in the open-source community. I have prior experience working on storage and networking and containers. I spend a significant portion of my time focusing on where container networking and telecommunications met. I still work with those particular groups on an ongoing basis as well to help define what's called the cloud-native network function. I also work in the zero-trust space quite extensively around these time periods where I help multiple groups and multiple companies try to work out what a zero-trust strategy would look like that is rooted on open-source technologies where much of it is available through Linux Foundation or cloud-native computing foundations. Some of those efforts also include I'm on the Technical Advisory Committee for the Linux Foundation Public Health and I'm also on the Technical Steering Committee for the SPIFI project, which is an open-source standard for how to deliver cryptographic identities to workloads so that they can then use that to gain access to other things without using secrets. Awesome. You said both of the magic buzzwords zero-trust and Sbombs. Everything in the last executive order. I'll introduce myself quickly as well. I am moderating the event. This is going to be shown out. If you're watching this live, you've probably already heard this. If you're watching this later, my name is Dan Lawrence. I'm an engineer and I work on a lot of container, cloud, software, supply chain, security-related things, which is how I got roped into according to this conference and Sbombs in particular. Let's jump in here. Sbombs get compared to a lot of different things. What are your favorite and least favorite metaphors for Sbombs? Let's let Alan start with that one. We've all heard the nutrition label. I'm going to reach the screen here and hopefully at Kubeton, I will have a new Twinkie drink so I can have something of the Twinkies and hold up. I think it's a natural approach. The list of ingredients is imperfect, but there's a very real question which is, hey, would you buy or use or consume something where they couldn't produce this? If someone said, Alan, Twinkie, it's got stuff in it. We know that it has stuff in it, but we hope that someone somewhere is actually keeping a careful track of it. That's really a big part of it. The other analogy I like is, and this is perhaps a little bit ambitious to think about given the role that it plays, is CVE or a vulnerability identifier, which is to say there's nothing magical about a CVE. Having a vulnerability on CVE doesn't mean it's fixed. You still have to do the work to secure your network and secure your software, but by creating this common framework and set of practices around it, and institutions around it, it has helped build a very effective and occasionally efficient ecosystem that allows us to make progress. It's one of these necessary data layers on which we're going to build a lot of cool stuff. An analogy that I've been using a lot with regards to container images is a sandwich and the way that I would talk about what SBOM brings to the table is the difference between a subway sandwich and no shade on subway, but what do you think is the difference in quality between the subway sandwich and a sandwich that was created by a farm to table restaurant? The farm to table restaurant probably has the higher quality sandwich because they actually know where all the ingredients come from, and they have working relationships with the communities that make those ingredients. That's because container images look like sandwiches, that's usually the analogy I use. I thought we were going to go somewhere with sandwiches versus burritos where you can see the insides of one, but not the other or something sticking with the Twinkie theme, but I like the farm to table one too. I have used container images like the sandwich versus smoothie versus parfait versus weird Asian candy. I'm looking forward to the first marketing campaign that advertises our Tiznoles bombs. Handcrafted. No, you're doing it wrong. We'll come to that one later. I don't really have any good analogies in this space, but I do try to bring things down to what are the core principles that people want to have. Part of it is when we talk about where did it come from? What's the quality of it? I think it all comes down to things like trust. Should I trust this thing? How long should I trust it for? And for what period of time? Are there any conditions where I should stop trusting it? The way that we have things today, we're trying to improve on that level of trust to say so that we can reason about it, and not just the human reasoning about it, but to be able to automate that reasoning, to automate much of that trust process. At the end of the day, we have all these technological measures, but it always comes down to why should I trust this thing? Makes sense. One of the first topics I wanted to talk through here is why now. It's 2021, supply chain attacks are all over the news. We've had food ingredient labels forever. We've had software for a long time. Why now? Why is now the right time to do this? Why are all of these attacks happening this year? Any guesses? Well, I was going to say, I think there have been folks that have been focused on trying to maintain software hygiene for a very long time. It just so happens that people really didn't see a need for it because at the time supply chain attacks were not really happening. But now that we have some that are very high profile and everybody knows about it, and I heard it on the marketplace morning report many times now. This is now top of mind and everyone's looking for a solution right now, yesterday. And I think there are a couple of reasons. One is in a rare note of optimism. We've gotten slightly better at software security and so that a determined adversary that actually wants to make some progress actually can't go and attack the front gate. They've got to sort of sneak in and find a way go upstream. And that of course expands the attack service. The other thing to acknowledge is of course, people have been talking about SBOM for a long time. And you know, there are champions out in the world, folks like Josh Corbin and I in the cavalry that have been talking about this. And in fact, there was even an attempt to put this in legislation back in 2014. And part of the challenge if you sort of want to understand the process to change is to understand why that didn't succeed. Part of it was, you know, and Frederick and you should know a lot more about this than I do. People weren't always great about tracking their open source obligations and licensing. And in fact, that's the origin of one of the two big SBOM standards is to track the licensing. But I think there's sort of this confluence events of people saying, okay, we know we're going to have to do something and this is something. So let's move forward. And then the high profile supply chain tax have given both industry push. And of course, your friends in the US government have said, you know, hey, what can we do to really drive behavior and investment changes? Yeah. And I think a lot of it comes down to economics as well. Like, if you're a determined attacker, you want to get an effect, whether that effect is to increase your bank account or to exfiltrate some information as part of a advanced resistance threat, then you have limited resources, even if you have a lot of resources that's still limited. So you have to look at where can I apply these resources. And to Alan's point, we've got in much more savvy at certain at defending against certain types of attacks, because we see them very often. And the supply chain is one of those areas where if you want to attack a hard and target, you don't attack directly, you try to find the the supplier attack the supplier and then indirectly gain access to the to the bigger target. There's also a couple other advantages as well, like we focus heavily on, well, how do we defend against the the company our suppliers from being broken into, but also taking into consideration that it's not just it's not just whether they're broken into you can have something where the supplier is never breached. But you still don't you don't have visibility into oh, I'm running a version of some XML parser that is that is at risk. It was statically compiled in my image scanner won't pick it up. I don't even know it's there. And then suddenly I'm reached. And so the supply chain is not just about defending against a supplier that's breached. It's also about knowing what's in your infrastructure. And one of the other advantages as Frederick mentioned on the on the market side of things is as we think about software creation and delivery, not as sort of a single actor, but as sort of a supply chain that we're moving down. SPOM is has the advantage of being discreet and measurable. You have one or you don't. And of course, there's many shades of nuance beyond that. But one of the things this can get us is it allows us to say, here's some progress that you can tangibly make that you can ask for your supplier that doesn't say, Oh, you need to go through a seven year process on it to show that you have that you've been reading from the DevSecOps Bible, and you've implemented your soul. It's an SPOM is a nice, discreet thing that is in economics terms, it's an efficient signal. If you have a good process, building an SPOM is cheap. If you don't, it's going to be more expensive. And so it allows us to sort of help reward organizations that are developing good software with good processes. So one thing I do want to highlight is the confusion between an SPOM and a package manifest. So most developers confuse the two because they think that a package manifest indicates what's in the software. And that's not true. That's just some information to the package manager to say, these are all the things that this particular component depends on. And I will just pull all of those things. And usually, package managers will say, well, all of these, each of these components have like other dependencies, and they'll pull those. So transitive dependencies are not usually visible in the package manifest. For that, you need an actual, like a full SPOM, because you don't know whether the transitive dependencies that you're pulling are vulnerable or not. And builds are weird things that most people like really do not have any visibility into. So this is, this I think is like some place where developers say, wait a minute, I do have an SPOM, it's right there in the package.json. And then there has to be like some education involved in there. So it's, yeah, part of our, part of our growth over here is to actually understand, like, why the things that we have right now do not meet the requirements that we want. Cool. Yeah. So this is KubeCon, CloudNativeCon. Everybody here is focused on containers, Kubernetes-y things. And Kubernetes, so we're recording this in September. Kubernetes, a few months ago, just shipped their latest release with SPOMs for the very first time. There's a bunch of work that went into making a new SPOM creator for the Kubernetes infrastructure. Do people here see kind of challenges with shipping SPOMs for container native, cloud-native-y things? Is it easier? Or is it harder? How does it compare to some of the other industries looking at adopting SPOMs today? So I'll chime in. Since I am native of the cloud, I'm going to give some of the sort of the contrast about what we need to sort of think about. And then I'll let the people who actually build things chime in on what we're doing. And I think it's important to acknowledge that a lot of the impetus behind software transparency started at the other end of the software world. It was around shipped software, it was around embedded devices, it was around legacy tech, it was around safety, critical world where it's actually really important to know, hey, since my rare industrial control system may not have a vulnerability in itself, that's easily stand for I have to care about what's under the hood. Now, another big part of it is it's assumed that the asset owner or the hospital or whoever is using the software on-prem is empowered to take some action. That once I know what the SPOM is, I can if a patch isn't available, I can tune my IBS, I can segment my network, I can work with my thread until there's a lot of security options there from the customer side. And so we talk about a lot of the benefits from the user side. In a cloud or SaaS world, some of the underlying mechanics are a little different. And this is something that I think the community still needs to explore is to sort of actually nail down what are the broader ecosystem wide use cases across the supply chain. But I also think that there's a huge advantage in the cloud native world in that there's the tools of DevOps allow us to sort of say we're already a little more aware of the importance of knowing what's in our ongoing build process. And so I think that's enabled much faster adoption and people sort of getting the value and hopefully you all out there do too. One thing I do notice is that in the cloud native world, software reuse is really high, much higher than what folks have anticipated for software. Usually it used to be like a small piece of code that was shared by a USB stick. And now it is like everyone has their own open source project posted on GitHub and it's free, everyone can use it. And oh, that's wonderful because that means that there's a diversity of innovation that comes with having open source projects and open code like this. But the downside is that there is also like the dependency chain is really complicated for cloud native stuff like hundreds and hundreds of software components associated with one container image. And it just breaks your brain. So abstractions become really useful to continue to develop. But the abstractions hide away all of those little software components that end up breaking your brain. So the challenge with cloud native is actually trying to organize the S bonds in such a way that they are easily accessible when they're needed and not like all the time. I think part of the challenge that we're running into as well is like when we were doing really early versions of Docker 2013-2014, the applications were generally simple. We explicitly ruled out things like people were going to run things that look like Hadoop or Erlang OTP, which have very specific networking properties. The idea was let's go and try to run something that focuses on small microservices that you would run within some platform as a service similar to Heroku or similar. Part of the, because of that very narrow focus early on, which I don't think was the wrong thing, it helped us succeed. But the thread model was not as extensive as we should have made it. Like we were looking at how do we prevent container breakouts? How do we protect the network? And how do you protect the Docker socket so that people can't send random commands to Docker? So the thread model did not include things like, well, what if my image was modified underneath? Would we notice? And it turns out that was an incredibly important question. And I know for sure that we didn't ask those questions because Docker save and Docker load is like, I wrote those. And so I was heavily involved in the creation of some of these things. It was the first format that was on disk for an image outside of the Docker push to a repo somewhere. And there's mistakes that we, if we had expanded our thread model at the time, these type of mistakes, many of them, not saying that they wouldn't be here, but there's a whole set of things around image signing. And where do the hashes come from? The Docker image hashes, historically, were just randomly generated. They didn't mean anything other than it's just a random number. And so there's a lot of mistakes that were made in the past that were paying for today because of that hyper focused. And like I was saying, it was a trade-off. The teams were focused primarily on things that were relevant at the time. And it was hard to predict that we'd get to the type of success that we see here today. Like we knew it was going to be big, but not this big and this important. So where we're now at a point where we have very complex pieces of software, like some of the amount that it takes to build some of these software would not be surprised if they were more expensive in terms of resources than building a cruise ship. Like how much goes into building Kubernetes itself? It's the second largest developer community in the world in the open source space. So you just look at the total number of resources. We're at a point where we have to fix things now because if we don't fix them, the software is just going to get more complex. The attack services are not going to get any better without significant work. And so this marks an important aspect of reducing the total cost of defending the system. Are you saying that we shouldn't be moving fast and breaking things then? Well, we should move fast and we should break things, but we should be mindful of what we have broken. He brought back all these memories of Docker in 2014. The subway sandwich metaphor is perfect because you wouldn't eat a sandwich from 2014 either. It's software expires too. I like it. Yeah. So let's say somebody does want to start producing Aspons. They're cloud natives. They want to start producing Aspons for their applications. Have you seen people kind of make this transition and start with the journey? Are there some tips you have? I think people should avoid. I should watch out for the right ways to do it. It's actually really easy to generate an Aspon right now. There are so many tools that can help you with it. In fact, Linux foundations automated compliance tooling group has a list of tools that you can use right now to generate an Aspon. What folks usually get upset about is that the Aspon is typically very big. And this is not surprising because that's how many components you have in your deployment. So naturally, it'll be really big. The other thing to note that it's also big because of the amount of metadata that is present in the Aspon. You need all of that metadata when you want to analyze your supply chain. And that's why it's there. So even though it can look very scary in the beginning, all it is is just a giant list. We still do not have tools that would take in the giant list and filter it out to get you whether you have these lists of denied things in the Aspon or whether it is an expired Aspon. All of those analytic tools don't exist right now. Mostly because the way folks have historically looked at this is point a static analyzer at this set of files and the static analyzer will crunch all the data for you. So Aspons have never really come into that picture before. And what we're trying to do is hopefully make sure that we have tools that work on this, work on these Aspons, and analyze it and generate the valuable information that folks need. I agree with Nisha. I think the basics of pulling this together today are there. There's, I love Cyclone DX as well as SPDX. I love them both equally. And both of the communities have done a good job of curating their tooling ecosystem again for both open source tools as well as commercial tools for folks who wanted to create them. We're seeing more and more things integrated into the existing build tools. And in this world, you may want to say, what's the Aspon of my repo, of my source, but really we want to move people to think about creating the Aspon at build. We all know that there's some dark arts in build tools and compilers. And so you're really only yet to get the ground truth of what's in your software from the moment your software is actually built. Where we're still learning is some of the glue that puts the different parts of the build chain together. And this has actually been great to sort of see people say on Twitter, well, how do I plug this into this? Oh, that's a good question. And then a few days later, hey, I solved your problem, check out this, you know, and so there's a great opportunity for people to actually come up with new things. And the last thing I'll say on one thing that's still known hard is software identity is an entity resolution is actually still a tricky issue, especially if you're in a slightly less well-trodden corner of the software ecosystem. If package managers are still sort of a rough hewn log cabin tech for your corner of software, then the challenge of an S where the goal of an S bomb is when you tell me what your software is, I should be able to know what that software is. And in well-defined areas, that's super easy, right? We have well-defined namespaces, but we don't always have that. And that's some of the things that we're going to have to think about moving forward is how do we make sure that we can actually map to the stuff we care about, we can map to the vulnerability databases, we can map to the license databases, things like that. You're telling us that naming is hard? I would say in the history of IT, we've successfully built a truly globally resilient distributed namespace exactly once. And the annual budget of ICANN is about a hundred million dollars. So, you know, we're going to need some help if we want to get to that level. I'm sure they couldn't name all the software they used to operate that. Yeah, and I think one thing to really highlight is what Alan mentioned is explicitly with compilers. Like we look at what the inputs are on the given system, and we say these inputs led to some outputs. And by inputs, we don't necessarily mean like this is file.c, like you actually have to drive down into the contents. And one of the examples I tend to use is if I had something that was called exploit.exe, and I rename it to principle.exe, is it still the same file, is it still the same thing, just with a different name? So names are actually metadata. But when you start looking at the compiler, the compiler represents an action or something that performs transformation. And you also have to know was the compiler in good shape, was it modified in some ways? You have to look at the environment that is there. And you also have to look at the process. And the SBOM does not necessarily tell you what was the process that this thing took in order to become that thing. So there's a variety of additional things we're eventually going to have to look at. The SBOM represents the first major turn of the crank, and it's a huge crank. So like don't get me wrong, that is super important. But it's not the only thing that we don't want to stop at SBOM. We want to take a look at how do we drive this towards so we can actually look at the contents and say this particular section of code, like here's a thought experiment. How many people in the industry have copied from Stack Overflow and pasted that into their source code somewhere, and then that's now shipping in critical infrastructure? Now let's talk about supply chain attacks. What if you had somebody who modified or added something that had a very nuanced bug in there, and now you need to go and update that bug everywhere it's been used. Impossible task today. And so if we can get down to the content and get down to the fingerprinting of that information, then it'll still be a difficult task, but then we at least have a chance of finding some of these things, saying like this specific code led to these CVEs, we're also running that snippet of code. And so we eventually need to find more savvy ways to do that, but that's way down the line. Like just step one, we have to learn how to crawl before we walk, and crawling is what went into our build in the first place through an SPOM? Both Alan and Frederick have touched upon a topic that's near and dear to my heart, which is build systems. One thing about build systems is that actually lot of Linux distro suppliers have kind of got a handle on these issues before, and they're actually making some really good progress in trying to reduce the build seed and make builds more reproducible and changes with packaging and functional builds, where the only thing to the builds, the only thing that represents the builds are the inputs and the outputs, and there are no side effects. So I think it would be awesome if the cloud native community took a look at what distro folks are doing nowadays. There are lots of folks that point to reproduciblebills.org, but there are many other distro tool suppliers that implement this in some ways, in many different ways, and we should actually take inspiration from them. Those are great points, and that ties into my last, sorry, go ahead, Alan. Go ahead, Alan. Go ahead. Last question. Sure. I was going to say that ties in the last question here, which I want to wrap up with. We have thousands of most talented engineers in the world here, NativeCon, KubeCon. If you could wave a magic wand and have them change something about software completely, get creative here, like to improve software supply chain security and Sbombs forever, what would you change about the way we build software? Oh, I can go on this. I often call myself code mom because I'll tell developers to clean their room. If your mom doesn't take care of your code, you take care of your code. It is your responsibility to know, to understand what your dependencies are, whether they're the right dependencies, and what exactly are you installing on your container that you're pretending is your desktop, and what versions of this code are you using? Are you downloading from somebody's GitHub gist or just some random neural that your friend gave you? Yeah, be careful when you code. Have some discipline. Tough question. I think part of what I would encourage is, first, when you write software, always consider where this thing is going to be run. It's always a trade-off because we have to get features out. We have to ship, but at the same time, we also have to make sure that we're doing a good job with what we're doing. There's a wide variety of tools and techniques that can help you do this that are available, and we're constantly improving and iterating on them as a community. I wish that people would be more proactive on some of this stuff to help find what some of these processes and techniques are. I found that people are more than willing to help. If you just hop on a Twitter, literally, and just say, hashtag cloud native, I need help with this, and how would I secure the system, or you tag a couple of key people who are in that space? You'll get a lot of visibility, and you'll very likely get somebody to at least point you in the right direction, but it still feels very ad hoc, very bespoke type of communication. There's not really a group that says, here's a set of patterns of cookie-cutter things that I can apply now that work across 80% of all projects that would help you set up that initial framework. We need to be careful that we don't kill innovation, but there's also basics that apply in most scenarios that would be fantastic for people to apply. I love Frederick's point about where something will be used, because there's such a huge difference in what we need from our software for a fun little web app that's going to help us better serve ads versus things that are in critical infrastructure. We actually need a lot more. My one plea, as I've worked with more and more folks in the cloud native world, is understanding the sheer diversity in the software community. There's a very long tail of folks that even if they have, and many of them do in fact have, the technical chops to pick up something like in Toto, it's fundamentally useless for their environment today, because it's not how the organization builds software, it's not how they ship software, and so we need to sort of think about that role. Things that I'm looking forward to, and this is where I get to have a plug, whereas if you'd like to get involved in the technical side and in the policy side, the global Sbom community needs your help. Looking forward, as I mentioned, there's a lot we still haven't figured out around sort of the SaaS model in general, even before we get to the great cloud native work that all of the KubeCon experts are using, which is, hey, tracking identity behind third-party APIs or microservices or dynamically generated code. What does transparency look like in those domains? These aren't areas where we don't have any answers, but we're going to need to sort of find a way to talk about it in a way that's sort of tech neutral in scales. And the last thing I'm going to add when I'm looking ahead into this is the more I'm in security, the more I realize that it's really the boring problems that are going to be the hardest work, and configuration management for me is sort of the next frontier, because we don't have scalable cross-tech, cross-sector ways of thinking about config management, where users and employers can actually sort of describe what they're doing in a way that folks who are in the compliance world or in the risk management world can actually understand and say, you've done it right. And even just as simple as, hey, you remember when I sold you this thing and we said, don't plug it into the internet? You weren't an idiot. Were you? We need a way to sort of manage that and measure that at scale. And so there's tons of work to be done. Awesome. Well, if you want to talk about config management, you came to the right conference. I'm sure you're going to enjoy the next couple days here at KubeCon.