 Hello, everyone, and welcome. We are so excited to have you join us today to talk about software supply chain security. I'm Emily Fox. I'm one of the co-chairs of the Special Interest Group for Security and the CNCF. With me today, we have Jonathan Meadows of Citi and Andres Vega from VMware. So why is cloud-native supply chain a problem? As we've seen with more and more organizations moving to the cloud, they're adopting more and more cloud-native projects to make that migration much smoother and easier, and in many cases, they are re-architecting from the ground up for the cloud, meaning that they are buying into cloud-native projects early and designing around them through modern software development practices. And with those cloud-native products and products coming more and more prevalent, they're producing software with a lot of these capabilities and with more cloud-native products being developed and produced, they're being used in deploying items to the cloud. So it's a very circular supply chain that we have just as with traditional supply chains. And supply chains in real life are complex. And in the cloud and in software development, they are just as complex and in some cases, much harder to track and navigate because they are intangible. There's no single product that exists that solves any organization's supply chain problems. Supply chain security is complex. There's all sorts of different layers, whether or not you're a producer or a supplier, or you're creating a raw material that you don't even know about. And there's a large gap in the existing security space for cloud-native producers and consumers. Nobody really knows what to do because they'd like to have that single product to solve all of their problems. So realistically, what is supply chain security about? Why are we talking about this today? Supply chain security is about defense in depth and it's not only about the layers of different security components and configurations and checks and verifications that you're doing on your images and on your signatures and on your hashes. It's also about the things that you are doing layered on top of each other, commensurate with the assurance requirements that you have for whatever it is that you're producing or creating, as well as the risk tolerance of your organization or your project itself. So why are supply chain security specifically in cloud-native? Well, we wanted to create some form of documentation or instruction for the community to be able to understand more about why supply chain security is so important in cloud-native ecosystems. As I talked about previously, everybody's going to the cloud and cloud-native is becoming more and more prevalent. But we can't just provide the community with a list of tools and say, here, go check this out. We want to do more than that. We wanted to provide them with recommendations specific to their risk tolerance, whether or not they have low risk acceptability, meaning that their environment is so important or whether or not they have a software assurance requirement. If they're just writing something to play with it or create a pilot or just learn a new capability, then they don't have the same level of assurance requirements as government or industry is going to have or even healthcare. We want to also provide an explanation as to why. We don't want to create a list of things for people to read through and try to understand why we're talking about the things that we are. We want to provide them with reasonable sound explanations from the community who have been doing supply chain security for as long as they care to admit or have recently gotten into the habit of instituting these tools and practices and have valuable lessons learned about why we should be doing things the way that we're recommending. And of course, we want to provide the list of tools that can help along this path. Like I said before, it's not just one. It's many of these things and many different layers of applying tools and configurations and techniques to provide that end to end level security of your supply chain, whether or not you're creating raw materials, you're a software producer or you're a software consumer. And I'm going to hand it over to Jonathan. Thanks, Jonathan. Thanks, Emily. So working with the supply chain working group, we've written a document detailing recommendations for how to secure your supply chain. I think it's really important as a document to show how we can leverage these cloud native technologies to start addressing some of the issues that we see at the moment. So if this is of interest, I definitely recommend coming along to the SIG security or the supply chain working groups and contributing. Now the white paper itself is written and split up into five main sections. And it really covers supply chain from end to end from where the developers start to write the code right through to the deployment and validation of that code. And that's really important because the supply chain really is only as secure as the weakest link within it. Now whilst writing the document, it became clear that we also needed to look at two different levels of security. One for moderate use cases, but also higher level use cases, such as where we're looking at a critical build component or working with a regulated industry, for example. So we have two different levels of recommendations within the paper. Now in terms of recommendations themselves, we really needed to start with the basics. So you'll start to see the starting with the source code itself. And this is really to ensure that we focus on who is contributing code to the source code repository. And the way of doing this is pretty basic. We just look to ensure that we have appropriate authentication and authorization on that repository. And all of the mainstream vendors provide core capabilities in the area. We just need to ensure we use them and configure that appropriately. But to take it a step further, we also looked at his working group and have suggested using MFA for access to those repositories as well. And additionally, for higher levels of security, we recommend ensuring that we sign those code commits that we put into the repository again using MFA. And this will mean that any time that developer credentials are compromised, an attacker can't contribute code without the MFA device providing that extra level of security there. So we move on from the source code, which is pretty basic, but then go on to the dependencies, which is a really important area for the supply chain. And if you think about it, it's similar to manufacturing any other product where you have raw materials entering the manufacturing process, and they check to make sure that they're of the right quality and you understand the provenance of those raw materials before using them. And it's just the same for software as well. We need to do the same. So at the very least, we need to scan those dependencies as we're bringing them into the supply chain, and that's using tools such as software composition analysis tools. Now, in some situations, that's all you really can do if you're loading in binaries and such, but we're really starting to recommend taking it a step further and beyond those heuristics and starting to look at getting a much better understanding of what's in the dependencies you're bringing in. And this is leveraging technologies such as software builds of materials or S-bombs and getting the dependencies or whoever is creating that software to provide that data to you as you're ingesting that software. And this really gives the end user a much more detailed understanding of what's within that product and what's been ingested into the supply chain. Now, there's a ton of work going on at the moment to evangelize S-bombs and to assist with supply chain security, and much of that work has been led by Alan Friedman from the NTIA. They're doing a lot of good work to evangelize the usage throughout the industry and have a number of proof of concepts running, particularly with the energy sector at the moment. But also, in addition to the S-bombs, we need to ensure we trust where the S-bomb is coming from and that it actually is associated with the software we're bringing in. That's a little bit of a gap we have at the moment or perceived gap, but Chris Blask and his colleagues are looking on a D-bomb initiative, Digital Bill of Materials, which is looking to create a secure channel that provides access to those S-bombs. They've put some early work together on that and it's definitely of interest to take a look on and follow through. So in addition to understanding those dependencies, it's also really important to understand the operational health of that dependency as well as the contents. And this really relates to the fact that CVEs are a trailing indicator of security. It shows that someone's identified a vulnerability, but it isn't in and of itself a guide to the security posture of that dependency. And to mitigate this issue, one of the recommendations we're looking at is really having a view of the operational security of the dependency as well as a CVE. And to do this, we could potentially look at a number of committers or the number of committers to that project or perhaps how long ago those contributions were made and really assessing that information before you ingest that data into your software pipeline. Now the Open Source Security Foundation are doing some great work in this area with the security scorecard work. And that's, again, a great project to follow and something that we'll be adding in as one of the recommendations within the paper. So still on with dependencies. If possible, we also need to ensure that the libraries really came from where we think they came from and that really requires us to validate the signatures on these dependencies or signatures on metadata that's provided as part of that build process when they were built using functions such as Notary or Intoto that we'll be describing a bit later on. And finally for truly high security dependencies, the recommendation is to build those dependencies from source and this is really to mitigate against potential that what is provided in the package managers isn't directly connected or has additional software in it compared to the source code. Now building from source is not particularly scalable. That's gonna provide or require a lot of additional focus to do that but it's very important to do so if you're really going for that high level of maturity and high level of security. So this brings us to the pipeline itself and essentially the way we look at it is a factory that takes all of the source code and all the dependent components, brings them all together and builds the end product. This is often a key area of compromise in existing legacy pipelines because historically they've been poorly configured or set up, poorly secured but it's a really easy place to attack and often it's a place where we have a lot of credentials for attacker to then take and pivot and move elsewhere. So there's a lot of focus on this within the working group and the approach we're focusing on really leverages a lot of the work from the Department of Defense Platform One team where they've really focused on providing a software factory and they've put out an excellent paper, the DevSecOps reference paper that is definitely recommended viewing. But that software factory, what it really describes is the ability to create multiple immutable and potentially ephemeral pipelines and each pipeline is tasked with building a specific type of artifact and that can be anything from a single product or a library or anything in between. Now the factory itself is built up by joining together multiple hardened build steps each one focusing on performing a single task and that really helps us reduce the attack surface. So there's much less ability to break into it. And each one of those tasks would effectively provide a single action whether that's bringing in the source code, scanning it, testing it or doing the actual build itself. Now from an implementation standpoint each one of those build steps is implemented within an image or started up as a container. Each one of them then connected to form that pipeline and that ensures that you have a very small attack surface built out of dedicated hardened components. Now the other point to think about there is those images and ensuring those images are hardened. So you have to build out a pipeline just to create those images and ensure that you have a high level of security for those images themselves. So put them through security tests, harden them, remove extraneous software and such and really treat them with a strong security ethic from the start. Now that details how the pipeline itself is constructed but we need to also take it a step further and ensure that the pipeline wasn't tampered with during the running of that build process and that's where we're ensuring or recommending we have a really high level of automation within the pipeline. We shouldn't have human intervention within it. It really does need to be immutable and that removes any issues we may have with people maintaining that pipeline and causing accidental security issues. But we also need to look at the pipeline in how we're operating it and that's where the recommendations we're focusing on around. We're making sure that there's no network access in or out of that pipeline except from to pull in the dependencies and publish that end artifact. And also we're looking at how we can monitor the build process of the pipeline itself. Now not every stage within that build is created equally. There's some parts of that stage, some start parts of that pipeline that are more important than others such as the build stage and that's where we're really looking at behavior analysis to really dig down and ensure that that build stage is doing a build and only a build. And this is where we're starting to look and recommend technologies around the PPF and such to really examine what's happening within that build. Now an additional area we're really looking into around the build itself is for high level or high security applications around reproducible builds. And this is an approach that focuses on ensuring that the build itself when it's produced is done in a highly reproducible manner. And this is to allow us to do a bit by bit comparison if we replicate that build somewhere else to ensure that the builds were exactly the same. And by using that approach we can have multiple different software factories creating the same software artifact. Then we validate it's the same software artifact before publishing and that proves that we haven't had any issues or compromise of the build platform itself unless potentially if someone's managed to compromise one of the tools within it. So to exploit that obviously an attacker would have to compromise all of the different build systems potentially three different build systems not just one. Now that provides a really high level of assurance that the build itself hasn't been compromised but it's not easy. Builds are often by just the way that they're set up non-deterministic due to timestamps being added to output files or differences in the ordering of dependencies within the file system. The Debian team have made a lot of progress in this area. They're really sort of leading the charge there and they've got some good tooling around something called the DIFFERscope that shows you the differences between different files and tries to highlight the issues that's making that build non-reproducible at that point. So that's an area we're looking at focusing on from a recommendation standpoint. They have a lot of work at the moment on the reproduciblebuilds.org site that we're looking at pointing to. Now when we have the build complete we need to sign that final build for deployment and this is where we have a number of CNCF projects already available to help us. Items such as InToto and Notary. Now these tools allow us, InToto for example, to sign the metadata from the build process which details the exact inputs and outputs from each stage and then chains that metadata together so we have real software provenance throughout that build pipeline and an understanding of what happened as you built that product. And as you deploy that artifact or even at runtime you then have the ability to validate the provenance of that artifact before you make those decisions to deploy it appropriately. Now also at that point just as you're validating the metadata you're then in a situation where you could sign that final software artifact using something that supports the tough protocol such as Notary and then publish it to anyone downstream. Now although we haven't really looked in detail at the architecture yet that supports these recommendations that is really one of the next steps we're looking at, we have started to look at how we can generate the key material for InToto and Notary so that we have a really small attack surface and a small window for compromise. And this is where we're starting to look at technologies such as Spiffy Inspire at ways of generating the appropriate key material so that we can use them to sign those InToto signatures with a very, very small time to live, potentially leveraging the S-Vids to sign the InToto metadata at that point and then finally using Notary to sign that end product before we distribute the artifact externally. So that wraps up the five main areas from source code right through to signing that output artifact and the metadata. There's still work to do to do community review and we're finalizing that at the moment and then we'll be looking at architectures to support that. But before I finish, we can hand over to Anders and he can provide some of the gaps perhaps we've identified in that process and provide a further explanation of the next steps for the document. Over to you Anders. Thank you John. Between the five main areas you've covered there are existing gaps identified that still require problem solving. While the software factory analogy is a helpful model to reason about, it is important to recognize supply chains can be complex or simple but do extend beyond the software factory and often multiple factories. For this reason some of the unresolved challenges include collection and storage of metadata about artifacts. This valuable metadata can in turn be leveraged to make policy decisions about artifact usage such as binary authorization. Second, another challenge becomes how to pass along software bill of materials across environments. Third, validation of signed artifacts backed by transparency law technologies. A solution that can potentially help here is a new product from the Linux Foundation that provides transparency laws. Sixth store can act as a source of provenance and integrity. While sixth store is a public service anyone can stand up and host the transparency law project known by the name of Rackport. So let's talk about what's coming up for this group of folks who have produced the work that we presented here today. As Emily Fox previously stated our reason for coming together was to create a set of resources to give the community direction and reference around how to secure supply chains. We have captured through a collaborative knowledge production experience the lessons learned from leading practitioners and turned those into recommendations and risk profiles. From there on we want to turn all of this information into something slightly more actionable that being a reference architecture that will ultimately serve as a framework of common tools and templates. And from it derive a reference implementation. You can take the templates and code examples and modify and customize those to meet your own requirements of your own organization. Now, once again, this has been the result of a collaborative effort. And we would love to call for action and get your participation into the work group. Best way to get involved is to chime in on the GitHub issue that tracks the secure supply chain working group under the six security repository. Additionally, we have set up a Slack channel on the CNSCF Slack workspace where most of the discussion has been taking place. Supply chains, as I've said, are complex or could be simple, but there's certainly a lot of moving parts. And we want to elicit thought from you. We want to see that light bulb turned on in your head and have you asked questions about how are you going about it today? Are there ways that you could improve that potentially from what we have contemplated and described as part of this working group? Are there things that you're doing that could potentially benefit others that you'd be willing to share? We have set up this arena precisely with that purpose for a knowledge exchange so we can all together safeguard applications and infrastructure from cyber attacks. I hope you have found this useful, educational and informational. We'd love to hear from you. Feel free to contact us directly through GitHub or Slack as well as once again, we'd love you to be part of the group. Thank you very much. We hope you enjoy the rest of the event. Thank you. Thank you.