 As he says, I'm talking about building images for the secure supply chain. I'm a product manager at ChainGuard, and I work on our images product. I think it's fair to say that people, and my people, I mean the public and politicians, are not very happy with the state of security in IT in general. And it's such a good seriously enough that politicians are now actually doing something and writing acts and executive orders to try and encourage some improvement. And you can kind of understand why. This was the sonotype report that came out a few days ago or a week ago or so. And they claim 742% average growth rate in supply chain attacks. I think this is like malicious packages found in package repository. So things like typo squatting attacks and similar. But at any rate, it certainly goes to show there are problems and they're growing and we need to try to address them. My basic thesis here is that things are kind of broken at the minute. We certainly could be doing a lot better than we are at the minute. One of the things that I think is definitely room for improvement is the state of provenance. So this is a sneaker that is clearly a converse knockoff. But at the minute, it's very hard to tell when you have an image where that image came from. Generally people base this on the name of the image because typically the image includes the repository and the registry it came from. So I might trust this image because it looks like it's a Docker official image. Or I might trust this image because the name indicates it came from my own organization's registry. But actually proving that it came from there or was built by the people you think built it is another question. And that's not really answered at all by most people or answerable by most people. And similarly, we can't prove that things haven't been tampered with either in transit or at rest or wherever. There's also a separate point about reproducibility. So even if I have like the Docker file and the source used to create an image typically if I run a Docker build again, I'll end up with a very slightly different image because there'll be things like different timestamps and build IDs and things like that will be different, which means they end up with an image that is bitwise not identical. Scanning is a big problem. So we have these great tools, things like sneak, trivy, gripe and they're really useful and they're very powerful but they give us a lot of output for the majority of images. And the thing is most organisations, they don't lean on what to do with this output. You know, you get like 100 vulnerabilities or 200 vulnerabilities. What are you meant to do with that? I mean, you don't have time to go and investigate each one and even if you do, but next week there's a dozen more. So it's a very difficult situation we're scanning at the moment. And on a similar topic, it's very hard to gauge exposure. So say like a new vulnerability drops tomorrow that looks quite important, we would like to be able to say, hey, you know, I have four containers when in production that potentially expose this vulnerability. But we can't and nobody really can. And that's definitely an area that we hope to move forward in as an industry in the near future. But yes, I'm going to talk about some approaches to trying to fix some of these problems. So a lot of this is what we can do today. There is also a little bit about what's coming in the future and what we're kind of working on. So provenance, this is a Detroit deep dish pizza. So in terms of provenance, you wouldn't want to be fooled into accidentally getting a Chicago deep dish pizza. And the way to look at provenance, or one way to improve provenance is this idea of signing. So we can sign our container images, which lets us prove that we know where they came from and who built them. The leading way to do this at the moment is SigStore. And we got to the stage where SigStore is now used to sign all the Kubernetes official images, which is a great step forward. A couple of nice things about SigStore. One is the signatures are actually stored alongside the images. So you don't need to run a separate server to store your signatures. And it's also something really nice called keyless signing. So I don't know about you, but I hate having a key pair, and I've got to try and keep this private key safe and secure and worry about it, et cetera. So I've got a couple of small demos that will hopefully work. Is that screen still off the side there? Or can you see the whole of that? It's good? Okay. Now you can see the magic. I've got to run the script. I should've run that before. Okay. So I'm going to build a Docker image here. This could go horribly wrong, but it looks like an automated. It does still run the commands. So we're building a simple Docker image. It doesn't really matter too much what's in it. I'm then going to push that to my repository on the Docker hub. Yeah, so it's got a similar image that already exists, so that was quick. I'm now going to try and sign this with Cosign. So as of today, I do have to type Cosign experimentally goes one to turn on the keyless signing. There may be something about that being fixed in the very near future. And then the image that I want to sign isn't like AMOT Cosign colon Detroit. I really want to sign the digest, so that I'm absolutely sure that I'm signing this same image and nobody else is like pushed to the Detroit tag in the meantime. So I wish I had time to go into the details of how this all works, but I'm going to answer yes here. This has actually opened a browser in the background, and it's asking me to authenticate with an OIDC provider. So I didn't give it like a private key to sign with, and so what it wants to do is use my OIDC credential to set up a temporary certificate behind the scenes. So I'm going to log in with GitHub. That looks good. And that's going to push the signature up to Docker hub. There's also some stuff with transparency logs and Wrecker and folks here that went on in the background. Unfortunately, I don't have time to explain all that. And we have Wi-Fi. Did something go wrong? I don't know what went wrong. I can't. It's in the script. So presumably this will fail. I think I just timed out. No, I didn't. I'm not sure what that error was about then. So because I rebuilt this image and it used the cache version, it actually has the same digest as another image that you've already signed. So we got away with this. And what this is saying is I used the GitHub OIDC provider and it was signed by somebody basically with the authority to access to adrain.adrain.mote.com account on GitHub. At that point in time. Oh, that's the next thing. I'm going forward. Yeah, so as far as the image sharing goes, please try and use it where you can. If you have a choice of images, try and use ones that are signed. Try out signing with your own images. One thing I didn't talk about is if you use GitHub actions or similar to build your images, it's really easy to use the keyless signing because it's already set up with the OIDC account. So really it's just a case of running co-sign verify and there's not much else to do and no keys to worry about. Yep, make sure you verify signatures. There's no point in just signing things. We also need to be checking the signatures and the typical way to do that in a Kubernetes cluster is use a policy management tool such as OPPA, or Caverno, or TrainGuard and Force. Also, if you get images from a third party vendor, do ask them if they've signed the images. Try and encourage them to sign the images. It's kind of noise. So this is going back to the problem of hundreds of vulnerabilities being reported. It is possible to cut this noise down drastically and get to the point where your reports are only, if you're finding any vulnerabilities, it's only a handful that you can cope with. And basically the answer is to cut your images down to the very minimum set of dependencies and to keep those dependencies absolutely up to date all the time. It's also in the future, there's something called a vulnerability exploitability exchange that I'll talk about later, and that may also give us a way to filter vulnerabilities through that and cut the noise down even further. So the first way to cut down your dependencies is to use smaller base images. The easiest way that you can probably do with just changing our front line today is to use something like Alpine or Debian Slim. And that can be a good alternative to using something like the full Debian distribution. But you do find there's still quite a lot of stuff in there, even with Alpine and Debian. I mean, Alpine is probably only at 55 megabytes, and I think Debian Slim is about 50. So there's still quite a few potentially vulnerable components in there. The other thing about these distros is updates sometimes take time, because what you have to wait for is upstream to either release a new version with a fix or to release a patch and then the distribution needs to update their package and then you need to wait for the image built from those packages to be updated. So typically, that's going to take a week or two before vulnerabilities are addressed. We can cut things down a lot further if we look at things like Google Container Tools distros images. I'm sure some of you have seen these. These are really small. These images, they contain a bit more than scratch, right? So they do have a few things in them, typically like an attempt directory, like TLS certificate, so you can actually talk to websites. But the Slim, they're cut down to the extent they won't even have a shell or a package manager inside them. So you won't be able to do things like app install and things like that. They're perfect for running things like statically compiled binaries that you can do with like Go or C or Rust. Google also has a couple of images, I think, one for Java, maybe one for Python, so you can run dynamic stuff like that as well. But there are some drawbacks. Typically, the Google Container Tools distlist, this list of is kind of hard to extend, even though it's based on Debian, so it's like a really stripped-down form of Debian. And you would like to think you could like app install stuff, but doing that is actually kind of difficult and you'll end up having to play with Bazel if you want to create your own versions. So at Chain Guard, we basically created our own base images using a very similar technique to GCT distrelist, so we have base images that are completely slimmed down to the extent that they don't have a package manager or shell as well. But we build our images with something called APKO and it's easy for us to extend these images and add in whatever we like. So we have both base images in the same sense as the GCT distrelist ones, but also some application images, so things like nginx, and I get images that I'll show you in a second. All our images also come with S-bombs, but importantly, we continuously are updating these images. So we're pulling and getting the latest sources and recompiling every night and building new images. So hopefully, all vulnerabilities should be addressed as quickly as we can. We basically have two flavours of images at the minute. There's images built from using alpine packages, but there's also images built using packages from our own distribution called Wulfi, which is built against G-Lib C. So now you can get very minimal G-Lib C images as well as Muzzle-Build ones. And I wish I had more time to go into Wulfi, but that'll be for another time. So I wanted to quickly demonstrate sort of what I mean in practice. So if we look at the nginx images in the Docker Hub and I run sneak or trivia gripe against them, the nginx latest image, I think that uses Debian by default, not even Debian Slim. And last time I ran it, this found 143 dependencies, so there's a lot of components in there, and there was 94 problems. Now, admittedly, the vast majority of these things are going to be critical, but the vast majority are going to be negligible and so on. But all the same, it's stuff, vulnerabilities that you've got to look at and think about and deal with, and it'll be much simpler if they simply weren't there. So we can do one better if we look at the alpine image. So this one had 143 dependencies. The alpine image only has 43 dependencies. So there's 100 less components in the alpine image than there is in the Debian image, for whatever reason. It's still found two issues, apparently to do with LibxML library. Finally, I run it against the chain guard image. This is one of our alpine images, but we're using our own build of nginx from the source. In this case, it's actually 46 dependencies. So some things are getting pulled in it shouldn't be. I need to fix that. But there are no vulnerable paths found, and I did look at this, and the reason is purely that we're using a newer version of LibxML for some reason. I'm not actually sure if the alpine one isn't updated yet. Where we do this is using our own tool called APCO to build our images. It's designed to be reproducible, so if you run the same build twice, you should get the same image out. It's declarative. And basically all you do, as you say, is use this apk repo, and I want to install these packages, and that's basically all you do. There's no equivalent to the run in Dockerfile, so you can't run random things to build up the image. The only choice way to get files into the image is to install them via apk, and that's basically what allows us to get to reproducibility and so on. So let's have a demo of that. This works. We've changed to a directory with a git image inside it. So this is the apco file to build the git image. This is the crux of it. We're saying we want to use the alpine packages in this case, and we want to install these packages inside the container, or the image rather. There's also a little bit of metadata, like entry points and work data, and user accounts, but that's really all there is. This is the same file, but using Wolfie base, so like a G-Lib C version. And again, it's almost identical. We're just saying what packages we want to install, but this time we're using Wolfie packages. I've got the small build script. So one thing apco or apco is it does use alpine tooling. So it's much easier for me on my Mac to use a container to run apco than it is to find vnit locally. So what this is going to do is build the alpine version, call it git local, and I put it to something called output.tar. So hopefully this works. It might take a second. And what this is going to do is build the output.tar file containing our image and also create an sbomb. Hopefully this will work. It's a wi-fi problem. The other thing didn't work as well. Okay, I think this might not work. It's not looking good. Okay, I might have to kill that. Interest of time. How do I go back? Oh, it keeps going anyway. So I think hopefully it still files from a previous run. We'll output an sbomb. And that sbomb will have complete details of all the packages inside that image. We can then load that into docker. The docker load command. Or normally what you do is just push it straight up to a container registry. So what I was going to show is that's the file that comes out. You've got an old version there. And I was going to show if you run chasum we get the exact chas for the file. And then if I run it again, which of course won't work again, you should end up with identical output.tar, but a newer timestamp on the file. So idea being shown that it does, that it is reproducible. At the moment this only works with Alpine based images. There's some bug somewhere in the G-Lib C build. That means that it's not always reproducible. So that's why I'm using Alpine build. We will fix the G-Lib C issue. So hopefully you will both be completely reproducible. Stop that. This will work, but I didn't actually build it twice. That's an old file. And that would have been a demo. The other thing I wanted to mention was runability exploitability exchange. I've many people heard of that. I guess we're a cloud native security coin. You put your hand at it. Because of the audience, at least a third of you have heard of it. Basically it's a standard file format currently being discussed. And it basically gives away her vendors to say we're aware that scanners are saying that this product, software or the image of whatever contains CVE XYZ. But actually you don't need to worry about this because it's not exposed in this container or actually it's a false positive and we don't have the runable version of the software or something like that. And the idea is that by cross-referencing like the scan of an ability report and VEX you should be able to further cut down the amount of noise from our scanner reports. The final bit I wanted to talk about is exposure. So what I really hope happens in the future and I certainly think this is possible is that if all our images have complete S-bombs, so at the minute if you use tools at the current level of tooling won't cover the entirety of most images. You quite often find like for example I'm trying not to name names but most of the tools to generate S-bombs at the minute run after the fact and they just interrogate the package manager. And so they can miss things like for example if you look at the redis image in Dr Hub the redis itself is installed by downloading it from the redis website. It's not installed by the package manager. So tools like sift and so on that try and create the S-bomb won't actually see that and so they'll have an incomplete S-bomb that is actually missing the most important package inside there. So that's one thing that needs to improve but really I would like to see why shouldn't the S-bombs push back to averse building the software. So in the cases like a doc of our building image it should really be the S-bombs created at that point and not afterwards. But assuming you do have a complete S-bomb that includes all dependencies and transitive dependencies if we stick all the S-bombs for all our containers in a database and we cross reference that with CVE information then theoretically we should be able to instantly tell our exposure to a CVE. So that's kind of what I hope happens in the future but we'll see if it actually get there. So that demo was a little bit quicker than I expected but to wrap up there's three things I kind of want to talk about one with provenance please look into signing your images if you're not already and verifying signatures on the images you're using. It is a lot easier now than it used to be. Scanner noise the way to defeat scanner noise or get around the problem is to just aggressively reduce the number of dependencies in your images and using chain guard images can be a good way to do that and also to keep your images as updated as possible. To be honest I think that's one of the key things we need to do as an industry is to keep things updated all the time. I know there's a reluctance especially at larger organisations to update sometimes because you update and things break but if you don't update you know you get hacked so there's not really a very good choice there. And finally we're guarding like figuring out our exposure where we're not really there yet but hopefully the future does lie in stuff like S-bombs and VEX. Okay thank you very much I think we have some time for questions. Hey is there any products right now to use VEX or that's totally in the future right now? Okay I'll let you take the mic in a second but at chain guard I was hoping to do a demo of someone called VEX CTL that we're working on but it's not really there yet basically the idea is you run VEX CTL and you'll have a VEX document alongside the container and then you can filter down the results but I'm not really aware of any too many tools but you're there's an old wasp the old wasp tools, the pencil track apparently is using VEX but I'm sure there's a lot of discussion and stuff going on with VEX There's another question Is there a competitive reason if you're going to enforce that everything's coming from a certain repo that you're confident on the access controls to also sign and require things that are signed? Yeah There is so I guess one thing would be you still got to account for the time between an image being downloaded and run so if somebody could tamper with it in between for example I think you can probably say more complicated things with signatures one thing I didn't talk about was attestations so you can start adding attestations to say like the tests we've run against this image or this image is suitable for running production or running on stage and things like that so you can get a bit more complicated So any other questions? Is it possible to reproduce APK art when the Alfa Linux upstream removes old version of APK packages so when you run APK art twice with can you reproduce exact same image? So I didn't quite catch that so when you run APK art twice between the first run and the second run the Alfa packages versions might be different so is it possible to reproduce the packages with all the versions? So the point is in that APK file I was just specifying packages I wasn't specifying exact versions of packages One thing we're looking at is basically some form of pin-in so you can pin to an exact version of a package So at the moment we don't have a good story there you're quite right, if it was to happen that a package was updated you'd end up with a different image but in the future we will have pin-in I'm writing very similar to so I'll open a digital issue right there Thanks Anything else? Okay Well thank you all