 Well, hi folks. How's it going? My name is Alex Goodman. I'm a senior software engineer at Encore. And I'm here to talk about problems that you might find in containers and how they get hidden. So in this image here, we have all the layers for this image. And the layer that's selected right now is most representative of the application payload. It's 13 megs in size. But the whole image is 500 megs. And so in another example here, so the application payload is about 230 megs, but the image is 3.2 gigs. The idea is that you're shipping a lot more than your application, especially now that we bought into container technologies. There's a lot more service area that you need to defend, and there's a lot more places that stuff can hide. So what are the kinds of things that can get hidden? Well, you have the typical secrets, et cetera, as well as unintended assets such as like you have development payloads. Or any blobs that may be sitting around. Or known vulnerabilities for your OS packages or any of the language packages that you're bringing in. So what are the common ways that the stuff gets hidden? So these are the top three ways that you'll find most of the time is folks in their Docker files going off and removing something with a remove command, but it doesn't actually remove anything. It's just adding a wide out file. And so if you were to push that image, you still have the payload as being pushed up to your registry. So not good. Also, if you're just in bulk fashion adding everything that you can in your Docker context, and you don't have the proper Docker ignore file in place, you may be adding something that you just didn't intend whatsoever. And the last one, maybe you're just installing packages, and this is probably everyone. So when you install a package, you may have a vast set of dependencies that are also coming along for the ride. So are you up to date? Do you have all the patches, et cetera? So how do we surface these hidden problems? Well, the best way is to try to describe what is in your image. So the best way to do this is to generate nest bomb. And then take that description and analyze it for either known problems or known expectations, or at least an audit trail left behind for incident response. So SIFT is a tool used to generate S-bombs. It generates S-bombs in SPDX 2.2, Cyclone DX 1.2. It knows how to generate S-bombs for container images, for file systems. It knows how to catalog a wide variety of ecosystems and also beyond just packages. And it outputs in both a table summary as well as rich JSON output. So it'll go on forever in that regard. Yeah, so when it comes to looking for problems, we have gripe. So gripe is a lightweight vulnerability scanner. It knows how to look for vulnerabilities for a list of packages, whether that comes from a container image, a file system, or if you have an S-bomb, which is really useful because you don't need to bring all those bytes with you, the entire image that you're scanning. You really just need the S-bomb, and that's a lot faster to scan. And this supports several different ecosystems for matching vulnerabilities. So yeah, so now we have these two tools. We want to tie this together into a workflow. So the idea is that you build your image and you generate your S-bomb. Then you use that S-bomb to run quality gates against it. So vulnerabilities, secrets, et cetera. Whatever it is that is important to your organization. And then after all that's done, you publish your image and your S-bomb. A lot of people couple step one and three together. So they build and push their image, and then they run tests against it, which is something you don't want to do, because if you have something like secrets, then you're going to have to spend all day scrubbing that from your registry. So what we're going to do is use Canico and SIFT for building an image and generating an S-bomb, gripe and a little bit of scripting for quality gates and Scopio and Cosign for publishing our image and our S-bomb. And we're going to tie this all together using Tecton, Kubernetes native CI solution. So yeah, all right, it's demo time. All right, so we have an application called CountGuber. It is an application, you know, given a number in a sentence, it knows how to extract it and replace it with other numbers. It's very useful. So now we have this very useful application. We want to build and, you know, validate the image. So here we have a Tecton pipeline made up of a set of tasks here for building and validating this image. And so I've already got this loaded up locally, and already run in a Tecton run. Sorry, in a Tecton pipeline run. So here's what this pipeline looks like. We're fetching our repo, we're prepping some assets, and we're building our image. When we build this image with Kanako, it stays local. We're not pushing it anywhere. Then with SIFT, we are generating our S-bomb. And this S-bomb, we get an S-bomb.JSON output at the very end here, which has all the information that we need. And we show a nice summary to show, yep, it looks like our CountGuber app is indeed there, version 010. And once we get to the quality section of our pipeline, like we have a vulnerability scan, and we're using the S-bomb as input, and we will fail if there is a severity that is of higher or greater. And it looks like that's what's happened, so the quality is failed here. And yeah, it looks like NLTK has a few high severity vulnerabilities that we need to remediate. So, okay, that's one problem. It looks like there's a secret here too, and notice that we're using our S-bomb as input. And if I look at the details, the assets config ENV has got a generic API key. And we don't actually have the key value here in the logs, because again, we don't want to have to scrub stuff, so no values are here. Okay, so let's go and start remediating stuff. So, in our application, we said NLTK needs to get updated. So it looks like far to head down to, yeah, it's at three, four, four, so... All right, so we're updating NLTK, and our other problem was we had a secret laying around. Assets config, yep, we have an API key, and don't worry, this is a fake one. You can take as many pictures as you want. And yeah, so let's go to what the culprit of this is. It looks like when we're prepping assets, we have this config ENV that's capturing all of our environment variables. We probably shouldn't be doing that. And that's not in our get repo, that's just in the pipeline. So, I commit, and I push this. So now I've remediated all my problems, I think. So I head back to our pipeline. I kick off another run, and just for time, I've already done that run. So yeah, green all around. Looks like we built our image, generated our S-bomb. Our S-bomb shows that NLTK has been updated indeed. Pass our vulnerability scan, and we have passed our secrets call to gate, and we've published our image, so it's been pushed to our registry. And we also have published out the S-bomb. It's also sitting in our registry. So, and I've already pulled down this image, but it looks like indeed it is working A-OK. And if I wanted to check out the S-bomb, I can always use cosine to pull down and take a look at specifically what the S-bomb contents are. So in this case, lots of JSON. So yeah, that's all I have. Let's see. Thanks. Yeah, if you're coming to find me, if you want to talk about SIFT, about GRIP, I'm happy to talk about anything whatsoever. I will be at Tom's watch bar tonight for Anchor hosted Happy Hour. Thank you.