 Hey guys, thanks for coming Shouldn't say guys. I'm trying to get out of that habit Y'all thanks for coming. My name is Paul Noverice. I'm a solutions architect with a company called Angkor if you haven't heard of us we Sponsor a couple of open-source projects all around container image security project called Angkor engine that's been around for about five years that Scans container images does vulnerability checks policy compliance checks and now we have a couple of new projects That it will be talking about today The title of this talk is like 600 miles long, which I I don't know how I got it I wasn't really paying attention when I submitted it. I thought I was going to go back and edit that for brevity But I didn't anyway I'm not gonna if you know if you want to find me on Twitter The slides are linked on my Twitter there in my we get hub repo, which is linked from there And if you want to know more about me personally, you know, most people don't care So I'm not gonna spend a lot of time on that Today we'll talk about I've got a lot of slides. I usually have no slides. I'm usually not a big guy with slides But we're gonna keep it moving We're gonna talk about types of Viva tax and why supply chain attacks specifically are a little bit different than what Most people are used to we'll talk about container image Getting visibility into those images why that's important. We'll talk about the S bombs themselves what they are You know what we can do with them and then finally we'll wrap it up with like a couple of Examples of you know how to do some of this stuff in practice So attack types I'm gonna start here with I'm not going to make analogies. I've been trying to get out of the habit of analogy So instead I'm gonna tell a little story this guy Douglas Hague was a Commander in the British Army. He was a Eventually a field marshal during World War one, but he had had a long career before World War one as a Calvary commander So going into World War one he became the commander of the British expeditionary force And all he had known his entire career was horses, right? So he goes into World War one thinking We're gonna, you know destroy the Germans with horses, right? And if you know anything about World War one, you know, he had a bad time, right? Basically barbed wire was kind of new machine guns Horses basically had zero effectiveness Hey Hague was pretty much responsible for hundreds of thousands of casualties at the Psalm Anyway, we'll come back to him a little bit later though Today I'm gonna there's a million ways you can kind of describe Different attacks today. I'm gonna kind of use this these two axes left to right here is this kind of Specific to mass scale. So what what I'm doing here is on the far right It's like one attack has one victim whereas on the left one attack has a multiple victims and then So at the top these deliberate attacks and at the bottom Opportunistic attacks where you you don't really know in advance who you're going to victimize essentially So what you'll see here these types of attacks we've got, you know The top left are these supply chain attacks, right at the bottom left These are kind of more traditional data breaches the top right These are kind of you know state-sponsored attacks or if you've seen war games like a David Leitman style attack Where you know a guy just kind of does a lot of research and does this homework and Very long drawn out attacks and then at the bottom, you know ransomware or like that I like the Twitter example You know like the biggest hack in in history almost and the guy like just used the time to beg for Bitcoin But these are the kind of things we'll talk about Supply chain attacks. You'll see I've got a ton of them kind of up it crammed up in that corner They're all recent to all of these are from this year except SolarWinds. I think was late last year. I guess it's But they're they're basically happening, you know every week now And the funny thing is Ken Thompson. I don't know if everybody knows Ken Thompson. He was one of the designers of Unix He also worked on go grep Won the Turing Award He predicted this back in the 80s. He wrote a paper Kind of showing some examples of how supply chain attacks could somebody could hijack a compiler or whatever, right? So these ideas have been around for a while. It's just all of a sudden They're becoming very popular one of the reasons is All of this stuff in the red circle is be is essentially relatively cheaper than the stuff outside of it, right? so You know the state-sponsored attacks that are getting more and more expensive as You know people's security becomes tighter as people just become more aware practices are better so you see more and more ransomware and supply chain attacks just because You know that's where the the bang for the buck is another thing about this circle is That this is where the people that say, you know, it won't happen to me They'll say things like, you know, my business is too boring, you know hackers or don't even know about me I'm under the radar. Nobody cares about it. Well, they don't care about you, but they're you're still going to get victimized, right? They're not on a crusade against you They don't care about you you're right about that, but they're still going to get you and we'll see why All right, this one the the iceberg I Hate this one. This is another analogy. It's actually a metaphor I guess but It's wrong, right? you know, everybody uses this metaphor where your code is at the top and All of the code you depend on is on the bottom that idea is essentially right, but of course Whoops wrong way Icebergs don't float like that. They float Like this over here on the side Whoops. I thought that was the laser button. So if you draw this there's a link to this website by the way If you draw the iceberg like this, it's gonna roll over onto its side, right? So really the iceberg is this funnel Right and all of this stuff is just getting crammed into your project, right? You and you think you're just kind of consuming these three items. You're actually consuming millions of different Pieces and if one of them, you know way back here gets compromised It just works its way through the chain and then all of a sudden You've got a problem, right? Now the opposite of this This reverse funnel from the attacker's perspective. It's the opposite, right? They just need to compromise one thing and all of a sudden all the dominoes fall and they've got, you know Basically traps set everywhere, right? So you can see why this is really attractive on a bang for your buck basis Especially for small-time kind of operators, right? They can really have a huge Amount of chaos cost for a minimal amount of effort Okay, so now we'll talk about Container images specifically and knowing what's in these images, right? So Yeah, RIP today on here, you know, we lost him a few months ago But this I think about this quote all the time, right? I mean amateurs think about vulnerabilities Professionals think about vectors, right? So vulnerabilities is more of like a tactical way of approaching things Whereas thinking about vectors is a lot more strategic, right? So beyond CVEs, right? That's what a lot of people think of when they talk about scanning they think about CVEs There's a lot of other things we can look for though, you know, there's you know malware types of you know Instrumented code or things like that. There's software overrides are really becoming Much more of a problem things like typo squatting dependency confusion We'll look at some of this stuff and then credentials, right? People leave AWS access keys in their image in their repos they end up in an image you publish the image and all of a sudden You've got another kind of problem, right? So all of these things are you know, see normal CVE scans won't catch any of that stuff So it's other stuff. We got to look for as well, right? So in the end the problem is a lot of people think of container images as this big opaque object And you cannot really tell what's in there, but in reality the container image format is pretty well understood It's very well documented container building is we'll look at all this in a minute, and it's really quite inspectable What we're going to look for in these images be things like we mentioned, you know typo squatting Dependency confusion things that will cause you to get something in the image other than what you expected, right? So in this example here, they just kind of misspelled a package name and all of a sudden you get some kind of compromise stuff in your images Rather than what you expected What could that be crypto miners, you know is a big one big problem I mean again, this is a perfect example of those that attackers don't care who you are all they want is your Electricity essentially right your hardware and your electricity, right? So these you know we've seen these get into things like kube flow, you know had a problem with this a while back other things right typo squatting this is a lot of the public registries and our registries are doing a pretty good job of this but the idea is Attacker will get an image into a public registry named like engine z instead of engine x, right? I mean that's the like the simplest form of this where you just type something you fat finger it and all of a sudden you get Something you didn't expect a little more insidious form of this is you know Registry or repository poisoning. This is where you actually get a compromise artifact into a registry in a Position where you're basically bumping something else that's known, right? So this isn't a typo squatting problem this is like an imposter problem where the image is actually named engine x and Either you fool the build process into pulling from the wrong registry or you've just broken into the registry one way or the other, right? All of these Dan Lawrence had a really good talk yesterday that covered a lot of this stuff about how to trust these Sources and vet those sources with signatures and things like that But essentially the the real the real clue or the real cure for this problem is just that visibility, right? So this is a mission patch from a pretty infamous US intelligence community spy satellite that was launched like right after 9-eleven I think and Basically their idea was they were going to suck up all of the global telecommunications, right? They basically wanted to have total control over Communication so they could see anything right so that's what we want to get over our container images essentially is once we have that visibility Right, then we can you know really get a handle on some of this stuff so What's an S-bomb then right a software bill of materials and how is that going to actually help us with this type of problem, right? So the problem is you can't protect anything if you don't know what's in it Right people don't know what's in their images people don't know what they're consuming You know, that's why supply chain the tax work, right? That's the only reason they work So the S-bomb is really an approach To solving some of this right It is I hear it a lot as the ingredients list, but it's not just the ingredients list, right? The ingredients list is just a piece of it It's really much more like the nutrition facts, right? You get a lot more detail about what those ingredients actually, you know mean for your caloric intake your nutritional intake Right, it's way beyond. I mean just in the bear ingredients list doesn't really tell you a lot about percentages or any of that stuff So there's really a lot more information than just what the contents are Now software bill of materials can be built from basically anything right we're talking about containers today, but you know Just a bear file system A lot of people do it from source code You know it could be a lot of different things, but container images do have a lot of really unique Aspects that make an S-bomb very well suited right so You know if you build containers properly they're declarative essentially right so all of the contents dependencies are explicitly or can be explicitly described right at build time The image itself is inspectable right a lot of people think of it as kind of a black box But it's not it's really just a bunch of JSON and tar files right so we can open it up and see what's in it Containers are in a sense immutable if you change an image if you update it rebuild it the digest changes So we have kind of a real quick check To see if an image has changed since the last time we looked at it and inspected it So with that digest we can tie an S-bomb to a particular digest And say okay, this is known good for this particular digest so we can lock those two together So that gives us a real good checkpoint for these Containers also I mean the containers themselves tend to be I mean This is not always the case right but tend to be short-lived Right, so that helps a lot too because the S-bomb if it's tied to the image digest It becomes less and less valuable the longer the container lives right as things drift There are things there are other projects in the works. I guess around Generating an S-bomb from a running container that you could diff against the S-bomb from the image that it was created from It's kind of look at what's drifting over time is somebody going into the container and updating packages rather than just rebuilding the Image is somebody installing new stuff things like that, right? Okay, there's a couple of use cases here. I mean a lot of things are around like security Incidents like doing some forensics, you know trying to make sure that what was in the container was what you expected or Doing these kind of cross-order compliance checks right license checks things like that And then application complexity like is is there just what we need in this container? Or is there a bunch of extra stuff that's unnecessary in here? So S-bombs can help with all of those. There's a lot of documentation on that S-bomb formats, this is kind of a Wild West right now although SPDX is kind of emerging all of a sudden it's gotten a few kind of nods as Probably the standard. I don't know if it'll you know, I don't know if that's completely settled But there's SPDX there's Cyclone DX. There's this other one called SW ID, which I don't really know much about If you're interested, there's a couple of slides worth of stuff here about you know where they came from what their Original use cases were etc. It's pretty dry Yeah, yeah, I think it's not I don't see a lot of activity around it I mean a lot of people I talked to are either using Cyclone DX now or are looking at pretty Seriously looking at SPDX those are the two that I see like as a practitioner What people are already using today or what they're using currently? So yeah, yeah, I don't SW ID came on here because in the SIFT project We were thinking about supporting it. I think but it's not a format that we support anymore Mm-hmm Yeah, yeah, that's a good point for anybody. It's watching remotely It's just some some detail around red hat and IBM were supporting SW ID, but it seems like it's kind of waning You know what's in the S-bomb there's a couple of just guidelines here, right? What's being cataloged? you know what's in here unique numeration of things and Kind of a some details on what actually did the cataloging right the S-bomb should describe what created it so you can go back and maybe try to reproduce it These things should be reproducible, right? Really good S-bombs can you know have a even more detail? What's in scope? What's out of scope? You know any kind of exceptional conditions things that we didn't that we encountered that maybe are unusual unanticipated And then metadata right I mean things like we were talking about earlier Things like secrets in a fire in a container image, right? Did somebody leave an SSH key in here? There's no reason you can't have that kind of stuff in an S-bomb Now sift is a Fairly new project that anchor is sponsoring It's written in go It is primarily intended to generate S-bombs for container images But it can work against just a regular file system. We I've had people Try to use it against an entire VM which It gets a little weird in some of the you know, the procfs or whatever That wasn't really an anticipated use case but in any case It can output Like I said earlier cyclone DX or spdx it has its own format as well Anchor had a previous project called anchor engine that used software bill of materials internally to generate vulnerability scans and Compliance checks and so the the other JSON format that it uses is kind of more compatible with what anchor engine was doing So it will output and handle any of those formats though There is also the companion piece called gripe so gripe takes the software bill of materials and spits out a vulnerability list It's really now that sounds like okay. I'm doing two things when I could just do one As we'll see in a minute though There's a couple of advantages to generating the S-bomb first and then doing your vulnerability check your compliance checks all that stuff Both of those in fact you can use gripe standalone. You don't have to feed it an S-bomb It has sift kind of integrated in it can generate the S-bomb on the fly, but it takes longer So now what how do we actually do this stuff, okay? This is like another story. It's not a not an analogy Probably the worst slot. This is like the ugliest slide I've ever made. I think it I've made some pretty bad ones but So things like airbags and seatbelts, right? They're incredible safety tools, but they only work when you have a wreck, right? They don't do anything to help you stay to avoid wrecks, right? whereas things like any like brakes traction control a lot of autonomous driving features are really in intended to lower the the frequency of an accident whereas survivability features like seatbelts airbags are really kind of intended to lower these the impact of one when it occurs I Don't think it's an accident that most of the money is going into things like autonomous driving Things to help you avoid the accident in the first place, right? That's obviously what we would all like, you know It's not it's harder to kind of think about right The the the thing the difference between what's seen and what's unseen is always kind of hard to Quantify sometimes, but I think everybody can agree that having fewer accidents is better than having more even if you survive a higher percentage of them So Not an analogy, but we'll come back to that in a little bit as well So what a spom's actually do for us, right? So two things one is speed of the checks Since if we generate an S-bomb we can go back to it over and over again instead of going back to the image and Cataloging all the things over and over again, so we can do vulnerability checks much more frequently as we get new Vulnerability definitions and it's not just for vulnerability checks either we can do compliance checks Check other things in there But it's a lot faster orders of magnitude faster to do those checks against the software bill of materials as Compared to doing it against this the source image, right? And we'll look at some of the timing there in just a second The other thing is the accuracy and precision of these You know some differences between accuracy and precision, right? But accuracy is you know how close to reality your results are Whereas precision is like how repeatable, you know is this consistent We can improve both of those with a software bill of materials because we can make adjustments to it so things like Let's say you're using angular, right some of the things get compiled down It's if you just look at the end result. It's hard to enumerate what actually went into it We can provide data to the S-bomb generator to tell it like hey for this artifact This is what went into it and it can kind of Put that in the S-bomb where it wouldn't be easily discoverable just by looking at the image itself Things like what this guy over here was saying earlier. I didn't get your name. I'm sorry John If you do multi-stage builds, right and you have a couple of build artifacts you drop those into a distrelas image You lose a lot of context, right? So in an ideal universe, you would be able to generate the S-bomb from the fat image Before you generate before you move those artifacts into the distrelas image And then somehow tie those two together, right some of these There's a little bit of work to be done, but that's the idea So the software bill of materials can provide a lot of that context that gets lost either by You know some of these techniques that are used to slim down the images Um timing this is just some example times I did on some junk images that I created. These are just some Test images that have a bunch of garbage in them. You can see The bigger the image the bigger the time savings as a percentage, but You know this is like orders of magnitude faster, right? So Generals my laser here this image here has just got a ton of stuff in it. It's like a it's I think it's just under a gigabyte The timing is hard to read here, but essentially Took like 15 seconds to generate a vulnerability list with gripe from the image itself One and a half seconds Doing it from the S-bomb, right? So it's a pretty big savings. The idea here would be Uh that if it's faster to do it you can build the S-bomb earlier in the process, right? If you go back to like waterfall development Security checks usually didn't come in until pretty late because it was an expensive process Right. You didn't you didn't you couldn't afford to do it Through the entire lifecycle of the project So, you know developers would be miserable because they would think oh, I'm about to ship this thing And then the security team like swoops in does their checks and sends them back to the You know way back to here Now what we want to do is kind of push this, you know all the checks further to the left It's because it's so much cheaper to fix things over here, right? Um, I mean just look at the difference between fixing something in production versus something in development, right? I mean, I think github came up. I think this is github data. They did some surveys on this And also the fact that you know There are a ton more developers than security researchers, right? So the idea is with these tools we can empower These people here to actually Contribute to this effort Instead of relying on this really small number of people over here um, okay, so Demos right everybody likes to demo I I hate demos right because what use this is what usually happens is I tweak it right before and It turns into a big pile of garbage, but We'll try one But if it does blow up, I've got some repos here These are linked on my twitter This is a real simple one It will you can set up a like a disposable Jenkins container And build some images and generate s bombs and do vulnerability checks on them. It's pretty simple There's a more complicated one here Um using github actions This one actually incorporates a project called cosine, which is kind of affiliated with six store So this one will generate s bombs sign them Use the s bomb to generate a vulnerability list attest the vulnerability list and um publish The attestations along with the artifacts It's a little bit more involved Um There's a big blog post that we've just published on it as well that kind of will step you through it um and Finally just like I haven't even built a demo for this last one But we just published a github uh s bomb action that will do kind of simplify some of the stuff as well um All right, so let's try it. Uh Let's see what happens here Let me mirror these and I will back out of this and Let's see what we got. Okay. So this is my github repo this Is I've got this one pinned Um, so there's a Jenkins file in here You can just you don't even have to clone this repo, but you can just follow these directions I've already got Jenkins up and running. This is just running on my laptop here in docker desktop. Uh I think let's make sure it's running Yeah, okay, so Where's my Jenkins? Oh, it's right here. Okay, so what we'll do is We'll just take this repo and I'll set up a new project Create a job we'll build a pipeline sift one And can anybody read that? Let me Let's do that All right, let's set it up Uh Pipeline script from scm. We're going to use get And here's the repo Save it. All right, let's build it. Is it building? Yep, there it goes. All right I think it'll work All right, so It built an image. Uh, it is now generating the sbomb right now And once it does that it's uh, oh it's already done. Okay, so that's going to leave behind Uh, a couple of things one I haven't actually run that the the vulnerability check twice So if we look in the console output, I can see the actual vulnerability listing Uh here And then it also left behind The sbomb is a build artifact and the vulnerability list is an artifact pretty simple if you go into the uh repo and look at the Jenkins file you can actually modify this a little bit There's some hints in here about how to like take the vulnerability check and actually cause it to break the pipeline if there's critical vulnerabilities or Actually, I think yeah right here this dash f option You can say hey if there's any high severity vulnerabilities break the pipeline or something like that real simple um So wow that worked amazing um The second one i'm not going to run this one takes a little bit longer, but um, you'll see here like what it's doing in this this Uh, basically when it builds the image it then will create the uh software bill of materials It signs the image which you know is kind of superfluous for our purposes Once it creates the sbomb then it will use that sbomb as an input to the vulnerability scan And then it will sign that vulnerability list and then it also kind of branches off here to sign The sbomb itself and then it publishes all of those things as artifacts right you can just scroll down here and see um Somewhere in here. Where is it? Yeah, there they are The artifact said it leaves behind Um, so if you want to check that one out that's in here as well It's actually cloned from one of my co-workers It might even be better to go back to his original one because i'm kind of always tweaking these things and may Catch on fire at any moment um, but like I said, that's linked in the slide deck along with the um Uh, the other stuff here this blog post this this is the the more complicated repo And then this details on that sbomb action um So, yeah, if you want to go to my twitter, uh, it's pvm There's the link to the slides and A link to the demos there Okay, so, uh any questions so far I know that was a lot of stuff really quick. Yeah Okay. Yeah, that's a good question. So the question is how is sif creating the sbomb? Uh, and what is it actually doing? Um So what it does is it basically untars the the image it will grab the image, you know It can be something local it will go out to a repo and pull it using scopio or whatever Basically just opens it up. It will use things like, um, if there's rpms or devs or whatever It will use that facility to catalog those It is compatible with, uh, language artifact packaging like node, you know npm or ruby or Python, etc So it will start with enumerating the packages both, you know, os type packages and language packages Then it will go to like files just going through all the files in the image Um, there are configurations you can do with it to have it look for Um patterns and files if you want to get that specific So like that's how you would find things like an aws access key or an ssh private key We can, you know, you can come up with pretty easily Regular expressions to look for those kind of things Um, is that pretty much what you're looking for? Yeah, yeah That's a good question. So the question is is it just looking at the the image itself? Or does it go backwards and look and to try to verify the the origins of those things? It does not do any of that work there. It's just one cog in that whole machine I would recommend if you want to do that entire Set of things, right? This is one cog in that that kind of tool chain Look at yesterday. Dan Lawrence had a really good talk on exactly that topic going all the way back to the hardware Right and verifying the hardware that the stuff was built on. So yeah, I mean, it's it gets really complicated really fast But yeah, this is specifically like, okay, I've got an image I just need to inventory it the question of where this image came from and stuff that's important stuff for sure It's a little beyond the scope of this tool Right Yeah, yeah Yes, this does not solve every problem we talked about today. Absolutely correct Yeah, yeah, there's Right. Yeah, that is that is 100 correct. Yeah, I don't want anybody to think This is a silver bullet and like all your problems are solved, right? Yeah, they're Yes Yeah, I mean I would it would probably There'd be a couple of Nobel prizes in it. I'm sure right Um Any other questions? I don't know if we had any online. Let me just check. Uh, it doesn't look like I don't think that those are for me. Those look like those are older. Okay. Um I will go back here. Just a couple of other things. Um, some takeaways and whatnot, right? I mean best practice. How do I get this big again? There we go. Um, okay. Yeah, so best practices here I mean, this is kind of, you know, uh, basic stuff, but it bears repeating, right? Um, centralize your CICD pipelines, right? Don't, you know, have people just setting up, you know Unsecured Jenkins servers all over the place Uh, build from trusted sources, right? I mean, that's not again It's easy to say that. It's harder to actually enforce it um Automate as much as you can though, right security testing vulnerability checks, uh policy enforcement So when you have s bombs, you can do things like look at Um, let's say like the You can have something like the docker file, right and look at where The base image is coming from right and say is it pulling from a known good source, right? Look at the digest of the base image, right? You can enforce If there's a list you could I mean the basic the easiest way to do this would be say I only allow base images from this known repository, right? Do not pull alpine latest from docker hub Only pull from, you know example.com slash alpine or whatever Right, that's better than nothing Having a list of digest that are approved was even better, right? Um, and then of course Right Yeah, and you can verify like so a lot of times, right people will put the the bear ubi image in their internal registry But they will vet it first before they let Everybody use it right and say, okay. This is a known good one. We'll bless it. Here's the digest It's it's okay to use this one, right? But don't pull it directly from Rel right because we don't know what'll happen in between on the network or whatever, right? I mean, there's a million things that can go wrong um So yeah, and then just you know deploy trusted images into production That could be something as simple as like a kubernetes admission controller to say before I deploy Do I have an s bomb for this image? Right if I have one great if I don't have one all of a sudden I think oh my god Something's off the rails somebody's trying to deploy something that we haven't seen before Huge red flag that the process is being circumvented, right? That's a pretty that's actually a pretty easy check to make So We did the questions already. So I'm going to continue on with takeaways here This one here. This is about We talked earlier about, you know satellites that are in intercepting telecommunications Since that happened, right Indian encryption has become a lot more prevalent which has made those kind of Surveillance operations a lot less effective, right? The people they want to intercept their their communications are not talking in the clear anymore Right. So intelligence agencies have had to move from these huge cheap mass scale operations to these really expensive Targeted attacks and they're higher risk because they can be detected right when you're just absorbing everything everybody You know everything is being absorbed You're not really sure if you're being singled out or not Whereas if you get targeted, you know, you find a gps tracker on your car. You're like, oh, they're really tracking me Right. So those are a lot more expensive. They're a lot higher risk. You can you know, you can be given away you get made So the idea here would be like if we can make these kind of supply chain attacks more expensive Right and maybe push the the activity somewhere else, right? I mean, that's the problem is It's like a balloon, right? You squeeze on one end. All the air goes to the other end. I don't know But anyway takeaways here, you know, don't be like general haig Don't rely on horses in a mechanized universe Wear your seat belt even if you have traction control. You should still wear your seat belt and have airbags in your car Now I did want to come back to this guy, right because Uh One thing about him. I mean, he did, you know, lead the british And not all the allies into like just huge amounts of casualties, but He did adapt. So by the end of the war, um, you know, he basically commanded the allies during the hundred days that It's probably the greatest a lot of historians think probably the greatest, uh, british military victory ever so he realized he you know that uh, he had kind of You know screwed everything up and Made the adjustments necessary. So it was kind of a redemption story there at the end, I guess Um, and then one last thing to just think about, right? I mean like we said earlier, there's no silver Silver bullet, right? We're not going to you know, uh, completely Eliminate risk. We're just going to be able to reduce it, right? I mean, that's the main thing is We've got to be realistic about what's what's possible and what's not All right, um That's it. Thanks a lot guys. I really appreciate everybody if anybody does have uh, we we do have more time. Um So if anybody's got more questions or wants to talk about anything else Uh, I'm I'm definitely I'll be here for a few minutes Thanks