 We're SIGHawk and this is malicious compliance reflections on trusting container scanners if you're here for malicious compliance Hey, so are we if you're not here for malicious compliance. You're in the wrong room, but we're happy to see you anyway I'm Brad geesterman my pronouns are he him and I'm a staff security engineer at ghost security And I enjoy hacking cloud and kubernetes with my friends here. I'm Ian cold water pronouns. They them I'm an independent security researcher a long-time community organizer and SIG security co-chair I love hacking weird machines and bringing people together and I'm looking for work if you're hiring Hi, everyone. My name is Rory McCune my pronouns are he him I'm a senior security advocate with data dog and I like to learn about security and write blogs Hey, everybody. I'm Duffy Cooley. I'm the field CTO at ice availing where I get to work on things like zilliam and tetragon and EBPF I'm excited to be here with all of you in person and I'm especially excited that we are all here finally all those all together on the same stage I'm gonna go drive this over here So SIGHawk isn't an official kubernetes SIG We're a hacker crew who have been poking at kubernetes for years And we work together to share attacker perspectives and help the whole community level up. So, why are we here today? We have a question for all of you real quick Who here has ever heard of containers? Of it we had a feeling By another show of hands who here has ever used a container vulnerability scanner before Wow, okay, great. Perfect So one thing you might have noticed is that governments auditors and your security teams are asking you to scan all of your container images for vulnerabilities so that you're more secure But we don't hear as much about the details and as it turns out the details are important So what do containers scanners actually do and what problems do they solve and how do container scanners work? Are they all the same? Can you always trust the results? Let's find out In this talk we'll be mainly for covering four widely used container scanners We chose trivy gripe docker scan and docker scout to test But what we'll be covering here isn't necessarily limited to those four as of april 13th of this year Docker scan has been deprecated to be replaced by docker scout Docker scan is powered by sneak on the back end which we thought was interesting and a lot of people still use it So we figured we'd still include it here today So to start let's use all four of these scanners to scan a vulnerable container image we created and see what results we get So let's show the versions Of each of the scanners that we're using today just for posterity So we have trivy gripe docker scan and docker scout So let's cap our base docker file As you can see we're using a slightly old base image. It's a composer 1.72 So it's got a couple of vulnerabilities. We know that that are in there and it's based on alpine 3.7 The next thing we're going to do is install no j s Then we're going to copy in some runtime dependency files So things like gem file lock package json yarn dot lock, etc And lastly at the bottom you'll see we copy in cube control. It's relatively recent, but it's not quite up to date and that's a go binary So let's quickly build the base image And scan that scan the image with trivy gripe docker scan and docker scout Let's check check the results in the summary format There we got some findings But let's see that on a graphical display So this is our baseline set of results and immediately we can see that the four different scanners trained on the same image Produced wildly different numbers of results. Why is that? Well, it turns out they don't always work the same way. Let's break it down a little bit So how do you can container vulnerability scanners work? Is it magic pixie dust? Now they work on a combination of several techniques that we'll have a look through now So files that hold operating system version metadata and release information like etsy os release etsy lsb release And etsy alpine release are typically one of the first things scanners look to to help inform them For their approach to scanning the rest of the base image From there container scanners also flag up outdated versions of packages provided by your linux distribution This is based on the scanner using the package database that's installed by the distribution And the security database that contains vulnerability information One thing that's worth being aware of is that in some cases there are os package vulnerabilities for which there is no available patch And the scanner can be told to either report them or not depending on your risk appetite And another thing to think about here is this means the scanner needs to understand the package database for your distribution So if you use a new or a niche distribution, it might not be supported Some of these scanners also lift for software language dependency files such as package dot json requirements dot text or cargo dot lock This use of the word cargo has not been endorsed by the rust foundation These files contain information related to libraries and metadata, which the scanners use to find vulnerabilities in software dependencies Another thing that some scanners look for is any metadata embedded in binaries Go lang since 117 embeds dependency data inside of the go binary and rush binaries can be scanned in the same way So now that we've talked about how these tools come up with findings Let's go back and look a little bit closer at our base image one more time the results So looking at the results again, you can see that we've broken them up into three different classes of finding os package dependency file and binary metadata And you can see that the numbers found per category for each individual scanner are wildly different as well as the total numbers of results I mean, that's a lot of findings There's at least 90 or more results that that we have to go through and fixing all of them is going to take a really long time To be fair some of these specific findings could likely be resolved by updating our base image Since we used an older one on purpose for the purposes of demonstration, but it's not always that simple Many of these findings are related to dependencies in the software itself And this can create compounding issues updating dependencies can change the software Can change the software's behavior and it may need to be modified to handle the change in dependencies And it's not just the one time fix we have to deal with this every time we have to update dependencies every time There's a new base image every time new cvs are learned about There's just a ton of things that can actually trigger having to go back through this And if this base image is actually from a vendor fixing it may not even be an option Okay, so let's say that we're the team who's responsible for this image And it's currently deployed as a core component in our production environment right now The security team just sent us these scan results And now we can't ship again until they're fixed Fixing all the findings that were raised means updating several components and maybe retesting our entire application So we're taking a risk of breaking production just by trying to do the right thing and actually fix them And that brings us to a choice What do we do about this? If we fix all of these findings to be compliant, we'll miss our deadline If we ship on time without fixing them, we'll get into trouble with the auditors None of those options are very good. There's got to be a better way So we need to address these scan results to be compliant, but That sounds like a lot of work Is there something we can do to just make the findings go away? I have an idea Now that we know what the scanners are looking for let's use a systematic approach To see if we can hide or bypass detection for every single finding for all of the scanners with no breaking changes to our containerized application If we can do that The next scan results will be compliant. Our app will still work and we can still ship it and make our deadline So remember when we said that the scanners can use the os release information to figure out which operating system version the image Is based on and that can how that can guide their approach to scanning the image What if we remain at report that it was an operating system the inversion the scanners had never seen before By changing the name and just deleting that information entirely Let's see what would happen So we're going to take that same original docker file We're going to change a couple things First thing we're going to do is make it look like it's our own custom distribution of linux and overwriting at the os release Then we're going to delete at the lsb release and at the alpine release just to make sure it doesn't get confused Okay, so let's rescan the image First we have to build it. Sorry so we can incorporate all those changes We're going to rescan the image Trivy right docker scan and docker scout Let's take a look at the results Interesting some things have changed here All right, let's go back to the graph. So it's a little bit easier to see We can now see that docker scan and docker scout Immediately stopped reporting operating system package issues. I mean that's pretty wild, right? All from just changing or removing a couple of files All right, that's pretty sweet clearly things have been improved, but I think we can do better That one took care of those findings from docker scan and docker scout But can we get the other scanners to no longer see findings from os packages either? So if changing or deleting the os release metadata work before We should also try to get rid of the os packaging metadata for the installed operating system packages That shouldn't have any effect on our application working, but it will prevent the scanners from seeing that information So What we're going to do is we're going to go back into our docker file and we're going to make another change We're going to delete the alpine os package metadata So down the bottom What you can see here is we've got a couple of lines that just do an rm minus rf on etc apk and lib apk Which is where that data lives Then what we'll do is we'll rebuild our image again because we need to make sure we're getting the latest version And that'll go away and do it and then what we've done now is we can now recap run our scanners so We'll scan the image with trivia with gripe with docker scan and with docker scout Let's see what kind of effect that change had Wow, if you look at that left column there, which is os packages. That's fantastic. We don't have any vulnerabilities anymore So that's a great change Let's look at them in graphical format So this is how many we started out with to remind everybody and this is how many we've got now So now all four scanners no longer show any os package vulnerability findings, but let's be clear The results are gone, but we didn't actually uninstall any packages. They're still in the image It looks to me like we're still seeing results from some of the scanners relating to software dependency files Like gem locks and those sorts of things. So our next stop should probably be to try and get rid of those So we know the scanners are looking for specific files with specific file names that hold software dependencies Used by languages like javascript python ruby and others So how are the scanners going about looking for those? Are they just looking for the file names because if so I wonder if they would try to follow a sim link In order for our application in this image to still work We can't change the names of files like package dot json and gem file dot lock But what if we renamed the files themselves to something else and then made sim links to them with the original names? Let's try it So we're going to take that original docker file But this time we're going to copy in all of those files with a different name Then we're going to make sim links to link them to the original file name So that when the application tries to look for a file like package dot json It just follows the sim link to the renamed file and the app will still work We also went ahead and deleted some python egg info metadata files because we don't really need them. So why not? So let's build that modified image now. Let's scan it with all four scanners again Let's take a look at the results. So to reiterate, we haven't actually deleted anything We've just made the scanners not see it anymore But let's take a look at these results again in graphical form That's all right. It's fine. It's all fine. Okay Again reminder, this is how many we started out with in the first place and this is how many we've got now It's clear that the scanners do not always follow the sim links And that made them find not not find any issues in the software dependency files And you know, this kind of makes sense Sim links have a rich history of being tricky to get right For a good example of that you can actually see our previous talk about kubelet vulnerabilities A slightly peculiar volume configuration from kubekan na 2021 All right, so it looks like we still have a few findings left and they're all related issues with binaries Ah, remember that in some in binaries the metadata can be embedded inside of it So I wonder how we can get the scanner to not see that stuff. What can we do about all of these binaries? Have you ever heard about upx packing? It's a way of compressing binaries and it also happens to obfuscate the internal structure of the binary itself This makes it very useful for things like making binaries smaller and Avoiding malware detection So let's see what happens if we upx pack all of the binaries So let's modify the docker file again We're installing upx as a package so we can pack some of these binaries and apply that same sim link and rename technique against one of the python shared libraries And we're going to pack both kubectl and busybox To see if that gets rid of all the binary issues So let's build that modified image again Now we're going to rescan the image with all four scanners And then let's check out the results. That was very effective Mostly, but we still have a few findings left. Let's take a look at that graph again Again, this is the original amount that we had And this is how many we have now There's still more to unpack here Three of them are still finding issues with the golang binary kubectl That's interesting though because that should have gotten rid of all the findings, right? Well, one thing we do know is the container images are built of layers. Perhaps we're looking in the layers Right, but is there some way to make it so the scanners can't see all the layers? I mean we are shooting for a hundred percent compliance, right? I mean we could try a multi-stage build. Let's try that So we're going to edit our docker file one more time And we can see that this docker file is a bit of a mess There's lots and lots of things that we're doing here lots and lots of layers that we're adding And lots of layers also that we're inheriting from the composer image as well So he'll be here at the top. I've just decided to call all of this work as builder And here down at the bottom. I've got my from scratch and copy two lines So we're going to go ahead and separate those build and the obfuscation steps and the final packaging steps And we are also going to copy in an icar sample test file into the image overriding bin bash And we'll talk a little bit more about that later. Where is that line? Here we go So that's where we're copying it in We're going to talk about that here a little bit later. So All of when we do this when we do this step of actually A multi-stage build we end up with a single layer image. That is the artifact that we're going to ship And this will result in a couple of interesting things When we build this new modified image and we compare the size against the previous one We can see that the old one was 306 meg And this new one Is 166 meg so big win right easier to distribute all that stuff. It's good stuff Our image is a bit smaller and any image that uses this one might also make make get benefit from some of the modifications that we've made So let's go ahead and re-scan these things And see what we see here view the results So let's put that back in the graph for him. That's where we started How it started How it's going And we did it. We are completely 100 compliant. I mean we haven't changed anything about this image I mean other than like how we build it and that sort of stuff So all of those findings in the first graph are all still in there. They're just not alerting anymore Okay, so this is exciting. We've managed to get to zero findings on this old image just by taking things out But how about adding stuff in could we sneak anything malicious in there and still be compliant? Well, we already did None of these scanners picked up the addition of icar because none of these scanners are going to look for malware or file changes If we want something that might pick back on the thing up, we might need to use a different approach What about something that generated an inventory of all software and dependencies in the image? Like an sbomb The white house and other governments have said that we all need to use sbombs to know everything that's in our software So sbomb stands for software bill of materials Which at a high level is a document often json formatted with a goal of telling users What's in their software or in our case a container image? So surely an sbomb would have to show us if there was something malicious that had been added to our container, right? Well, let's find out We can use trivia and sift to generate some sbombs for our images And then use trivia and gripe to scan those sbomb documents for vulnerability results Sbombs have a couple of different formats, but in this demo, we're going to be using these tools to output them in spdx json So let's go ahead And generate sbombs for the original base image our starting image and our multi-stage build image We're going to use trivy And sift and sift and gripe work together. They're both made by encor So sift generates the sbomb and gripe scans that sbomb But trivy scans trivy So let's look at our sbomb results And right away you can see something's up because in our base images the file sizes are quite large And in our multi-stage build images those files are kind of tiny So let's dive into those So if we look at the sift sbomb for the original base image We can see this is the spdx format by the way. It's just a lot of json. So if you love json You can see it captures all the packages, right? They're all there And captures some of the files Including bin bash, but you remember we copied icar on top of bin bash But what do we see for the shahash all zeros? That's interesting, right? So let's look at trivy's sbomb for the original base image And as you can see similar format spdx, but There's only one file shown and all of that metadata is in the packages So right away you can tell that they're not quite exactly right and not quite uh compatible And lastly we'll look at trivy's sbomb for that multi-stage build image the one that we had zero findings for And this is how long that document is That's it. That's the final sbomb So if we look at a trivy sbomb generated for the multi-stage build you can see that's pretty light And there aren't a lot of results there which is incredibly interesting But did it pick up anything malicious of ours? As it turns out All the same obfuscation techniques still work to hide the findings from being reported from the sbombs So all that work worked And again, we still haven't changed anything about the software in our image And this really calls out how important it is to understand how these tools work This sbomb records that there are no vulnerabilities in the image and we've shown you that that's not true Which doesn't stop this image and the associated sbomb for being considered compliant A few of these techniques are legitimate things that probably some of you have used during to manage the life cycle of your images in the past I mean who hasn't built a multi-stage build at this point? You probably you perhaps did not know that you were being maliciously compliant at the time and this stuff is hard And even it is compliant. It's definitely not secure The demo gods are not smiling upon us today. So we're going to go backwards one slide So what interesting thing we found is that the results are not always consistent Between when some of these scanners scan images directly and when they scan sbombs generated from the same images Let's demonstrate So here are the results of trivys original scan of the original base image again And now we're going to scan the sbomb documents. We generated before with trivy So let's scan the sbombs trivy generated for the original base image And our multi-stage build which had no issues Okay, fair enough The trivy stands are consistent with the prior results when we scan not only directly with trivy but with the sbomb document The original image has the same number of findings and the sbomb is still being fooled by the multi-stage image We built which of course has no vulnerabilities at all Let's look at the original gripe scan results From when gripe scanned our original images directly Okay And then let's scan them again with the sbomb results that were generated by gripe's partner tool sift These two tools are designed to work together. They're made by the same vendor. This is a feature that's advertised So we're going to scan the sip generated sbomb for both our original base and our multi-stage build images with gripe As you can see, they're not at all the same not even a little bit That's weird, right? You would think they would be consistent, but they're not The number of findings gripe finds from the sbomb generated by sift is totally different from when it scanned the original base image By itself directly that said the whole thing is still fooled by our zero finding multi-stage build image Because as you can see this image has no vulnerabilities at all So looking at the results of the sbomb scans, we obviously got some surprises there for a few reasons We would expect it to get the same number of vulnerability results from scanning an image directly or scanning an sbomb of that image But that wasn't always the case But also if I expected to have some indication that a file had been overwritten We expected that the sbomb would include metadata for every single file perhaps in the form of a hash But when we were but when reviewing different sbombs, we found that not all of them do that Another surprising thing we found is that although these tools allegedly produce an sbomb in the same format Like spdx there is no defined standard for how that format gets implemented So these tools and documents aren't interoperable If you use one of these tools to generate an sbomb document and feed it into another one of these tools For example, generate an sbomb with trivy and have gripe scan it or the other way around with sift It leads to unexpected results or frankly no results at all It's even inconsistent when they come from the same vendor which was surprising too So we've shown that sbombs don't capture everything in the image And we've seen enough interesting behaviors here to suggest that there's more to dig into Sbombs clearly need more research We don't have time to cover it all in this talk today, but we have a lot to say So stay tuned coming soon to a kubecon near you So what can we learn from all this? We've shown that image scanning and sbomb generation tools are quite sensitive to the changes where the metadata from where they get their results The quality of those results is also directly related to the quality of all of the steps involved in how the image is built And you can't always determine just by looking at the output that something is off It's one of these tools generally tend to fail in a way that is pretty opaque So if you can manipulate the results of these findings, this has potentially significant downstream effects And it doesn't even have to be done on purpose Inconsistent results from these tools or their lack of interoperability can cause problems for organizations Where different groups use different tools for vulnerability assessment Many organizations, perhaps maybe even yours have siloed teams that use different tools and maybe don't communicate very well with each other If you can't always count on consistent results It's really hard to know who or what to rely on Yeah, for example Many organizations use vulnerability scanning results when determining which workloads to admit to their production environments So if the results can't be trusted that could be a big problem So here are some things that we think these tools could do better to prevent against attacks like this One of them is to provide a more restrictive mode where detection coverage is the focus and at the expense perhaps of performance Following things like sim links throwing warnings and errors for clearly anomalous configurations Like you found a bunch of files that you can't associate with packages or a particular copy step A scan that returns no results can be a false negative And it causes these tools to fail successfully as we've shown you in this case Defining a consistent standard for using the same fields in generated espom document formats across tooling could help these tools become more Interoperable and could help their results become more consistent While we realize that many vendors might not have a lot of incentives to collaborate with their competition It would make it easier for everybody's users of these tools to trust their results Also, we've only scratched the surface of potentially malicious techniques So over time it would help if the tool makers adopt a more adversarial approach thinking about how people like us are going to try and bypass them So you're watching this and you're like, okay. Well, that was all cool, but I'm not involved in the development of these tools So what can I do about this? One recommendation check for unusual behavior What are you expecting to see in your environment? Does everything look like you expect? Everyone's going to have different answers to this because everyone's environment is different. Do you know your environment? If you don't starting to do that would be a good first step If you do know what to look for are you expecting to have zero cvs as a result? Are you expecting to have upx packing or sim links in your build process? If not, that's weird and you should probably go check out what's going on You can and should validate all of the inputs and processes for how you build container images to ensure that they are following your policies for what a well formed artifact might look like So that they can be observed correctly by these tools And compliance teams who use these results to assess the security of third party software That they should be using should pay attention to what's in the image not necessarily just the output of the tools Yeah So when implementing these scanning tools and ci pipelines to enforce policies that block image builds or gate deployments just based on the results It's important to consider what is being asked of the folks who build and maintain these images As the number of images grows the number of scans and results needing resolution increases exponentially So this directly translates to extra work and development friction for those image builders Which can cause an adversarial relationship to develop with those security teams And that might be enough motivation to make it so that they choose the approach of malicious compliance like we did today We're all in this together And ideally teams are working with one another to achieve larger goals So let's talk about that. What are those goals? What problem are you trying to solve? A lot of toil can be reduced by considering your threat model and how these tools fit into it If you can make policy decisions strategically based on that your work will be much more effective This will look a little bit different for everybody because everyone's threat model is different But if you can place the results of these tools into that context, whatever that context may be That will help In some cases it might be helpful to consider whether or not the problem you're trying to solve can be solved by these tools at all And I think it's really important to understand what these tools do And what they don't do and how they work They're not magic boxes. They're pieces of software and they work in relatively predictable ways When you understand more about how they work, it helps a lot when you try and understand where and why they fail And how you can best use them Another thing that I think all of this shows is the importance of trust in the supply chain You know what quote I like about trust is a trusted person is someone who has the power to betray you Tools like scanners and s bombs assume that you can trust the entity that generated them And they really can't help you if the people that generated them turn out to be malicious or if they just didn't do things right I spent a ton of effort lately in the tooling To define and improve the security of your infrastructure and supply chains But it seems like there's still quite a lot of work left for the adopters to figure out how to get all of these things to work for themselves We are often incentivized to be compliant, but not necessarily secure And these maligned incentives can make it easier to choose the path of malicious compliance When what we really need to do is choose the path of security Security might be the harder path, but it's the right one and it's one that we can all walk together The effort to improve the security of images and artifacts in our environments is something that each of us has a hand in So let's go make things more secure together Oh and uh one more thing Thank you all so much for coming everybody Here are some links for the repo and base image for our demos if you want to play along at home As well as some references that we think might be of interest If you scan this goosey qr code in the corner that goes to giving us feedback on this talk We'd love to hear from you and if you're in this room Don't leave because immediately after ours is b corbsis talk on pod security admission And I know for a fact that one is going to be a great time So thank you all so much for coming come find us if you want stickers because we have them. Thank you Thank you everybody appreciate it