 Hey, so thank you for being here. This is my first Q-com I've been to several amazing talks where the speaker said that it was the first time Speaking in public after COVID. This is my first time in speaking in public Period, so let's see how this goes. Thank you Thank you to the CNCF You clearly don't know what you're doing So a bit about me what we are going to See how you can explore the layers of your container images To find some sneaky vulnerabilities and previously we'll go through some more common mitigations Very quickly about me. I'm Pablo. I'm a software engineer at VMware. I joined Like one year ago. I started working on containers and home charts And now I'm more on the Java microservices world What by the way, I I've also studied law before computer engineering So if someone has a problem here in Spain talk to me The story behind this talk Was in November of last year when I moved from the team Maintaining the binami open source catalog For those that don't know binami offers a huge library of containers and home charts built tested and published by an internal pipeline I'm now part of the team that's developing the product that will supersede this internal pipeline And offer many of its features as a service So in this new team, I was tasked To establish a vulnerability baseline for the Java microservices under development We needed a way to tell if under a new circumstance we were worsening our security posture and A baseline is what allows you to to make that judgment So we had a very good starting point almost love environment. We had Very a very limited number of microservices built with up-to-date technologies Most of them used build packs But as you may know build packs require your app to fit into a set of requirements and when that can happen The classic docker file approach is unavoidable So we also had already in place a vulnerability scanning phase in CI running aqua strivi, but it was just informative and non-blocking All of that to achieve a sound goal to have no creative no fixable critical vulnerabilities in production Why do I think this entry level talk might be relevant because as you know containers are an excellent tool to is the Deployment and to is deployment and development But they also bring their unique challenges when it comes to securing them to people that Never had before a similar responsibility if you are working on a team. That's similar to mine developers are now responsible for the patch management of the OS base image and Also this shift happened While containers are marketed as boxes that just work. So why would you even want to look into them, right? Finally many of the open source vulnerabilities scanning tools are noisy by default have noisy outputs by default and Requires some tweaking to give you the useful information on what you can act So I'm going to go through the steps of fixing the vulnerabilities in a Java Spring Boot Maven project to show that that's a task that can be Done by anyone with a minimum knowledge of containers and even going one extra mile It's not that difficult once you remove part of the of the magic of containers So let's go quickly Removing that magic. So containers as you know Container images are sets of layers. We'll see in a bit. What are those layers and You may remember that there was this common phrase of Every instruction in a Docker file generates a new layer But with bilkit, that's no longer true now the layers are cached and If no changes are detected The cache layer is used so effectively only run copy and add instructions create a new layer in your image If you run a Docker build with this simple image with the simple Docker file, sorry you'll see that the Really the only layers that are created are the ones that are highlighted highlighted here And the other ones are just skipped Layers are just folders and configuration files if you run that command You'll see An output similar to this I've highlighted in green yellow and orange the layers from this Docker file and You find in one the mini dev file structure in it up is just a stripped-down version of Debian maintained by Vietnamese on the other the empty file and On the last one the cool binary with the CA certificates and path So I've set up a playground for this talk that are two Java projects simulating a plugin system and Container library that in this case just holds the image for Building the demo service Okay, so let's see life Okay, I hope everyone in the back can see this is a strange layout so The builder image The demo service uses multi-stage fields Multi-stage fields help us Implement the builder pattern in which we separate well in which we divide the process of bringing source code to production into two images the first image with all the Build chain tools and the other one just holds the binary generated or the bytecode to have a Slimmer image So here we'll just run a Docker build and call it local maven column maven and so on So in the demo service, this is One of the the skeleton of one of the microservices that I had to look into I kept the Original structure, but no code. So I have the API core infra some ports and adapters pattern and in the boot module the boot module is the one that holds It acts as a glue of the project because Has the has a dependency on all the other modules And also the the application entry point so in in here is where we are using the Docker file maven plugin to create the image of the app and the Configuration settings are very simple The repository is going to be the artifact ID the tag is going to be the version of the project and then as build arcs We are sending The jar and some other environment variables So if you are curious the this is the Docker file Here we have the builder image some magic to install helm and And here the real image that is going to be deployed so we lose the maven rubber and the typical clean install command and Hopefully the network is going to Be okay Okay, so while this is building we can go now To the process of refining the 3d output We'll start by running a naive 3d image and Use yeah, and use the image generated here sorry for those in the back because This is not going to be seen very well But trust me Nothing to worry about just 200 vulnerabilities in a container image So this is Not set not the output that we want because we want just the critical ones So for that 3d has flag that we can use Called a severity so we can do dash dash severity set it to critical and If you notice 3d the output of 3d was now instantaneous because 3d cache the result from the previous run so this is nicer, but We still have here some vulnerabilities that Are not relevant because if they if a package doesn't have fixed version we are not going to go to To develop the fix for that package in the upstream in Debian, so 3d has this nice flag called Ignorant fixed That will give us the actionable items that we need So with this With this we are good to go If you are okay with it, I'm going to start by with the base image vulnerabilities This is an open SSL Critical vulnerability we will have to go here and I have bad news I think the only way to to know in which image This vulnerability is resolved is just manual manually checking if a new revision of the base image or if a newer version Doesn't have this vulnerability so Hopefully running again Install will generate an image that doesn't have this vulnerability Okay, yeah So now 3d is detecting that this is a different image from before so it's going to take just a little bit. Yeah This is done. So we can go now with this Very odd runtime dependency this was Helm was one of the reasons why we couldn't use build packs for for this service among with other More rare runtime dependencies again more Bad news to know in which version of Helm This one this Version of container D is bumped. I had to go to the Helm repo searching the issues and see in which pull request the they bumped the the container D version so Again, I know for a fact that in the 381 version of Helm This vulnerable this vulnerability Was mitigated so With that rebuilding We can See if that's Really out. I can't believe everything's working Okay, 3d again. It's going to take a bit and we are left just with the Abulnerabilities It's debatable if Mitigating the vulnerable the compile time vulnerabilities Well if a scanning at the container image level for compile time vulnerabilities is the most efficient way 3d in fact has another option. That's not image That's file system that will look into the palm dot XML instead of the jars But if you ask me, this is just a Good last mile verification so Somebody may know which CV is this is the spring for shell RC and Fortunately We have this very well organized and in In the original service, we had only to we only had to bump this maven property So we could bring the the latest Springboard version You may have noticed that I'm using Java 16 because that's the version that we had in November That's deprecated and And We now use Java Java 17 so Okay So now we should be We should have Get rid of spring for shell and the one that's left is This X stream library So if you know made them If We are using this X stream in our project. We will we could just look for it, but We may we may be bringing it through a spring so Maven has this Plugging called the dependency plug-in with with the gold cold tree and we can see if The X stream is here And it's not Again, if you are if you ever worked if you ever had the fortune of or this grace of working with Maven you also know the Plag the help plug-in and the Effective point goal that's going to merge all the different forms of your build and that's what Maven effectively uses to to run the build so if That's not here If they're not if they're not that's not here We can know for sure that in this project this X stream library is not because maybe it's our source of truth, but In some place of the container image There's this X stream jar because 3d is flagging it so what we can do With 3d is instead of using the table format that just prints the most important bits of information There's an extra flag that we can use to unleash all the information that 3d Collects so with this Jason flag, I'm going to show it. Yeah with this Jason flag We can find here the X stream Library and it says that we are bringing it From the hello plug-in That's in the plug-ins Folder so with this information we can go here look at the docker file slash plug-ins Probably is going to be in Here and we are bringing something from the builder image to this second one by looking at the Name of the folders we Can be pretty sure from which of these Instructions we are bringing that hello hello plug-in But trust me the original docker file wasn't so simple. So we can be sure by doing something like What we did What of what we did in in the previous slides Something like a docker save to a tarp package and then looking to into the the contents for This hash This one the one that starts with for a b a But there's There's a more convenient way by using dive So dive is a layer explorer tool That will give us a ton of information about our image So probably yeah, I have to Sum out a bit sorry for those in the back Yeah so that's a lot of information and well, I love it but If you maybe you can read it in the right it says current layer contents I Usually toggle the unmodified because Having all the files from the Base image Clutters the screen a bit and this way we can have we can see only the the files added or the files That have been modified So in the left we have the layer Explorer the layer details of the one selected and the general image details if Someone here has good eyes Here we are leaking build time secrets because we are using The arg instruction In a way, that's not meant to be used args are just to be used for Environment variables that you want available at build time but not at runtime But we came here for another thing we came here for Knowing in which of the layers we are bringing this hello playing and obviously in the 120 bytes layer shouldn't be so we are now sure that it is in This line of the docker file that we are bringing The plug-in with vulnerabilities so we can go And be sure here Yeah Here it is. So basically what what I had to do in real life is Called the maintainers of the plug-in they they bump the extreme vulnerability and then in the Demo service will have to use a newer version. So TV doesn't flag it So Conclusions of the stock you shouldn't be afraid of Going one extra mile seeing the contents of your images because they are just files and folders Many vulnerabilities can in tools need some fine-tuning to provide a useful output and As you saw mitigating the reported vulnerability. This is very often an easy task You just have to tune the scan set sensible expectations and get the job done So, thank you, Bethes Do you have any questions? There's a mic in the center of the room Do you have trivi implemented in your pipeline? Yeah, we we had I step in see I that just use the trivi CLI Yeah, hi, and thanks for your talk. Thank you. Is there a reason why you had the helm? Binary in the final layer of the docker image because usually I Think you don't need it at all Yeah, that was Tactical temporal thing we are now we we now move that from the orchestrator service and we had it in the Just in the execution. I know how to explain it But yeah, it was a temporal thing now now this Orchestrator service uses build packs. It's just sending the What we want to execute to a tecton cluster and in there it's where we have Helm and the other runtime dependencies Thank you Hi, hi, this might be a bit off topic. I was just wondering why do you use Debian images as your base? Yeah That that was also temporal a we Yeah, I should have asked why to the developers They used these slim version of Debian But for me it was strange also that they use the the open JDK instead of the GRE so I know I Don't have an answer a clear answer for that If I had to guess it's probably for size using the Debian slim pack Yeah, yeah, we make use of a lot of the bit now me containers and I'm responsible for fixing the vulnerabilities And the way I deal with 99% of them as I replace the base image with a boon too. Hmm Yeah Yeah, thank you