 Hello, zusammen. Bonjour à tous. Welcome, everybody, to our talk, Vaccinating Your Container Images. Today, we will tell you how to use open source tools to make your container images more healthy. But first of all, let me introduce myself. My name is Dina Truxios. I'm working for the Federal Office for Information Security in Bonn, Germany. This is our mission statement. So we are a subordinate authority of the Ministry of the Interior. And we have the luxury situation in Germany that we, as BSI, Federal Office for Information Security, cover everything that has to do with information security, no matter if it's automotive or industrial control systems, the branch I'm working in, vulnerability management, or whatever you can imagine. So what we do is we shape digitization through prevention, detection, and reaction, as said for nearly everybody, which means government, businesses, and society. So with this, I hand over to my co-speaker, because we have a joint talk. Welcome on stage, Jose. Thank you. So hello, everyone. Bonjour, Gatuis. Buenas tardes a todo. My name is Jose Antonio Carmona. I'm a software engineer. I come from Spain. And I've been part of being aware for the past four years now, Broadcom. And actually, more specifically, I'm a part of the Bitnami team. Some of you may know Bitnami. Some of you may don't know Bitnami. So for those of you who don't, let me just say that Bitnami is one of the largest, if not the largest, catalog of open source software application. We currently have around more than 250 applications in it. And we account for more than 1 million deployments per month. So we actually also have the enterprise version of it. Its name is Tensui Application Catalog. It comes with enterprise features like S-bomb and VEX files, which will be covered in the session. And yeah, but basically the main idea behind Bitnami is that you can deploy anything, like in any environment, local cloud, data center, anything like we have virtual machines, we have containers, we have deployment templates. And in any major platform, so if there is one main and key thing that I want you to remember about Bitnami and is really connected to the talk is that we produce a lot of content, like a lot. So with that, and without further ado, let's jump into it. OK, I jump again on the stage. So we promised some content. So this is a very huge number, 29,065. Any ideas what I'm talking about? Oh, you're too good, shit. OK, spoiler, it's not my monthly salary because I'm working for a governmental institution. And we can't afford that. But yes, it's CVEs. So just coming to my next slide. So I tend to not talk about problems. I talk about challenges or even spiky challenges. So one challenge is the amount or the growth in vulnerabilities over the past years. So what you see here is an increase in vulnerabilities. And the number you just saw before was the number of CVEs reported last year. What you also see from the graph, so because I'm a scientist, I trust in numbers. I'm sorry for that. I hope you were with me. We will reach more than 30,000 vulnerabilities or CVEs reported this year. As I'm a very lazy person, I can tell you I don't want to go through all these vulnerabilities manually. I hope you're with me. Cool. So let's automate things. Let's make things better. And I hope that we can convince you how to do it with open source tools. What you also see on the slide are some of my favorite VIP vulnerabilities, like Spectrum, Meltdown, and Lock for Shell. Who entertained him or her themselves with Lock for Shell? Raise your hand. Wow, it works. Cool. OK. So coming to the next challenge, because I don't think that vulnerabilities are the only challenge that we experience. So the next challenge and the spiky challenge is dependencies. And I think we have here the right audience that you understand what I'm talking about. So what you see here is just a small snippet, just a little part of the Kubeflow dependencies. And what do I mean with dependencies? So there is, so the complexity is growing. And just by statistics, when you have 1,000 lines of code, it is possible that you have at least, sorry I don't have a hand, but three vulnerabilities in it just by chance. You can do good coding, you can do best coding, you can do whatever coding just by statistics. They can be security relevant. They are not necessarily relevant. And when we're talking about dependencies, it is a matter where this vulnerability sits. And if it really has an effect on your system, on your components, on your products. So this is the next level of complexity, spiky challenge, that we add to our talk. As I said before, we need to automate things because me, we are lazy. So one thing to fight such dependencies is the usage of S-bombs. Who is aware of S-bombs? Please raise your hand. Oh, wow, great. Fantastic. So it's an audience. It's good. So S-bombs stands as most of you know for soft bubble of materials. And I love to cook, I love to eat, and I love to make analogies between software, hardware, and food. And what you see here is a very local dish. We will come to that. I will spoiler it, Salmorejo. It's a Spanish soup. Yes, it actually comes from my region of Spain, Salmorejo. So when you cook, you know the ingredients. When you write an S-bomb, you have to know the ingredients. So you have to know which libraries you're using and how other dependencies. What you just heard before, which tools are you using and what products are integrated and what pieces of code you have licensed or you haven't. So the cool thing is that an S-bomb also shows the relationship between components. So we also cover the dependencies. But what S-bombs do not cover are vulnerability information. So nonetheless, having an S-bomb shows that you know what you're dealing with, so what you're cooking, what kind of soup you are making. You can apply security because you know when to update, when to patch what to do, which dependencies you have. You are compliant to current and upcoming legislation. I will come to that later. And you can also do risk mitigation. As just stated before, we are lacking vulnerability information here. So let me proceed with the food example, so the analogy. So what you see here is two times salmorejo, the same soup, the same stuff, the same ingredients. One is made by Jose and the other one is made by me. And you also see that there are eggs inside. As stated before, I'm working for a governmental institution and I received a food alert telling me, listen, BSI, there is salmonella contamination in some eggs from some regions. So as a federal person, I'm asking Jose, like Jose, can you tell me if the soup that you made is contaminated with salmonella? Well, I'm really concerned about the salmonella thing. And this is a very serious thing. I'm happy to hear that. But if you're coming to my restaurant, I do have an S-bomb of my ingredients. Oh my god, he has an S-bomb. I'm so happy. I do know where my tomatoes come from, where my eggs come from. And I know that my eggs are coming from a reliable source. It was not an infected market, farm, sorry. So yes, I am very concerned with the salmonella thing. But if you come to my restaurant, there is a notice in the entrance saying that you can safely eat salmonella because my eggs come from a reliable source. So we entertained you with soup, at least I hope so. But you're here to hear some technical stuff. So let's transpose the food example to a real world example. And therefore, I hand over to you. Thank you. So yeah, for that, I will use another real world example. This actually happened to me like a couple of months before. As I said, in the introduction, I come from Bitnami. And in Bitnami, we love to produce content. And as part of the publishing pipeline we used to publish assets, we have a verification stage before anything is published. So in the verification stage, there is one step that actually checks for the vulnerability that are in the containers. So we are using Trevi as the vulnerability scanner. And as I said, this actually happened to me. Let's just say this is a container I've just built. It is based on Bitnami Apache Cassandra. And I've just built it. As I said, let's just scan it with Trevi. And the result says that it has 16 vulnerabilities or 16 entries, which is, wow, quite a lot, right? So if we were to inspect that, it lists some dependencies, like it says the faster XML Jackson and the CVE and also the severity. So what happens if we try to investigate more about that? So if we go to one vulnerability database like the NVD and we search for the CVE, we actually get that it affects faster XML as we saw before, the dependency. But it says that it only affects when certain feature is enabled. So is that feature enabled in our container? Are we really affected by that? If we keep investigating, we finally reach to a point in which we find an official ticket from the Apache Cassandra ticketing system. It is actually marked as resolved. And if you were to read the ticket, it says that it can be suppressed. It can be dismissed. So we've been through a lot of stages. At first, we thought that we were infected by this vulnerability. But we have actually proven that we are not. So don't get me wrong, Trevi was not wrong. We have this dependency inside the container. And this dependency can be affected by the vulnerability. But vulnerabilities and cybersecurity in general, it's kind of a tough thing, tough matter. So it's not easy. It lacks context. And this is really what VEX is about. VEX stands for Vulnerability, Exploitability, Exchange. So now I will stop giving examples and give you the fancy definition. It's a form of security advisory which allows to assert a status of a specific vulnerability in a particular product. That's the fancy definition. As we like to cook, I have cooked my own definition of VEX or the way I like to understand VEX as VEX is just a way to add context to vulnerabilities. So in this case, for example, we wanted to say that we were not affected by this. So here is an excerpt of a VEX file following the CSAP format in which we actually say that. So as you can see it, there is entries in red that actually make reference to the affected file, affected dependency. In this case, it was faster XML, if you recall it. There is also an entry saying the vulnerability that we want to suppress. It's in blue, CV1. The yellow one is interesting. It is the actual statement saying, OK, we know that this is known to not be affected. So we actually state in that. And we can also have some metadata or some other comments that's in green. We can actually say that, OK, Cassandra, consider this something that can be suppressed and linked to the official ticketing system. So some of you may resonate with this and say, OK, this is a security advisory, kind of a security advisory. Yes, VEX is kind of a security advisory, but historically, security advisories were targeted to human, was human-oriented. This one is machine readable. And when you have machine readable things, you can automate things. And automation enables doing cool things. We will cover that a little bit in the demo. But yeah, actually, VEX provides context and actually tells, like, can enable to tell the vulnerability details, analysis, exploitability status, and those things. So there is one more thing that I want to mention. Recall, I always like to say that. VEX is not only meant to just say that something is not affected. As I said before, VEX is all about context. So you can use a VEX statement to actually say that something was fixed, something is affected, and what you can do in order to not be affected, something is not affected, and the reason why it is not affected, or maybe we don't know. Maybe we need to investigate that, but we have to state that we are investigating that. We are aware that we are investigating that. So you can also state that. So I've covered what VEX is. Now I'll hand it over to Dina. She will cover what CSUFF is. Yeah, thank you so much for bringing all the content here. So I hope you're already vaccinated a little bit. You feel vaccinated? Good. So you've heard about VEX, vulnerability, exploitability exchange. You've heard about SBOM. You've heard about dependencies. You've heard about vulnerabilities. Maybe that's nothing new to you. But as Rosé introduced, I'm now going to talk about the jack-of-all-trades things that you need to combine everything that we just mentioned before. It is called CSUFF. CSUFF stands for Common Security Advisory Framework. It is an open standard that was published last year in November, so it is pretty new. It is an open standard by OASIS. This standard has the beauty that it just does not a single thing, but multiple things. So one thing is we talked about automation or the possibility to automate things, to lay back, enjoy your life, don't entertain yourself with too much manual effort, and let the IT security people do what they should do, which is IT security, rather than going through all the pages, crawling for security advisories, assessing them, evaluating them, and prioritizing them, no way. So this is a solution to that. So as said, the standard specifies the format, which is a machine-readable format in JSON. And it also specifies, this is the second part of the standard, the automated retrieval, dissemination, consumption of security information, and also of CISF documents. I think this is a pretty cool thing. As we are here on KubeCon, I can tell you there are several open-source tools already available to expand the CISF version. I'm not kidding. I mean, things that are new need to be pushed. We need contribution. We also need the community. And I'm happy if you raise poll requests, if you raise issues, if you contribute to that, if you use it for it, make it better. When you demand and promote CISF, we can expand this universe and use it and automate things. So CISF was the first standard that included VEX information. So CISF has in total five profiles. One of them is, for example, the informational advisory. So things like, okay, there is some issue with wiring, just really wire things, which has nothing to do with security. There are security advisories, and there is also the VEX information that Jose just talked about. So basically, as said, a jack of all trades. How does CISF look like? So here you see an example of a classical part of a CISF document, of a CISF advisory. In blue, you see the document level metadata in the brownish-yellowish part, which is really cool. You see a product tree. So you can enlarge it, and you can go down to a single version. You can list all the products. You can also have relationships between hardware and software. This is what you can communicate in this machine-readable advisory. And you also have the vulnerabilities linked to each and every product, and also the information, what your management or an authority is interested in, like, are we affected or not? How much money does it cost, blah, blah. And also the mitigation measures, the usual communication why you use security information. Having a proper asset management, which is a prerequisite for using such things, you can automate things. You will only consume information that is relevant for you, and find out whether your product is affected, not affected, or whatsoever, as you have seen with the statuses of VEX. Sorry for that. I have to entertain you with legislation. Are you aware of the Cyber Resilience Act? Please raise your hand. Wow. I'm impressed. So the Cyber Resilience Act is an upcoming legislation. It is a market access regulation that will demand cybersecurity or security, information security, in products with digital elements, no matter if it's software or hardware. So whenever you do any business in the European Union, CRA, if you don't, maybe no CRA. So the CRA does not only require requirements across the whole life cycle, but it also requires an SBOM. This is literally written in the law. So you have to have an SBOM in order to be compliant. There will be market surveillance from authorities, whoever that might be. They can ask you for an SBOM to prove that your Salmorejo is not contaminated with Salmonella, and most important, you have to have vulnerability management. And why am I telling you this? I told you, use CSF. This is one solution to be compliant to current and upcoming legislation. So just nine days ago, the Cyber Resilience Act was approved by the European Parliament. So it is coming. And maybe one last thing, I don't want to entertain you with legislation. I know how to entertain people with paper and myself. But so the Cyber Resilience Act is immediately enforceable. Then it's too directive, it's a directive and has to be transposed into national law. That one is there, and you have to be compliant to that whenever it's there. Okay, enough about paper, enough about soup. Let's do some life cooking, and I'm really excited. Yes, we'll do some live demo. Well, I can use this one. No, it's okay. This is a live demo. Oh, let me just... Yeah, yeah, I'm making it bigger. It's a live demo. Can you see it properly now? Yeah. So it's now, okay, it's now enabled. So the demo is actually what I've covered before. Let's just say I have a container. This is, it really is a simple container. Here I have the docker file. So let's just print the docker file. And as you see, it's really a simple container. I'm actually using PhotonOS. You might be familiar with that. PhotonOS is an optimized OS for containers coming from VMware. I'm using that because it reduces the footprint of CVEs. It's also hardened and everything. But it's really simple. I mean, I am pulling and downloading one library. You, based on the raised hands before, I guess you're familiar with the library. It's Apache Log4j. It's an old version. And I'm just unzipping the library. So if we build a container, it's an innocent library really. Now let's just scan it with 3B and see what happens. Well, yes, I guess it's too big. But yeah, it has identified six entries. Two vulnerabilities, one of them being high. Now, as we are the developers of this container, we know from a fact that this does not affect us. So we might just create a CISA file that states that this does not affect our product. So I have already a template of a CISA file here. You can see that it has metadata at the top. There's also the product tree branches and everything. This is pre-filled for me before the demo. But actually, you have an entry here identifying the product that was flagged. This case, Apache Log4j. And down there, we have the entries for the vulnerabilities. So in this case, the CV here, I just paste the CV that we want to suppress. And very important, the product status for this one, as I said before. In this case, it is known not to be affected. And yeah, you can relate this to the product we defined before here. So let me just save this. Yeah, can you? Thank you. So now if I scan it with 3V again, 3V allows for a new tag to pass in a CISA, a CISA, yeah, a VEX file. We have actually contributed to 3V, we were to support that CISA format. So, boom. Now the vulnerability does no longer appear. And if you were to inspect the logs here, it says that the vulnerability was found, but it's actually filtered out because there is a VEX file indicating that the status is not affected. So you can get an idea of how useful it is to have a catalog come in with these CISA files, with the VEX files. So this is actually what you get with TensorFlow Application Catalog. And that was really what we had to show you today. It was a brief introduction, but if you have any questions, if you want to also come and say hi, will we be more than happy? Yeah, so we are not biting, at least not initially. So as we could not cover the whole CISA version and everything that is there, you can reach out to csafe.io and find all the tools and open source tools and upcoming things. So there is some more information, particularly for CISAF, and for sure also left some information about Bitnami, and we are now happy to take your questions because we have some minutes left and it would be really great if you use a microphone because then it is also recorded and people who are looking at the videos later can also listen to your questions. Okay. So the microphone. So thank you for the presentation. If you go back to the VEX file, I see that the product status is actually an object. So I guess you could put multiple statuses there. How does that work exactly? And also it contained like an array of some code. So what do these represent? I mean, in the demo, I guess you refer to the demo one, right? I cannot open the demo one. Yeah, the demo one. So VIM and or FX. Do you refer to this one, right? I don't see it here. Where it said status not affected. Oh, the status not affected. Yeah, this one. Okay, yeah. So a vulnerability can actually affect more than one product. So here you are referencing like the products that are known not to be affected. So let's just say I have another dependency that is also affected by CVE, this one, this big number. So instead of redefining another entry that makes reference to that CVE, I just can put it here. So this is P001 makes reference to the product I have defined above. This case you see here that says Apache log for J and the product ID is P001. So if we had P002, like let's just say another, I don't know, another dependency, yes, we can define it there as an array. Okay, it's clear now. Thank you. Okay, so maybe I can add something more. So this is really helpful, especially when you have different version numbers and you have to discriminate whether a particular version number or a version range is affected. So this can be shown here. So this is the intention behind that. Yeah, we are happy to take your questions. Cool. What do you know about the coverage of this standard on distributions like Debian, for example. So you get CDs for a product and then you are using Debian and you need to check also Debian advisories. Are they already supporting it so that you can easily filter out? So hopefully I got your question correctly. If not, please correct me. So as I said, the standard is more or less new. It's like 17 months until the Big Bang, or since Big Bang. There are several vendors and manufacturers, bigger ones as well, that are already publishing all their advisories in the CISA format. So you can switch from, there are also some, let's say advisory viewers, where you can switch from the JSON file to a human readable file. There are several vendors that are, let's say converting their human readable or extending the human readable to machine readable ones. But for sure, not everyone is yet using CISA because this is not in the law yet. But as I said, if you demand and promote CISA and you ask for that, and this is what we are currently doing because we have kind of an aggregator that is mirroring everyone who is publishing these advisories where you can consume them from in order to automate the process, it is getting more and more and it's coming. So I'm very positive about that, but for sure not everyone is doing it. So I would be happy if you push into that direction. Thank you. Thank you. Very informative presentation. Thank you for that. How do you feel about malicious actors providing this context, writing their own vague statements? This is, I think, a very, very good question. Thank you for asking us that question. I totally agree that there is malicious potential, I will not say as with everything, but particularly here. Anybody, really? Maybe they have the good intent, but not the right context. I know, if they do it wrong, they just publish something unintentionally that is wrong. Yes, for sure, to be honest, this is not yet solved. Maybe there are some ideas how to, let's say, approve things better or to have a more trustful, let's say, environment, but I think this is something that we can all solve together. So we need swarm intelligence to maybe think about that and make things better. So I did not go into details. There are several roles how to publish CISA with different, let's say, security or levels of trust, which would compensate for parts of it, but the unintentional case, for sure, we will not cover it with that. Yeah. Thank you. Let me also add that it also has to do with how trustful your sources are. So if you pull a container or you get the container from Bitnami, you also can get the CISA files. So if you trust Bitnami from containers, you can also trust that the information is coming from us. Right, thank you. Yeah, and this is why I mentioned this aggregator that we mirror all things and that we, but nonetheless, we can't say it's 100% secure, which we can't ever. Hi, thank you again for the great presentation. One question is, Bitnami willing to provide these VEX files for their containers? And the second question would be, are there tools in place already to create these VEX files because they are machine-readable? They should be also machine-vitable? So for the Bitnami part, I would say that Bitnami is already providing the VEX file, but as part of the enterprise catalog. So actually in the Densu application catalog, not on the open source version. So for the other, do you want to? We are sharing, so we are also sharing the answer. So as just mentioned before, VEX includes several, CESAF includes several profiles, including VEX. So there is, for example, a tool that also proves for standard conformity for CESAF and you can use the, so for this foundation, yeah. And it's called SAC Visogram. And you can use it to produce CESAF with VEX profiles in it. And this would be one tool that you could use to produce these machine-readable advisories that have conformity with the CESAF standard 2.0. Thank you. Hi, I have a question for Bitnami, maybe. Is Bitnami doing something in direction of hardening the images because your images have shell in it, everything in it, which is useless. How then in what? Can you repeat that again? Shell. Shell. Yeah. Are you doing anything in field of hardening the images like chain guard is doing? Do you know chain guard? Chain guard. I know chain guard, yeah. I don't think I am the right person to know because I don't know the plans really in January to answer that question, really. Thank you. But maybe just come to us here on the stage because I'm really sorry. I'm not sure we will take the last question but I think this question is a very good question and I'm happy that you asked it. So just come by and we can discuss it here. And maybe we just take one last question. So three, two, one, go. It's you. Thank you. So it's not a non-technical question. You talked about cyber resilience act, right? And it will be imposed by you. Do we know when? And then regarding the S-bomb, is it if I have 1,000 images in my registry, should I do it for every image? Even if there are dependencies between the images? So thank you for this question and I'm happy to also take non-technical questions which are related to techniques. So as just said in my, or just mentioned in my talk nine days ago, the Cyber Resilience Act as it is, I think it was proposed in 2022 and it was approved by European Parliament nine days ago. So it has to pass, let's say some minor decisions until it is entered into force and then you have like a time span until it enters into force. So I think it is. So some of the stuff is not clear yet because it has to be done by standardization. So the Cyber Resilience Act will be an umbrella and it depends on your products and on everything that you use, how to behave. But the frustrating answer from my side, I mean for God's sake or yeah, I'm not deciding on that, but worst case scenario would be that you have S-bombs for everything that you're using and you have to show it to the market surveillance authority upon request, especially in case if they have any doubts. So maybe to calm this a bit down, I think the initial intention was more like the talking, pet sitting, child care, self-ordering cleaning robot that the European Commission was afraid of having such a thing in your house and being a security or being a risk in terms of information security. So it is not likely that every product is that heavily regulated. Thanks. You're welcome. So as we run out of time, if you have any other questions, you are more than welcome to come by here. We can just answer them here. I can, if you have all the specific questions like you, I can redirect you to someone you can speak with. So yeah, that was it. Thank you very much. Thank you.