 Hey everyone, welcome to the CNCF webinar where we are talking about how to generate Vex documents based on application behavior using Cubescape. Now there might be a lot of words you haven't heard before yet, so I'm going to explain the whole. So be with me in the next minutes. So welcome here. So who am I? Just a few words. I'm Ben, a co-founder at NCT WhatArmo, but as how I'm related to Cubescape, I'm one of the maintainers and project initiators of Cubescape. We'll talk about a little bit further on what is Cubescape. I'm also coming from a security background before coming into the cloud native world if I could talk a little bit in this way. And in general, I love innovation, I love innovative things and I love security. So this is about me. So what is Cubescape? Cubescape is a Kubernetes security project. We started Cubescape from two to half years ago by enabling users to scan their clusters, their YAML files and hand charts from security misconfigurations, security potential security problems based on NSA's hardening guide and the project evolves into a general Kubernetes security posture project which can meet you in different phases of your workflow. It meets you as we started from the Kubernetes runtime side where we are running a cluster and workloads but we are also supporting also the left side. So the development side, you can create even in your visual studio code, we can find potential configuration problems from a security perspective or find vulnerabilities. You can embed Cubescape into your CI-CD processes. You can monitor your cluster and in general it became a security security posture tool finding configuration issues in your cluster, finding image vulnerabilities, finding different aspects through EBPF in your cluster which you need to fix. So just before we are going further, receiving shock and if you don't know what is CVE shock then you might need to do a little bit more explanation from us to that. So one of the best definitions I've heard up until now that CVE shock is actually a state of total helplessness when facing the overwhelming list of CVEs returned by the vulnerability scanners. So this is true, I completely designed on this you know, when we are working with vulnerability scanners finding vulnerabilities in our container images in our clusters, the results most of the time are overwhelming, we have tons of false positives of different vulnerabilities existing in packages inside our containers but most of it, actually what we are looking for is to know whether these vulnerabilities can be exploited by an external attacker or in what ways they can be exploited, we don't know that. So therefore we are getting a bunch of results and we have hard time managing them. So just to give you a feeling about what are the different vulnerabilities and what are the accounts. So we took a few, for the sake of this presentation we took a few well-known applications, containers and applications I guess most of them and you can see that they are today delivering these container images are shipped with tens, if not nearly hundreds of vulnerabilities by the time you're deploying them in your cluster and this is like something that obviously vulnerabilities are not good and we need to keep the limit them but also it's sometimes very, very hard to limit them and fix them or wait for fixes from external vendors like in this case of these images. So sometimes it's very hard to manage them therefore it is very important to understand which of them are really posing a threat or risk to your environment and which of them are not. We had a CubeScape, we started to work on a feature a year ago, a little bit more than a year ago which we called reachability. So reachability is a super interesting feature because we've noticed that most of the vulnerabilities that are returned by the scanners are belonging to packages, software packages inside your container images that are not even opened during the runtime. So this means that when the container images used and running those pieces of software are not even loaded into the memory this means that they cannot be exploited directly by an attacker. This means that when the software is running and accepting packets from the outside world through a network, these software packages are not running and therefore we devised a way to differentiate between vulnerabilities that are belonging to packages which are not run inside an image, inside the container, sorry. And vulnerabilities that are belonging to software packages that are running inside your container. And the way we do it is CubeScape itself uses another CNCF project called Inspector Gadget. Inspector Gadget, if you don't know them, you should. It's an EPPF project, I think one of the best around and they give us visibility over what's happening inside workloads, what is happening inside the workloads and between the kernel. So if you have a pod which is running we can use Inspector Gadget in order to trace what kind of system calls the pods are doing. And if we are looking at what kind of system calls they are doing against the kernel, Inspector Gadget helps us to understand what files are opened during the runtime of the pod. So if we know what files are opened and we know during our vulnerability scanning we get to an S-bomb which we are generating using gripe. We understand what software packages are inside that container image using this tracing information about what files were touched by the pod inside the container helps us to understand which software packages were used and opened during the runtime of a pod. This helps us to differentiate between software packages that are dormant or software packages that are actually used during the runtime of an image. And using this S-bomb differentiation we can only cross check those packages that are used in runtime against the vulnerability scanner and understand and mark those vulnerabilities are what we called relevant or reachable vulnerabilities. These are the vulnerabilities that are actually can be exploited at all and during the runtime of a container image. And this is a very big difference. So just to give you high level understanding given a reddish image we tried during the preparation of the data of this talk we found that there are more than 150 vulnerabilities in the reddish image scan. But if we are subtracting from them the all those who are belonging to software packages that are not loaded into memory as it turns out we have less than if I recall correctly than less than 40 or 30 vulnerabilities. This means like a 20 or more percent reduction to 20 or more percent like reduction by at least 80%. And this brings me to the next point is how to pack this information. So during the last year more and more people are talking about a format called VEX. VEX is around for a few years right now and multiple industry organizations are talking about it. Among them the Linux Foundation OpenSSF and also tech security in CNCF. VEX stands for Vulnerability Exchange Document and it helps you to explain and give a log about what are the vulnerabilities that really matter in your software product. So it's a document format which actually enables you to devise a package that you are using in your software delivery and what is a vulnerability that is bound to the same product and what's the status whether it is something that needs, it's fixed, it's affected or not affected really. So it's something that a vulnerability that matters or doesn't matter. And this is something that we are, we've been started looking at. So how do you get VEX documents? How do you, how do you attain them? That's the hard part. So for a long time people were talking about VEX and I wouldn't say that there is a very clear winning standard of how to describe this document but on the other hand more and more people are understanding the importance and I think that in the, it's a prediction by me but in the upcoming years this is going to be more and more important just like S-Bomb become a standard delivery artifact for software renders. VEX is most likely will become as well because somehow software renders will be, will need to able to communicate vulnerabilities to their users. And, but the problem is that it is very tedious and hard to maintain such a document. You know, every time that there is new vulnerability exposure somewhere in some, one of your dependencies as a software render you have to start to evaluate whether this is, this vulnerability affects your outputs or end product or not. And this is very tedious and a lot of effort involving job to do. Therefore, we thought of the best solution for that would be to start to generating VEX based on, you know, automatically and without human involvement. And therefore we devised way in Cubescape to generate VEX documents automatically for application behavior using our reachability capability which on one hand we are able to tell what are the vulnerabilities inside a container image. On the other hand we are able to say which are among those vulnerabilities which are those who are not loaded into memory. Therefore we can mark them in VEX document as vulnerabilities that are not affecting really the product. And this gives us an automatically a way of generating good VEX documents. So it started nearly a half year ago. There was a discussion with the open SSF team at Linux Foundation about the importance of ideas around it and among the way came up the idea of Cubescape supporting the production of this format. And we started to manage this project immediately. You know, I hear, you can see from the ticket that there were other contributors to this project outside of the original Cubescape team. And you know, we started supporting in the beta version like two months ago I think, not more than two months ago. So as part of this webinar, I want to show you how it works. So let's go there. So you can see I'm going to do a live demo here or at least recorded live demo. So hope that everything works. So the first thing I'm going to do is I'm going to install a Cubescape operator inside my Kubernetes cluster using Helm. As you can see, I've needed to turn on the best generation as a capability. This is something that right now we need to enable explicitly during, you know, the testing of the feature. And when we see that it will work stable for everyone we'll enable it by default for every use case. So I can see that still my containers are starting up. Cubescape is in the starting up phase. So let's wait a second for that. So as the pods are nearly there, they're nearly running. In the meantime, I'm also starting up. Yeah, everything is running here. So I'm starting up a demo deployment and deploying an NGNX from the Kubernetes examples. What will happen right now is this container starts up. Let's check this out in K9S. So you can see that there are two NGNX pods are running in this example. So what's happening is now that during this time, the NodeAgent component in Cubescape operator it does, it starts to monitor the application behavior, trace the application behavior as I told you before and is looking into how the NGNX deployment behaves in the very same time we're doing the NodeAgent is monitoring. Actually, the cubulin here, this pod is this one is creating an S-bomb for the NGNX image running inside this pod. So it is just for you to see. So as you can see, it's a pretty old NGNX is NGNX 1.14.2. And it will must have a lot of vulnerabilities. So during this time, while we are monitoring, you can already start to check whether we already have the VEX document for this component. So let's wait a few seconds for it to come up. So we're watching and waiting. While we're waiting, I'm just giving another explanation. So Cubes, oh, here it comes. So for both pods, we got a VEX document. So let me stop here. So what you see here is actually Kubernetes object, Cubescape emits all its scan results as Kubernetes objects. You can talk to them through the Kubernetes API and get them. So just for you to show you, if I want to try to get this object here, let's try to get it as JSON file. And yeah, here it is. So what you see here is the row VEX document. You can see here, I just showed you that you have here, for example, a CVE, that when we're looking at the impact statement, which is one of the main things in a VEX file, also you can see the status. So the status of this CVE in this image is that it's not affected. It means that in our case, Wilhelmberg component is not loaded into the memory. As I just, we are giving this information to you. But the question is okay, that this is a hard thing and what you can do with it. So as you know, we are going forward. More and more tools we're managing when our bit is will be able to ingest VEX documents. But I want to show you to already how to use that. But even beforehand, let's, I've just emitted this JSON into a file, showing you here, few interesting things. So if I want to check in this JSON file, how many vulnerabilities are what we call affected? You can see that only four of the vulnerabilities in this container image are affected during our tests. And to check out who's are not affected, how many of them are not affected, you can see that 392 vulnerabilities are not part of software packages that are not loaded into the memory and only four of them are. So you can see that it's a huge, huge gain because instead of 396 vulnerabilities that you would need to manage, you have only four. So that is about the demo. For your consideration, think about how do you think that this can help CNCF project? How, you know, how this can drive VEX adoption and to go away, okay, I will be really, it will be very helpful if you could, you know, give feedback on Cubescape and VEX, you know, use our CNCF Slack channel or open an issue, we'd love it to be in contact if this is something that interests you. So thank you very much.