 Hello, everyone. First of all, this is a very good time slot. Usually, all my talks are after lunch, and people are sleeping very well in my talks. So I hope that right now it's a better time slot, and you will be more engaged. But maybe it's me. Let's see. So we're going to talk today about prioritization of vulnerabilities. And I think this is going to be, I'm going to show you a few new things, or maybe you're already knowing them, which is good. But I think that we have to start to look at vulnerability prioritization in a different way. So just a little word before I start. This talk was meant to be given by Oshrat, who is a dev rail at Armo. And sorry, she couldn't make it because of the situation at home at Israel. But shout out to him for making this presentation happen. So a question here. Who is in a CV shock? Any hand? Whoa, love hands. So the second question is, OK, what's a CV shock? And the third thing here is that you can see this definition from Twitter, which is totally understandable. And to be frank, it is kind of funny story, which I want to go inside. But it turns out that I had already talked two years ago about using this term. And later on, someone used the same term. So I was really happy to see that on Twitter, someone coined, used my term, which I coined. But in general, the idea is that CVs, which were vulnerabilities which we are getting for different vulnerabilities, are really overwhelming and it's really hard to maintain our software with them. So just a few words about me. I'm Ben, co-founder and CTO at Armo. Also a CNCF project maintainer and contributor to other projects coming from the security world. This is more or less about me. Happy to chat with anyone if you want to talk more. So when today, we are looking for CVs, looking for vulnerabilities. How do we do that? And as I see it, since a lot of people of you are in shock, I'm sure that you are doing already in different places. Now, many of us are using different public images. And actually, these public images will make sense a little bit later more, but just to give you a hint, we are talking about these public images because they are very frequently used in many organizations. So I'm sure that you know all of the names here, but you can see that they are very, very widely adopted. And we are scanning for them for vulnerabilities to manage our security posture, but for more to manage our security exposure. But if we're going through these public images, we can see that we all know that they are coming with caveats. These caveats are the security issues they are coming with them. So we are enjoying all these open source softwares and publicly available technologies, but they are coming with their issues themselves, and we need to handle them. Now, if you're looking at the same images, what are the number of vulnerabilities here? And there are different versions of this, I know, and different times. So this is just a snapshot of a given version and a given time that I just brought this together with Osha to give you a hint of the idea. So you can understand that you have tens if not more than hundreds of vulnerabilities which are coming with these, no vulnerabilities that are coming with these images. And it's really hard to manage them. Now, where are we looking for vulnerabilities? We are looking for vulnerabilities in our images, in CI-CD pipeline, when we are building our own images, or when we are delivering it to our registry or inside our registry. But as I see, and as we see it, one of the most important things to do behind, and of course, it's very important to block images with no critical vulnerabilities in your CI-CD pipeline, but it's even more important to monitor the vulnerabilities you have in your production software. Why? Because every day there are more than 100 new CVs which are being published by NVD. It means that every day that your software is running more in your production system, there is a higher chance that there is a vulnerability that you are not knowing of or you missed during the delivery time of your software, and it's lurking around in your software and attackers can exploit it. So it's really important to monitor the actual happening. But to tell the heart's truth is that there is a clear difference between vulnerabilities and exploitations. So vulnerability can exist in your workloads, but there is a good chance, a very, very good chance, and I don't want to go into numbers, but it's an overwhelming number of them cannot be exploited. It means that, from my point of view, as a guy who's coming from the security field, I don't see a point in investing to fix them too much because why to fix something that cannot hurt me? And everyone of us know that everyone who practiced this knows that fixing vulnerabilities and updating images comes at a great cost. And today, also as an entrepreneur who are seeing how much time my DevOps is investing into fixing vulnerabilities in our production system, I can say that I only want to focus on those vulnerabilities, which can really hurt my system. And there is a big difference. So how do I get to this? How can I say about the vulnerability? How can I prioritize the vulnerabilities? What to fix and what to not? So there is a great deal of the context where your application is running. So can the attacker come from the public internet and use that API, which is vulnerable? This is one question. It depends on your setup. There are multiple things. Whether this vulnerability is reachable for the attacker in the sense that whether the software runs using our scanning containers and will return to this thing a little bit later. But there are a lot of software in your containers which are actually not used. And if it's not used, and it's really hard to exploit them. So there are multiple factors going into the prioritization. And it takes time and takes resources in our work to decide them. And if you have tens or if not hundreds of vulnerabilities per image, all these checks and tests are taking time. If you are coming from a big company who are investing in a lot, you might have security analysts in your team who are doing this is their daily job to do that. But if you're coming from another big company, you don't have that. You don't have simply that function in your company whose job is to go after and deal with all these things. But you still have, you might have exploitable vulnerabilities. But for sure, you have vulnerabilities in your images. But you just don't know what to do then. So the first and most classical way to prioritize vulnerabilities right is CVSS. I don't think that anyone who's used any kind of vulnerability scanner doesn't know this technology. CVSS is a way to describe the effects of vulnerability and give them some kind of a score for prioritization. NVD, if you're talking today, they're saying that it was never really meant to be a very good prioritization metric. But in a practice, there is nothing else. There is up until not so long ago, no one really came up with something that is much better than this. And the way it works is that when I find a security issue of vulnerability in some software, I'm as someone who found a lot of security issues and submitted it to NVD report. I'm just self-filling out a questionnaire about different aspects of vulnerability, whether the attacker needs to have network access or the attacker needs to exploit the vulnerability or it needs to be run on the same machine, whether the attacker needs user interaction or not, whether this vulnerability affects the confidentiality, the integrity, and the availability of the software. Depending on these parameters, we are getting a number between the range of 0 and 10. And obviously, 10 is the worst. You have to, I would say, you could be dealing with them first. And 0 is kind of OK because it's most like it's not a real vulnerability at all. But there hasn't been a widely adopted any other metric to deal with this. So there has been, lately, there was a new way to see whether there are exploits for vulnerability. So there is KEV, which is also by CISA. It's known for non-exploited vulnerabilities. It's a list of vulnerabilities which they know that there is their reports, that in the outside, there are actual exploits targeting this vulnerability. So it's a very, very good way to describe whether your vulnerability can be exploited because someone filled a report that this vulnerability was used in an actual attack or there is no exploit example out there in Metasploit or any other kind of tool. And if you know that this vulnerability is exploited out in the wild, then obviously, this is something that you have to take into account when you're prioritizing your vulnerabilities. So we'll think that's cool, really great. I have KEV. I can disregard CVSS. I need to solve all those vulnerabilities in my production system which are in the KEV database. And all the others are, for now, OK, because they are not normally exploited. Now the sad truth is, and this is something that's coming from our user statistics, these are customers where you see that under a total column you see a number of vulnerabilities in their system. And you can see that on the column of P0, you can see that out of the 23,000 vulnerabilities, only 41 of them are in a KEV database. So you can see that there is a very low number of actual vulnerabilities in the KEV database. I don't know what's the latest number, but if I'm combining into, we are at KubeCon, right? And we are talking mostly about cloud native applications. So I can safely say that there are few vulnerabilities which were submitted into KEV database which are related to our environment. And this is like, obviously, if you have some kind of vulnerability which is in KEV, you should be running to fix it. But on the other hand, there are too few vulnerabilities there, so I wouldn't suggest using it just as an only parameter for prioritization. So there has been a great deal of work lately around EPSS. EPSS is a new way to prioritize vulnerabilities. EPSS is a machine learning-based scoring system for vulnerabilities where I'm not going to go inside machine learning, among other reasons, because I myself don't understand it. But they are taking into a lot of parameters around the vulnerabilities, which are then translated into a number between 0 and 1. This is a fuzzy output, meaning that there is no 0, 1 exploiter or not exploitable. But the exploitation output is in a scale. And in this scale, the higher you are in the scale, the more possibility that the vulnerability can be actually exploited. And what they mean is that they are trying to understand from the scoring and from all the vulnerability parameters what's the chance that this vulnerability is going to be exploited in the near future. It sounds really, really good. It's really interesting. Myself, I have some doubts about it. And it's just because maybe because I myself don't understand it. But it turns out that actually it's not really a bad scoring system. And I will show you a little bit later why I got to this conclusion. But I want to see you something that will show you. And I think it's one of the main lines of my presentation today. And so we'll be taking time going over it. I'm going to explain why EPSS is a good tool and what they are trying to solve. So you can see on the left side, you can see how the scoring system worked with CVSS. So you can see that I would say the light blueish gray circle representing all the CVs in my system. And out of the system, I put a threshold, or actually this is from the EPSS documentation. And they put a threshold of they want to solve out of all of my security, all of my vulnerabilities. They want to solve only those who have CVSS score 8.8 and above. They said that this is an effort of solving 253 vulnerabilities out of 1,000. And then to say the thing, they also took into account which of those, manually they created statistics, which out of those can be really exploited. And the reddish pinkish circle shows those vulnerabilities that can be out of the old vulnerabilities can be exploited. You can see that around half of these, they've covered nearly half of these vulnerabilities by the threshold. So there are 50% coverage of all the exploited vulnerabilities. They say they are fixing by giving this threshold. And it gives them an efficiency of 5%. Because the point here is that they are investing a lot of time, a lot of time fixing a lot of vulnerabilities. And still, they are only covering only half of the vulnerabilities that they should have been from the beginning. So out of the 1,000 vulnerabilities, only 5% they hit out of all the exploited vulnerabilities. So if we are going to the two EPSS, different versions of the EPSS scoring system, you can see a big difference in the way they have the effort they put in. Why? Because they put another threshold, obviously a threshold between 0 and 1. And they said that they are going to fix in the first version 93 out of the 1,000. And they are still covering around 50% of the exploitable vulnerabilities. And this gives them an efficiency of nearly 13%. If we are going to EPSS v.2 when they give another threshold, and they are only fixing 47% out of the 1,000, they still have a coverage of fixing among all the really exploitable vulnerabilities. They are fixing, again, nearly half of them. But they were much more efficient because among all those they have fixed, they hit 42.5% of the exploited vulnerabilities. So I know it's hard to understand for the first time, but what you have to take from here is this. They created a scoring system where they are saying, we are better identifying the exploitable vulnerabilities. We might not covering them all of them. But with a much less effort using the scoring system, you can still cover all the same thing you used to cover with CVSS. So with CVSS, with a really high threshold, you worked really, really hard to fix hundreds of vulnerabilities. Now you are fixing only below 50 vulnerabilities, but you are getting the same coverage, which is actually a saving time for your company and for your workforce, which is a big difference. So here I want to introduce you to something we at Armo and Cubes came up with. We call it reachability. We used to call it relevancy, but then I saw that people don't understand what I'm talking about. So I started a Twitter thread about how to call this feature, and actually the first time in my life that the Twitter discussion was actually fruitful. So we changed the name to reachability. And the way it works is that we are using eBPF to identify which software components are loaded into the memory during the runtime. And using this information to filter out all these vulnerabilities which are not loaded into the memory. And since they are not loaded into the memory, they cannot be exploited. So now I want to go through a research and statistics we created. So again, I'm returning to these well-known software I've showed you at the beginning. So just to consider Redis, I don't remember actually which version of the Redis this was. This looks like an old version because it has more than 150 vulnerabilities inside. You can see on the left side that what is the static scanner output gives us. Again, we see a bunch of vulnerabilities, like 150 vulnerabilities. But if we are only considering those who's been loaded into the memory, we can see that we get considerably less of the vulnerabilities because most of the components which were vulnerable are not loaded into memory. They can come from different places. They can come from unused models in the case. But the most common case is actually coming from the base images. You know, these products are not yet coming maybe from scratch or these robust images. Therefore, the OS-based images are loaded with a lot of vulnerabilities. And it's really hard to disregard them. And through this technology, you can disregard them. But wait, because I want to show you a few more. So in NGNX, in the testbeds, we've used only 25% of the vulnerabilities were reachable, actually. And you can see that only with other components, I'm just deliberately going a little bit faster. So with Argo CD, you can see that less than half of them are actually reachable. There is a huge difference between different kinds of applications that are, for example, Go applications, which are built into monolithic binaries, tend to result in more components that are loaded into the memory. But for example, CC++ applications or Java applications, which have tons of modules or applications with very bloated base images tend to show more effectiveness if we are talking about reachability. Just for example, MySQL see that less than 20% of their static vulnerabilities are actually loaded into the memory and, again, saving time. So going after these things. So seeing that also some cross-checking, but see that there is a great variance among all these applications. And the reason why it's very, very important to monitor also the runtime is to actually detect what is really happening inside our applications. So we cannot really build up heretics, which are general, to every application. Just another example of Kafka and Jenkins, where you can see that less than 60% of them are reachable. You can see that in Grafana, we brought a difference between two big versions, 9 and 10, where there is a huge difference in the reachability of their vulnerabilities. You can see that actually the number of unreachable vulnerabilities has shrunk between these two versions. And the reason for that they have swapped some of the things in the base image. Now, getting back really to EPSS. So beforehand, I told you that EPSS is a very effective way to address vulnerabilities in comparison with what we saw before with CVSS. So we showed that using EPSS, you can invest much less time in fixing vulnerabilities, but get more or less the same results. Now, if I'm comparing EPSS and the reachability, there is a difference between these two. So I can already show you that there are vulnerabilities in EPSS, which are above the threshold of 10%. They are not loaded into memory. And it makes sense because, again, sometimes there are applications and maybe exploitable things that are simply not used in the container image during runtime. And EPSS will never be able to tell it, because EPSS doesn't know this information. Therefore, what they are doing is they are based on the description of image. They understand how exploitable it is in general. But on the other hand, you can see that if I'm taking all those who are reachable and only concentrating on them, I might also cover a little bit more vulnerabilities that maybe would be exploitable, because actually reachability is a great way to filter out a lot of noise, but it doesn't make sure that it doesn't prove to you that the actual thing is not a false positive. It might also trigger you saying things that, well, this vulnerability is loaded into the memory, but there are, since as I told you in the beginning of the talk, there are multiple factors of going inside where actual vulnerability can be exploited. The reachability is only one of them. And if we are focusing only on the reachable, which is considerably less than all the vulnerabilities, it still might cause us to work more than we should. So in general, what we are trying to do in the near future is to combine all of these data sources and enable the cloud native users to prioritize even better using both reachability, CVSS if they want, but also EPSS in the way they are prioritizing their vulnerabilities. So again, to limit the investment of the users of how many things they have to fix in their environment. Just a few words about how this reachability calculation is working. We are based on the Cubescape project. Cubescape project was started by Armo, and today Cubescape project is part of CNCF. It's a sandbox project, and we are eyeing incubation next year. The way Cubescape is working, it installs a node agent. This node agent is using Inspector Gadget, which is yet another great awesome CNCF project, to hook into different EBPF events. It uses file activity information to understand what files are opened in each of the workloads that are running in our Kubernetes cluster. And this information enables us to cross-check the S-bomb together with this information and see what are the software packages that were touched by the runtime and understand that if they are loaded into the memory of not. So if a software package, a Python package, was opened, we'll know it from the EBPF that given Python file was opened. We'll take the Python file and see if it's in the list of the S-bomb, and it belongs to one of the packages. And if it belongs to one of the packages, then we'll mark that package and say that this package is in use, and it's reachable. And when we are feeding the vulnerability scanner with the S-bomb, we are feeding it with a filtered S-bomb, where you only see in the S-bomb only the packages which were touched by the runtime of the container. This is how it works, and this is how we're doing it. So within the ARM-O platform, where you see the results of Cubescape, you can already see that it may be a little bit small. But for every vulnerability, you can see whether they are reachable or not. Hopefully, very soon, we'll be also able to enrich it in the next two months with also the EPS score. And you will be able to filter out, based on these two factors, only the vulnerabilities you think that you should obtain. And you will see that it causes a major difference in our daily work when we need to fix vulnerabilities. So as a conclusion, I think that there is a clear problem definition around CV assets and around overwhelming numbers of vulnerabilities. This problem has been around for a while. And as we are getting all the IT communities getting more mature and we are trying to make our security posture and exposure better, we understand that we have to invest a lot of time. But on the other hand, sometimes this time is going to waste because we are doing things which are not really helping us. Therefore, NIST also sees our trying to come up with new proposals like CAV, like EPSS. And we, as a part of the cloud native community, we are also trying to do our best to be able to have us focus on the most important things and solving things that really matter and not trying to go after all the false positives. Thank you very much. If you want to meet Armo, you can meet us at booth F12. We'll be happy to discuss it if you're interested. If you want to use the open source Cubescape is available for everyone. We'd also love to have feedback on that as well. If you want to give feedback, use this QR code. And thank you very much. Are we taking questions? Yeah, yeah. If there are any questions, I would love to answer. Yeah. Question, do you know if any of companies in the regulated kind of industries are picking up using reachability and EPSS? Like I'm in health care. Yeah, yeah. So there are multiple, there are two kinds of regulated industries. One is that where you have to define your only regulation and you have to stand by, right? And there are the others who are really like told to not to have SLAs for fixing CVS score 7 plus. So as part of the letter kind, no. I don't know of any kind of. These things are too new to get into the hardly regulated areas. But again, I think that there is going to be, where these things are going to prove themselves. And I think that they are already proving themselves. So I really think that in 2024, there is going to be a big push on the regulators to take into these things into account. So I guess there will be a change. In the self-regulated areas, it really depends on the understanding of the security teams. And again, since there are so many times it's going to waste, therefore I do really think that there will be a high pressure everywhere to adopt these things. I hope so. Yeah, I agree. Hi, thanks for the great talk. I have an in the weeds question and a higher level question. In the weeds question is, can you comment on does the application need to be? How long does it need to run in order to get a reliable understanding of it? And one specific use case we have is Argo workflows, which are wonderful. And they do some work for us. But they run for varying lengths of time. So yeah. So great question. So I don't have a magic number. I don't have a magic number. And I can tell you that there were companies and people who tried to do reachability even in the past two or three years. But they were, because of different product reasons, the way they implemented it, that they started to monitor applications not from the beginning of the application. And therefore they missed a lot of information. And they didn't filter right. So at least when the way we are talking about reachability is that we are saying that we have to see the application from the start. And I cannot tell you for sure because there is no answer to how much we need to see. I can comment that in our Armor Production environment, we are saying that around 10 minutes from the beginning, we are expecting the results to be fairly accurate. But we are also monitoring the whole runtime. So even if a module is being loaded a month after the pod start, we'll get notified that it become reachable and to address the issue. So OK, that's great. So for the workflows, like I was describing, they're typically short life cycles. So the product would see the whole life cycle. So it would know. But for an application, you're finding that 10 minutes is a reasonable rule of thumb, mostly accurate picture. Is that right? Yeah, so this is. Are there any significant variances between languages? So I don't have a very, very specific what I can say. But I say that usually the reason we chose 10 minutes because our system tests, for example, are running less than 10 minutes and we are expecting the code coverage around 90 something percent. So we said that since we are getting to pretty high code coverage in our tests, it's really hard to think that something is going to be loaded into the memory. But you are like, for example, Argo, and not just Argo. Think about Jenkins, right? Jenkins, like month after the deployment, I can install a new plugin and something will become reachable month after it. So there is this special kind of applications which are really problematic from this behavior analyst thing. And the only thing I can say is to always monitor your environment. Thank you. See there are not other questions. So brief one. You told us about the new scoring system. I was wondering if you could just expand on how the conversation is in the industry and the community when first hearing about it, when digesting it, comparing it to CVSS. So you're asking about TPSS? Yes, about people's acceptance, people's mental journey to accept it, things like that. So we've been running for tests on EPSS for six months. And we see some adoption by commercial companies who already put it in their environment and producing. And after a month of tests, we decided that we are going to join the working group who's working on the EPSS. I cannot tell that how. The only thing I can say that from all of my discussions, people are looking at it as something that is so important for them that they want everyone to accept it. So everyone sees that the problem is so dire that they are so waiting for something really good to happen that they are ready to hear about everything new in this perspective. So it will really depend, I think, the security industry, the decision makers, which will take time. It will take time. As we talked about it before, the highly regulated industries, it will take time for them to accept it. But since I think that since CISA is behind that, it gives a fairly good start. So just like, and I really don't want to go into security politics, but head like an open source or organization would come up with this and push this. I'm not sure it would succeed. But since CISA is pushing it, I have a good feeling about that that is going to help. And by the way, CISA is not saying that it replaces CVSS. It says that it should be used together with CVSS together. Thank you very much.