 So welcome everyone, as you can see, I'm not Craig Boggs. Hope you didn't expect him to come, okay, because he couldn't make it here, but for a good reason. And I'm going to present you research with it together about open source security in general and around open source project we well know in the Kubernetes and Cloud-native security space. And we're going to compare them against the security posture of the whole world of other projects. So I don't know who's been to the keynote session today this morning, so Priyanka actually did me a favor, okay? I'm not sure that she knew, but when she started to talk about the security posture and the good management of security issues around open source projects and CNCF projects in general, okay, he actually really did me a favor, okay, because this is one of the things we are going to unwind today and we are going to talk about comparing them one to another based on our statistics we made at Armo. So just a few words about me. I'm Ben, I'm co-founder and CTO at Armo. Armo is a Kubernetes security company. We are the creators of Cubescape. We'll talk about it a little bit later for those who doesn't know. But we are working in general, okay, we are working in the Kubernetes security posture world, okay? In my previous jobs, okay, I used to work in offensive security and also in this defensive security for a long time. So let's just jump in to the presentation. Okay, so Cubescape, which I'm one of the maintainers of, okay, is a Kubernetes security posture tool which tells you, okay, about misconfigurations inside your cluster, how you can improve your security inside your cluster. And it's both a CLI tool and it's both a operator which you can use in different ways. And it also scans your containers inside the Kubernetes cluster and can tell what kind of vulnerabilities your containers have. And it does the same also in your, what we call the shift left, the left side of the world, okay, when you are preparing your Kubernetes deployments, okay, in your GitHub repo, in your VS code and so on, and can scan your GitHub repositories and container registries to find out things about your security postures. There's also very important thing here to note, okay, one of the most important goals of the project is not just to dump the data on you and have you like trying to work out how to solve these issues, but also to prioritize and show you how to fix these issues. ARMO is providing a platform, okay, behind Cubescape in order to store the data of the results of these scans and help you analyze them even further and gives you a bigger perspective, okay, around these issues. Now, what it makes us, okay, it creates a ability, okay, to look into this data, okay, and this is what you're going to analyze today together. Okay, we have a lot of data and we wanted to like create a comparison between projects of the world, okay, and the general population and in our very specific case, the population, okay, I've graduated project inside CNCF and see how they are fair one against the other. So there are two data sources we are going to talk about today. In the ARMO platform, we are receiving data from Cubescapes for, as I told you before, security issues, configuration issues around Kubernetes objects and vulnerabilities from image scans. Okay, the image scans can come either from container registries or the Kubernetes cluster itself and security issues can also come both from Git repos and Kubernetes clusters as well, but we are today going to focus on data, what we are getting, okay, from Git repositories and container registries. Now, about the dataset we are going to talk. So we've seen container images, okay, from nearly 180 registries. We've scanned 44, or nearly 44,000 container images. We've seen nearly 2000 Git repositories and inside those Git repositories, we've scanned YAML files and Helm charts in the size of nearly 164,000. Files, okay, so it's not a small dataset, but it can bring us some really interesting things. So the first time, we are going to look into image vulnerabilities scans. So we're going to, as I told you, we're going to compare two samples again, one to the other. The first sample is the general population. The second sample is the sample of graduated projects, okay? We are going to look at the distribution of severities, okay, if we have vulnerability, how severe they are? We are going to look at what are the most common vulnerabilities in both samples, and we are going to dive into the relevancy of these vulnerabilities. If we're looking at what were the most prevalent image repositories, okay? We are going in the general sample. We are going to see that even the first one of the image repositories is already a CNCF graduated project, okay, Argo CD, which gave us the most image scans, okay, nearly 20,000. You can see that the second is Redis, okay? We all know, although it's an open source project, it is not under the CNCF umbrella. Okay, the third one is also belong to Argo, okay, the same project. Then you have Prometheus, okay, which is also a graduated project. Then you have a very interesting project, okay, also open source, okay, the sidecarp from Kaivi Grid, which is, again, an open source project, not under the CNCF umbrella, but it's also, we saw it in a lot of clusters. In general, it is a tool for those who don't know it. It's a tool to update config maps during the runtime, okay, of a container, update them inside the containers, okay, who are using that config map. So again, Prometheus, we saw a lot of MongoDBs, again, Prometheus and the Datadog agent. So, among the graduated sample, okay, the top repository is obviously Argo, CDM, Prometheus, we already saw, okay, in the general sample, because they were also very high up in the general sample itself. We can see also Kuproxy and Kupestate Metrics Exporter, as part of the top images we've seen in the general sample. Obviously, the general sample has much more images which are unrelated, okay, to open source, and within the graduated samples, okay, this is the setup, okay, and after the first 10 images, okay, you can see another core images of Kubernetes itself. So, the first thing we've compared is the distribution of severities. So, the first one, okay, in this chart, you can see that it is ordered by the general sample and the prevalence of the distribution of the severities inside the general sample. So, you can see that the most, okay, vulnerabilities were medium in the general sample, and in the second was what we call negligible. So, it might confuse you, okay, because there are, not every image scanner uses the severity, okay, but we are, Kupescape is using gripe, okay, and gripe differentiates between low and negligible severities. Negligible is from zero to two in the CVS score and two to four is low. Okay, there are a lot of scanners who are like marking everything from zero to four as low severity, but for ourselves, I think it's more interesting to look also in, on the scale of zero to two. And we already can see that for graduated projects, okay, the most common vulnerability, severity is negligible. So, it's a very low, most of the vulnerabilities are negligibles and the lowest priorities, severities, okay, you can see also that in the general sample, okay, after it, you have the third, most common is high vulnerabilities, and in the graduated projects, there is here something interesting. So, in graduated projects, you have nearly as much high vulnerabilities as medium vulnerabilities, and if you are looking at low severities, we can see that in the graduated project, it is quite few and the general sample is above that. Okay, if we're looking at the critical, okay, the critical is nearly head to head in the both samples, and last but not least, okay, the others severities are belonging to those who are, those vulnerabilities who haven't yet been scored, and therefore they are just like not categorized yet, okay, in the time of when they were scanned. But this can show you already a difference in the distribution, okay, but it doesn't really, honestly, it didn't really make us a lot of sense, okay, yet. So, what we did, okay, we started to look at the top vulnerabilities in the general sample, and we looked at the CVs, and these are the top 10 CVs in the general sample, and you can see that there is one vulnerability which is outstanding, it is relatively new vulnerability in busybox, which was in nearly 37,000 images we've scanned, okay, and it's a high severative vulnerability in busybox, then we saw another, okay, in LibJ crypt, which was like nearly 15,000 high, and then you can see a few which are like, also in around these 15,000 scans, and at the end, from eight and down, you have less. Now, we started to look into each of these vulnerabilities because it was really, really hard, again, to make sense, okay, to understand something better, okay, something more interesting take out of this data, so we started to manually go through these vulnerabilities. Now, this is the time when, you know, my experience kicked in in the sense that I really used to look into these vulnerabilities and report a few CVs myself, so, look into how they are making sense around the world, around the cloud native world. So, I took, you know, the CV, first CV from the list and started to look at it, okay, I don't know how are you in reading CVSS vectors, but I can't, if you are not reading it that well, so I can tell you that this is a vulnerability which can be exploited through network connections and relatively have a big impact, okay, factor, because it can cause a takeover of the client process. Now, I started to read the description and started to read a little bit more about this vulnerability. So, it's a busy box vulnerability. Busy box is, I don't know if you know what busy box is, yeah. So, busy box, if you're using that stat, okay, to read DNS records, okay, an attacker with a mal-crafted DNS record can take over your VT compliant terminal, okay, and alternatively, the attacker also can change the colors of your terminal. So, I have to say it was really funny to read this description, okay, because I don't want to go there, we don't have enough time, but it was really funny. So, I've been around, okay, Linux, okay, and all these network tools for a long time, even more longer than I want to admit, okay, I've never used net stat, okay, to read DNS records. I don't know if there is anyone here who did that, but in general, okay, the attacker, if he wants to exploit this, he needs to access, be able to poison your DNS servers, okay, your DNS records, and have to cause you to read, use net stat to read that DNS record, okay, into a terminal, and in order to really exploit, to create the exploit, the thing which is going to be taken over is actually the terminal process, which doesn't belong to your container, okay, having what it means, in general, if I want to understand what does it mean for a cloud native application, this thing, as a security expert, okay, I will say that well, my ability to exploit this issue is very, very low, okay, so let's go to the next one. Libcrypt, Libcrypt, okay, in specific versions, okay, in very, in specific versions, okay, mishandles a cryptographic algorithm, a symmetric cryptographic algorithm called LJML, okay, and it enables, okay, the attacker from a side channel, okay, to extract the private key, okay, which is actually a serious problem, okay, the point is that the reason why this is so prevalent and so up in our list, in our top 10 list, because it is part of the GPG protocol. Where do we use GPG protocol? Anyone recalls, GPG protocol is used in our package managers, so if we are downloading, installing packages, okay, their signatures are verified with GPG. Are they verified with private keys? No, packages are validated with publicly keys. When I'm signing packages, okay, then I use private key. The very, you know, small percent of container images which contain private key are, which is like a very bad thing to have a private key inside your container, okay, is very small, and the reason why you are getting, this is so prevalent because the package managers you are having in your base images, and this thing, the most, in 99.99% of the container applications, this thing is not running inside your Kubernetes cluster, okay, but you still have it in our top 10. So let's go to something else. SQLite, SQLite, when everyone knows SQLite, right, there is, there was, okay, a vulnerability in SQLite, actually a critical vulnerability which enabled if someone, if an attacker could craft a very specific SQL query, he could take over the SQL, the process which runs SQLite, okay, with this crafted query, okay, which is a real serious thing, but if you think about it again, in order to exploit it, the attacker needed to have an SQL injection beforehand, right, because the way from outside the attacker is coming into your workload is to inject a SQL query, takes, you know, a weight that it has to access and inject this query. So again, it is really, okay, it is not really simple because you already have an SQL injection problem beforehand, and if you are, again, looking at why this is so up in our list, this is so up in our list, because there in SQLite is part of all the Sentos and Red Hat images by default, okay. I don't remember why, maybe it's, again, the package manager, I don't know, okay, but the fact is it's there. No one says that it actually runs in your cloud native application. So again, this is something that's really hard to see how it's going to be exploited if you're looking at it. So I went through, okay, all the top 10 images, and like I had this made up, this really bad okay opinion, but it's like it's trying to make here a point that I only seen two of them, which I can see of the top 10, which can be in some case relevant for cloud native applications. All the others are really hard for me to see how they can exploit it, okay. And again, this is not, I'm not completely like true here, okay, but I'm trying to make a point that most of these things are really, really hard to see how they are exploited, even if at all. And I wanna see here too, one is which is a Zilip vulnerability, which I saw that there are, I think no GS implementations, HTTP server implementations where they are using Zilip with this vulnerability and actually can be exploited in very rare cases, and also in libSTN, ASTN, which is for SSL. Again, not easy to exploit narrowly long, but at least they have some, some percent of them can be exploited. So I looked at also that the 10 most vulnerable, most vulnerabilities in the graduated project sample. And what I saw here, okay, I saw here one which was really, really outstanding, okay, is an issue with Protobuf library, which is a vulnerability which goes dates back to 2015, which was like a high vulnerability, which is around for a long time and was really, I was really curious about it. And I saw here Golang issues and Prometheus issues, which is again, makes sense, okay, because Prometheus is one of the most common images we've used to scan and also Argo. And so on at the end of the list, you can also see, okay, the same libGCrypt and SQLite issues we had before. So I returned to this vulnerability, okay, of Protobuf, okay, which made me think it's really strange that in the graduated projects, we have a high vulnerability for, which is so old. And after a while, I started to look at it, so it's a real issue. And after a while, it turned out that actually, there is a bug open on GRIP, okay, and it turns out that actually GRIP misinterprets the Golang implementation of Protobuf, and it thinks that it's the same as the C implementation, and the vulnerability actually only exists inside the C implementation and not in the Golang implementation. In other words, okay, this is simply as a bad entry, okay, and the scanner creates here a false positive, okay? So this wasn't an issue, but in general, I went through the same list and looked at also I found, okay, that only from a point of view of five of these are, can be vulnerabilities that can be in some circumstances exploited in our world. So then I looked at, okay, what are the stats of per image vulnerabilities? So if I'm taking a random image inside the sample, okay, what are the potential, okay, of number of vulnerabilities, okay, in each category, okay? And you can see that in the critical vulnerabilities around, you know, nearly, I think it was like seven to eight critical vulnerabilities in the general sample, okay, sorry, there is something bad in, I just noticed that in the colors were swapped here. So the black is the general sample, okay, the blue is the graduated project, okay? So the legend is wrong, I have to be fixed. So as you can see, this is something very, very complimenting to the graduated projects, okay? Because if you are taking any random image, okay, from our sample, it most likely have so much more of vulnerabilities than the graduated projects, okay? And it's also the same for what you've seen this morning in Priyanka's keynote. So I thought, well, okay, this is very, very nice, okay? Because we are doing an awesome job in CNCF, okay? But then it made me think, okay, well, what's happening here? Because this can be so good, okay? And it's returning to the same thing we already started to talk about, okay? That a vulnerability image scanner returns is not something that actually an exploitable security issue always. Okay, and then I started to decide to look into, we have a CubeScape, we have a working progress project, okay, called Sniper, which enables you to use EBPF to understand which software packages are loaded into the memory in the container. And using, okay, this information, what we can do is we can filter out the S-bomb, okay, of the container image and only create an S-bomb which only shows you the software packages which are actually inside your container running, okay? And using this filter S-bomb, you can feed the vulnerability scanner and get the result. Now the interesting thing, I am just returning for here a minute, okay, you can see here an NGNX container, okay, the image vulnerability scanner returned near 400 vulnerabilities, but only four of them are in the memory, which is like amazing, okay? And now it took, again, all the top 10 vulnerabilities and now recalculated the images, okay, with this relevancy and what I saw, okay, that actually still graduated projects are much better, okay, in the general sense, but the difference is not so big, okay? So if we had like this 10 to one, like general number beforehand, we can see that if we are looking at general sample after we filtered out all these packages which are not really running inside the containers, we can see the difference is not so big, okay? So you just focus on the critical and high vulnerabilities, you will see that there is no big difference between the number of the vulnerabilities, which kind of, again, made us, okay, think that, well, still, the graduated projects are doing much better, okay, but they are not so far as we thought before. Now, why this is happening, okay, and I think it's an interesting discussion, okay, and we could, like, engineers among ourselves, we could talk a lot about this, but in general, okay, I think that one of the reasons, okay, that in the graduated project, in general, the CNCF projects has much less vulnerabilities, okay, because we are using newer technologies, most of our projects are not, are really created for this ecosystem and they are using Go, okay, they are usually built up on base images which are empty, okay, and not like base images of different Linux distributions which are bringing in their own vulnerabilities even before you've added your own files inside them. So, actually newer technologies are, I think one of the, you know, a big, have a big effect in general, okay, in these numbers, and also, if we are looking, okay, the way we are managing things at CNCF and these projects, I think, the transparency and is very, have a big effect on solving issues really fast and I think there is a really good, you know, we can see that it has a big effect in general. So, about scanning configurations, okay, configurations of Kubernetes objects. So again, what we are going to do is comparing these two samples, okay, of repositories of graduated projects and repositories of the general population, okay, we are going to look at what kind of issues we are seeing in both samples, okay, and what are the most prevalent issues, and then we are going to see, okay, how these issues are affecting, okay, in general, these populations. So, Cubescape uses this concept of controls which are actually, think about them as tests, okay, of different properties of objects. So, if you like, you have, let's take it for, whether you are using an immutable file system in your Kubernetes workload is one test, okay, we are checking, okay, the security context where you have this property, and you can see that the most failed control among the general population, okay, was this immutable file system. So, we've seen a lot, a lot of workloads which didn't use immutable file system. So, it kind of makes sense in a way that immutable file system is not easy, it takes time to configure it, okay, and although it is very good from a security perspective, I wouldn't say that it is the most important, have the most effect on it. So, it kind of makes sense that it's among the first things. It was quite surprising that we've seen, as you can see, the second one was resource limits. There are a lot of workloads which doesn't have resource limits in our sample. We have a lot of who are running as root containers, therefore, they are failing the control of non-root containers, and also around the same numbers are allowing privilege escalations inside, okay, your containers, and also, after it, you can see that a lot of them are not configuring memory limits, resource limits, common labor usage, which I don't think it's a security issue, but in general, I can see that white has a point, and I saw after it that we had a lot of CIS compliance issues down at the end of the sample. It was quite interesting because I didn't understand at the beginning, okay, when I looked at this sample, why we have so few security contexts as opposed to root file system and so on, because I would expect that more people are not configuring their security context than, for example, putting the resource limits, and since I'm looking at the sample of the last three month and Cubescape only released CIS compliance only a month and a half ago, at the end I understood that, actually, this wasn't, you know, the reason for it that they are at the end of the list is because, actually, the sample is tainted in this direction. So on a graduated project, it was very, very, very surprising to see that the most common issue was that resource limits are not configured and after it, okay, readiness probes and immutable file systems a little bit later on memory limits were missing, non-root containers, which is always showing that non-root containers, they're more configured in a graduated project and privilege escalations are more configured, but a lot of them, again, were missing things like readiness and probes and resource limits, which was, again, kind of strange, but as it turns out, this is how it goes. But in general, okay, we've seen that there is this distribution, which is a little bit different, but we have a lot of issues. Now, we try to compare, okay, if randomly we take any workload from both samples, okay, what are the number of controls, how, what are the percent of all our controls which would fail on a randomly picked workload, okay, and it turned out that 35% of our controls would fail if we pick a random workload in the graduated project and 38 in the general sample. And from my point of view, I can say that it's real clear that these numbers are really close, okay, one to each other, so I can say that there is no big difference, okay, so as we could, we saw that in the image vulnerabilities, we saw a big difference between the two samples, okay, of vulnerability distribution and number of vulnerabilities inside the image. In case of the way that the workloads are configured, we can see that they're, it's still better, okay, the graduated sample, but not by far, okay, by a little, so if we are wrapping up, okay, this presentation, it is really okay, it is really hard if I'm getting back to the vulnerability discussion, okay, it's really hard, even though I presented you numbers which are showing you this direction, it's really hard to say that graduated projects are less exploitable, okay, definitely okay, because the very reason because I'm, and I'm sharing you with this, with YouTube and this small number of people we are having here, my opinion that it is, vulnerability scans are really bad today, they are not really representing real exploitability, okay, and therefore all these numbers are, even if I'm taking into account the Sniper results which showed filtered out a lot of issues which cannot be exploited definitely, all with all the vulnerabilities I've stayed with, I didn't have any real number about whether they can be exploited in a real life workload without going to one of them, to each of them and start to check it, so I cannot state for sure, okay, that the graduated projects are less vulnerable, but there is a good hunch that they are less, okay, vulnerable, and I think that, you know, we have to work in general, okay, we have to work a lot of vulnerability scanning to make it much better, okay, and much useful for us. In the misconfiguration side, okay, again, which I already told you, I think that graduate projects has a slight lead, okay, on that, but again, this is not a big, you know, big difference and I think that one of the, this research we are still doing, one thing it made us to think that how we can, we could generate a pull request to graduated projects and in general, okay, open source project to send them, you know, fixes automatically in GitHub, and, you know, we are looking for contributors in the Cubescape projects for those who want to join this effort, yeah, and this was my talk, okay, rated, and thank you very much, and I'm going to, in a half an hour, I'm going to talk in the next room from here about prioritization of workloads and how scan, prioritization of scan results can help you to solve big issues. Thank you very much.