 will seem to have trickled in already and we'll let the others join in as they come in. My name is Raghavan Srinivas. I'm a Lead Container Developer Advocate at Sneak. I'll be moderating today's webinar and I'd like to welcome our presenters today. That's Matt Ferina, who's a health maintainer at Samsung SDS and Haley Denbraver, who's the Developer Advocate at Sneak. A couple of housekeeping items before we get started. During the webinar, you will not be able to talk as an attendee. There is a Q&A box at the bottom of your screen. So if you pull it up, you should be able to drop your questions and we will try to kind of monitor it as we go. We will have some time at the end for doing the Q&A as well. This is an official webinar of the CNCF and is subject to the CNCF Code of Conduct. And essentially what that means is just be respectful to each other, to fellow participants and presenters. The recording of the webinars will be posted later today at the CNCF webinar page, which is at cncf.io vac webinars. With that, let's jump straight in. My name is Raghavan Srinivas and basically I work as a Lead Container Developer Advocate at Sneak. I worked at Sun for a long time. So I'm a Java guy, but I've been doing container stuff and Helm as well. I come from a Cloud Foundry background, which means I expect things to just work and I think Helm makes it possible. But the onus is also on the developer to make sure that Helm does not, the Helm charts and all that do not have any vulnerabilities and so on. And what Matt and Haley will be doing during this webinar is kind of address some of those issues. So having said that, let me turn it over to Matt. Matt, please take it over. Hi, my name is Matt Farina. I am one of the Helm maintainers. There are many Helm maintainers. In fact, there's three of us named Matt. I'm the co-chair of Kubernetes SIG apps and I used to chair Kubernetes SIG architecture. I've done many software libraries. I'm a former open stacker. I used to use Cloud Foundry a lot. So I'm incredibly familiar. I've used Drupal and contributed to those and been working in upstream open source for more than 15 years now. I currently work over at the Cloud Native Computing Team at Samsung SDS. And I'm gonna kick things off here after Haley introduced herself talking about Helm security from a Helm maintainers point of view and a charts maintainers point of view. So with that, Haley introduce yourself. So my name is Haley Denbraver. You can find me on Twitter at the link there. I'm a developer advocate at Sneak and my particular passions have been security education and Python security but I'm new to the CNCF ecosystem and happy to be here. I also wrote a report last fall about the state of security within Helm charts themselves. So we're gonna be taking a look at some of the findings from that report and that'll be a bit later. So back to Matt. All right, so before we get into the report, I wanted to talk about Helm and charts and security with Helm itself. Before that, it's good to kind of level set on what Helm is. Lots of people have lots of ideas and as a Helm project, Helm is a package manager for Kubernetes. You can think of it like app, yum, homebrew or chocolatey. If you're not familiar, that's the Windows package manager. But in this case, it's for Kubernetes clusters because Kubernetes is like a cluster operating system to run your applications with schedulers and things like that. And so Helm is the equivalent to those but for Kubernetes. And so what is a package manager? Because package managers are really useful, right? I use them on my desktop systems. I use them in my clusters and I've been using them for a long time. And this is my definition of what a package manager is because it's about enabling. It's tooling that enables somebody who has knowledge about an application, right? Maybe you know Postgres and you know it really well and you're gonna package that up so that somebody else who doesn't have all of that business logic and knowledge on it can pick up that package and install it and use it without needing to learn all of that domain knowledge. I remember a lot of times where I use something like apt and I said apt get install and it was my SQL or Maria or some package. And I didn't know the exact configuration files that needed to live everywhere on the system or the exact business logic or the right commands to start up. I didn't know that. How do you get the root password? You know, when you're installing this and put it in the right place. I didn't know all of that. And yet I as a consumer was able to use those packages and get a lot done. But somebody who did was able to. And so that's the real power of what a package manager does. It allows somebody who knows that knowledge to take and make an application accessible to others. So I wanna talk about a few ways that this is important and really useful to people. The first is obviously taking a package up like Maria, Postgres, WordPress or many of the other things out there. And one or a few people packaging it up so the masses can go forth and use it. That's a lot of the traditional case for package managers and what you see with the Helm stable repo incubator repository, the Helm hub and other cases. But then there's the case of say, you're at an organization and you've got a complicated set of applications and you need to install that somewhere. Maybe it's somebody who you're passing it off to your different developers because they need to have the other microservices they don't work on and install that into their local Kubernetes clusters. Maybe you've got a complicated application and you need to deploy that into Kubernetes just as part of your CI or your DevOps or whatever else you've got going on. Package managers lets you do that. In fact, even before Helm and Kubernetes I used to use apt this way with older systems whether it's public clouds, infrastructure as a service, VMs, this was the way we use package managers back in the day and the same still holds true with Helm now. And then of course some of us deploy continuously and some of us have cycles where you've got to take a package and a version of something and move it from dev to QA, it's QA, that version gets launched into production and charts and packages let you go through that really well because you've got an instance of something in time that you can take to these different environments, pass them off to different people. Now, many of us are living on the edge of CI CD and you can use Helm there too but I do understand that some of these other environments exist as well still and many people use them. Another way that you can use Helm is actually as a building block for other tools. In this case, I've grabbed a diagram that comes from Flux. Flux is another project in the CNCF right now and Flux is GitOps and so there's a Helm controller you can have that runs in your cluster that gets configuration updates from Git and applies them to your cluster and it can do that for Helm charts. You can do both Helm 2 and Helm 3 by pulling in information and using it. And so this is a case where you can use Helm as a building block as part of higher level constructs and so Helm can very much be used in those places as well. And so that's kind of gives kind of the core building blocks of what we look at when we look at Helm and a lot of the security applies to that but I do know that people use Helm in other cases. It's the reason we have Helm template. Helm template is a function that originally started as a plugin but turned out to be really popular and it can take a Helm chart and turn it into a Kubernetes manifest that you can then take and use in other places. It basically templateizes, treats it as a template and then spits the output out so you can do other things with it. And when we look at this case, it's useful and it made its way into Helm and I love that people use it but many of the core security features weren't designed with this case in mind. They were very much designed with package management in mind and when you start to see what those look like, you'll see it. Now there's some things that still work here but not everything works in some of the cases that stretch Helm into a template system which is a little different than the normal package management system. So let's kind of level set for a minute here. One of the things that I like to level set on is what some of the terms are so that way we can understand how they're useful. And the first thing is kind of a hash or a digest. You can read the definition up here which comes from Wikipedia but in a practical sense, it lets you see the content that you got is what you expected. It doesn't say where it came from. It doesn't say who created it. A lot of times you'll go get a hash and you'll get the thing from a downloading server and then when you get it you can compare the hash to what you downloaded. And one of the, what you can see then is did what I download change in between because if something in the middle of downloading it over the network, over the internet, transformed it or modified it, checking in the hash will fail. And so you can see that but it doesn't say anything about how it got where it's going. And a good example of this here is if you go look at a Kubernetes release. Here's release version 1.17.3. And if you go look at the release notes you can get the Kubernetes source and you get two hashes. One is the SHA-256, the other is the SHA-512 of those files. And so if you download it from GitHub you can now compare it to the hash to make sure you got what you got. But is this enough? Is this all that we need? And in KubeCon, CloudNativeCon last November, Justin Kapos, Justin Kapos is a professor at NYU who works on lots of security projects. And Justin did a pop quiz. And so I'm stealing his pop quiz here even though I know you can't answer here of what do these organizations have in common, right? These are all organizations that have security teams or have had security teams over time are known for handling their security fairly well. And one of the things they all have in common is that their users were attacked via some issue with software updates which is a problem. And you can look at ways that those vulnerabilities can happen, right? If somebody gets into a server and swaps out a hash and a file, then you can download it and you can check the hash but it may not be from the actual team. It was maliciously put there. And there's lots of ways where the software process can be hacked. Now Justin talks about Tuffin and Toto, other CNCF projects designed to help alleviate those problems on a larger scale. And so we're gonna talk a little bit about how do you go beyond that today? And one of the most known forms of being able to do this is what's called provenance. When I first started getting into security I had not heard of provenance but the whole idea is being able to track the lineage of what you've got. Where did what you get come from? Did it come from who you said it was going to be? And this is different from a hash or a digest because it doesn't just let you see as the content what I expected to get. It's did it come from who I expected it to come from? And that's important because if somebody else swapped something out from a security standpoint you can check, did it come from who I expected? No, it didn't. And that's what provenance tells you. And here's a provenance example. If you go look at this is for the Linux kernel. If you go to get the kernel you're gonna see a tarball and a PGP signature. And that PGP signatures where they signed the package and it's signed by one of the maintainers of Linux. Who it's signed by. They've got documentation on this. And so you can actually get their public key and check the signature and make sure that what you got came from who you expected it to come from. So what does this mean to Helm and containers and charts and all of that stuff? I know it was level set. How do we actually apply this? And the first thing we can look at is downloading Helm. Here is a snapshot of the release notes from Helm and it's a section of the release notes that we're gonna go download Helm. And you can see you can download it for different files. You can also download a checksum which contains the hash. And you can see the hash is here for each of the different files. And if you look at the bottom this release was signed with. This is a signature for a key. This was me in this case that can be used to actually validate the download. So you can check, is it what I expected and who did it come from? In fact, Helm does this with all of our releases. So one of the very first things as far as Helm security goes is you can check to make sure when you download Helm, did you get what you expected and from whom you expected? And an easy way to do this, you can see highlighted here. I'm using Wget in these things to fetch things on a POSIX system. You can see we download the Linux version of Helm and we download the checksum. And if you actually check the link for the checksum on the last slide, it would be to the same file dot SHA 256 sum. And then at the bottom there, there's a command run. This is on Mac, this is on Linux, it's on POSIX systems called SHA-SUM. And in this case, we say dash A, which tells us it's a SHA 256. And then we pass in the SHA-SUM file and the SHA-SUM file lists different files and their SHA's and it checks it. And here it says, hey, that file is okay. It checks out. It is what you expected to get, right? And then if you look at a failure example here, if it didn't match, it would tell you not that it's okay. It would tell you that it failed and you'd be actually be able to see that it's because the checksum didn't match and the exit code. If you ever wanna look at the exit code for an application, you can use echo dollars like question mark. It'll tell you the exit code and you get a non-zero exit code. So if you use these commands inside of your CI system and it wasn't what you expected, your CI system would fail if you're always downloading Helm and CI saying, oh no, this isn't what we expected to see, there's a problem. And you get a non-zero exit code and you can actually see you're downloading Helm and there's a problem. But this gets into hashes and diagests. Let's talk about signatures. If you go and look at the release notes for Helm, for every release, not the release candidates, but the releases, you'll see that there's a dot ASC file for every single file we have. And those contain the signatures. And so you'll see, if you look here, there's for the tar, there's for the zips, in the case of a zip file, you also see SHA-SUM-256 has a signature. And you're gonna see another file that's just a SHA-256. If you're using SHA-SUM, there's some extra material in the file, not just the SHA. And so we do have a legacy file that just contains the SHA that people were using, but we recommend they use the SHA-256-SUM file with the SHA-SUM commands now. And so these are the signatures and they're uploaded to GitHub because that provides them from an entirely different domain hosted in a different place through different processes, right? This gives you the ability to get your signatures from someplace entirely different from your files. So if one location ever got was ever hacked or something nefarious happened there, the other location isn't, your stuff would still, it would fail, right? Nobody could get both locations. It was very difficult for somebody to get both locations to stop that. And so we do have our signatures here. And you can do things such as here on the first one, you can go ahead and get my key because I signed it. And then you can see you download the file and then you download the signature. And so I'm using curl here. So you can actually see using curl to download and get things. And then the last line, you can use GPG to verify the file against its signature and it'll tell you, hey, the signature, it's from Matt just like you expected, it checks out, okay? And you can actually see now, it's not just what you expected to get but it actually comes from one of the maintainers on the Helm team. It came from who you expected it to come from. And if the signature didn't go well, here's an example of what happens if it fails, right? It's gonna tell you it's a bad signature or a bad problem and it's gonna give you a non-zero exit code. So if you're using it in CI or someplace else, it's gonna fail, you can see it fail. And there's lots of ways you can get keys. On the top here, you can see four ways to get my public key because my key is gonna be the same everywhere that you look. And that's because keys have a fingerprint. And here you can see the GPG command to see my key on my fingerprint. And earlier on in the release notes, I pointed out the fingerprint was in the key. So you can verify, it came from who you expected it to come from, the key. There's the whole verifiable loop. And in granted, this is more security than most folks use, but it gives you those assurances that it comes from who you'd expect it to come from. And if you look at the four ways to get a key, if you look at the third one down there, you're actually in GitHub, everybody who does releases for Helm has their public key in a keys file at the root of the repository that anybody can get important to GPG and use it in your verifications. So let's talk about charts. We've talked about now about getting Helm and securing Helm itself. Let's actually talk about the charts themselves and what goes in those. Now, what goes in charts is part securing the chart and securing the content. So we're gonna talk about both of those. And one of them is let's start with charts and registries because that's normally the recommended way to distribute your charts. And it's a core way that provides some security. You can get it elsewhere, but it provides some security features. And so here I've just got an imaginary chart foo and I package it up and you can see it's packaged into foo zero dot one dot zero dot TGZ. And Helm repo index creates an index file. You can see this is often. So here we've created a very simple application and repository. We'll say we uploaded this to a static web server. And so that's kind of our case from this point on to level set with we've got a package. And so in a normal case, you would use something like Helm pull or Helm fetch or Helm install. And you would tell it where the repository was and the name of the chart here. My repository is called demo repo. And in this case, if I use Helm pull it just pulls down the package. And if I look at it, I've got that tar file that I can expand, I can install, I can do things with. But that isn't really a way to secure it. You can actually just pass it around this way. Well, it turns out that charts can be signed and there's kind of two methods you can use to cryptographically sign a chart. Helm package that originally creates that tar file for you has the ability to use PGP, pretty good privacy, to sign the chart. And so if you tell it to sign and you point it at a key and a key ring and a key ring, it can go ahead and sign the key ring. Now PGP is a little different from GPG. GPG GNU-PG is the GNU implementation of that. And here, so you've got the two different implementations. And so Helm knows how to talk PGP because that's what the libraries are, but it doesn't talk GPG. So in the second instance here, we have a plugin and you can see how to install a plugin because Helm has plugin support, Helm plugin install. And this one was created by Matt Butcher, Technoselfos. He is one of the co-creators of Helm, one of the other mats. And so you can package up a chart and then you can go ahead and sign that chart. And when you sign a chart here, if I do, I'll ask, and this is the original structure where I have the chart, the tar file, I now have a provenance file, a .prov file. And that provenance file contains information. And so we'll take a look at what that information is. If you look in a provenance file, there's kind of three important pieces of information here. First you see the information, the API version, app version. This is the metadata that comes out of the chart.yml file. The next thing down you have is you have the SHA, the SHA-256. And this is your verification of is the file match a hash? It doesn't say anything about who it came from, right? If a nefarious attacker gets into a server you downloaded from, changes out the SHA file and changes out the actual contents of the package, you could download it, you could do a SHA some on it, you'd see it all passes. Because a SHA tells you you got what was on the other end of it, it doesn't tell you who it came from. And then you have the signature, which is where a person or automation signed it using cryptography, using a key, right? And that key, they sign it with a private key, then you get their public key and you can use that to verify it came from who signed it and who said it came from, right? Even if the server, somebody hacked into it and swapped out the SHA some and even swapped out the provenance file, if you know which key to expect and it comes from a different key, not the one you were expecting, you can see, hey, this isn't who it came from, it didn't come from who I expected it to come from. And so when you go to verify files, if you use Helm pull and use dash dash prov, you can see, hey, I pull a provenance file. And here you can use Helm GPG verify and say, did it come from who I expect, right? And here you can see, this is who it came from, the SHA matches and it's good. And with GPG, you can actually output a public key for somebody into a file and then use that. And so here's a command where I could take a key out of GPG and export it into a file called keyring.pgp and you can pick and choose the keys you want in this file. And so if you only expect it to come from one or two or three people, you can create a PGP file for just that. And then when you do Helm pull verify and you point it at the key ring, it'll only check the package comes from people listed in that key ring file. And so if it somehow got swapped out or changed on the other end and it was signed by somebody who's not one of the keys listed in that file, it'll fail verification. And so the whole idea here is you can see who it's coming from. So let's talk about security in the charts. And now we've talked about getting charts between places and being able to verify who it comes from. There's security in the charts. The first thing when it comes to security in the charts and these are Kubernetes things, you should be doing irrespective of whether it's with Helm or other places. And the first one is actually limit the resource usage to just what your application needs using resource quotas. Because one of the ways that you can attack a system is to cause a service or a microservice to use lots of other resources which impacts the other things and causes other disruptions. And if you limit the resources of your pods, then you limit the ability from an attacker to cause resource problems in one service to affect other services in your cluster. And so the first thing is use resource quotas. That's one way to limit an attack factor. The next is actually use secrets and not config maps to store configuration. Secrets are really powerful and I'm gonna dig into a little bit of depth on them to show you what kind of power you can get in your applications. Because usually we just use them as another form of config maps. And you may think of them as storing things in a base 64 encoded manner, which isn't all that secure. It's not encrypting them or anything like that. It just stops them from displaying in terminals and other places like that very quickly. Exactly what the secret is. Base 64 encoding is not a level of security here. But use secrets to store your credentials instead of config maps. And secrets have this really neat power. You see secrets are typed, not in a validation sense but you can actually specify a type on a secret when you create it. Here's an example of a secret file I created and we're looking at the output of the contents. And you can see there's a type line at the top level that's different from the data that's stored in it. And you can actually use tools. And in the lower section here, I'm using kube control to get my secrets within a namespace. And I actually tell it just my type and I could have 30 different kinds of secrets in this namespace. I could have 50 secrets and it's only showing me the secrets of this type. Because the ability to query and understand structure is built into secrets and secret types. In fact, this is one of the things Helm 3 now uses by default to store its information in a cluster. It uses secrets. And we use secrets instead of CRDs and CRs or other things because they have the capability of being more secure which I'll talk about in a minute. As well as they're available to everybody. Not everybody has the ability to install CRDs in a cluster. In fact, lots of people don't because of security controls and lockdowns for clusters. And so they're widely accessible in the ability for people to use them and install them in clusters. But the other neat thing about secrets is there are actually ways cluster administrators can set up secrets to be encrypted at rest. Here you can see there's a documentation page I'm pointing you to in Kubernetes that talks about how you can do this. So it's not just base 64 but when they're at rest they can be stored in key management systems in a totally encrypted way. Then there's RBAC, right? Roles, cluster roles, cluster role bindings. You can do things like RBAC in order to limit the ability of that application to do things within that cluster to make changes, to talk to things. And it provides you the ability to kind of lock things down. So if something were to go wrong it again limits the surface area for those attacks. And one of the Kubernetes maintainers, Jordan Leggett has created this wonderful tool called audit to RBAC that allows Kubernetes it allows you to point it at audit logs run an application and build your RBAC policies from that. So in a development environment you can run your application, exercise your application and create those RBAC policies that now you can deploy via charts into production and in other environments and help lock down your applications. Then there's the use of service accounts. You can use service accounts to say who is the user that's doing whatever it is inside of the pods. By default, your applications in a namespace get a default service account. But here you can actually specify application specific service accounts. So you can limit and control, okay, this application is this service account with these policies under RBAC. It just helps you limit the surface area for what can happen for things coming out of those containers. And pod security policies, right? This is the last big one I'm gonna say pod security policies provide a way and people do package these up in charts for their applications so that you can specify what the pods can do. Are they even allowed to run in a privileged mode? Are, you know, what kinds of things can they do? And so it gives you that ability to again control some of those policies. So that way, if something does happen somebody is able to get in and mess with things they're limited again in surface area. And all these things are things that are normal, good Kubernetes things that we should do. They can all be packaged up in charts. And if you go look in things like the stable and incubator, mostly the stable repository you'll find there are charts doing many of these things today. And so the next thing I wanna talk about is image security because, you know they're securing helm and how you get helm to your system and making sure it came from who it says. Then there's the charts and how we can pass charts around and do that in a secure manner and making sure nothing nefarious happens in between at their sources. And then we look at Kubernetes, right? We can say, how do we get the configuration right? But your workload is ultimately running in images and containers based on those images. And so we need to talk about image security here and how you can have good security around that. And one of the first things that I learned was there was a paper published a few years ago now about Docker Hub and what they found was among the many things that they found in Docker Hub is over a period of time. And here they talk about days the number of problems found inside of images. And you're actually going to find that there's stuff all over these images that are bad and are problematic and have issues. There's CVEs in so many of the images. There's a security expert who I'm friends with who points out he goes, give it a couple of days and there's a CVE in your image because they're being discovered all the time. Software's insecure and we need to look at how we can have processes and policies and tooling to help us with that. So the first thing I want to say is scan your container images and fix CVEs. It's one of those things that we can implement in tools and I'm going to talk about that here. And it's something that can make a world of difference in practice and locking down potential vulnerabilities. And so we're going to look at two ways to do this. The first I want to look at is Quay. Quay is a repository other than Docker Hub and it's one of many options out there that you can use to do this. That you can use to store your images in whether it's on-premise or not. The CNCF has Harbor. There's lots of tools out there you can do this. But I like the visualizations as far as I was talking in a presentation. And here if you go to Quay and you've uploaded an image it can actually tell you about the vulnerabilities in that image. So in this case I grabbed the Kubernetes ingress controller image and I grabbed an old version of it not the latest because the latest doesn't have any security vulnerabilities in it. And you can see Quay is telling us there's 79 vulnerabilities and 10 of them can be easily fixed. And here it's listing out those ones that can be easily fixed. And so Quay as a container registry actually will help you discover when there are vulnerabilities in there. And they're not the only one other registries do I just like the visualizations of this one. And if you actually look and expand to not just the 10 that are easily fixed the all 79 of them you get a nice long list of here's the layers, here's the CVE because you can actually look up the CVE learn more about it and understand it. Because as I said before it said or shows that 10 of them have patches available and the latest version of this image there's no vulnerabilities in it. And that's because all of them could somehow be handled in some way shape or form even beyond what the system says. And by understanding the CVEs you can learn how to fix those. But the neat thing about Quay here is it actually has the ability to notify you when a vulnerability is found. And in the settings you can actually go and say hey notify me email a webhook webhooks are great if you want to write some other form of automation on top of it slack notification. And you can actually set the vulnerability level. So if one of vulnerability comes in at a high level or whatever you can maybe get an email notification and they'll tell you when vulnerabilities come in so you can go off and start acting on fixing them. And so I think this is a wonderful feature again Quay isn't the only one that has it but I like the pages I could show you. Then the other way is Sneak who is gonna be talking here in just a minute but they actually provide tools and one of those tools that you get allows you to analyze your stuff in a different way from when it's in the registry. And I wanted to talk about this other method of analyzing and that's in your CI workflow. So one of the things you can do with Sneak and on GitHub itself is they have a GitHub action. GitHub actions if you haven't used them are the CI system for GitHub. They're like a Jenkins or a Circle CI or a Travis CI but it's baked right into GitHub itself. And they have an action which is a step you can import into your workflows to use Sneak to scan for vulnerabilities. And one of the ways it can scan for vulnerabilities is in Docker files. And so on the left, it's in the marketplace for GitHub where you can see Docker and Ruby and PHP and all these things and how it can scan. You can see Docker and on the right side is an example in their docs. If you go look of how to do that configuration in your GitHub action to pull this in and scan your image. And then as part of doing pushes to your repository to your pull requests, the images can be scanned and problems can be found at that point. And so that's really what I wanted to show you about Sneak. So before we jump to Hailey, did anybody have any questions? And I know there was a question about what's the difference or preference in terms of PGP and Chawson verification. I've said it a bunch of times and I tried to incorporate it in, but the difference is with PGP, you know who it came from. No matter where it comes from, you know it came from somebody. So if Linus signs the Linux kernel and you download it from Linux or you download it from a mirror or if somebody hacks into a server and it replaces it with their own nefarious file, you can always check whether it came from him, right? It allows you to see who it came from, not just the source. And if a source gets compromised, well, you can detect that compromise. With a Chawson, you can't detect the compromise because if they swap out the Chawson on the other side, but it will tell you if something has been changed in between, like in the network traffic, if there's a proxy somewhere that detects it, changes it to their own file, you can detect certain things like that with Chawson verification because it said did what came from the other side come, you know, was what I expected. But if the other side was compromised, you know, what you got for a Chawson is compromised. So I would suggest that a PGP provides a much stronger guarantee of who it came from than something like a Chawson. Were there any other questions? If not, I'll hand it off to Hailey and she can talk about a bunch more fun stuff. All right, thanks, Matt. So I'm going to share my screen here. Just give me a second. So let's see and presenting. Does that look good to everybody? Looks good. Yeah, okay. So I want to open what I have to say with just a bit of context about CNCF's investment in security. And there were some exciting things in the last year and two that I found on my radar were the security audit for Kubernetes and the helm security audit. And the fact that they open source these and so you can actually go and take a look and see what was found from a security perspective for these two technologies. And in fact, CNCF has a commitment to require this kind of security audit for project graduation. So that's a part of a project going from incubating to graduated. And just real quick to review, Helm is the package manager for Kubernetes and we learned a little bit earlier about what that means and how you can use that. And I think Matt did a really good job about breaking down the different ways security impacts a package manager. And so you have to consider the manager itself. You have to consider the packages and or in this case charts and how you get them. And then you need to consider the base images that those charts are built upon. And that's gonna be my focus today. So last fall, I researched and wrote a report on helm chart security. And you can find it here at this link, sneak.io slash helm dash report. And I'd encourage you all to go check it out. It has some insights on vulnerabilities that we found in all the stable helm charts. So to give you some context on how this was put together, this is the helm charts repo up on GitHub. And you can see here that you have two different kinds of charts. First is incubating charts. And then the second is stable charts. So this report is only concerned with the stable charts that were there as of October of last year. And so this is what that repo looks like. So we went through and scanned every chart that was in this public repository. And we did that using a tool that an open source tool that sneak built last fall called HelmSneak. And if you go to the helm report, sneak.io slash helm dash report, there will be a link to this and you'll be able to use it yourself, try it yourself. So that's kind of how it was put together. And I just wanted to quickly share a few findings. And was it not? Okay, wanted to share a few findings and we'll kick that off now. So the first thing is that we found 277 stable helm charts at the time. And more than half of them, 68% of the stable helm charts were using an image that contained a high severity vulnerability. So if you are using a public helm chart, you most likely have a high severity vulnerability in your project. And it's important to know that and it's important to be able to act upon it, right? And so more about these images that contained these vulnerabilities. There were 416 images throughout the charts. So most charts had a few images that they were using. So you get more than 277 or whatnot, right? And what I found really interesting and kind of good news, bad news sort of way is that six images accounted for nearly half of the vulnerabilities found in our study. And the other 410 images accounted for the other half. So most of these charts are using images that have a high severity vulnerability, but it might not actually be that difficult to remediate because you can like highly target what to address, if that makes sense. So 15% of these charts utilized the bats image and that was the image we found that contributed the most vulnerabilities. It's a popular testing tool. So even though it contributed a number of vulnerabilities it might be difficult to actually exploit or compromise in a meaningful way. So that's kind of good news too. So a little more on the vulnerabilities themselves. The most common types were out of bounds, reads or writes, access restriction bypasses and null pointer dereference. And I'm going to talk really briefly about that in a few slides just to give you some background of the most common problems that we're cropping up. And then in total, more than 40,000 vulnerabilities for phones in these 277 charts, but each vulnerability is only being counted once per image when it appears. Meaning that sometimes things would crop up twice in a single chart, that kind of thing. So we didn't want to double count. So the good news is that there's a lot you can do to make your use of these charts and of these images even better. 176 stable helm charts can benefit from an upgrade. So there is something that can be done. And in total, there are 261 image upgrades that can be made across like ecosystem wide to help improve the security of helm charts. So this is a breakdown of the vulnerabilities that we found. And you can see here, nearly half of them are high and almost the other half is medium. We've found very few low severity vulnerabilities. And then this chart will show you that there were a number of different types of vulnerabilities. The 30% in the dark blue is just kind of everything else. And then the rest of the colors represent single issues. So out of bounds accounted for more than a third. And then access restriction bypass and null pointuring reference made about another 20% combined. Now to take a look at those just really high level brief overview. So in out of bounds vulnerability can take two different forms. First is a out of bounds read. And the risk there is that your data is exposed. So this basically means that someone is able to see something that maybe they shouldn't be able to see. And then the second type of out of bounds vulnerability is a right vulnerability. And this one can be potentially more hazardous. You can corrupt the data. You can crash the data. You can maybe execute code. And the two of these together account for more than a third of the vulnerabilities we found. And then the second highest amount is access restriction bypass. And this generally occurs when the user identity isn't checked correctly. So maybe you're getting around permission somehow. The user's action may not be logged correctly. So there could be bad actors that are kind of flying under the radar and doing things that you're trying not to let them do. And you may not have the footprints to prove that they were there. So that's the class, the second most common vulnerability found. And then the third is a null pointer dereference. And this happens when a pointer with the value null is treated as though it pointed to a valid memory area. And generally this is thought of as a reliability concern because this can cause things to crash. But it also has the potential to be a security issue because depending on how things are situated, a user might be able to see the stack trace or print out logging. And that information can always give them a clue to your system and they can further penetrate it. And so that's why we generally want to avoid these as well. So yeah, I hope that gave you a taste of what's in the report. I'd encourage you to check it out. And just some final thoughts from me. And I would say my takeaway from this report, writing it last fall, was that it's always better to know than to not know, right? You wanna know what's in your images. You want to be able to make a conscious choice as to whether it matches your risk tolerance or you wanna be able to remediate, but you don't want to go into something not knowing. And I think that Matt's presentation jived really well with that as well because it was all about knowing where things come from. How do you trust something? And so yeah, I really appreciated his perspective on that too. And I think at this point, we'd like to direct you to some resources. And again, these slides are gonna be posted. So you will have access to all these links. And I would say too, that if you're going to KUKON.eu, both Rags and I have a talk and Rags has a panel. So please consider coming and checking it out. And at that point, I think we may switch over to questions. Rags, do you? Thank you. Yeah. Just give me a second. Like I said before, even though if you can please put any of your questions in the Q and A box, we'll get to as many as we can. There's a question about except for static scanning, does sneak provide dynamic scanning policy to cover SAST and DAST? Yeah, I'm not so sure that. Do you know Rags? I know we have some monitoring in production, but the vast majority of it is designed to be part of your CI CD before you deploy. Yeah, I mean, you can, for example, look for misconfigurations and in the Kubernetes config, because as they say, Kubernetes is not really secured by default, right? So, we do have some static analysis tools too, which kind of tries to shift it as left as possible, right? But at the same time, you can also kind of monitor them in production as well. I don't know if that answered your question, Faye. So let me, I think we still have time for a few questions. I really wanna thank Matt and Haley for a really great presentation. I learned quite a bit myself. We do have time for questions, just drop it in the Q&A tab, like I said before. We don't need to go for till the top of the hour if you don't have any questions. But feel free to either put them in now or interact with us where Twitter or whatever you deem fit. And like what Haley said and like what I said before, the recordings will be available later for you to view sometime as early as today. Right, and all the links and things to relevant stuff will be up there. All right, so let me kind of hold for a couple of minutes to see if there are any questions. Any other closing thoughts from Matt or Haley? You know, in the meantime, see if we'll monitor the Q&A box anyway. I don't see anything right now, but... Matt looks like he has something to say. I was just gonna say, if you had more questions about helm security, feel free to come by the helm issue queue and just ask a question over there. We have somebody every week who goes through and we triage them usually on a day-by-day basis except for maybe weekends. And so somebody will get around and attempt to answer your questions or point you in the right direction if you've got more questions about it. Great. One, two, three. I think we've done questions. I really wanna thank the foundation CNCF for providing us with this opportunity. I really wanna thank everyone for participating, you know, the questions that were asked. And I really wanna thank Matt and Haley for a great presentation. So that's all we got time for today and thanks again for joining us. Have a good day and talk to you soon. Cheers everybody.