 Hi, and thanks for joining the CNCF webinar about Cubescape. I'm Craig Box, I'm the VP of open source and community at Armor, the original creators of Cubescape, and the company that contributed it to the CNCF. Today, I'm going to tell you a little bit about the Cubescape platform, what we at Armor who built on top of it, and what we're doing now, and in the next few months to make the product better for everybody. Let's start by asking the reasonable question of, why do we even care about security? This is the first of what I like to call the scary numbers section. IBM Research last year says the average cost of a ransomware attack was $4.54 million, and that doesn't even include the cost of the ransom. There's an entire industry predicated on providing scary numbers, this should be fun. 97% of organizations say they have concern about Kubernetes security. Looking at what risks people are most worried about, the highest number were people worried about misconfigurations or exposures in their environment, followed by vulnerabilities, and only then actual outside attacks. That tracks to what people have actually seen in the last 12 months. Red Hat Survey says that 53% of its respondents had detected a misconfiguration in their environment that caused a security flaw. 38% had a major vulnerability that they needed to remediate. Gartner, by some measure the king of scary numbers, say that 99% of breaches in 2025 are going to be traceable to customer mistakes. So some of you will have an inner monologue that's going, I am but a developer, I don't have to worry about any of this. I have an entire team of professionals whose job it is to make sure that I don't have to worry about any of this. Well, you should go and talk to them, because it turns out that they don't think that's why they're there at all. Cloud and Kubernetes have been great at moving responsibility around, so that developers have to do more things. Developers are shifting right, while security and the responsibility for it is quote unquote shifting left. So put simply, today developers and DevOps teams need to care about security. Some people call this DevSecOps, I'm just going to call it extra work. Half the people Red Hat surveyed said that security got in the way of them deploying at the speed that they wanted to work at. Citing some foundational research in the industry, the tag security white paper says that 85% of defects are introduced during coding with a cost of $41 to fix at that point, compared to a post release fixed cost of $26,542. Let's try and catch those vulnerabilities early, shall we? So hopefully I've scaring numbered you into thinking that you need to do something. How should we think about security for Kubernetes? Well, we turn to the documentation and it talks about four Cs we need to consider, code, container, cluster, and cloud or co-location provider or central IT team, if you like. These all build on top of each other in a very specific fashion. If your cloud is compromised, there's no amount of security you can add to your code to help protect you. Clouds have many more services and concerns than just the ones of a Kubernetes environment, so for now, we simply have to accept part of the shared responsibility model and say, we have trust in our vendor. But the cloud vendor doesn't know what you're running unless you install some sort of agent to tell it. And in general, they don't want to be responsible for what you are running or how you have configured it. Let's then look at code. There have been books written on how to write good code. There's a sea change happening in the industry at the moment, in terms of memory safe languages, which I will glibly summarize as rewrite it in Rust. But ultimately, you're going to have to do your own work to make sure your software is secure. What then is the software supply chain? Well, I like to think about it as the gaps between the boxes, specifically by what process is code taken and made into a container, or by what process our container is taken and instantiated to run on the cluster. And we'll look at that a little bit more later. So back to our four Cs. We are predominantly concerned about the container and the cluster. What we're going to run and how we're going to run it. What we're going to run is reasonably self-explanatory. Code has dependencies. Those dependencies have occasional vulnerabilities. Those are documented on the common vulnerabilities and exposures list, CVE, with coordinated disclosure being performed. How we're going to run it relates to the thing people are scared of. Kubernetes is basically its own operating system, with many different options to trip you up. The thing that makes Kubernetes different to a PaaS is the fact that you must, or at least can, make all the decisions yourself. What sort of storage, what network configuration, what permissions, etc, etc. You submit these in the manifest, synonymous with its data format, YAML, and then you hope you didn't make a mistake. Our first move needs to be to put in some guardrails to stop people making mistakes. This is a quote from the Tag Security Cloud Native white paper, saying it is vital to scan application manifests in the deployment pipeline to identify things that could potentially result in an insecure deployment posture. There are a number of tools that can do this, but ultimately it's not really helpful to say something is wrong. If I knew something was wrong, I would have written the manifest properly. What you really want to be told is something is wrong, here's why, and here's how to fix it. You can't just compare everything against the checklist and say I am secure or I am not, but that doesn't mean that such checklists aren't helpful. The good news is that there are a number of government agencies and nonprofits that have developed benchmarks or frameworks which you can measure yourself against. You can use these frameworks in a couple of different ways. As sets of tests, you might want to run against a cluster to check you have a good security posture, or to validate to some external compliance team that you are running your infrastructure in an acceptable way. None of them are law, but they're a good baseline. Take for example the Kubernetes Hardening Guidance developed by the United States NSA and CISA in 2021. It introduces a threat model, talking about three possible sources of compromise. Supply chain risks, malicious threat actors, and insider threats. It then spells out things that you should be doing as mitigations to those common threats. Some are configuration knobs that should be set in a certain way. Some are best practices for the configuration of workloads. And some are cluster settings that will allow you to limit attack surfaces and help diagnose the problem if you ever are attacked. Just after the release of the NSA guidance, Elmo released Cubescape. The first piece of the security landscape that Cubescape addressed was that NSA framework, but we've continually added support for new frameworks and more ways to provide a view of your Kubernetes security posture. We're creeping up on 200 controls or potential vulnerabilities to check across the three major Kubernetes security frameworks and their own Elmo best practices guidelines. Cubescape is an engine, a command line tool, a set of cluster components, and a SAS backend for defining and enforcing best practices, whether it be against one of those frameworks or something that your business wants to define itself. Identifying and preventing drift, continuously from deployment of applications through the lifecycle of an application and production, and continuously hardening and reducing the attack surface of your environment with remediation advice and contextual insights on things that you might want to change or improve about your environment. The first way that most people interact with Cubescape is through the command line tool of the same name that takes us and put the control library, which it will download and make sure is most up to date when it runs, and it can be run against YAML files, Helm charts, a running Kubernetes cluster, or a Git repository that contains manifests. Output can either be directly to the terminal in useful formats like XML or JSON locally, or submitted to a backend service like the HALMO platform. Installing Cubescape is as simple as this hypersecure curl pipe bash command, but you'll be pleased to know that you can also install Cubescape with Homebrew with the crew package manager, and we are currently implementing support for Deb and RPM installs. As long as I have Cube CTL set up with a context to connect to a cluster, then I can run Cubescape to scan that cluster. And in this case, let's use the NSA framework. There we go. That's as easy as that. You'll notice that there were some things that we were unable to scan here because we did not install the host scanner. There are some things we can't get from the Kubernetes API server. We actually need to run an agent on the host to query some of the state of the machines. So to install that, we simply run with dash dash enable host scan. The host scanner is a daemon set, which is run on each machine only for the period of time which Cubescape is scanning it. Once the scan is complete, the daemon set is uninstalled, and you don't have anything continuing to run. Previously, we had some vulnerabilities which would have been critical had they failed, and it wasn't able to check those. With the host scanner installed, it is able to say that everything's okay and those are no longer a concern. Here on the screen, you can see the URL to copy and paste to view this data on the AMO platform. AMO platform is a hosted version of Cubescape's cloud components, which allow you to not only view your security information but define your frameworks and policies, define exceptions and configurations for them, and make them available to Cubescape wherever it's being run. AMO platform is a SaaS service available for free, up to 10 worker nodes, and with plans available beyond that. We intend to open source this and have it be available as the Cubescape cloud backend for everybody later this year. Let's have a look in our cluster now. Scan results uploaded here are visible, showing us all of the same controls and information we saw previously, but also allowing us to easily view the parts that need remediating, the severity of them, and understand which ones we need to address first, and the fixes required. For example, the NSA framework suggests that all pods should have resource limits set. So we can see here that there are four resources that have failed, seven exceptions that are set for system pods generally. And if we click on fix, we can see there are four components here. They are part of the Docker desktop setup that we're running, but they are not correctly configured as far as the framework is concerned. Clicking the spanner allows me to open the remediation view where I'm able to see the definition of the Kubernetes objects and also see the places where it suggests I need to perform a remediation. In this case here, I would need to set a limit for CPU and memory for this particular pod. I can take the output here, I can download the ML file, I can make those changes locally, and then I can put that back into my CI pipeline and say for this particular object, the remediation has been made and now we'll apply going forward. Most users will want to run Cubescape in their cluster so that it is continuously scanned and security posture information can be uploaded to Cubescape Cloud. To do that, you install a Helm chart. With the add cluster button here, you can see the information that's required. You simply install the Cubescape Helm chart and configure it with your account identity in order to connect the two together. In this case, you'll see that we have a cluster here named Docker Desktop. That meant I didn't have to worry about any of this because I used our new Docker Desktop integration. Installing Cubescape into your Docker Desktop environment is as easy as going to extensions and searching. However you have the Helm components installed in your cluster, having them gives you access to Cubescape's vulnerability scanner. It will scan all the containers that are deployed in your cluster and upload their CVE data to the Cubescape backend where you can visualize it. We can see vulnerabilities over time, excluding of course that this is a Docker Desktop cluster and it gets turned off occasionally. And you can also see vulnerabilities broken down by their criticality, whether or not they're a remote code execution vulnerability, and whether or not there is a fix available for them. Let's have a look for example at one of our critical vulnerabilities, the recommendation service. There are two critical vulnerabilities here, but only one of them shows up as having a fix. So if we remove the filter here, we'll see both of these. And we can also use Cubescape's exception or ignore rules feature, whereby we say we've looked at this particular vulnerability, we know it doesn't apply to us and we don't want to be notified about it again. We can mark the vulnerability as accepted and then it will no longer show up in results in the Cubescape UI or if your Cubescape client is connected and downloads the exception rules from the backend, it won't show up in CLI scans either. Here's something new we're working on to help people understand which vulnerabilities matter and which are just noise. Sniffer is a POC project which will filter out irrelevant vulnerabilities from Cubescape scan results. What we do is watch what a container actually does for a short period of time when it starts up and then from that point we can evaluate whether the files on the disk that it doesn't touch are likely to be relevant. We can't rule them out completely of course but we can say in this case of the 76 potential vulnerabilities here are the 12 that are actually in code that the container used while we were observing it. The evaluation is done with EBPF we're currently experimenting with engines like Felco and Silium and we're looking forward to bringing this to be part of the Cubescape product very soon. Under settings and then the frameworks and controls section we can add new frameworks. We can customize the controls configuring for example which container registries are allowed to be used and which are problematic and should be denied. Now thanks to the magic of AI we also have the ability to create custom controls with our new GPT integration without having to know any rego. We just need to name the control let's call it check replicas and define what we would like to have happen for example let's say fail if a replica set has more than four replicas we could give an example object to test against or a description if we want or we can just trust the machine will generate it correctly. We hit generate and here we go it has provided a remediation in the case that there are more than four replicas reduce the number of replicas to four or less it has provided the rego code snippet which will test and then fail in the event the replicas are more than four and now we can download this control file here's our downloaded JSON file containing the rego rule set to make sure we have no more than four replicas here's a manifest file which asks for three replicas we'll open a terminal and run cubescape to use this JSON file as a check and check against the replica set file you see none of the resources failed and we have no risk but if we update this to a number that's greater than four watch out there you can see the resource has indeed failed and we are 100% at risk something else you may notice here is that there is integration with cubescape in visual studio code for example we can look here at the image flag and it says that there is a control indeed a CIS control which says minimize container registries to those who are approved the recommendation from the benchmark is that you configure a set of repositories and say only these repositories are allowed and if those repositories are indeed configured then you will no longer receive the error in BS code here's something else that we're cooking up in cubescape labs at the moment support for a new kubernetes feature today to do admission control in kubernetes you have to run an external webhook that now becomes part of the infrastructure that you must operate maintain upgrade etc if your webhook is down the control plane is unavailable either by way of allowing anything to be admitted which is insecure or allowing nothing to be admitted being able to move this functionality into the kubernetes API server process has many advantages we achieve this by using the common expression language or CEL which is currently used in other parts of kubernetes for example for validating CRDs here is a rule that's quite similar to one we looked at before let's say we don't want to have more than five replicas in a particular deployment we can define that in the CEL and upload it to the cluster as a validating admission policy we've converted a number of our existing controls to CEL so that you can use them with the validating admission controller feature these are scans that you would currently do on your cluster to see if you have insecure objects and you'll now be able to use them to actually stop insecure objects being deployed to your cluster we look forward to working with the kubernetes team on bringing this feature through to availability later in the year last december we were accepted into the CNCF sandbox as the first security and compliance scanning project we're in some pretty esteemed company when it comes to security tooling but all the other vendors seem to want to have a proprietary platform for security perhaps with some open source cluster tooling we want to be open source end to end which is why i joined ammo we have to separate out a bunch of the kubernetes back end from other ammo code and integrations we use and build a version that you can install in your own cluster and in a few months time we will have fulfilled our promise just like mario open source projects are powered by stars we're very happy to see that we've shot straight to second place we need to keep building features users love so we can catch up to our friends at cert manager here are a couple more features we've built recently we mentioned software supply chain before the cubescape team wants to help users make sure they're using the right container images in their environments so we release the capability to verify container image signatures we can now verify against a list of trusted public keys as with any other configurable control we have two separate controls one for testing if an image has a signature and one for actually verifying it that way our users can get immediate metrics regarding which images are signed and which are not even before configuring keys the tooling enables users to start to align with supply chain security requirements and verification of image signatures if you're using amazon's elastic community service you might want to compare against the amazon specific cis benchmark it's designed to understand the fact that you have a shared responsibility with the vendor who in this case is responsible for the control plane we recently added support for scanning against the cis eks benchmark we developed custom controls or mapped our existing controls to the benchmark and as of today we support 50 of the 53 controls you can scan through the cubescape cli or you can view the results in the cloud back end with armo platform where they're broken down by the section headings defined in the cis documentation today we're giving you a whistle stop tour through what's new in cubescape and a few things that are coming soon once we've released our vulnerability relevance calculation with ebpf we can use that same engine to do so much more things like generating kubernetes network policies based on application behavior keep your eyes peeled for that coming soon we're going to open source the back end that runs armo platform as i've mentioned we're going to add the aks and gke flavors of the cis benchmark and we're going to write the kind of documentation that you'd honestly be surprised to see from an open source project thanks very much for taking this journey through cubescape with me today i'll look forward to answering any questions you may have in slack on the hash cubescape channel if you want to get involved in contributing the channel is hash cubescape dev and we look forward to seeing you in the community soon