 Hello and welcome to our presentation on enhancing Kubernetes with the security profiles operator. My name is Colleen, I'm a contributor to the security profiles operator and I'm a software engineer at SUSE and it's a pleasure to be here with you today. Hey friends, I'm Sasha, one of the main handers of the security profiles operator and it's a pleasure to be here today. And I work for Red Hat. What will we cover in this talk? First of all, we would like to speak about how to enhance default Kubernetes workload security. So we would like to speak about which goals we would have in mind and how to achieve that in such a rapidly evolving project like Kubernetes. Secondly, we would like to lay out the current state of the security profiles operator. So we will cover the operator features in detail and how they can be used to make workloads more secure. And then we will see the operator in action. So this is time for demoing the operator while keeping common use cases in mind. And after that, last but not least, we will lay out our plan for the future of the project. So for example, we will discuss our long-term goals and we'll leave some room for some Q&A session afterwards. How to enhance default workload security in Kubernetes. Kubernetes does not provide strong security defaults out of the box. And so this means that Kubernetes has to be designed for usability over security. So for example, if we decide to restrict the discourse of a workload, then we can have to keep in mind that this may break applications while it makes others more secure. Other than that, security related decisions have to be considered by the system administrators, the site reliability engineers, as well as application developers. So different roles make it hard for the community project to decide which position may be the most important one and which should be considered as the default. The security context of Kubernetes holds a security configuration that will be applied to a pot or a container. So this native API field makes the usage of the security feature more comfortable because the API server can validate the fields directly by applying them to the workload. Some security related features like app armor are not graduated to the security context yet. They are still using the annotation based syntax to apply settings to workloads, which is not obvious to use, and which is kind of error prone because containers have to be referenced by name. And second, that this graduation to the security context in Kubernetes 1.19. So the feature is now fully integrated into the security context and will not support annotations anymore from Kubernetes 1.23. Our first goal is to make features easier usable by providing native fields. For example, if you look at the second profile of the security context, then we have a native type, which can be local host runtime default or unconfined. If it's local host, then we have to specify a relative path. And this makes it easier for the users because the absolute path has to be taken into account anymore. And if we compare to the former annotation based syntax, then we can see that it's way easier to read. And that we don't have to actually reference the container name anymore because we have the security profile field as part of the container or as part of the complete pot. And the problem for our back and limitation of that is that we don't have a local host profile validation. So the relative path is still error prone, and there is no solution about how to distribute second profiles to all nodes. Our second goal is to increase the user adoption of the feature. So how to provide stronger defaults for second in Kubernetes. Can we make the container runtime default profile to need default for all workloads, or how to distribute second profiles across thousands of nodes. So we just get a mechanism to easily distribute those second profiles. And the last question is how to create second profiles for my application. So is it really necessary to collect all this calls by hand, or is there a way to record profiles. So what do we want to do, let's write a Kubernetes enhancement proposal. No, features can be already quickly out of the Kubernetes source code tree. So the development cycles can be shorter, we can enrich faster testing and feature graduation is even easier. So really it's less effort for the community as well as for the project maintainers to do it out of three and Kubernetes six can sponsor such a project. So this provides community infrastructure access as well. This provides full CI testing and proud image promotion and distribution and repository access management. This is exactly the point where the security profiles operator initially second operator was born. So please visit skates.io slash security profiles operator. And the project is sponsored by Zignore that lives in the Kubernetes six namespace, which is kind of nice, because then we can leverage browse already mentioned. The project already turns one year in April. So we already have two releases and plan for the next one containing many new features and enhancements. And the idea is to cover common use cases around security profiles. So around second, say Linux and up our more and all that bundled into Kubernetes. So we have a list of user stories and you can give them a look. And those contain the three mentioned perspectives of the administrators decide reliability engineering and application developers. So we have a huge roadmap, but the project at all is still very tightly scope. So right now we still focus on feature completeness for second and say Linux and how does it work in detail. So let me pass over to Colleen to answer that question for you. All right, thank you Sasha. So I'm going to discuss a little bit about how the security profiles operator works and talk about the current features that we support in it. So the operators driven by a few different custom resource definitions that we can kind of break into three general categories. The first is the profile CRDs. These are the set comp profile and SE Linux policy CRDs. And these are resources that define the actual security profile that will get installed on the underlying Linux host. Then we have what I'm calling composition CRDs. These are the profile binding and profile recording CRDs. And those are resources that will help application developers to build and use security profiles. And then finally we have more operational CRDs. These are resources that help cluster administrators with the running of the operator itself. So going into a little more detail on each of these, we have first the security profiles themselves. These are kind of the cornerstone of the operator. We started out with set comp and set comp restricts a process to executing only a finite list of allowed system calls defined in a set comp profile on the host. And so the set comp profile CRD provides a Kubernetes native interface to create and validate these set comp profiles. This will get managed on the back end by a daemon set that converts the set comp profile resource into a JSON configuration file that gets picked up by the Kubelet and used to launch these containers. And then more recently we added support for SC Linux and SC Linux defines more complex access controls for processes and files via policy rules and object labels. And so the SC Linux policy CRD provides again a Kubernetes native interface for creating and managing these SC Linux policies. This will also get managed by a daemon set that drops these SC Linux policies into a directory on the host and then there's a daemon that will watch this directory and invoke SC Linux tools to install these policies on the host. And then we have some more advanced resources that help support application developers. We have the profile binding resource which creates a binding between a workload image and a profile resource. And so with this there's no longer any need to change the YAML for a pod to set a security context to bind it to one of these profiles. Instead there will be a webhook that will inject a reference to a profile into a pod security context when you create the pod. And so this is especially useful if you're running some kind of workload that, for example, if you installed an application from an open source Helm chart, you would no longer need to fork or edit the Helm chart. You can just create this profile binding and the pod and the profile will automatically be linked. And then another exciting feature is the profile recording resource. This is for Setcomp and it generates a Setcomp profile for a running workload. And so with this there's no more need to write these complex profiles out by hand. No more guess and check of running the application and checking the audit logs. Instead you just create this profile recording resource and it automatically creates a Setcomp profile Kubernetes resource for you that can then already be used by your application. This does depend on using cryo as your container runtime and it depends on the most recent version of cryo. And then finally we have resources that help support cluster administrators running the operator. The security profiles operator daemon resource is kind of a configuration point for the controller that runs the daemon on the nodes. So it lets you toggle different configurations or set node tolerations for the daemon set. And then the security profile node status resource is a read only node level view of the status of installed profiles. So since profiles are cluster level resource resources this gives insight into the availability of a particular profile on any given node in your cluster. And then finally it's also worth mentioning a couple other features in the operator. We have this log and richer feature which supplements the audit logs with annotations that help give context to audit events. And lets you trace events back to the Kubernetes workloads that generated them. So that's extremely useful. And then we have this profile stacking feature. And what this is is kind of an inheritance model for Setcom profiles. So it allows you to compose application specific profiles on top of standardized base. So if you wanted to create a base profile for all the system calls that run C would need to launch a container. Then you could also create that as a base profile and then create a slim down profile with just a short list of system calls that your application specifically needs that wouldn't be necessarily being needed by other containers. And with this profile stacking feature these can be composed into one single profile. And so with that I can break into a demonstration of some of these awesome features. So the first thing we can look at is the deployment of the operator itself. The operator webhook depends on cert manager. So that needs to be installed first as a prerequisite. And then it's a simple matter of applying the operator manifests to your cluster that will create all your custom resource definitions. So set up your RBAC and run the main deployment and the deployment is what will spawn the daemon set that does the actual legwork of managing the profiles on the nodes in your cluster. With that done the first thing we can look at is profile reconciliation or how a profile resource in Kubernetes actually gets rendered to disk on the nodes in your cluster. So if we take a look at this example setcom profile resource, this gives us a list of system calls that an application should be permitted to use. We can deploy apply that resource in Kubernetes and then if we go and examine it, that will give us a relative path where the profile is installed on the nodes in the cluster. And we can even go and examine a node and see that the profile has been rendered into a JSON profile under the kubelet where the kubelet can find it and use it. And then for a pod to make use of this setcom profile, we need to define a security context for it with a reference to that relative path to the profile under the kubelet. So if we go and create that pod, the pod should be created successfully, but if the pod goes and tries to use a system call that wasn't expected and wasn't explicitly permitted by the setcom profile, the host operating system will forbid it. So we decided that this is an expected system call that our application should be allowed to make. We can edit the profile, add that system call to the allow list and reapply the profile and restart the pod and then the pod will be able to use a system call successfully. Since we're managing these resources as higher level resources, it allows us to add logic that kind of enforces relationships between profiles and workloads. So for example, if I tried to delete a profile, it will not actually delete the profile from Kubernetes while it's still in use. It will mark it as deleting but leave it existing and it won't actually delete it from Kubernetes until all of the workloads using it have been removed. And once that happens, it will get removed from Kubernetes and it will also get cleaned up on the back end because the Kubernetes controllers give us the tools to be able to clean up on the back end when something is deleted, when a resource is deleted in the Kubernetes API. And then another feature that we can look at is profile stacking. And so what this allows us to do is create a setcom profile to use as a base and that could define a limited list of system calls that we think that any containers should be allowed to use. And then define a much more refined slim down profile that's specific to just our application. And so this gives us a much smaller, easier to use, easier to read profile. And then if we create a pod that references just that child or create both profiles, the base profile and the child profile. We can see that when it gets rendered, the composite profile contains all the system calls from both sets of profiles, the base profile and the child profile, including this application specific system call. And so if a pod wants to use that child profile, it will be able to do so successfully. And then the final thing that we can look at here is profile binding. And so what this allows us to do is define a binding resource that links a profile resource like a setcom profile or in the future an SE Linux profile to a image tag. And then we can define a pod that no longer needs to explicitly set a security context. We can just have a basic pod spec defined here. So if we go and recreate the original setcom profile, and then create that binding resource and go and create the pod, then if we go and examine the pod, we can see that the setcom profile has been injected into this pod once it was created. And so this is really useful for running like external workloads, for example, an open source Helm chart where you don't want to modify the Helm chart. You just want to use your custom setcom profile. This allows that to happen really easily. And so with that, I will hand it back to Sasha, who will demonstrate profile recordings. In this demo, I would like to show you how we can record setcom profiles with the security profiles operator. So to be able to run this demo, we would require cryo 121, which may not be released yet. But the latest cryo master should be sufficient at all. If we look at the output of pseudo cryo version on my local machine, then we can see that we run cryo and the latest 121 development version. It is also necessary that we set up the OCI second BPF project on the local machine. And this is an OCI pre-start hook, which locally compiles a BPF module, attaches it to the workload, and then traces its syscalls. If a workload load gets deleted, then the syscalls will be put into a file which can be picked up by the operator. We have to configure cryo that it points to the actual hook. And for that, we can look at a hook steer configuration option, which points to the OCI second BPF hook in my local machine. So everything should be set up. What's next? Now we have to create a profile recording, which is a CRD. And a simple profile recording may look like this. So we create a profile recording with the name demo. Right now we can only record second profiles, and we choose a pod selector, which matches our label's app equals demo, and all those workloads which match those pod selector should be recorded. So then let's create the profile recording and put into our cluster. If we now look at all our profile recordings available in our local namespace, then we can see that the profile recording is up and running, and we can also verify that the pod selector still matches. So let's lose a new pod to create some syscalls recordings. So for that, we just run a simple pod, which gets immediately removed after it has been run. And we name it my pod, and we add our labels app equals demo, and then we create a directory test. And let's see what happens now. The pod gets deleted, which is fine. And now we should have a second profile available locally in disk. So let's look at our second profiles and outlusters. Here we go. Our demo my pod second profile has been successfully reconciled on every node. If we now look at the second profile in more detail, then we can also find our mkdir syscall in it, beside other syscalls which are necessary to actually run any workload in our cluster. This is the least amount of syscalls necessary to run a single workload in a Kubernetes cluster on my local configuration. We can also see that we get some events, so the second profile successfully got synchronized at disk. Now let's delete the second profile again. And recording single pods is nice, but recording multiple containers within a single pod is even better. So let's create a pod. So we also add the labels to our pod, and then we run two containers in it, one engine x and one radius container. So this should work at all. And if we create the pod, and then we wait for it to be ready. Then we can look that the recording is now in progress. So we should now test our application so we can, for example, load balance the engine x or the radius container. And then we can actually see that all syscalls or all code paths of that container application gets triggered. Now for test purposes, let's delete the pod again. Now we finally got two additional profiles instead installed instead of just one. Let's look at our second profiles within our cluster, and then we can see that we have our demo radius and demo engine x profile successfully installed into two different second profiles. So let's look at those profiles to ensure that they are actually different. And here we go. We can see both syscalls are actually not the same for both second profiles. So the recording succeeded at all. Now let's delete those profiles again. We can finally bring this to the next level because the operator is even able to record whole deployments. If we, for example, create a deployment which looks like this, that we have two replicas of our engine x and radius containers and still have applied the same labels at Epicals demo. Then we can create this deployment and wait for it to be available within the cluster. Now if we remove the deployment from our cluster again, then we can look at four profiles indeed. So we got a profile for each replica. So they may differ because different load balancers may reach different replicas of the deployment. But we are now able to actually assemble our second profiles by just dipping them and looking at our workloads. So this is our main strategy how to create second profiles for actual real life workloads. And this was my demo for recording second profiles with the help of the security profiles operator. And this is my demo for the future of the project. Well, there are many things planned, but the few of them are just here. So the first thing we would like to do is we would like to introduce a note status for profile reconciliation to avoid data raises. So we are working on a fully transparent notes that is to determine if a profile has been reconciled to this go not that this helps us to avoid data raises and make the mobile implementation more robust, especially for large scale deployments. We would like to identify the deployment in favor of automatism. So the webhook deployment should be part of a single all in one deployment file. We would like to add custom deployment logic to keep the rollout of the operator in sync. We are also considering to remove the sort management manager dependency completely. We would like to work on audit logs for the score recording. So we already have the log file enrichment implemented for the operator. We are also able to collect internal messages directly. And we also plan to enhance this log enrichment mechanism to be able to record profiles directly from this data. We would also like to add full feature cellinox and almost support. So, right now the security profiles operator is targeting to also enable the cellinox policies to and upon support will come in the future releases as well. The last one, but not the least one is metrics, our plan is to provide a metrics endpoint for the operator to expose security related information to prometheus. This gives cluster operators more details about the actual security policies and what is going on in that cluster. So we would like to love to see your contribution if you're interested. And if you want to get started working with any of those topics. With that, I would like to thank you all for listening this talk. And now we are happy to answer any questions. See you.