 Okay, so here we are then, welcome to this presentation. And now as you asked, I work as a software engineer for Microsoft, I work part of the QIN4 team there, so we did, you know, in-spec project and other projects using the BDVF. Today's presentation is about how to use BDVF to generate some security policies for our cluster. So yeah, in today's presentation, I will show you a quick introduction of different security policies, probably this is something you already know, but as in case we are already in the first, we are going to show how can we generate those policies by using some of the BDVF tools, then we'll show you how to audit those policies to others around the cluster, and then I will present the different implications of this approach. Yeah, I will show you a quick introduction to something. This talk is to be together with Alban, but Alban is not here, so Alban will be here and I will show you some of the most traditional of these presentations. Okay, so let's get started with the security policies concept. We are known that for the software is the very kind of security policies that we can use to make our cluster more secure. There are so many of them, but in this case, we concentrate on ZCON, both capabilities and network policies. They are changed by open policy engine and other security related things that we are going to cover on this presentation today. Well, the problem that we have with the security policies and core netizens that they are difficult to configure. So in order to configure those security policies, we have to understand what our application is doing. And this is something that we don't understand all the time. For instance, if we want to configure a second profile, we have to know what our assistant codes are, are secured by our application. The same applies for our latest capabilities and in the case of network policies, we have to understand how the different network services that we are running there communicate to each other. Also, yeah, this is something that unfortunately is true many of the times the person that is defining those policies is not the person that is developing the applications. Maybe there is one person that develops the application and then there's another person on the team or maybe a different company also that has to configure the security policies for a different application. So this is, for this person, this is very difficult to understand what are the security policies that has to be built. How to configure those security policies because many times that person doesn't have a lot of knowledge about the application. So yeah, in today's presentation, I want to present you a different way to generate those, or a different way to define those policies. So what about if we observe the application, if we use some tools of certain application based on the behavior, based on the activity that application is doing, we can define the security policies that we want to use for our application. So the idea is actually very simple. There are two steps. The first one is we have to observe the application. We have to understand what the application is doing. There are different technologies for that. In this case, for sure, we are going to use EVPF because it has a very low overhead. So when we are observing the application, we are not affecting too much the performance of that. And also EVPF is very, very flexible. So we are able to do a lot of things that we need to do there. So yeah, that's the first step. We observe the application. We capture the activity of the application. We know what the application is doing. And then by using that data, we generate a policy. This is something that sometimes is really easy to do. For instance, for a second profile, this is just to capture the system calls and to put a list of system calls. For both capabilities is the same, but in other cases, like network policies, this is really difficult because we have to generate the network policies in a way that are easier to understand for the developer or the operator of the platform. So yeah, depending on the kind of network of policies that we are generating, it could be very easy or it could be really, really difficult. So yeah, let's go ahead and start with the first kind of security policies that I want to cover today. This is second. So this is a mechanism on the kernel to limit the system calls that a given process can make. And yeah, how it works is that we define a second profile, we put a list of system calls there and we can tell the kernel, okay, if a given process executes this call, perform this action, and the action that we have are for instance, just get the process. We can send a signal to the calling process. We can deny the execution of the system call and return an error. Or we can also execute the system call and lock to the system. Actually, yeah, there are other actions that are support, but my intention here is not to go into the detail of those ones. This is what a second profile looks like. So we have a default action. This is what happened by default when our, when a given system call is not a secure it. Or better, when a given system call is a secure it and it is not on the list that I have defined, then I can define different list of system calls and put an action for that. So in this case, I'm allowing the few tests on the red front system call. Yeah, actually in second profile, I can even define different list of system calls with different actions. So I can have a sort of mix, allow, and deny this together. In Kubernetes, we can define those profiles in the pod security contest. This front coordinate is 1.19. Actually, before it was also possible to do, but it was defined by using an annotation in the container pod, but now this is the precary, so I'm not going to talk about that today. We can configure the second profile at the pod or at the container level. So it means we can have the same second profile for all the containers on the pod or we can even have different second profiles for each container in the pod. So this is what it looks like. You have the security contest. We have the name of our security profile and that by default is looking in this bar lift cubelet path. So yeah, actually handling security second profiles in Kubernetes by Han is a little bit difficult because you have to be sure that the second profile that you have is installing all the different nodes so you have to keep that aligned on the different nodes. So this is sometimes difficult to do. So for this reason, there is the security profiles operator. So in this case, you can define the second profile as a custom resource and this project, this operator will take care of synchronizing the second profile on all the nodes in your cluster. This is just to show you an example of how we define a second profile. So here we have defined the second profile and yeah, in this case, this is a very simple one. We are only going to log all the decision calls that are the security. And this is how we associate a path to that second profile. Actually, the SPO project has different ways to do that but this is not so relevant for this presentation today. So yeah, let's get into the interesting part of this talk. So how do we define a second profile for our application? Of course, and a low list is the preferred approach. We only want to allow the decision calls that we know that are safe. There are times that new decision calls are added to the kernel. So if we are using a block list, maybe one of those dangerous decision calls is implemented, we are not going to block that. So yeah, in this case, this is better to have an allow list to be in the more secure side. But the problem with this approach is that we have to be sure that we include all the decision calls that our application needs on the profile that we are defining. And then the question is, okay, so what are the decision calls that my application needs? Well, how can we know that? The first option is, yeah, if we are developing an open source application, if we have access to the code of the application, we can check and we can try to read the code to understand what are the decision calls that the application is using. But actually, this is really difficult if you are using any standard library, maybe from Go, LAN, C, anything, you are doing a call to the standard library but you don't know what is the decision call that a standard library is doing to the kernel. For instance, in some cases, you can just call open but the standard library but it implements an open app call to the kernel. So there is no like a one-to-one mapping there. So yeah, it's a difficult approach. Another idea is to use a trace. We could run an application with a trace. A trace is a tool that brings all the decision calls that are executed by a given application. But this is like a more in-box approach because we have to... We need a trace to be available in our container. So if you are running a disk release image, you are not going to have a trace there. So yeah, it will need some more work to make that useful. Another option is, yeah, we can define a simple set-com profile by using the login action and then we can check the audit log of the kernel. But actually understanding what is on that audit log is not so simple because for instance there, we don't even have the name of the decision calls that are executed. We only have the number of the decision call. That number depends on the architecture. So I will have to map those numbers to the name of the decision call. So again, even if that's possible, that's not so easy to do. So the thing that I want to propose here that we want to propose here is we can use a tool that records all the decision calls that are executed by my application and based on that, we can generate the second profile. As far as I understand, there are two different projects that allow you to do that. The first one is the security profiles operator that has an EVP recorder and the second one is the second advisor from Inspector Gadget. So let's look at the demonstration of the first approach by using the security profiles operator. Hello, I will show you the security profile operator on how its EVP recorder can monitor system calls and generate the second profile. So first, I need to enable this feature. So I'm checking the enable EVP recorder feature is set to two, that's cool. Then I need to enable in the feature in the namespace how we do my testing. It has this level set to two, that's cool too. Then I need to create a profile recording resource. Here I have a profile recording resource in the namespace of itals and it has a podselector with a selected level I want to test. Okay, so I apply this profile recording. Now it should be installed on my cluster. And now I deploy my pod. This pod will have a level that match the level selector of the profile recording. Okay. So now let's see my pod, it is deployed. I will generate some network activity on it. For this, I use port forward to enable it to my app and I use some calls command to connect to my app. So this is working fine. Now I can stop the port forward and let's delete the problem. Okay. So let's see if my second profile was generated. Yes, it is installed. Let's see what is inside. On here, I see generated a second profile with a default action of returning an error except for the system calls which are not. Thank you. Okay, so that's the first way to generate those security second profiles. And the other one is by using this inspector gadget second advisor. Yeah, actually the mechanism that this uses is very similar. This is something that we implemented almost in parallel, two different, two very similar features implemented by two different teams of parallel. So yeah, just let me show you the second one. So in this demo, I will use the second advisor to monitor my workload to see which system calls it is using. So let's start the gadget with advice second profile, ask it to monitor the second demo on this face on the flow pattern pod. I get a trace ID that I can use later to stop it. And now I can deploy my pod. This pod is unconfined, so it doesn't have any second profile. I see it is running already. And I will not generate some network traffic to it to exercise different code paths. I use path forward, so I can use code directly to it a bit later. So now I run a couple of codes command. I can see the result with the webpage. And now I can stop that. I will stop the path forward and then get the result from this gadget. So I reference the trace ID and ask it to return the system calls in this file. So if I look at this file, it will simply be the list of system calls in the JSON format. So you see there are different system calls for socket TCP connections and so on. I love this. I have this file in the ML format as well. This is a second profile from the security profile operator. And here by default, it returns an error for any unknown system call or it allow the list of system call it has detected before. So I can deploy this second profile that I generated. Now it should be installed on my cluster. And I will deploy a pod that reference this second profile. So now it's no longer unconfined, it just reference the second profile I generated before. So let's deploy it and that's it. Okay, so now we understood how to generate this second profile. But yeah, once we have those second profiles in place we have running our applications with those security profiles. How can we know if a given second profile will break my application or better? How can I know if a given second profile is blocking some system calls that my application is trying to execute? Again, one option is to use this option lock. Sure, in this case it will not block the system call but will give you some understanding that the system calls are being executed. Then we can check this is locked but again as before this is difficult to understand. So an alternative that we have is to use this second audit gadget that provides you, that tells you when a given second profile blocks a system call from a pod. So let's look at another demo. Hello, this time I will show the second audit gadget. I will use a different version of my second profile. This second profile contains the same list as the whole system called us before that are alone. But the default action will no longer be to return an error but to log them. So the audit log will be able to display them. So let's deploy this second profile. Now it should be deployed on my cluster. Okay, before deploying my workload I will start the inspector gadget. So you can see the stream of system calls being executed. So let's start this command inspector gadget. I will monitor the second demo name space and display the system calls on the return code and so on. Okay, so far there is nothing because I don't have another workload. So let's see the workload. The workload reference my second profile as I showed before. And let's deploy this and let's see if it is running correctly. It is already running but so far the only system calls that are executed are the one I wrote. So there is still nothing to display here. I will try to generate different network activity to see if something happened. So like before I will use port forward on some curl command. So port forward is running. So I can run curl and see if anything happened. I get the result, I see the webpage but still nothing here because as before the only system call executed are the one that were in my all list. Okay, so now I can stop the network activity and I will try something different. I will exact into the port and see if something different happened. I will execute bash and bash should execute different kind of system call that were not done before. So now the audit can notice them. And if I use this command, M cannot, that should, although on this M cannot system call that is, that were not in the all list. So I see it in the audit log. Thank you. Okay, so that's it about set com. So let's move to another kind of security profiles. Those are Linus capabilities. And yeah, science Linus 2.2 privilege process doesn't have all the capabilities of the system. So those privilege operations were divided into different blocks, if I can say that way. So the idea there is, if there is a, if our process gets compromised, if a privilege process is compromised, it is only able to do some things on the system. So yeah, for instance, if we want to change, if a process needs to change the owner of a file, it has to be running under this caption capability. If it has to be buying at poor, less than 2,000 24, it has to have this net buying service capability and so on. So the idea there is to have, to limit the capabilities that are all process have. Of course, the list of capabilities in Linus is very long and actually some new are in the different releases of the kernel. In Kubernetes, the container runtime give us some capabilities by default and we can add or we can drop the capabilities that we want to give to our container by using this security contest. So this is an example in this case we are dropping all the capabilities, but we are only adding the capabilities that we need for our container. So yeah, the problem here is very similar to the previous one, how do we know the different capabilities that a given pod needs to run? And actually in this case it's actually more complicated because this is not only about system call but the capabilities are like an internal concept on the kernel. So sometimes those are used, sometimes those are not used depending on the kernel version that you are running on different configuration parameters of the kernel. So let me show you how you can generate, how you can understand the capabilities that a given application is using on a Kubernetes cluster. Hello, today I will deploy it to RSI and see what kind of capability it use. So first look at the original version of RSI in the specification, you see it use all the privileges and so far I have not installed it. So let me install it first. I will deploy the inspector gadget capability to RSI gadgets. I deploy it on the RSI system and M space, looking at the specific pod for RSI and looking at all those columns and I don't want any duplicate capabilities and so I specify the unique flag. Okay, so now let's start RSI and it will start the reason. Okay, it's already started and you can see at the bottom the capability it has exercised, setPK, Ptrace, and this admin, this resource. Okay, so now we can delete it and let's make some space. And now we'll have a prepare the new version of RSI. Instead of using privilege equal to, it use the following capabilities and you can see it will work in the same way as before. And so this gadgets allows me to find which kind of capabilities I can have in my pod spec. Okay, thank you. Okay, so yeah, that's it about the Linux capabilities and the last one that I want to cover today are network policies. This is just a clear view, I will go very fast on that. So network policies are a coordinated mechanism to limit how the pod can communicate with a network entity. Network entity is another pod, coordinated service or an external endpoint. Coordinated network policies operate above the IP and the port level, that's layer three, layer four. And we can use that to restrate the ingress or the egress traffic from a pod. This is an example of what it looks like a network policy resource. So we have the pod selector, which means the pods that the policy should apply to. This is where we define the policy. So here we are saying, okay, our pods should be able to communicate to other pods, other name spaces, other pods and name spaces or IP blocks if those are external endpoints. And in the part we define the layer four the layer four of our network policy that's the protocol and the part. So yeah, it was just a quick reminder of network policies. So how do we define coordinated network policies? Well, usually those should be created when we are designing our architecture. But yeah, as I was telling you at the beginning, this is not what happened in many cases. So in some cases you have to define the network policies after your application is already running. And yeah, in those cases using a policy advisor, using a tool that suggests you send network policies to be applied can be a motion. And yeah, let me show you very quickly how we can generate those by using the a network policy advisor from Inspector Gadget. Hello, I will demo the network policy advisor. So I will deploy a workload and observe the network traffic done by this workload and then generate network policies thanks to the network policy advisor from Inspector Gadget. So first I create this demo namespace and so far there is nothing in it. And I will deploy this Kubernetes manifest.tml. Before that, let's start the network policy advisor. Start it with this command. I use a kubectl-gadget-advice command to monitor the network activity in the demo namespace and redirect the log in this file. Okay, so far there is no recording because there is no network traffic. But once I deploy the manifest, at some point it will create some network activities. It will take some time to prepare all the deployment to be ready. So let's wait some minutes. After some minutes, my deployment is successfully deployed. So now I can stop this. And then let's ask Inspector Gadget to generate the network policies from this log. And then the network policy will be saved into this file. Then we can have a look at this file. And we see the first network policy generated is for the add service. We noticed some equals traffic to DNS. So power 53 UDP to the kub system, kub DNS service. And some increase service from the front end to some TCP port. And you can see other network policies for other services as well. Okay, and finally, how we do audit the network policies. So how we can know if our network policies open the packets from our application. So yeah, depending on the CNI provider that you are using, maybe you have that support implemented there. If you are using Celium, they have this audit mode through or this flat better. And when you enable that, the Celium doesn't force the network policy, but it will tell you when it eventually drop the packets there. If you are not using Celium, there are other tools that kind of the same at the IP table level. But yeah, in this case, you have to find a way to correlate the IP table rule with the network policy that is dropping the packets. So yeah, just to finish here, I want to discuss the limitation of this approach. So I will say the biggest limitation here is that our application has to generate all the events that it needs to work when we are monitoring that. So if there is, for instance, a given system call that our application needs, but is not used, why we record that to generate the second profile, the second profile is going to block that system call. So when the application is, we'll try to use that specific system call. When running, for instance, a production environment, that system call will be denied and our application won't work. So how can we ensure that our application is generating all the events that it requires in the real scenario? Well, maybe one option is to deploy our application in a production environment, send real traffic to that and monitor that there. Of course, there is some risk on running an application in production without security policies. Another option is to run some integration tests to try to go through the different parts of the code. So those are one of the options that we have, but actually this is like the biggest limitation and it's difficult to do. So there is no like, this is the way to do that. There are so many different options. And the second limitation, this is, I will say, limitation warning is that when we use this technique, we are assuming that our application is trust. So we are monitoring the application if there is any malicious activity going there, we are going to include that into our security profiles. So basically what it means is that we are going to allow the malicious activity in the security profiles that we are generating. So yeah, we have to be sure, I don't know how to do that to be honest, I don't have an answer, how to be sure that our application is trust, but we should be sure that this is why we use this approach. And yeah, just to finish, there was a, I will say similar talk this Monday in the Cloud EVPF negative day. That is also how to generate this security profile by using EVPF. But in this case, it was by the ISO Valentine. They were using Tetra for that. So if you are interested on that, maybe you can check the recording of that. It should be available on YouTube in some weeks. And yeah, that's it. So happy to take any questions that you can have. Thank you. Thanks for the great talk and not just great but comprehensive overview. My question is, it's all in the limitation about during the recording phase, you can create policies only what you saw in the recording phase, right? So what you haven't seen, you cannot define. Have you been looked into or researched how could you decide when to stop recording? I mean when to, or how to signal to end recording and you say, I saw enough of the application activity that now I can turn this into a policy? Yeah, that's a very good question. We haven't done that. So far, when to start, when to stop recording is totally controlled by the user. So you as an operator, you say, okay, I want to deploy my application, I will start recording, then you, I don't know, you wait for one hour, 24 hours depending on an application, and you say, let's stop. But yeah, maybe that will be something interesting to explore how to start and stop recording in an automated, smart way. Thank you. Thank you. When, if SETCOMP stops an application because it breaks security policy, what is the experience of either the APTA or the operations team? What do they see when that happens? Oh, sorry, I can't understand. Will you mind removing your mask for a bit, please? I can't understand what you're asking. Yeah, when SETCOMP takes some action on an application that's running, it breaks a policy. What is the experience for the operations team or the application that created that? Is that just an error, or what do they see to know that this app broke one of the policies? Yeah, so when the second profile is violated, depending on that, it could kill your application. So what you see is a pod that is crashing off because the application is being killed there. Does it say, like pods crash sometimes frequently? So does it say a particular reason why the pod crashed? You have to check the logs from the application. I mean, if you are using the audit second that I was showing you before, that will tell you what is going on. So that will tell you that a given second profile sent a kill signal to a given process, and it will give you information about the container, the pod, and so on. If you are not using that, the only information that you have available is the log of the application. So you will have something like the process was killed and that's it. Probably there is also something in the audit log from the operating system about that, but in that case, you will have to correlate the PID with the pod. So in that case, if you are running a lot of pods in a node, this is not easy to understand what is the pod that was killed. Right, yeah, exactly. Okay, this is not great. Okay, cool. Thank you. Thank you for the talk, it was really helpful. It looks like the tools that you showed are really helpful for developing the initial policies that you wanna enforce. My question is, as applications change over time, they may have requirements for new links capabilities. They may need to make new system calls. Do you have any thoughts on how you can maintain these policies over time such that it's not blocking application developers from making progress? Yeah, I will say that this is a rather difficult question. So the idea that we have with these tools is, okay, you don't have any policies. We suggest you what could be an initial version of those policies. So the idea is not to use these tools automatically. So the idea is you, as a developer, you use the tools and then you have to understand the output of those tools and you have to be sure that the things that are in the security profile make sense. I'm not totally sure that it will make sense to update those profiles automatically when the application changes, because what about if there is a malicious activity there and you are updating those profiles automatically to allow the malicious activity that is going there? How can I understand? Okay, yes, I updated the application, so I have to update the profile because my application is doing something similar. How can you be sure that that's the case and that this is not about that you are having some malicious activity there? So yeah, I mean, this is very complex to understand if you can update that automatically without human supervision there. I was pretty interested in the network policy generation and I noticed the network policies that get generated there, using pod selectors with labels to clearly identify the pod that is involved in the network policy. How intelligent is that network policy generation? Is it trying to create like the simplest possible network policy or how does it decide what labels goes into that? You have a point, very good question. Yeah, so the general logic that we have is that we capture the traffic. So the traffic that we capture has the, I mean source destination IP addresses and we enrich the data with Kubernetes related information. So we do a transformation like from IP to pod name, service and so on. And at that point we capture the labels of those pods, services and so on. And then we try to consolidate that to generate the network policies. Actually what we are doing right now is like try to be as granular as possible. So we are generating very small network policies. So actually if you are running so many different services you are going to have a very big generating network policy and this is what I was talking at the beginning. I mean, we have a limitation there because how can we consolidate those network policies and less network policies with more labels? This is something that we don't know yet how to do. So probably more work has to be done there in order to provide network policies that are consolidated and that are easier to understand for the users. So actually one very simple case is when there is a pod that is reaching an external host, I mean by IP. So we only generate a slash 32 rule. But what about if actually the pod is trying to access a app signlet. So how can we consolidate those into app signlet? This is something that we don't know how to do that and this is something that we require more research on that. Thank you. Thank you. Last one. The output that is used to create the network policy, that log file, is it human readable? Like, could it be used to audit the frequency of traffic and create a profile so that if you're trying to create a secure network policy based off of that output, you have information to use. Like, obviously as a security sys admin, you have more information about the expected traffic between services. So if you can use that, not actually the generated policy, but the log file to inform the network policy that you're creating. Like, is that possible? Is that a use case that's been considered? So do you mean only to capture the data about the network packets and then use that for? Yeah, like you've got that log file that is being used to create the network policy rather than just creating the network policy, use that log file to create a traffic profile that you can then use to inform the network policy? Yeah, actually, those are two tasks. I mean, capturing the traffic and reaching the traffic we coordinate this information is like the easy task. The difficult task is, okay, how do we convert that network traffic that we have captured into a network policy? Currently we have a very simple algorithm there that just tries to consolidate that. But yeah, there are many options that we haven't explored yet. Yeah, it just seems like it'd be very interesting to output a profile rather than a policy. So something you could run that advisor several times and compare the traffic over a period of time to see like, oh, we're getting hit by this request at this frequency, which is not something we expect and maybe is a malicious actor that we should look into. Yeah, that could be also an option. Also, thank you. Thank you, everybody. Thank you.