 Hi everyone, here I am Carol Valencia. I used to work as a DevOps, currently I am working in Aqua Security, and I am focused more in Cloud Native and all the best practice that we call Applied Insecurity. Today I will talk about runtime security with EVPF. I will do an introduction. As we can see here in the agenda, I will talk a brief introduction about challenges in container security, the built states and the runtime. Next, we will talk about EVPF, what is this technology that is so many companies that are investing in applying this new technology. And we will see some runtime security use cases, like some attacks on the wild that is happening right now in Cloud Native. Let's continue with the first topic that will be talk about container security. Well, here we have the container life cycle. As you can see here, we have the code, the source code, that it could be like open source code, proprietary code, third party. Also, it could be binaries and libraries, plugins. And we will take this source code and we will try to build another library or in the case of containers, we will build an image, as we can see in the second stage. And after that, we will have a pipeline, continuous integrations pipeline that it will generate another artifacts. It could be infrastructure as a code artifacts like terrafore, terraformations or even our image, container image. So if we have continuous deployment, we could automatize this. And in the case of our image, we will deploy it in some built-in machine or Lambda or even a Kubernetes cluster of Kubernetes. We will have our image in the state of running. Our container is running right now. And in this process that we have, especially the pipelines, we will have some intermediate steps like, for example, validate and try to push the image. We have this step of container registry. When we built this image and if the date is built correctly, we could push and save this image in the container registry and we use it in the next stage. We also have, if you notice the artifacts, that could be the CI artifact and CD artifact. It's another element that we need to take care that we will see in the next stage because this is like an overview of how we built the image and we will see what about security. Well, here we have the container security life cycle. As you can see, we have like new steps. For example, when we will build, we will in the stage of, in our continuous integration pipeline, we could apply static analysis tools. Maybe like the most famous in the past with traditional applications will be SAS, Das, or ACA, the compositions analysis. But right now that we are using cloud native applications, for example, container, we also want to do image scanning to find the vulnerabilities that we have when we build our image, that will be also with the dependencies that we have. And we could also have another scanning tools that could be like, for example, for Terraform or Proformation, that will be the infrastructure scanning that will help us with the misconfigurations are also with some hard coding data or hard coding secrets that could be find with secret scanning. So we have like in this stage of the static analysis, we could add in our inside of our pipeline as a step to do these checkings. And when we talk also right now, we have this new box that is supplying chains because you maybe you heard about the commons attacks that we already having in supplying chains. That means that in this stage of supplying chains, we will need to protect the source code and also the artifacts that could be, as an artifact could be also our CI pipeline or our CD pipeline. This is artifact that we need to protect, for example. When we build our source, we need to prevent some commit from malicious code. If we are talking about to build integrity, we need to ensure sign-in metadata in this build process. So we could analyze the pipeline to really validate if we don't have some misconfigurations or issue insecurity. And also in the artifact of the deployment, we need also to release integrity. Supply chains will help us with these checkings that try to do some controls to create some standards that help us in this stage. But if we see that we have this stage of supplying chains static analysis, but we also will need some tools or steps in our runtime stage that will be the runtime protection. Maybe you already hear some tools like RASP or WAF that they are protecting at the runtime, but they are very specific in some usage cases. But for example, if we are talking about some, the container and we have some attack that is happening at the runtime, we need other tools that help us to identify, for example, some process in the Linux or the kernel that is happening a bad behavior or suspicious behavior. And we need some tools or we need some protections that prevent, even prevent not only the attack. We will see, because the talk is happening, it's the talk that I will explain. It's more about runtime and EVPF and it's in the latest stage. This means that means that we need, it's good that you have these steps in the static analysis that you use SAS, DAS, that you use vulnerability tools, that also it's good and it's necessary that you use some tools for supplying chains, that you are signing, you are creating a bond, you are validated your pipelines. These are good practices that we needed, but we also need tools protection at the runtime because it's in another state that we will need. Together it will be like a more protection from the source until to the runtime. Okay, let's continue. Well, as you can see, we see it in the before slide. We always will have new vulnerabilities that is now as a zero update. So it's always will be, find it also a new misconfigurations that could be in our cluster or in any technology that we are using that we will find that it's some misconfigurations. It could be a whole insecurity. Critical vulnerabilities also it's fine. Continuously it's updating the database with vulnerabilities. That's the reasons that we need some tool for runtime because even if we do the best practice in the supplying chains or in the pipeline, the depthcups, the shift left, we always, we are finding some new things at the runtime because our applications is already running and we need in this stage some protections that help us to mitigate the future alerts, the future attacks that could happen that always is continually discovering new threats. Let's continue with the introduction to EVPF. Here we have a phrase from Brandon Greer. He wrote a lot of material about EVPF. He already has two books that if you want to know more, you can consult. As you can see here EVPF is more than 10 years inside of the Linux kernel. It's already inside of our applications, Linux applications that he's using. Linux, what problem does EVPF try to solve and how it's working? I will try to describe this and how can it help us with security, observability, etc. If we want to understand EVPF we need to understand more deeper the Linux kernel. As you can see here the main two parts, especially to do some difference how it's working our process is the kernel space and the user space. The user space is everything that we are interacting like we have some visual code that is here. We are using maybe Docker that is a client but we are using from the user space. Kubernetes is another client or even when we are trying to create a program like as we can see here we have a Golang and we will create some binary we build it and create some program with Golang or Python or Java but we are in the user space these applications could interact with the kernel space. For example we create some program that they call they need to call some devices, some device drivers or keep some memory, use the memory in the CPU we will interact with the system calls. The system calls is like an interface as you can see it's in the top of communications it's in the middle help us like a bridge between the user space and the kernel space. Here is the importance of the system calls we can think about like as an interface and in the kernel space we can imagine all the main modules all the kernel core that could be like I put some functionalities that it's interesting from the point of view of EBPF like key probes or tracing points that it's functions from the kernel that help us for example you probes interact with the user space and the key interact with the kernel the kernel functions and trace point with some system calls that is interactive we have also some well the kernel core it's very big but I put some interesting parts that will help us with some topics that I will talk in the next slides for example the Linux security module maybe you already heard about App and More or Seccom and these are security models that help us like creating some list like some access controls to create some white listing that help us with security for example try to block some specifics try to block or some special system call that interact with some device they will try to create some rules they help us with security and we also have these low at ball kernel modules that it's another way that we could extend the kernel core functions functions and with these low at ball kernel modules we could try to low at dynamically that this could extend is generally using with device that for example you have some hardware or some device driver and in this common case is when you use this LKM okay this is like an overview that you try to think about when you try to understand EVPF you have some kernel functions in this kernel space and we have the user space and we have the system calls like as interface between both the space and with that I will put some samples about that will be more clear about how is a system called okay for example here you can use the S-Trace command that give you a resume about how many systems calls you need for any activity that you are using for example I am putting cats on text you can see we have like almost 100 system calls that it means like open try to open file right and we have like many systems calls that it has to call when we are using our for example this presentation or using any problem soon we are acting so many with throw many system calls with the kernel so this is like a sample that only can give us an overview that a simple call can call many system calls so as you can imagine the events that is happening at the kernel is thousands of thousands and by seconds so many activity or events and this is the kind of information that EVPF is receiving it's trying to tracking and yeah it's like a lot of events that it's at the kernel level okay so here we have a program imagine the sample that they did before that you have a program that is calling some system calls as you can see this is happening in the user space but our program can communicate with the kernel using the systems calls you can call indirectly like when you the binary it's doing it's not maybe directed like you can see here that I am calling some system calls but for example if we are open a file behind these sections we are calling many system calls that we see in the previous slide in this case I am directly calling using this Unix point to call it can happen in many ways so we can interact with Linux kernel let's continue here we have a diagram about the Linux kernel architecture we will see that the application is in the user space and in the kernel space we have several modules that is specialized in some parts like for example in networking we will have several modules that is specialized in some topics that will interact with the hardware mainly the device drivers and depends on the CPU the characteristics of the architecture of our machine we will need some special core functions kernel functions to interact with this hardware and in this context this is the diagram about the Linux kernel we will try to think about how is creating a container what happens when we create a container in Docker RAM basically when we run the Docker run the Docker is in the user space because it's a client that we are interacting with our shell and when we create the Docker these other container runtimes they will use the system calls to interact with some kernel components like name space and control groups and try to isolate this process because we will run like a few processes inside of our containers that it will be isolated because of the name space and control groups that is belong from the kernel from the and in that way if we can think about at the end the containers is like more process that is running in the kernel in the Linux kernel and we can explore and try to see these events with eBPF so that it will be well let's try to understand how is working eBPF power in the before slide we see an overview about all the main parts about the kernel core but right now we will try to understand the eBPF as you can see here we have like a complete new section that we can start to think about the user and the kernel space and in the user space as unusual we will have some section for eBPF that we could create a program try to interact maybe with some try to create some eBPF code or maybe go along throw some wrapper but just for simple at the beginning we are using some problems when we use some problems we are in the user space but the difference the main difference when you think about in a simple way at the beginning we have some problems that we create some binary that is in the user space that when we create some problem with eBPF this program lives in the kernel space as you can see we have some eBPF program but it is living in the kernel space and living in the kernel space is like he is receiving all the eBPF events you can think about like an event driving in the kernel space so eBPF is like receiving all the events that we see in the before slides they interact with the systems calls that is happening with all the programs but we have some eBPF program that is living in the kernel space getting these events and we could create some programs that is that could be specialized like networking, observability and security when you create some eBPF program you will have like these steps that are in this section that you can see that we will have before start living as a program in the kernel space they will go for an eBPF verifier that will do some shakings and after that we will compile with this kernel that you can see is just in time and the kernel and after compiling we will have an eBPF program in the native code in machine code that after compiling after go for these steps after approve it it will be an eBPF program that is ready to receive these events and work with eBPF map that is like you can think about like a key value that will start some important parts to interact with the old systems calls and we have this eBPF help color will help us to translate these calls that is happening in the kernel but it will as an input with the eBPF programs well I think I try to do in a simple way this definition I hope you understand but if you have some any doubts if it's not clear maybe also with these designs please contact me I would love to receive your feedbacks well here in this slide we have like a time life of the features that was added to the Linux kernel and many of these features that is in the tracing part it's that some of the main components that we you will see in the eBPF programs for example keyproops, uproops, trace points and all these components that is part of the kernel it's using for the eBPF so you can think about that eBPF it's like born like in this year basically because we call the create some compile some program in the kernel space they can use all the other the before features that it's already belong to the kernel to the Linux kernel well here you can see some basic sample about some eBPF program interacting with the networking because at the beginning eBPF it was the next step for the TCP dump the first uses cases it was focused on networking as you can see here we are in the user space and here in the kernel space the eBPF program is is reading the assistance calls about networking and trying to do some sniffer so at the beginning eBPF was focused more in networking but with other features that eBPF maps and also other features CDFP it will be more complete that could interact with also with the Linux security modules and the others parts of the Linux kernel for example here in the next slide well here in this slide we can see uses case for eBPF we will have in the kernel space our eBPF program after is doing the verifier and just in time compiler with this eBPF program could use the kernel functions the kernel modules like sockets you prove that this belong to already to the kernel it's part of some functions that belong to the kernel they will interact with this kernel functions interact with the eBPF program and with that this eBPF program could help us in container security observability interaction detection because we are in the kernel level we have informations about everything about all the systems calls we only need to create a program and start to interact with the specific system calls that depends on the use case that we want to resolve for example container security maybe it will be very important to try to track in the namespace and cgroups other actions like privilege also networking depends on what you are interested you will try to hook this important part from the kernel from the main parts of the the modules that the kernel is interacting for example networking or user space or kernel space I think with this overview about eBPF we will try to describe some security attacks Tom, we will hear some eBPF use cases with security talking about Kubernetes security and with eBPF because we have access to the all system calls and all the kernel events through eBPF programs we could detect or even better prevent several common attacks that is happening in Kubernetes like for example some explotations for credential access someone has access to the service account and with this they could do lateral movements for example and try to steal information because they have already the credentials and they could try to go to the host for example some lateral movement or even if we have some privileged container someone will maybe try to do some mounts or some actions and these actions could be detected with eBPF some container escape because we can see everything that is happening is a system call an event and because eBPF is reading the testing these events we could create some programs focused on security so this is like a use case in Kubernetes we will see here we have another use case that in security maybe you will hear about rootkits that this kind of attack as this meaning of this word root it's someone that has some privileged administrator that with this privilege can do some actions in the levels of our operation system or just a space or current space and all the levels of privilege that we could get access well here maybe we will have more clear about the levels of the rootkits or the types of the rootkits because it will depend of in which part this rootkits is created to restart and we have several hierarchical protections maybe the protection rings that is very popular that you can define how you can get access to the privilege but in a simple way you can see that if a rootkits some program that will have some access to our system could be installed in the user space or maybe using some programs like low adder or using some resource in their own memory or using their own models of the kernel we have some extensions models like the linux security model or the low add wall kernel that they could use also this resource to create some rootkits and also they own the EVPF technology that we use to create some rootkits because at the end any technology also could use it for the adversary ok in the next we will see some protections how can you detect the rootkits with EVPF well here you can see that EVPF could help you to detect a rootkits for example if someone hook as is called table an adversary gains control on the song specific system function it could be like reading a file or writing a song file system and after that the adversary may also hijack the execution flow of other process and the system called hooking is considered a malicious behavior with EVPF you could identify this kind of behaviors and it generally is used by rootkits because EVPF has these ppf events you can recognize this kind of behavior generally in cloth native environments attackers usually execute rootkits on the host if you want to know more about how is happening I am leaving these links that is with more detail how is happening the attack change here and here you can see some rootkits that already happening it's using some open source that people is trying to learn or is from security that it's like testing some feature or maybe you are trying to learn some technology and you build your project and the adversary is attackers use these friendly projects that is not with hacking purpose and they use these repositories that they have for example here is a research about rootkit and that it's used in some real attacks this common thing that you see some repositories that someone is trying to learn that it's published and this is reusing for attacks purpose this is a sample for example you can find it it's a project that is trying to explain how to create a rootkit with load ball kernel modules and it was used in real attacks I will change here here we have another attack that is happening in runtime the file is attacked that it's creating some descriptor it's running the program the malicious program at memory with this if it's running at memory some of the tools like AVs, antivirus or even network tools like endpoints maybe it's common that not attack these kind of behaviors because they only detect base set and file system because in file is you are creating load in the program at memory you can detect or use these traditional security tools it's a difficult attack to detect you have here some explanations how how can you detect using linux for example maybe could use using ld preload or using the own system calls of linux like mainfd create or pitrace and with that they can have the creation of this process at memory here I put this example how easily you can create with the own linux features mainfd create that it was in the before slide you can only call in any program and in that way you can create an anonymous descriptor and you set this to run it to run in a file is mode if you want to be more detailed how to execute it I give you the details in the slide here you have a demo about that I am executing but I am executed in the file is mode using this program run it and as you can see with using Tracy I am detecting this suspicious behavior all the details are to you can do it at your home you can try it it's not a malware it's a benignus process so you can try it at your house yeah here we have that using the explanations that in the before slide mainfd create this call it's happening real attacks as you can see that here is described that using mainfd called even the same program that they use main run but they was used it in real attacks more details you will have about in this block and it was using shuri that basically use encryptions to to be more difficult to detect and with that they use with a malware in the before before demo I use only running a date a benignus process like it's been dated but in that case you can execute any other program in that like a malicious program that could be a malware well that is the end thanks so much this is the tool that I was showing here in the sample thanks so much for being here any doubt that you have it here you can contact me in the link in the description I will be happy to talk more about this content bye