 Hello, everyone. I'm Sungjin Kim, the second speaker of this session. Implementing container privilege escalation detection using EVPF for cloud-native security. EVPF is an innovative feature of Linux systems for safely run a user's code in the kernel space. This is widely known for performance instrumentation and system tracing. But it could also be useful for the container security. In this talk, we discuss how EVPF could be used to detect malicious privilege escalation that causes a container escape. Containers are relatively lightweight and portable virtual instances compared to virtual machines. It packages applications and all its dependencies into a container image. Containers allow us to package and deploy large-scale application workloads quickly. These features make containers more popular in microservice and DevOps environment. From a security point of view, it has a thin isolation layer. It relies on the namespace and a C-group mechanism provided by the Linux system. The namespace provides containers with the process, network, and IPC space separate from the container host. And the C-group restricts the resources on the host that containers could use. Those isolations create minimal barriers between containers or between containers and the container host. However, since they share the same host corner, containers additionally leverage the Linux system's kernel security mechanisms. Capabilities at Amor and the second are the security mechanisms for restricting system level functionalities. The following figure depicts a system architecture with containers. It shares the corner of the host operating system. And the container runtime performs an isolation by using the namespace and the C-group when a container instance is created. Hardware virtualization by the hypervisor is not performed in a container environment. Unlike virtual machines, it does not require a guest operating system, so the performance overhead is relatively small. Instead of a slow virtual machine boot up, you could have a fast application boot up times in seconds. Sharing the kernel without doing a hardware virtualization offers performance advantages. However, sharing the host corner could cause a serious security threat. For example, if there is a security bug in the kernel, an attacker process inside the container could break out the container and compromise the resources of the host system. This is called container escape attacks. Specifically, it refers to an action or state that goes beyond the bounds of a given resource or permission for a container. Among them, a malicious privilege escalation is a prime example of container escape. The host corner may have multiple vulnerabilities. Many attack factors exist, such as memory corruption, cold injection, and race conditions. Not all kernel vulnerabilities support container escape. Recently, many security researchers checked the container's impact when they discovered a kernel vulnerability. And when passable, they refer to that as a vulnerability that could cause a container escape. Also, some exploits are blocked by the isolation layer or a security mechanism. However, some security settings do not mandatory or the vulnerabilities still affect container security if you disable several security settings, such as a default second profile. For example, it can be safe if there is a security bug in a system call, but the default profile blocks that system call. However, in some container runtimes or orchestrators, the second profile is not mandatory and may cause a container escape attack success. So let's talk briefly about the container escape scenario. Suppose the kernel has a buffer overflow vulnerability. First, a remote attacker with access to container number three exploits the remote code execution bug of application three to execute a kernel exploit code. And then a part of the exploit code could be executed with root permissions. And then an attacker could use privilege escalation to disable security settings for the post-exploitation procedure or execute a backdoor. In this flow of attacks, the privilege escalation is one of the attacker's primary target. The figure is MITRE's container attack metrics. We could see that after the initial exploitation, there is a stage where an attacker requires a privilege escalation in order to continue malicious behavior, such as data exfiltration or lateral movement. Here, we briefly describe how exploit codes for Linux systems, including container environments, perform privilege escalation. They contain the processes credential structure and two main kernel functions. There are some things that need to be done before a privilege escalation. But we will only summarize the required parts to explain the defense mechanism. The prerequisites include hacking methods to bypass protection mechanisms, such as SMEP, SMAP, or KASLR. The figure shows the data fields of a credential structure. Attackers focus on tampering with values, such as UID, GID, or CAP permitting. The prepareCred function is used by the kernel to change the privileges of a task. That function creates a new credit structure containing the credential information you want to change. If the demo passed as an argument is null, an initCred structure with credential information set to root privilege is returned. The commitCred function must be called to update the processes credential. That function is responsible for installing new credentials into the current process. The process refers to a new credential structure through the RCU readCopyUpdate method. The reference count is decremented while referencing the newly created credential structure and releasing the previously referenced credential structure. This single line of code is well known to be used by attackers. Many exploit code gets root permission by securing that code. The prepareCred function takes 0 as an argument and creates a root credential with all capabilities enabled. Then reference the newly created root credential via the commitCred function. As a result, an attacker process could gain root permissions along with a shared kernel vulnerability. Many previous studies countered previous escalation through kernel exploits in Linux systems. Both studies account for the widespread abuse of the single line of code discussed on the previous page. The first study, ACCO, noted that mandatory access control on Linux system is insufficient to combat kernel exploits. In general, the credential of a user process is changed by the system calls such as senuid or setgid. ACCO performs a series of observations by modifying the system call handler. It detects if the credential data field changes when several system calls other than legitimate ones such as senuid or setgid. The second study covers the container environment. The container rarely calls the commitCred function or changes the credential data except during the initial boot up process. So it modifies the commitCred function to check whether the process that is called that function is in the container or not. If the process is inside a container, you may suspect that malicious behavior has occurred. The third study implemented a reference monitor in the corner that monitors changes in the credential structure. The monitor verifies access to protected data according to predefined policies. For example, you could allow only a high-listed process to change a credential structure. The fourth study uses a method similar to ACCO to detect tampering with sensitive data through system calls other than legitimate ones. The following table is a summary of the four previous studies mentioned. All studies focus on the tampering of credential structure and the detection logic is secured at the time when a system call or a corner function related to privileged escalation occurs. This table shows the implementation of previous studies. All studies have modified the corner and in order not to affect the functionality or performance of the corner, the location of the code changes carefully selected. And only the minimum code is added. However, these attempts can increase operating costs in a production environment. Although existing studies have minimized the side effects on the corner, a build or test system is needed to maintain the changed custom corner and respond to corner updates. Moreover, there may be a risk that the detection logic won't work depending on the corner update details. In other words, the implementation of the previous studies has fundamental limitations in practicality. EBPF could help practicality when running user code in the corner. It is a default feature built into the corner and users could write code that runs in the corner while complying with some constraints of EBPF. It also has a verifier that checks for a corner crashes, which is better than no verification, at least in corner modules or custom corners. For the detection logic implementation, the user could attach the EBPF code in K-Pro to hook the commit crisis function. Hello, I'm Inhyuk Jang. I will talk about implementations in this presentation. With the rapid proliferation of cloud-native environments, the corner's role has become more important. In particular, in a cloud-native environment, it has become common to isolate and operate multiple containers on a single host. So the consequences of corner vulnerabilities or excessive privilege assignment may be more serious. From another point of view, it can be seen as a better way to monitor the corner's privilege management in a cloud-native environment. As correlations of privileges within a container are less common than those on a host, because each container has what the application needs to do and what permissions it can have. This is because assuming container best practice is normal that no work is observed during the container execution that demands new high privileges and undermines container immutability. From this perspective, we think it is meaningful to observe the critical data structure in which the corner's privileges are managed and the privilege changes in the container process when the privilege change takes place. This is especially true because EBPF enables realistic host-based monitoring. We apply the following principles to observing the elevation of privilege in the corner for container security. Here, we compare the current privilege with new privilege when common create is called to check whether privilege escalation occurs. Also, check whether the new privileges are more escalated than the lowest privileges observed from the creation of the container process to the present. Finally, check whether the new privilege is greater than the parrot processes privileges. Serious method can be considered to monitor for violations of these principles. First, consider tools using EBPF. Also, for direct implementation, you can consider implementation using BPF Trace, BCC, and consider BPF Core for portability. The first thing that comes to mind as an EBPF open source tool is BCC tools. BCC tools provide various tools for observing system performance, but unfortunately, there is no tool for observing privilege changes. However, there is a capable that monitors CAP capable. This may allow some degree of monitoring of privilege requests. However, it is necessary to add the corresponding function because the return value is now observed by default. Also, it does not show information about the container process we are interested in. There is an option to exclude container process. The following is an example of the results of running the capable tool. You can observe which process requested which privilege. It is hard to tell if a process actually has privilege because the return value is not observed here. You can also consider FARCore, a CNCF incubating project as an EBPF open source security tool in a cloud native environment. FARCore performs monitoring, mainly on system calls. It defines detailed patterns for individual instance as rules. It is difficult to directly observe the corner's credentials we are trying to do. The following is one of FARCore's privilege escalation detection rules. It detects the well-known pocket, which is written to detect running PKEX EC by unprivileged users. You may also consider TRACY on EBP open source security tool. TRACY consists of TRACY EBPF, which observes and outputs various events of the corner using EBPF. And TRACY rules, which generates a security running by receiving the output of TRACY EBPF as input. First of all, TRACY rules doesn't seem to have any rules for credential change monitoring. This figure is a list of signatures provided by TRACY rules. There are various rules, but we don't see the rule we want. TRACY EBPF observes changes in UID, GID, and capability when COMIC CREZ is called. If the current CREZ differs from the new CREZ, when COMIC CREZ is called, an event is raised. The following figure is TRACY EBPF called which TRACY is COMIC CREZ. And you can see that an event is sent if there is a difference between the previous CREZ and new CREZ. If TRACY EBPF is executed with the TRACY event equals to COMIC CREZ option, only related events are output. TRACY EBPF also provides functions for tracing various events and acquiring artifacts. Therefore, it seems that using TRACY EBPF properly can provide the wealth of information. The list of events provided by TRACY EBPF can check with the list option. PPF TRACY is a high-level tracing language, and you can use the EBPF by writing CREZ conveniently. PPF TRACY is relatively well documented and with many examples, one-liners, and even books. PPF TRACY is mainly used for performance measurement. With PPF TRACY, you can read and compare the current ID values with the new ID value when COMIC CREZ is called using K-Proof. And output C if there is an evaluation of privilege. The main part of the source code is extracted. Because it is a high-level language, it is very easy and straightforward. It was very helpful when I first encountered EBPF and it is very useful when you want to simply check what's happening in the corner. The following figure is the result of executing the PPF TRACY code. You can see that the changed ID values are printed along with the process information when the user ID or GID is escalated. PCC is a toolkit for creating efficient corner tracing and manipulation programs. It provides good documents, examples, and tools. Using PCC, you can write corner instrumentation in C and front-ends in Python and Lua. In order to detect CREZ changes that violates the three principles presented previously, in addition to comparing the current and new CREZ by tracking COMIC CREZ, a map that maintains the lowest CREZ of the container process is used. When a task is created, if it is not a host process, the process information is inserted into the map and when a task is finished, the information is deleted from the map. This may maintain basic information and create information of processes, such as PID and CREZ. When lower CREZ values are observed for the same process, using this map, we compare whether the new privilege is higher than the existing lowest privilege or higher than the parent's lowest privilege. Additional tracing points other than COMIC CREZ can be used to compare current privileges with previous CREZ privileges. In addition, we compare current privileges with previous CREZ privileges. The default detection tool detects and prints container and host privilege escalation events according to the options provided. When the privilege escalation detection tool is executed and privilege escalation is attempted in the container, a warning event is generated, as shown in the red part of the figure. At this time, type 1, 2, and 3 are printed together, depending on which value is compared with the new value. Although a function to detect a security event was created using eBPF, two distribution information collection pipeline and user interface are required to use in an actual environment. You can develop these yourself, but I thought it would be nice to have an open source already prepared. I discovered the Pixie project by chance in early 2022. Pixie has everything I mentioned earlier and is an open source of the CNCF Sandbox project. Pixie's data source layer, Strolling is a module that collects the data needed for Pixie from the corner. This Strolling is based on PCC and developed in C++. Strolling is implemented as a data source that utilizes eBPF and a source connector that allows the data connected from the data source to be utilized at the top of Pixie. Therefore, if the tool developed as a Strolling data source and a source connector is created to show it, we can use Pixie for security observability, allowing us focus on implementing security features using eBPF. So far, we have looked at ways to implement privileged escalation detection using eBPF for container security and methods that can be considered for practical use. Thank you for your concentration.