 Hello, everyone. My name is Charu Lee. Thank you for joining my presentation. Today I will share my experience about rebuilding security critical program binary in Linux with compile time hardening options like safe stack and CFI. Introduction. Why compile time hardening is important. Think about zero day. When program developed, some vulnerability introduced. The problem is that until the security patch, some security hole period happened. Security hole starts from exploit released in the wild and in the end patch deployment completed. The problem is that many attackers knows the vulnerability and they make exploit code. And they use this exploit code to corrupt some new system. If set attackers, nobody knows the vulnerability. So this means in that period, the program remains under attack. And how we can protect a program having exploit the compile time hardening provide protection mechanism. So by compile time hardening, the binary have some security technique inside, so reducing some attack possibility and actually remove attack point. In the end, we can raise the bar for attackers. In this chapter, I explain why I choose safe stack and CFI compiler hardening option. First, we have to understand what is control flow hijacking. Left picture. You can see the control flow hijacking flow chart. For helping understand, I prepared some memory map. For control flow hijacking at first, the color data pointer. Data pointer is BOF buffer with size 10. If we want to corrupt this data pointer, BOF buffer, we input oversize like 15. And next step is modify code pointer. In this slide, code pointer is return address. So we modify return address to some address of a shell code. This is step two and three. And for executing injected shell code, we use return instruction. At the end of function, return instruction is called. So if we override return address by shell code address, we can jump to address which store a shell code. Of course, shell code is in a executable memory area. Finally, we change the original flow of the program. In this page, you can see the some protection mechanism against control flow hijacking attack. There are many, many research and techniques, but we only choose mechanism, fully implemented as part of GCC or CRAM. First is memory safety for preventing corrupt data pointer. The address is used for memory safety testing, modify option, modify code, more secure, or we can develop some system programming by memory type safe language like Rust. Second, for preventing modifying code pointer, check stack check code pointer integrity, specifically in the back of the edge, like function return. Third, for making hard to search or address of gadget or shell code, which can be used to attack or we can use randomization, ASLR or SAP LANDO. SAP LANDO is not fully implemented in GCC or CRAM, but check stack tool, check the SAP LANDO, so we include it. And fourth, for preventing using some pointer, like return pointer or indirect call pointer, the bad guys or attackers using this point, bad purpose, called control flow integrity technique is needed. The backward edge stack gallery is widely used, but control flow integrity, specifically in indirect call, is not popularly to use for hardening. Finally, for preventing executing abnormal code, we can set memory property to non-executable by NX-BIT or VELO options. This is the default compiler hardening option in Debian package. For memory safety, Spotify is enabled. For randomization, ASLR is enabled. And for control flow integrity, stack analysis is enabled. And for non-executable data, NX-BIT, VELO option is enabled. So, what is the problem? By this default Debian hardening option, we can only stop this path. This, this, and this, and this. We cannot stop this path, modify your code pointer and use point by indirect call and jump. So, Debian default option is not enough to prevent control flow hijacking. For raising the bar for attacker and reducing possibility of attack success, we add more, more hardening option at this point, this point, and this point. Save stack option make awesome bar at this point for preventing modifying a code pointer and save by make a bar for preventing some indirect call violation. There is one question. Where is the bar at this point? Making harder for preventing, executing available gadget or function. This is very, very difficult problem. So, purely practical solution is not yet implemented in GCC or CRAM. Let's talk about what is safe stack and control flow integrity. This is important to understand how build error and testing error arise. So, we first understand a safe stack and control flow integrity. Safe stack is part of code pointer integrity project. So, theoretically, GPI is stop all control flow hijacking attacks, but overhead is very, very high and not implemented in GCC and CRAM. Safe stack is only to stop backward edge, but overhead is almost none and only implemented in CRAM. Let's see the concept of LFM safe stack. Safe stack divides memory stack into two. Unsafe stack and safe stack. Safe stack object is stored on unsafe stack and other object like RIP, IEP, functional argument, or a safe object is stored on safe stack. Let's think if there are buffer which can be overplowed like this. The buffer is decide unsafe object and stored on unsafe stack. Two stack, unsafe stack and safe stack is separated. So, unsafe object server flow doesn't impact safe objects like return address. And then reduce the ability of attacker to redirect program execution to an attacker controlled destination. The full control program is constructed in link time optimization. A perfect control program is undecidable problem so we can get approximately CFG. If program go abnormal flow, CFI impulsements mechanism detects a violation according to the generated CFG. CFI is fine grained CFI. Intel CTIBT technique is also coarse grained CFI but precision is low so fine grained CFI is better and precision is better. LVH CFI is used type-based control flow integrity. This is concept of LVH CFI. LVH CFI make CFI jump table like this. In this source code there are function but, but, but, but. When link time optimization for making CFI jump table, functions are founded with same function type. This is function pointer have type void to void. And CFI have to find void and void type function type. But, but, but have to excluded because both are not assigned in the program context. But, but and but is assigned in the program context. So more precisely function pointer analysis performed this work. By this function pointer analysis removed the but's choice. So in the end, final CFI jump table have but, but functions. Now we only check whether a function pointer is between but and but functions. If buzz function is assigned in the context and buzz function is defined in dynamic library. How can we deal with this situation and what if it does program link dynamic library. LVH CFI support to protect indirect core across dynamic shared object boundaries. But this option is experiment but work well. But honestly many build errors are non-symbol problem. Target packages. Practically, rebuilding all packages is impossible in Debian 10.9 1363 packages installed. So there are so many packages. So we set higher priority in a aspect of security. In Debian community provide some guideline higher priority audit guideline. So there are four categories. The first one is CIG binary which is installed send UI or CIG and anything which provides a service over a network and any remotely accessible CIG PHP script. Anything which contains a cronjab or other automated script which runs with root privileges. But we defined higher priority because we focus only focus executable binary. So this is our redefined higher priority. Any binary which is installed send UI or CIG and anything which provides a service not only over a network. So for send UI and CIG we found 29 binaries 28 packages. And for service we found 123 binaries and 36 packages. And let's rebuilding with hardening option. Generally Debian package building process have three part. One is this part, the package source. This command get we can get by this command we can get source code. And second is MK build depth by this command we installed dependence packages. This difficult build package command actually building Debian packages. For enabling compiler option. And we can append like this. The package build package is option. Depth, she flags up and and CXX flags up and and LD flags up and the first two one is safe stack build a safe stack compiler option. And at the last, last one is CFI compiler hardening option. Our first target is a system the package. As you already know that system these very, very important. And they have PID one. Now I'm talk about how we build a system D and solve the build error and testing error problem. The first problem we meet is link time error. During the link time CFI related function can't found. This CFI slow pass undefined the reference. So this is this error is alright by no undefined the defined option. This is Mason default option system D use Mason deep build system. So when building shared library undefined symbol will be shown as link error. For this for solving this error, we append on reserve the symbol option link. This is link linker option. And we can disable no undefined options. So we solved this problem. But there are another problem testing error. After solving build error we encountered so many testing failure error. If we do not pass this testing we do not get Debian package files. There are many 52 fail and that commonly related to CFI violation. For example, the testing of system D TMP files is cute like this command and this output STD error, but this is Python script error. The actual error is illegal instruction error. This illegal instruction error is realized by CFI violation. If the CFI check fail, the UD2 instruction is cute. On the right side, you can see where violation arise. In this case, base bucket hash is the location function. In the base bucket hash function, hash is called the hash is function pointer. At this point hash refers to string hash function. The string hash function is the target function. At this time, CFI check whether the string hash is in the CFI jump table like this. hash has hash function type hash have hash function type hash function t type hash function t type have some function type get two argument void structure and return void type. But string hash function have function type character and structure argument and return void. So, this is time mismatching. This is CFI violation. Also, string hash function is not in CFI jump table. Let's think about this is really wrong. This is general interface design in C program language. This callback architecture is broadly used in C programming. But in aspect of security, void type argument is not recommended. The table shows the headers related to CFI violation. Many callback functions are in the header. Most of the CFI violation arise in the header hash function h. There are 46 violation. These are all related to CFI violation. And there are many, many callback function like string hash function. And last, this is testing error with save stack. And 74 fail happened. Like CFI case, no pass, no depth. This is this error of segment fold. This error arise in RD land function. Simply said, this assembly code is very, very old. Follow me. RD land save random value to register RAX. And set B instruction save the result of RD land fail or success to some address saved in register RAX. This is a strange point, RAX used as address, but RAX is random value. So segment fold arise. This RAX register must be replaced to another register. Why this happened? This is because of instrumented code by save stack. In some reason, when compiling through save stack instrument pass, register replacement problem remain unsolved. Actually, this is save stack problem, not a system problem. And how can we fix CFI violation? Simply, we modify time mismatch part, character time to void. But this is not a good solution because the source code have to be modified. Or we have another option, we can change callback architecture, but it also have many cost issues. So we handle this problem by using LFVM sanitizer special case list. This is blank list, which have function and source code name. The function and source code written on blank list doesn't be checked by sanitizer like save stack or CFI. Right side is syntax of sanitizer blank list, a function name or source code name. Right side is sanitizer blank list for system D. Finally, a panned blank list option like this, we can pass the all test case, which is failed by CFI and save stack. And we get Debian package pile. Additionally, and we checked whether the binary is really compiled with CFI save stack compile option. The tool is, it's a bash script to check the security properties of executables. Many attackers or security support use these tools. And you can see the check set result. C length CFI and save stack is implemented in the binaries. And now we rebuild packages in virtual environment. And each package have each one virtual rebuilding environment like this. APT, bash, color D, call utils. After building package comparatively, we snap shut the virtual machine, especially D bus package have dependency on system D library. So we have to rebuilding process after building system D and after installed hardened system D library. We building, rebuilding D bus library, D bus packages. This table showed the result of our result. And we identify 123 services. Each services have a binary. And each binary is included in some packages. We rebuild each packages and reach almost 87% and 86%. And another option 90, almost 100%. But that is not a 100% result. There are many reasons failed to rebuilding with the CFI and save stack. The reason explained later. This is a senuide set GI the result. And there are 20 29 senuide set GI the binaries. And we reach it almost 90 93%. And in CFI, save stack, another options are reached almost 96%. Last chapter. We have been get many, many experience from rebuilding lots of packages. If you try to hardening this package, you may meet the following problem I already made. First is GCC dependence problem. Some packages that default CC is GCC. So we have to modify default CC to C-Lang. And another problem is build error by C-Lang. Some features are not compatible with C-Lang. As you can see the Debian Wiki, this Wiki, Debian community continuously revealed testing with C-Lang. And there are many types of errors actually to solve the problem. In some case need to modify source code. That is so hard to modify source code. So this is hard to do. Another issue is testing error. In the back system case, if there are callback function and indirect call, CFI violation arise. Yes, this can be solved by Sanitizer blacklist. But we have to choose. We use blacklist or we modify source code. Blacklist is another security hole. So we have to decide. And another problem is not all packages have test case and test suite. System have a lot of test suite and we can find many errors before install and execution. But if there are not test suite, after install packages error may occur during execution. So these errors, these issues, just dependency issue and testing error issue, you may encounter during rebuilding packages with CFI and safe stack. Conclusion. Most packages can be revealed with safe stack and CFI. But that is not perfectly rebuilding error. But it was a try. And for more secure Linux system and raising the bar for attacker and reduce some possibility of attack success, you have to try an error. Thank you.