 Okay. Morning everyone. I'm Xiaocheng Zhou, a Ph.D. student at the UC Riverside. I'm honored to be the first speaker in this year at the Linux Secure Summit. My talk is about my recent project, Cisco. The inside of Cisco is that we systematically measured 75 percent of Linux bugs from Sysbot and find out that a lot of minor bugs like wording or invalid pointer dereference, they can still have critical security impact and even lead to kernel compromise. A little bit background knowledge here. Our study focused on Linux kernel. So the most efficient way to find bugs is fuzzing. Fuzzing is the automatic process that tries to discover the software bugs by giving concrete test cases. The state of our father for Linux kernel is Syscaller. So recently the continuous fuzzing platform is taking more places of the bug discovery process. For example, Sysbot is the biggest continuous fuzzing platform targeting the Linux kernel. In the past four years, Sysbot had found more than 4,000 valid bugs. 3,000 of them had already been fixed and 1,000 of them is still pending for patch. Among the 3,000 fixed bugs, the biggest portion is the memory bug, including K-some bugs or KM-some bugs. There are like around 1,000 of them. The next biggest portion is a warning and info bug, so it reports some minor issues in the kernel. It's around 900 of them. And the third portion is the general protection fault and some invalid pointer dereference. It's around 500. So in some sense the general protection fault and some invalid pointer dereference, it might also belongs to the memory bug, but here we just separate them from the general memory bugs, like used after free or autobomb, we separate them. So the kernel bugs can be simply categorized into sanitizers or kernel assertion. Sanitizers seem to catch several specific type of bugs, like used after free or autobomb or double free. So we have kernel-adjust sanitizers. The title showing on C-spot is like K-sum, used after free read in some function. We also have K-sum captured the data response and KM-sum captured the uninitialized use or UB-sum defined behavior. Besides the kernel sanitizers, the kernel assertion is some sort of code that embedded in different kernel places to capture some abnormal behavior. For example, if a variable shouldn't be less than zero, so the developer can put a check to see if the variable is less than zero, then we can throw a warning. This is the meaning A type of bugs. They occupy around 75% of bugs from C-spot. Oh, so there's another strange category just called bug. This bug contains a lot of small type of bugs, like unable to handle kernel page or sleeping function or least entry not found. So yeah, it's just contained a lot of small type of bugs. And general protection for it is usually happened when you access the user space memory from kernel or you write to a read-only data, it will trigger general protection fault. Now let's take a look at the security impact of each type of bug. Memory write bug has severe security impact in terms of compromising the kernel. For example, by overwriting the UID GID attacker can achieve the local privilege escalation. Or if one can overwrite the function pointer, he or she might hijack the control flow, which is also considered as a top risky exploitable capability. Therefore, we think it's fair enough to claim that the memory write bug high risk bugs. And invalid free is another critical impact that may compromise the kernel. So if the object of freeing is controllable by the attacker, it can be easily converted to a user after free. So you can allocate the object in between the two free. So it's also high risk. Other than that, we have memory read bugs. The memory read bugs can leak kernel information like KSR offset or other kernel address. It's useful sometime, but they are not mandatory in kernel exploit. We also have some non-security bug, like warning info general protection fault and bug. They were non-security bugs. So we think the memory, the warning, the info general protection fault and the bug, they were low risk bugs, which means by using their capabilities relate to their bug type, like read, like invalid pointer dereference, we don't think they are strong enough to compromise the kernel. So we think they are low risk bug. Let's talk about the motivation of our project. The first is there are two million bugs to fix. These bugs continuously report two or three bugs every day, but sometime it took us weeks or even months to fix one. Besides the reason of the bug's complexity, we also find there is a prioritization of risk level in bug fixing process. For example, more serious bugs like zero day will be fixed as soon as possible. From the delay between bug found day and the first patch attempt day, we find memory write bug took relative less state than the memory read bug to fix or other non-security bugs. So therefore, we believe how people correctly understand the severity of a bug is necessary and important. Imagine a warning bug that can compromise the kernel, but somehow people mislabeled its severity and didn't patch it for a long time. The consequences could be terrible if a bad guy managed to exploit it. The next motivation is there's too many patch to port. The main line is the upstream kernel and the downstream kernel like Android, Ubuntu, Fedora, they will fork a long-term support version and create their own distributions. Usually the patches that were applied to the downtown support version will also apply to their corresponding distributions. Sometimes the distributions like Ubuntu may also cherry pick patches from the upstream kernel directly. However, the patch propagation is slow and inefficient. The maintainer may not know which patch are urgent and which one are unnecessary. For example, the CVE 2019-215, it was initially reported by C spot as UF read and fixed in Linux upstream in 52 days. Unfortunately, it took over a year for the patch to propagate to the downstream Android kernel because of lack of knowledge on its security impact. In fact, it's only until a bad actor were caught exploiting this vulnerability in the wild, then Google start to realize its severity and obtain a severe number. The reason is that the maintainer do not know the correct security impact of a bug. So we will see if a bug clearly shows a more critical impact like auto bound write, the patch will propagate to their distribution sooner than a bug does not show its security impact like wording. Therefore, because of misunderstanding the severity of a potential high risk bug, a downstream kernel may remain unpatched for quite a long time or even forever. So what is the specific problem Cisco want to solve? We prepare three questions. Are all seemingly low risk bugs actually low risk? Do bug reports always reveal the real impact of bugs? Interestingly, in our research, the answer turns out to be double no. And can we convert a seemingly low risk bug to high risk bug automatically? Yes, we can convert some low risk bug to high risk bug and even automatically. And the original bug report from C spot or other conventional kernel fathers sometimes incorrect or misleading. The goal of Cisco is to reveal the high risk impact of seemingly low risk bug. And Cisco does not aim to produce the end to end exploit automatically because this part of job has already been done by the following two works, Kuby and Fuse, they target both auto bound write and use after free bugs. Okay, let's jump to our insights. The first insight is the first impact of a bug that appears in the execution path. In this case is the wording here. Even though we can capture another two, but just let the fuzzing continued execution executing without stop. However, the conventional father does not do that because does not do that. They was just stopped execution, panic the kernel. This makes sense because the father only cares of bugs, no matter what bug it is, secure related or not. Because all bugs need to be fixed. So the very first impact only, the very first impact is good enough to say, hey, there's a bug, you have to fix it. But this methodology ignores the fact that lack of resource to fix all bugs before they got used by the bad guys. So correctly understanding all impacts of a bug definitely help us in the bug fixing process. However, the only impact of a bug on C spot is the one showing its bug title or the bug report like this. It literally showing no impact, like no details. And it will ignore the following impact behind the warning. So first of all, the first bug impact does not always show the most risky impact. Therefore, we may be misleaded by the bug title and ignore the potential high risk impact that just having a low risk bug title. Another problem is even we let the buzzing continue without stop. Sometimes it still won't catch the following impact behind. For example, the UF read and write here. The reason will be presented in the motivating example later. And so we knew that UF will be right and the inventive three are high risk bugs. We just talk about it. Now we consider the following three impacts are also high risk, the country flow hijacking, the arbitrary or constrained value write, arbitrary or constrained address write. However, these three impact cannot be detected by adding sanitizers or current assertions. Therefore, we call them follow up impact. We will talk about the reason why buzzing cannot detect these three follow up impact in the motivating example two. Okay. So this is the real case from CISBOT, the motivating example. But I simplified the source code. The root cause of this bug is because Cp hash is bigger than the length of the perfect array. So when the loop index I is bigger than the length of the perfect array, the EXTS, the EXTS here would be autobond. And because the EXTS is autobond, is autobond, the case on will capture the autobond read at 98. So when the EXTS tries to de-reference the data pointer action, so it will capture by the case on and report to the CISBOT as slap autobond read in the TCAF EXTS destroy function. So what if we let the buzzing continue instead of stop at the first bug impact? We will enter the function here, TCAF action destroy. It looks like we can find an impact N916 because of the action comes from the EXTS, which is autobond memory. So the action, its value can be arbitrary. Therefore N916, we write zero to an arbitrary memory address. So let's just call it arbitrary address write. So if we let the buzzing continue instead of stop, can we capture the arbitrary address write later N916? However, it's still unlikely for buzzing to find arbitrary address write, even we let the execution continue. There is two reasons. The first is N914. The array pointer actions comes from the autobond memory, so its value is unpredictable. If the pointer, so the EXTS is in the autobond memory, and the action comes from autobond memory, so its value is unpredictable. It could be invalid memory like 40 or a valid memory. So if it's an invalid memory address, it will panic the kernel directly. Even though the action is a valid memory address, as long as the action i is zero, it will still not enter the correct branch because its force, it will enter the false branch instead of the true branch. Beware the action comes from the autobond memory. Fuzzing has no power to control the memory content in the autobond memory. So the father cannot determine what value the action could be. So the second reason is that even the father is really lucky to satisfy the check N914. Father captured nothing N916 if the action i is on a legitimate memory address. So like the action is in the EXTS, and the EXTS if it points to a legal memory, and we write zero to a legal memory, so everything is normal, nothing happened. The best scenario for the fuzzing is that the action happened to point to a to a illegal memory like autobond memory. Then the fuzzing can capture an autobond right at here. So as long as the action is pointing to a legal memory address, the fuzzing would capture nothing. It will just consider it's just a normal right. So eventually the fuzzing still failed to identify the arbitrary address right at N916. But if we are the attacker, it's easy for us to bypass the check N914, because the action comes from autobond memory. We can use hipspring to make it point to a valid memory address and with a long zero value in it to satisfy the check over here. Then we will find the arbitrary address right, because the hipspring we can literally control the entire EXTS which is autobond, and we will control the action. Therefore, we can put any data here. We will satisfy this check. And if we look further down, we can even find a function pointer. We can look, so we enter the action clean up function, and the A comes from the action I, and becomes a P here. And the P dereference loops, and the function pointer clean up. So if we look further down, we can even find the function pointer to reference N923, which gives us counter flow hijacking. And of course, by using the hipspring, we can also control the loops and the clean up, because they all come from the action, and the action comes from the autobond memory. But so far, all the further impacts, including the arbitrary address right, and the function pointer dereference, just they were found by human expert. We want it be automatically. Is there any automated solution that achieves the same results? So let's start from the problem itself. The first problem is the father has no power to control the content in the UF OB memory, but the human expert can't buy the hipspring. The father, and the father may be blocked by the kernel panic or unsatisfied constraint, for example, the one at my 14. But the human expert won't be blocked because by using hipspring, he or she can construct a dedicated action pointer to satisfy this check. So as long as there is a technique that allows program to determine the value in the OB or UF memory, it will find a correct one for the action. The technique is the symbolic execution. So symbolic execution is a program analysis technique that use symbolic value instead of concrete value during the program execution. So let's skip the technical details of the symbolic execution. Let's just say the symbolic execution can achieve the exact same goal of the hipspring here. We symbolize the entire OB memory, and the symbolic data will propagate to the action and the ops and then the cleanup. So if we want, we can solve the symbolic expression and we will have the concrete value for the entire OB memory, which is the EXTS here. We can have the concrete value for this, for the action, for the ops, for the cleanup. So basically, we will know what value to put on this memory will lead us to the function pointer dereference. So for example, the symbolic execution will tell us if we want to enter this branch, the true branch, the action must be a valid memory address, and the action I must not equal to zero. Otherwise, it will enter the false branch. So the symbolic execution will tell us this knowledge. This is for the motivating example. Let's talk about the modes of Cisco. We have two modes for the upstream open box, Cisco package static analysis and symbolic execution. For the fixed bug, Cisco package fuzzing static analysis and symbolic execution, excuse me. We will notice that only fixed bugs have the fuzzing process. This is because the fixed bugs have the patch, which we can use to eliminate the new bugs, because the fuzzing will tell us a bunch of new crash. And if we don't have the patch, we don't know which, if the new crash is still belongs to the same bug or just a totally new bug. We will talk about the detail later. The workflow for Cisco has three components, the vulnerable context exploration, hidden impact estimation, and the exploration validation. They both relate to fuzzing, static analysis, and symbolic execution. But here, we won't feed our talk in 40 minutes, so we ignore the static analysis part because it used to accelerate the symbolic execution. It's not mandatory. So we just keep the fuzzing and symbolic execution. So the fuzzing will try to find more new crash, or we can call it a new context. And each new context of the bug will feed to the symbolic execution. So the more context we find, the more chance our symbolic execution could convert it to a high-risk bug. We will talk about the fuzzing first. The main goal of our father is to find more buggy contexts that share the same root causes of the original bug. And we do not stop at the first impact. Like the original bug is the warning. So we do not stop at the warning. We let the father capture multiple impacts without stopping. So we may find the UF right behind the warning. And the strategy for our fuzzing is we use the original POC, the original POC from C-Spot. We use it as our input test cases and apply a conservative mutating strategy. For example, we do not, we do not enable all the system call. We enable like only system call in the corresponding module like network or like KVM. So now we have three, we have, we also find two double three. Let's see, we also find two double three crash. Now we will determine if the new crash still belongs to the same bug. Okay. This is the new context verification part. Now we have two buggy contexts. We will verify if they are, if they are not some new bugs. Therefore, we use a patch to eliminate the new impact. We first apply the patch to the vulnerable kernel to make it invulnerable to those original POC. It means the new patch belongs, if the POC of the new context, we try to rerun it on the invulnerable kernel. If it still triggers a new impact, which means it's a new bug, we will abandon it. So otherwise, if it's nothing been triggered, we consider it's still belong to the same bug, but it's a new context. We will keep this context. So this is how we verify the new context. And we will let father capture multiple bug impact along one execution path and pick up the most high risk one. In this case, it's a, it's a UF right here. And we also have impact feedback. We consider each new test case that trigger a new bug impact is interesting test case and the worst to mutate. So this is based on the code coverage feedback. Please note that code coverage feedback is not all the goal of our father because we want to, we want to trigger different context of the same bug, but the code coverage could sometime mislead us to some unrelated code region. So we have our impact feedback. Let's talk about the architecture of our symbol execution. Before we do a symbol execution, we will actually trigger the bug in Qemio. We set a breakpoint as a case on report to freeze the kernel. So we launched the anger and our symbol execution engine. We symbolize the UFOB memory by inspecting the case on report. We symbolize them. And we transfer the register's value from Qemio to the anger and dynamically pick up the memory content when we need it. Our symbol execution will verify the following five high risk impact. The UFOB write, we will check if the write, if the write2 address is in the range of the UFOB memory. We just got from over here. We know the UFOB memory, the range of it. So we will check if the write2 address is in that range. So it is, it will be the obuf write. For the invalid free, we will check if the arguments of the k3 is symbolic value. If it's symbolic value, which means it comes from the UFOB memory, which means it's contruable. We will consider it's invalid free. The control flow hijacking, we will check if the k2 address is symbolic. The arbitrary constraint value write, we will check the write, the value will be right to this address. So we check the value if it's symbolic. And does it have any constraints? If it has any constraint, it will be the constraint value write. Otherwise, it will be arbitrary value write. And we have the arbitrary constraint address write. It's the same theory, but we check the write2 address if it's symbolic. Okay, now the dataset of experiment from C spot. We pick up all the memory read bug and wording info and general protection for N bug. So they occupied around 75% bugs on C spot. After deduplicating and rule out the cases without a valid POC or the one don't target on the upstream kernel, we finally got around 1100 valid bugs. We conduct our experiment on such configuration. And we ran 3-hour kernel fileing and 4-hour symbolic execution. So this is the overall result. We totally have 147. Lowest bugs have been converted to highest bugs. So one high-risk bug could have multiple impacts. So this column is about how many high-risk bug we found and the rest are about each high-risk impact. We discovered 178 arbitrary address write and 654 arbitrary value write and the 51 counterflow hijacking, 8 invented 3. If we consider the counterflow hijacking as the most critical impact, we will see that not only those two type of bugs but also the general protection for bug, the wording, the info, they all have chance to have the counterflow hijacking. So to verify our finding, I manually pick up three cases with counterflow hijacking impact and manage to write exploit to target the vulnerable kernel. So for our finding, the finding alone discover 61 high-risk bug, which is 53 percent of high-risk bugs we find with 167 impact, which is 5 percent. All impacts found by finding only a UFO be write or invented free. This is because the follow-up impact cannot be detected by finding. And the average number of the impact for fixed section is 27.9 versus the open section is 16.9. This is because the open box does not go through the finding, so it has less context. And the buggy context in the fixed section is 1.37. Context in the open section is only one context. The simple execution, besides the 61 bugs that can only turn to high risk by finding alone, the rest 52 bugs in the fixed section, they can only turn to high risk by simple execution. And the simple execution discover 95 percent impact, including all the counterflow hijacking, arbitrary value write, arbitrary constraint value write, arbitrary constraint address write, 95 percent. Then we disclose our finding to the CVE maintainers, which received eight CVE assigned, and we noticed that some high-risk bugs were ignored by several vendors like Ubuntu, Fedora. After our CVE assigned, they find their kernel vulnerable to such bug, and they apply it, apply the patch immediately. And our paper will appear on usenic security 2022, and you can download our paper online if you find any interest. It should be available soon. Thank you for listening. Cisco is now open source and such GitHub repo. I'm ready to answer any questions.