 Hi everyone, Myself Akash Jain. I work as an Engineering Foundry software team, Samsung Semiconductor India R&D. The topic for today's presentation is deep dive into Linux kernel live patch. So let's get started. So the agenda for today's discussion is about live patch and why do we need it? How code redirection is achieved using ftrace? Live patch implementations using kpatch and cage graft. Live patch in mainline on Linux kernel. Driver structure of a live patch. Limitations in driver structures approach. Why do we need ELF sections? Kpatch build and its example. And at the end, limitations of live patch. So what is kernel live patch? So live patch is the ability for the kernel to change the flow of code execution from a broker or vulnerable function to a new or fixed function during runtime. In most cases, the new function is the exact same as the function it is replacing. But with minor changes such as adding a null pointer check or changing the order of some locking mechanism or adding a quick code fix. The code redirection is achieved with ftrace. So ftrace is a tool which allows you to trace kernel function call, but it can also add and remove instructions from function as well. Live patch is implemented by compiling the new or fixed function into the kernel module and loading it into the system. Then ftrace is used to redirect call from the old function to the new function in a kernel module. So there are two ways with which we can write a live patch as of now. First one is writing driver using predefined live patch APIs and compiling it as a kernel module. Second is using source diffs using kpatch binary. So in our presentation, we will be covering both the approaches. Next is, why do we need a live patch? So it is pretty common to see bug reports from Linux machines which have very high uptime such as 6 to 12 months or sometimes even longer. These machines normally learn important workloads which can't be interrupted for a reboot since they might be a part of critical public infrastructure or a busy byte system. The Ubuntu kernel team typically releases a new updated kernel for each distribution release on a 3-week SRU cycle with additional updates always within a day of a new CVE being released. Machines with important workloads are not going to reboot in every 6 months or in every 3 weeks for each new kernel release. So keeping these machines safe and up to date with security fixes is a must and this is the motivation behind the live patch. Code redirection with ftrace. This diagram shows how ftrace is used to redirect calls from the old function to the new function in the kernel module. Here ftrace patches out the knock instruction which is no operation instruction with a call which points towards the new function. If you see carefully the knock instruction is located before the function starts manipulating the stack which means everything is consistent and when the control jumps to the knock instruction ftrace redirects it to the new or fixed function thus bypassing the buggy or original function. Kpatch. Kpatch is developed by Red Hat and uses a simple consistency model. Kpatch operates pretty much as explained before by using ftrace to change the knock instruction in the old function to a call instruction pointing to the new function. So as we can see from the diagram which is just similar what we saw before for ftrace. Here just an addition of Kpatch block is there. So how Kpatch works is Kpatch keeps the system consistent by first stopping all running tasks. The stack traces of each task is then examined. If the old function is not found in any of the tasks stack traces then ftrace applies the patch and all future calls to the patch function will now use the new function. So this approach is atomic and safe since there is only one view of the function at a time. It is either old or new. There are no consistency issues that arise if a new function changes data structures differently to the old function and a structure is passed to tasks which haven't been migrated to the new function. The limitations of Kpatch involve not being able to modify data structures as if a process is still using the patch function patching fills and all the tasks are restarted again to attempt the patch at a later point of time. So there is some overhead in stopping and starting all tasks which result in a small loss of service as those tasks are stopped. The next implementation is Kgraph. Kgraph is developed by Soos and thus is far more complex consistency model. Kgraph employs a per-task consistency model where all tasks remain running on the system and tasks are passed one by one. So this gives no downtime at all. Since all tasks keep running during the live patch and patching can never fail. So if we see a diagram we have taken an example. So let's say we have a user space process making a syscall and a live patch request come in midway through the syscall. If the syscall involved calling the function which will be patched multiple times on a subsequent calling of the patched function the semantics might have changed since the first time it was executed. If the locking mechanism have changed we might be facing a deadlock situation which will add in a certain failure. So as we can see in the diagram here is what Kgraph does in this situation is it inserts a trampoline. So in our case it is a reality check function which is the target of the calling function which is replacing the knob instruction of this buggy function. The trampoline points to both the old function and the new function. If the task has not been migrated to use the new function the trampoline jump to the old function or bug function and execution continues. If the task has been migrated then the new function or the fixed function is called this means that any user space process in a syscall or kernel task or interrupt handler still in kernel space will always use the old function. This continues until each user space process finishes its syscall or kernel task completes or interrupt handler completes. So this diagram shows that all bug tasks are then migrated over to a new function when all tasks have been migrated the trampoline is removed and the call instruction is updated to point directly to the new function. The benefits of Kgraph is that all tasks are kept on running during the live patch. Downsides include keeping two different implementations of the same function around the same, around at the same time. So this can cause problems when long running processes like those waiting on disk or network IO get stuck in kernel space and won't be patched until they complete. This can lead to inconsistencies if the new function changes internal data structures differently to the original since both functions can still be executed in site by site. So coming on to live patch in mainline or Linux kernel. The live patch goes mainline into the Linux kernel during the 4.0 development cycle. Live patch migration is a hybrid between the Kpatch and Kgraph implementations taking the best ideal from both. Live patch uses the Kgraph per task consistency and syscall exit migration alongside Kpatch's stack trace based switching. Patches are applied on a per task basis one task at a time as tasks do not need to be stopped. This also means that the trampoline based solution is used so the consistency model for mainline operates in a set of steps. So first is the stack trace of all sleeping task is checked. If the function to be patched is not found in the stack trace the task is patched to use the new function. If this fails for a particular task it will re-examine the stack trace periodically and attempt to patch at a later point in time. The second step is to patch the task once it completes and exits from kernel space such as syscall finishing or interrupt handler completing. So we will be considering Ubuntu in our context Ubuntu uses the live patch consistency model which has patched both Kpatch and Kgraph. All code is the same as what is shipped into the mainline kernel and there are no custom changes. So the upcoming three slides tells about the driver structure of a live patch. Please note the live patch API has been changed over time. So we will discuss the live patch API as found in Linux 5.4. As you can see since the live patch is a kernel module it follows the same process required when writing a kernel module. So here we include the kernel module header file such as module.h, kernel.h and declare our module underscore init and module underscore exit function pointers. Here first we will map the new function to the old function by defining a struct of type klp-fund in this case called functions and fill in the members such as code name and dot new underscore fun. Since we might need to replace more than one function in our live patch we can create many of these function mappings since functions is an array. We then tell a live patch what to patch with struct klp underscore object. We set dot funds to our array of functions and set dot name to be another live patch module this has a dependency on or simply null if we want to target VM Linux. So live patch init function will contain APIs for registering and enabling our patch module. Live patch underscore exit function will disable and deregister the module underscore xyz is our patch to function and at the end this is wrapped into a struct klp underscore patch where we declare the module name and the object struct. This is the struct we pass as a reference to when klp underscore enable underscore patch is called. We can build the driver with the following live patch module which is similar to any kernel driver make file so for that first you need to install a compiler and a kernel header for your running kernel as apitget install linux headers hyphen unium hyphen hud apitget install byte essential then go ahead and run make so after patch module is generated insert live patch module using INS mode after that you can see your new function getting live patched without even need of a reboot. Coming on to the limitations in driver structures approach so we can use this approach when we want to write a complete new basic function say for example changing kernel parameter this approach won't work when we want to patch the existing code but if we try to patch the existing code with this approach during build time we may face error because when the module object build it cannot be linked because the compiler does not know the offsets or locations of the functions which decide in the unstripped or stripped VM linux binaries so to resolve this issue we need to add ELF sections to the object file which will tell the kernel live patch system how to apply relocation for each of these functions into the kernel that we are targeting so here comes the need of ELF sections so there are two ELF sections that need to be add on so first one is shf underscore lila underscore live patch second is shn underscore live patch so first one shf underscore lila underscore live patch is used to deliver the function which need to be redirected with ftrace that is the functions that are actually being live patched shn underscore live patch are all the local symbol that the fixed function calls and need to be fixed up each section needs entries of the form .klp.rilla.objname .section underscore name where .klp.rilla denotes the relocation section name fixed with the string .klp.rilla .objname denotes the name of the object that is vmlinux or the name of module to which the relocation section belongs follows immediately after the prefix and section name denotes the actual name of the section to which this relocation section applies all for shf underscore lila underscore live patch would be this .klp.rilla.vmlinux.text.meminfo underscore proc underscore show here meminfo underscore proc underscore show in the patched function these other sections need to know the addresses and offset from the vmlinux binary coming onto the kpatch build so it is an automated build program which can generate live patches from the source div and programmatically patches and insert these illab sections which contain the simple relocation table. kpatch build is a collection of tools which convert a source div patch module they work by compiling the kernel both with and without the source patch comparing the binaries and generating a patch module which includes a new binary version of the function to be replaced the primary steps in kpatch build are first build the unstripped vmlinux for the kernel patch the source tree rebuy the vmlinux and monitor which objects are being rebuyed so these are the changed objects recompile each changed objects with hyphen f function hyphen sections hyphen f data hyphen sections are writing in the changed patched objects unpatch the source tree then recompile each changed object with hyphen f function hyphen sections hyphen f data hyphen section are writing in the changed original objects so for every changed object use create hyphen div hyphen object to do the following tasks first one analyze each original patched object pair for patchability then add dot kpatch dot funds and dot rila dot kpatch dot funds hyphen sections to the output object the kpatch core module uses this to determine the list of functions that need to be redirected using ftrace then add dot kpatch dot dynamic rila and dot rila dot kpatch dot dynamic rila these are the two hyphen sections to the output object this will be used to resolve references to non-included local and non-exported local symbol these relocation will be resolved by the kpatch core module generating the writing output object containing the new and modified sections link all the output object into a cumulative object and at the end generate the patch module so we will be seeing the example in the coming slides so the steps needed to build kpatch build source code is first we need to install some packages and dependencies then we will clone the kpatch source code and build the source the next step is to download the ddev package for the kernel we wish to make a likepatch module for a list of all kernel ddev packages can be found at the ddev package repository we will be targeting 5.4 kernel so we need to download and install linux image unsigned 5.4 debug sim ddev package thus resulting debug vm linux will be placed under live debug boot directory so ddev package is generally a debug symbol package for debugging symbols from our vm linux binary so here we have implemented one functionality which will give some of all the undirect kernel memory allocations such as vm log per cpu kernel stack paste table socket buffer etc so for that we have taken a kernel 5.4 source and using mem info.c file under the fs proc directory from the kernel source code here we won't be discussing about the patch functionality implemented our focus here is to implement the changes and get the diff out of it as we discussed before kpatch build operate on source diff so we will take a diff for our patched function so now we got the source diff here we need to take care of one important point which is the source that we are targeting we should take a diff from that kernel source code only if we don't follow this approach our patch won't build and it will throw error during the compilation time so now we will compile our patch file using kpatch build binary kpatch build works by first downloading the source archive of the kernel that you are targeting which is determined by the vm linux package you pass in from there the standard vm linux is built normally once that completes the source tree is patched with the patch you specified and it is rebuilt again as we know that only mem info.o got changed so the single object is compiled again with hyphenf function hyphen sections hyphenf data hyphen section are flagged in both the patched and the unpatched form then each unpatched and patched object set is then analyzed by create hyphen diff hyphen object to determine what functions have been modified and to extract the changed functions this program also checks for like patch compatibility so the special part of create hyphen diff hyphen object is that it creates the necessary yellow symbol relocation sections to the patched object file it adds kpatch.funs and .rilla.kpatch.funs symbol which tells ftrace what functions are actually going to be left patched it adds .kpatch.dynamic.rillas and .rilla.kpatch.dynamic.rillas symbol which are used to fix up symbol relocations for local function call in the fixed function to symbol in vm linux so from there kpatch build generates a new kernel module containing all like patches which is ready to be used so we insert the module using inns mode also we can check our like patch function status with dmessage logs as you can see mem kernel functionality got like patched and gave some of untread memory kernel memory allocations so for permanently installing a like patch we should give kpatch install to our patch module for resisting all installed like patches we should give kpatch our list for those irrep sections we can examine the kernel module to see them using read irrep bin binary with hyphen hyphen sections and hyphen hyphen results flag with our patch module so coming on to like patch limitations so like patching is only for critical security problems linux kernel patches can fix linux kernel like patches can fix vulnerabilities if the problems can be isolated to a small and specific portion of kernel code however if the problem is complex and affects many functions or affects data structure like patching can't be done only functions that can be traced could be patched linux kernel security fixes must be written by experts even simple patches require an advanced knowledge of linux and c if the patch is intended for production servers it must be thoroughly tested across a wide range of platforms and kernel versions this takes enterprise level equipment and skills to do properly this comes to an end of a presentation I thank you all for being a part of this presentation and I thank the whole linux foundation committee for giving opportunities to showcase our work if you have any doubts regarding this presentation please feel free to ask in chat thank you everyone