 Hi everyone, myself Akash, I work as an engineer in Foundry Software Team, Samsung Semiconductor India R&D and the topic for today's presentation is deep dive into Linux kernel LightPatch, so let's get started. The agenda for today's discussion is about LightPatch and why do we need it, how code of redirection is achieved using app trace, LightPatch implementations using K-Patch and K-Graft, LightPatch in mainline Linux kernel, driver structure of a LightPatch, limitations in driver structures approach, why do we need ELF sections, K-Patch build and its example and at the end limitations of LightPatch, so what is kernel LightPatch, LightPatch is the ability for the kernel to change the flow of code execution from a broken or vulnerable function to a new or pitch function during runtime, in most cases the new function is the exact same as a function it is replacing, but with minor changes, such as adding another pointer check or changing the order of some locking mechanism or adding a quick code fix, the code redirection is achieved with app trace, so app trace is a tool which allows you to trace kernel function call, but it can also add and remove instructions from function as well, so LightPatch is implemented by compiling the new or fixed function into a kernel module and loading it into the system, app trace is then used to redirect call from the old function to the new function in the kernel module, two ways with which we can write LightPatch is, first one is writing driver using predefined LightPatch APIs and compiling it as a kernel module, second is source diff using kpatch build, so in our presentation we will be covering both the approaches, so next is why do we need LightPatch, it is pretty common to see bug reports from Linux machines which have very high uptime such as 6-12 months or sometimes even longer, for example factory control machines, so these machines normally run important workloads which can't be interrupted for a reboot since they might be a part of critical public infrastructure or a busy build system, so the Ubuntu kernel team typically releases a new updated kernel for each distribution, release on a 3 week SRU cycle with additional updates always within a day or two of a new CVE being released, so machines with important workloads are not going to reboot in every 6 months or 3 weeks for each new kernel release, so keeping these machines safe and up to date with security fixes is a must, so LightPatch eliminates the need for unplanned maintenance windows for high end critical kernel vulnerabilities by patching the Linux kernel while the system runs, so the LightPatching infrastructure that is introduced in the Linux kernel helps in updating the running kernel with security fixes by LightPatching it without any disturbance to the machine critical workloads running on a system or a need to restart the applications, all the running processes in the system start seeing the fixed functions or versions of the affected kernel functions immediately without any delay after the kernel is LightPatched, so this allows you to benefit from improved service and application availability while keeping your infrastructure secure and up to date and this is the motivation behind LightPatch, so now code redirection with ftrace, so this feature is named function tracer or ftrace, it is a powerful framework to measure several aspects within the kernel like events and interrupts, for example it can measure the latency of specific functions like writing to disk etc, so LightPatching is achieved by redirecting the call to a kernel function to its patched version with the fix and it depends on the ftrace tracing framework that is available in the Linux kernel to profile kernel functions during their invocation, every redirection requires a callback to be registered with the ftrace framework to redirect any call made to a function to its patched version, this diagram shows how ftrace is used to redirect call from the OID function to the new function in a kernel module, here ftrace patches out the knob instruction which is no operation instruction with a call which points towards the new function or the replacement function, if you see carefully the knob instruction is located before the function starts manipulating the stack which means everything is consistent and when the control jumps to the no operation instruction, ftrace redirects it to new or fixed function, thus bypassing a buggy or original function, kpatch, this is the first implementation of a LightPatch, kpatch is a Linux kernel patching feature created by Red Hat, once installed, kpatch executes light patching of a running kernel, with kpatch a Linux kernel patch is applied without nessie stating a system reboot or restart, as a result kpatch empowers system admins to apply crucial security patches to the kernel as soon as they are needed without having to worry about users logging off, scheduling reboot windows or completing long running tasks, kpatch effectively amplifies system uptime and availability without sacrificing security or stability, kpatch uses a simple consistency model, kpatch operates pretty much as explained before by using ftrace to change the knob instruction, no operation instruction in the void function to a call instruction pointing to the new function, so as we can see in the diagram which is just similar to what we just saw before for ftrace, here it's just an addition of kpatch block, so how kpatch works is, kpatch keeps the system consistent by first stopping all running tasks, the stack traces of each task is then examined, if the void function is not found in any of the task stack traces, then ftrace applies the patch and all future calls to the patch function will now use the new function, so this approach is atomic and safe, since there is only one view of the function at a time which is either old or new and there are no consistency issues that arise if a new function changes data structures differently to the old function, the structure is passed to tasks which have not been migrated to the new functions, so the limitations of kpatch involve not being able to modify data structures and if a process is still using the patch function, patching fills and order tasks are restarted again to attempt the patch at a later time, so there is some overhead in shopping and starting all tasks which result in a small loss of service as those tasks are stopped, kgraph, so this is the second implementation of lab patch, kgraph is developed by SUSE and is by far the most complex consistency model, kgraph is a live patching feature for runtime patching of the Linux kernel without stopping the kernel, so this maximizes system uptime and thus system availability which is important for mission critical systems, so by allowing dynamic patching of the kernel, the feature also encourages users to install critical security updates without deferring them to a scheduled downtime, kgraph employs a per task consistency model where all tasks remain running on a system and tasks are patched one by one, so this gives no downtime at all since all tasks keep running during a live patch and patching can never fail, so if we see in this diagram we have taken as an example, so for example let's say we have a user space process making a syscall and a live patch request come in midway through this syscall, if the syscall involves calling the function which will be patched multiple times on a subsequent calling of the patched function the semantics might have changed, since the first time it was as included because if the locking mechanism have changed we might be facing a deadlock situation which will end in a certain failure, so as we can see in the diagram here what kgraph does is it insert a check function which is the target of the call instructions which is replacing the knock instruction, the check function points to both the old function and the new function, if the task has not yet been migrated to use the new function the check function jumps to the old function or the bug function and execution continues, if the task has been migrated then the new function or fixed function is called, this means that any user space process in a syscall or kernel task or interrupt handler is still in kernel space will always use the old function, this continues until each user space process finishes its syscall or kernel task completes or interrupt handler completes, so this next diagram shows that all the bug tasks are then migrated over to the new function, when all tasks have been migrated the check function is removed and the call instruction is updated to point directly to the new function, the benefits of kgraph is that all the tasks are kept running during the live patching, downsides include keeping two different implementations of the same function around at the same time, so this can cause problems when long running processes like those waiting on disk or network I.O. get stuck in a kernel space and won't be patched until they get complete, so this can lead to inconsistencies if the new function changes internal data structures differently to the original since both functions can still be executed in side by side, so live patch in mainline Linux kernel, live patch was mainline into the Linux kernel during the 4.0 development cycle, the live patch implementations is a hybrid between the kpatch and kgraph implementations and taking the best ideas from both, so live patch using kgraph for a task consistency and syscall exit migration alongside kpatch step trace based switching, patches are applied on a per task basis one task at a time and there is no downtime as tasks do not need to be stopped, this also means that the check function based solution is going to be used, so the consistency model for mainline operates in a set of steps, so those steps are first the stack traces of all sleeping tasks is checked if the function to be patched is not found in the stack trace, the task is patched to use the new function, if this fails for a particular task it will reexamine the stack trace periodically and attempt to patch at a later point in time, and the second step is to patch the task once it completes and exit from the kernel space such as a syscall finishing or interrupt handler completing, so we will be considering ubuntu in our context, so ubuntu uses the live patch consistency model which has the patch of both kpatch and kgraph, all code is the same as what is shipped in kernel mainline kernel and there are no custom changes so the upcoming three slides tells about the driver structure of a live patch, please note that the live patch api have been changed over time, so we will discuss the live patch api as found in linux 5.4 focal, so now you know that the live patch is a kernel module it follows the same process required when writing a kernel module, so here we include the kernel module header files such as module.h kernel.h and live patch header file such as live patch.h and declare our module init and exit function pointers, so here first we will map the new function to the old function by defining a struct of type klp underscore func, so in this case called functions and fill in the member such as dot old underscore name and dot new underscore func, struct klp underscore func is defined for each patched function, it describes the relationship between the original and the new implementation of a particular function, the structure includes the name as a string of the original function, the function address is found via call sims at runtime, since we might need to replace more than one function in our live patch, we can create many of these function mappings since functions is an array, we then tell live patch what to patch with struct klp underscore object, struct klp underscore object defines an array of patched functions, struct klp underscore func in the same object, where the object is either VM linux or a module name, the structure helps to group and handle functions for each object together, we set dot func to our array of functions and set dot name to be another live patch module this has a dependency on or simply null if you want to target VM linux, so live patch underscore init function will contains api for registering and enabling our patch module, live patch underscore xz function will disable and deregister the module and new underscore xyz is our patched function, so thus when the original function is called it is redirected to the patched function which is new underscore xyz, at the end this is wrapped into a struct klp underscore patch where we declare the module name and the object struct, this is the structure we pass as a reference to when klp underscore enable underscore patch is called, so struct klp underscore patch defines an array of patched objects which is struct klp underscore object, this structure handled all patched functions, the whole patch is applied only when all the patched functions, patched symbols are found, the only exceptions are symbols from objects that have not been loaded yet, a function that is non traceable by the app trace framework cannot be live patched, live patching depends on app trace for redirection to a patched function of a patched version of a function, as the next step in the registration process the callback for the original function xyz is registered with the app trace framework and after the registration is successful the patched version of the function is enabled through the klp underscore enable underscore patch application binary interface ABI, the klp underscore unregistered underscore patch ABI unregistered the app trace handler for the patched version of the function and restores the call to the original function instead of the live patch version of the function, so we can provide the driver with the following live patch midfile which is similar to any kernel driver midfile, for that first you need to install a compiler and the kernel headers for your running kernel, as apt-get install linux-headers-uname-r apt-get install build-essential then go ahead and run make, after the patch module is generated insert a live patch module using ins mode, after that you can see your new function getting live patched without even need for a reboot, after the module is inserted a directory with the module name is created in the sys kernel live patch where the patch can be enabled or disabled. So, coming into the limitations in driver structure approach, so we can use this approach when we want to write a new complete new basic function say for example changing the kernel cmd parameter, so this approach won't work when we want to patch the existing code but if we try to patch the existing code with this approach, so during build time we may face error because when the module object played it cannot be linked because the compiler does not know the offsets or the locations of the function which reside in the unstripped and stripped VM linux binaries. So, to resolve this issue we need to add elec sections to the object file which will tell the kernel live patch subsystem how to apply the relocations for each of these functions into the kernel that we are targeting. So, here comes the need of the elec sections, so there are two elec sections that need to be add on, first one is shf underscore lila underscore live patch, second is shl underscore live patch. So, shf underscore lila underscore live patch is used to declare the functions which need to be redirected with ftrace that is the functions that are actually being live patched. shn underscore live patch are all those local symbol that the fixed function calls and need to be fixed up. Each section needs entries of the form dot klp dot lila dot obj name dot section underscore name where dot klp dot lila denotes the relocation section name which is prefixed with the string dot klp dot lila, then obj name denotes the name of the object which can be vmlinux or the name of the module to be which the relocation section belongs, follows immediately after the prefix, then section underscore name denotes the actual name of the section to which this relocation section belongs. So, as an example for shf underscore lila underscore live patch would be this dot klp dot lila dot vmlinux dot text dot meminfo underscore proc underscore show. So, here meminfo underscore proc underscore show is the patched function. These elec sections need to be know the addresses and offsets from the vmlinux binary. So, here comes the question why a live patch need to write its own relocations. A typical live patch module contains patched versions of the function that can reference non-exported global symbol. And non-included local symbol. So, relocations are referencing these types of symbol cannot be left in as it is since the kernel module loader cannot resolve them and will therefore reject the live patch module. So, for the live patch module we have to apply the relocations that affect modules which is not yet loaded at patch module load time. So, live patch is solving this problem by embedding the special elec section which is dynamic-rilla. So, live patch is solving this problem by embedding the special elec section which is dynamic-rilla module elec output. So, using this dynamic-rilla section, live patch can resolve symbol while taking into account its scope and what module the symbol belongs to. And then manually we can solve this problem. So, now coming on to the k-patch build. So, it is an automated build program which can generate live patches from source diff and programmatically patches and insert these elec sections which can generate live patches from source diff and programmatically patches and insert these elec sections which contain the symbol relocation table. k-patch build is a collection of tools which converts a source diff patch to a patch module. They work by compiling the kernel both with and without the source patch comparing the binaries and generating a patch module which includes new binary versions of the functions to be replaced. The primary steps in k-patch build are first one is, first one is, build the unstripped vm linux for the kernel, patch the source tree, rebuild vm linux and monitor which objects are being rebuilt. These are the changed objects. Then recompile each changed object with hyphen f function, hyphen sections, hyphen f data hyphen sections, resulting in the changed patched objects. Unpatch the source tree, then recompile each changed object with hyphen f function hyphen sections, hyphen f data hyphen sections, resulting in the changed original objects. So for every changed object, use create hyphen diff hyphen object to do the following task. Analyze each original or patched object pair for patchability. Then add .k-patch.funds and .rilla.k-patch.funds, elec sections to the output object. The k-patch core module uses this to determine the list of functions that need to be redirected using f trace. Then add .k-patch.dynamic-rilla and .rilla.k-patch.dynamic-rilla elec sections to the output object. This will be used to resolve references to non-included local and non-exported global symbol. These relocations will be resolved by the k-patch core module. Generate the resulting output object containing the new and modified sections. Then link all the output objects into accumulative objects and at the end generate the patch module. So the steps needed to build k-patch paired source code is first we need to install some packages and dependencies. Then we will clone the k-patch source code and we will build the source. The next step is to download the d-dev which is debug-dev package for the kernel. We wish to make a live patch module for. A list of all kernel d-dev packages can be found at the d-dev package repository. So we will be targeting 5.4 kernel. For that we need to download and install linux image unsigned 5.4 generic debug sim .d-dev package. The writing debug vm linux will be placed under debug boot directory. So d-dev package is generally a debug symbol package for debugging symbols for a vm linux binary. So here we have implemented one functionality which will give some of all the untread kernel memory allocations such as slab vmallop per cpu, kernel stack, paste tables, socket buffers etc. So for that we have taken a kernel 5.4 source and using mem info.c file which is under fs proc directory from in the kernel source. Here we won't be discussing about the patch functionality implemented because our focus here is to implement the changes and get the d-dev out of it. So as we discussed before kpatch blade operates on source-dev so we will be taking a d-dev for our patched function. So now we got the source-dev. So here we need to take care of one important point which is the kernel source that we are targeting. We should take a d-dev from that kernel source code only. If we don't follow this approach our patch won't bed and it will throw error during the composition time. So we will compile our patch file using kpatch blade binary. So kpatch blade works by first downloading the source archive of the kernel you are targeting which is determined by the vmlinux package you pass in the cmd. From there the standard vmlinux is built normally. Once that completes the source tree is patched with the patch you specified and it is remade again. As you know that only meminfo.org got changed so the single object is compiled again with hyphenf function hyphen section hyphenf data hyphen sections in both the patched and the unpatched forms. Then each patched, unpatched and patched object set is then analyzed by create hyphenf hyphen object to determine what functions have been modified and to extract the changed functions. This program also checks for life patch compatibility. The special part of create hyphenf hyphen object is that it creates the necessary either symbol relocation section to the patched object file. It adds the kpatch.funs and .rilla.kpatch.funs symbol which tell ftrace what functions are actually going to be life patched. It also adds .kpatch.dynamic rilla and .rilla.kpatch.dynamic rilla symbol which are used to fix up symbol relocations for local function call in the fixed function to symbol in VM Linux. So from there kpatch.bind generates a new kernel module containing all the life patches which is ready to be used. So we insert the module using INS mode. Also we can check our life patch function status with rdmessage logs. So as you can see in the mem kernel functionality got like patched and gave some untread kernel memory allocations. So for permanently installing a life patch we should give kpatch install to our patch module. For listing all installed life patches we should give kpatch list. For those illef sections we can examine the kernel module to see them using read illef bin with hyphen hyphen sections and hyphen hyphen relox flag with our patch module. So coming on to life patch limitations. Life patching is only for critical security problems. Linux kernel patches can fix vulnerabilities if the problem can be isolated to small and specific portions of the kernel code. However if the problem is complex and affects many functions or affects data structures like patching can't be done. So only functions that can be traced could be life patched. Linux kernel security fixes must be returned by experts. Even simple patches require an advanced knowledge of Linux and C. If the patch is intended for production servers it must be thoroughly tested across a wide range of platforms and kernel versions. And kpropes in the original functions are ignored when the code is redirected to the new implementations. So this comes to an end of a presentation. I thank you all for being a part of this presentation and I thank whole Linux foundation committee for giving opportunity to showcase our work. If you have any doubts regarding this presentation please feel free to ask in the chat. Thanks.