 Thank you for this. Yeah. Hello guys. This is Vandana. I'm a Linux kernel engineer. I've been working with Linux kernel like more than two decades to really 24 years and I've worked on various subsystems as a more and not just limited to device drivers. In case of device drivers, I've worked with embedded systems, SOC, on-base developments, as well as networking device drivers, network subsystem storage, as well as the other subsystems related to memory management and security as such. So here in today's session, what we are going to do is going to look at the kernel tracing functionality, some of the, how to look into the kernel activities that are happening, but then using the EPP framework as such. So to get started with, let me give you an oral idea of what this today's topic is going to contain and how we are going to proceed with this. So the agenda is to start with the basics of the EPP framework, understand the architecture and then look at how this framework is more valuable or useful compared to the existing architecture, existing tracing functionality as such. And the way it has been used for doing the dynamic and the static instrumentation, particularly as the session is more for the kernel processing, we'll look at the kernel instrumentation techniques and then we'll look at from time to time. I try it and to use the tools that are part of this BPF tool chain as such that help to demonstrate some of the features and functionalities that can be traced or the data can be collected from the kernel using these tools and tenderly use some VCC tools or BPF Trace tool as well as the programming using the BPF programs as such. So the oral scope and expectation you can have is the high level introduction to this architecture and all the processing can be done with the help of this set of tools, as well as from the programming as such. And also provide the useful pointers for the information that has been referenced as such, okay. So to start with, let's have a bit of background of BPF that is the Berkeley Pocket Filter Framework, which was originally being created to speed up the filtering of the network packets as such. And eventually, in 2014, Alex designed the whole of this BPF frame with the language of the BPF framework, adding extensible functionality to it, so that it turned into a more generic general purpose existing engine, and which was not limited only to packet filtering but more functionality was being added to perform variety of operations, such as profiling or debugging or enforcing the security policies as such. So some of the features that got included with the efforts that has gone into 3.8 in kernel included like an introduction of this BPF system call that would be used to load the various BPF programs that would potentially do different functionality profiling debugging and looking up at the various kernel functionality as such. And based on those different hookups, there is the set that would be added for different types of programs, not limiting to the packet filtering as such. And that's how the BPF framework evolved what we call as a, now we call as extended BPF functionality, EBBF as such. So basically, BPF or EBBF, they are like interchange in the being used as such. So, EBBF is basically a revolutionary kernel technology that allows the developer to customize or customize the code that once when it is loaded dynamically into the kernel, and that would help, that would help. That would potentially change the way the kernel behaves, means if you want to perform something additional as to the kernel code that is there loaded dynamically, then that can be achieved through the BPF programming as such. So basically it provides a more flexibility to add to build up the tools or to customize the policies as such. And that is dynamic runtime as such. So, basically the tools that are based on APBF, they are, they are being developed and they have been evolved to do the, potentially observe the events, any kind of events that are happening across the kernel, as well as applications, either on the virtual hardware or physical hardware or even in the patronized environment. Basically, in a short, what we can say that BPF is a technology, the compose of the instruction set that is the BPF program that is run. In the storage objects, how the information, if it is collected information, and if it is shared between the user space and that information, and the set of helper functions that helps collect this information are to write the policies or how that behavior kernel functionality can be changed. And there are the set of library functions that has been defined in define as such that means that those things can be taken help by this life and functions. So basically, it can be considered as a virtual machine running in a totally isolated environment as such. That's why this eBPF is important. For that, let's look at the difference between the basic behavior kernel and user space in Linux. As we are aware that the application they run in the un-privileged layer called user space means which does not have to direct access to the hardware as such. And in that case, the applications make use of system called to make the request kernel to do its work on its behalf as such. So really that might be like accessing the hardware or changing some policies as such so like reading writing file or sending receiving packets or accessing memory or allocating memory and several other activity. So that application from the user space that is done by invoking the system calls. Whereas like the kernel runs in the privileged layer and acts as a bridge between the application hardware basically it provides the abstraction of the hardware, physical or virtual as such. And that is done through the set of the system subsets or layers handling all those different resources. The use of this subsets typically allows for some level of configuration, level of configuration based on the needs of the users as such. So, like, if the desired behavior has not been configured in the kernel as such, then or if it is not present as such or it is like some changes are needed to the current functionality as such. So the weight can be done and the kernel change would be needed to add that support. And it has been particularly defaulted to the two approaches that was adding the native support that is adding the change into the kernel, or like adding the change in such a way that it can be dynamically loaded, thereby taking the functionality of kernel modules as such. So each of each of each of this method has its course pros and cons as such, like if there's some change has to be added into the kernel that is an eight make it as a native support as such. Then the change has to be implemented and then it goes through these iterative cycles of reviews with the kernel community means you have to convince the current community what that change is necessary as such. If there is several iteration and waiting, then that change might get would get reviewed and accepted and then available in the upstream. So there is like the waiting cycle, if the change has to win as a native as a message. Another option is the kernel module where you make the changes as form of formal module. The downside is like you need to maintain maintain that change and fix like as in when the kernel changes or the apis that have been used that changes as such. And also there is a risk of correcting the kernel your system, if it is not been implemented properly and having some security issues as such. So in that case, that option might not be always workable as such. So that's where we have this another functionality that comes to the BPF rainbow, which allows the behavior of the kernel of which allows to add this functionality without making changes to the kernel sources or loading the kernel module. And taking the help of this functionality, we would be going using this functionality to do some tracing of the subsystems as such. So here, this is a block diagram or simplified way of projecting how this architecture can be viewed as such BPF architecture so here. The board that you want to load load and running the kernel they they have been implemented as BPF programs and this BPF programs can be loaded dynamically and executed in the kernel as such. So when this programs are done loaded and executed there are there are the other components of this framework that ensures that ensure the security and safety of the program that is to be loaded and runtime as such. So that's where this BPF verifier component it does the work of ensuring the programs are safe and cash free. And it does does some work of checking whether it's breaking the security policy in some way like the program would be rejected if it is deemed to be unsafe. Such as like some of the features that is considered as unsafe is not as having the program has in for like a loop or loop as such are backward branches and those kind of function to the program has that kind of functionality and the verifier would reject it as such. Once the verifier ensures that the program can be loaded and that they have the other component is the engine compile just in time compiler basically does the work of translating the micode into the machine level instruction that would be executed as such. And once that is done, the program is attached to that particular function or particular event as such and as and when that up the particular event offers them that program would be triggered as such. And like when this programs are being designed, they will be designed to provide some functionality as well as collect some data as such. And so for collecting the data or this one of the cons the constraint construct been developed is called as the BPF maps as such, which basically provides the facility ability to store the collected information and share between the user space and the common space. And based on the requirement, like there are different types of maps like hash tables, arrays, buffers, stack, and stack, trace as such and of course there are a lot more. So you are in this case this example is this but particular by a program. It is to be attached for but this network related system calls as such. So here in this case the programs get attached to send message and receive message will see how this has been done with the examples aside. And whenever the applications makes the socket program, so network programs that calls the second socket system balls in that case that particular BP program would be executed as such. So, yeah, so here we talk about loading the BP program which is done dynamically as such. And this program can be loaded and removed from the kernel dynamically. And once they're attached to the event, the they are triggered regardless of the cause of the event as such. So it can be caused by any application or maybe a process getting executed or maybe process communicating with each other. But with that particular event has been triggered that BP program would be executed. And the programs, they can be attached to at various places at various hooks as such to give some of the example is like the system calls are any kernel functions as such. It can be a particularly if you see system call like the entries system call entry or system call exit. Similarly, for the other kernel function entry exit can be the disprograms can be attached. Again, another is these trace trace points, the various trace points that are there. Then the different subsistence particular network events or other events of the subsistence events. This BP programs can be attached as such. So to give an example like the BP program can be attached to the open system call edge point entry of the open system. And whenever any application tries to open a file onto the system, then this BP program would be executed as such. The example as for this diagram to say that the BP program is attached to the exact system call. Like whenever a new application is executed, wherever it is, then this program would be executed and based on what functionality has been implemented. And this BP program that would be executed so basically like one way is like using the health function and the various constructs. The arguments passed to those system calls can be collected on the credential process credentials can be collected. And from the availability point of view, the system observatory can be done like what's happening on the system and who is doing what and how the information, how the parameters are being passed. So all that thing can be collected in the collected through this BP program on as needed the changes could be made as such. So basically, this functionality. This, this leads to one of the greatest trend like observable observing our observatory or security tools that the tooling that can be done using the BP framework as I so basically what it does like it gives you the instant visibility or everything what is happening on to the system as such. So the BP framework is one of the latest framework that has been used for this observability and the security tools as such that you can enforce the security policies using this BP framework. So, oh yeah so this is the way the dynamic loading and programming can help to get you the get you into the system into the system know what is the health of the system or what exactly the system is up to as such also like fine figuring out some debugging issues or performance issues that can very well be are done with the help of this BP of programming as such. So here in this case before getting into the details what we would we could come run back and forth with the examples as such. At this point you might not to get all this. Each of the function that has been used or it might be like you are already using, but like to start with what we'll do is like just go through a program or hello world program hello world. We will return in a python python code as such and how with the help of this python code how the BP program can be attached to some system call and then see how the system call can be invoked as such again. So basically, the python program can be considered as, let me say it consists of two parts as such, like, yeah, like it contains the actual BP of program and then the, then the python code that does the work of loading the loading the BP program into the kernel as such, compiling and loading as such. Okay. So here in this case. So just cover slide and then go back to the code as such. So, as we said that it consists of two parts that is the BP program that will run in the kernel. So it's just basically nothing but a C file a C function as such and it has a simple help function that does the work of printing just similar to print K as such so here it is the BP trace brain key doing hello world print as such. Okay, so this, all this all together is your BP of program so which is a C like functionality as such and the rest of the part is the python code. So the python program that does the work of loading the BP program. And basically the BP of tools tool chain that is BCC one of the two that does the work of for compiling the C code before executing it and then loading it into the kernel. And then it attaches to a particular event as such so you're in this example taken exact system call as such a proof function. Okay, so you're looking at this diagram like the year this is for a program python code which and execute that what it does it's does the work of loading the BP program on to into the kernel. And that loading and attaching it to a particular even that would be triggered as such. And whenever that even is triggered whenever the applications as such makes a call as such like exact system call application new application is executed. The exact system call gets executed and every execution exact system called the BP program will be executed. And whatever the functionality it has done now here in our case it is nothing but the print statements, it prints this string as such, so which has been collected in some kind of offer, and which is passed back to the application and it prints that on the screen as such to hold. Okay, let me go back to the pool as such. Okay, so here this is the be a redefined the BP program and then the rest of the Python code does the work of initializing this BP program by creating the BP object as such. And then we choose to attach this BP program to the exact system called event as such. So here these are the set of helpful function that have been used as such. Here the exact system call is to be used. So here what we do is we tend to make use of the health function to get the function corresponding to the exact system call as such. So, and then this exact system call function name. We try to create, we try to attach it to the attach this attach the function, the function that we program that has been defined to this exact system call event as such. And that has been done using the cable cable functionality as such. And once that is done then the print print trace print does the work of outputting the new cross trace put alphabetic kernel as such and we could eventually go and show up on this screen as such. So basically, to go and say, basically the output is stored into a seed of wireless. So this the running running this program. Depending on what the load is there what is happening on the system what the different types of application that are running. And whenever they make the exact system call them that is this event will be triggered as such and then you will see this hello world string as such. So here in this case, what we'll do is just execute some command as such BS command and then we see if we are getting anything printed. So here we see that there is one more print as such. And let's do this. Again, you say so you're in this case when we are on the another terminal, some application is being a process is been executing so that execution happens to the exact system call as such so whenever that has been tracked through this Python code as a Python application which loaded the program to trigger the events whenever the exact system call is executed. So when we are doing that on another terminal, we get to see a log be generated that through the printer statement as such and you get to see this hello world string as such as part of that. And also you get to see it's not just the string that is coming but there is on a other contextual information as such particularly like the PID process name and the process ID as such. So, basically, this output that is coming that is actually coming from the from that file or from the terminal log that is the that that basically what it does the Python that goes and reads this trace pipe. So the file as such the all those traces are then fetch from that as such and another phone information we can note down as the other contaction information so that is basically that is basically that is added by the current tracing framework as such infrastructure as such and giving the information process ID and process name and other information so it's not very specific to BPF as such but it's a part of tracing tracing framework functionality as such. So this is like a very simple hello world kind of application we just control see and terminate it as such so you define and so yet the programs are defined as a C function and it has been passed to be a baby of object as a string as such so you just pass the whole of this function creating the baby of initialize the baby of and then that was a function of functions, you would go and attach the BPF program to this particular event as such that might be set of system calls, then trace points for events and other events as such. And basically, yes, particularly using the key proof functionality, which is like, which is used for doing the dynamic instrumentation as such, and then there are other ways, the events can be attached as such. So this, like, getting started helps with this. Now, one more thing that this trace pipe, here actually you can open this file and see that you would be able to see the trace output as such so basically this does the information that is a string based on simple strings as such. But at times, the based on the functionality and how the BPF programming has to be done to do a lot of other operations as such. Now you're what we are doing anything as such but in real use case, a lot of data data is being shared with the from the kernel to the application and based on that data, the corresponding actions are to be taken so in that case the stress pipe might not be a sufficient mechanism for getting the data as such. So therefore this BPF maps are being used as such, which are, which is a data structure that is being accessed from the user space as well as kernel space from the programs and as well as from the user space and basically provides the key values tools as such. So, if the data has been shared has to be shared it's like in some structure data structure as such, maybe like a socket socket structure or network packet as such so in that case, you cannot use this trace pipe it would be an efficient for tracing or scanning that output as such so by making use of the BPF maps. And that that information maybe a whole structure or nested structures and if related information. They, they are being shared that keys this BPF map is a useful constraint to be used and based on the requirements and the definition there are different types of maybe maps that are implemented. You can figure out that in the BPF dot hr file, and it was commonly that has been used are the hash tables suing buffers areas, various others as such. And also like there are other types, the different types to be defined based on the type of operation like five foot five foot use tags and let's reuse the LRU data source as such and also like based on the different subsystems or information specific like particularly for the networking as such. There are different BPF maps that are constructed for doing this sort of a storing and sharing the socket and network device information so this is just an example of a socket and that will do the sock maps and the dev maps as such. So we'll see some of the examples how this maps are being used as such. One minute. Yeah, is there are several questions in the chat. Okay, probably would be better answered. If you can take a break. If not, let me know we'll we'll get to the question later. Okay, okay. Yeah. So you will, I think the first question is I think that a lot of conversation happening here. Some of the questions are answered. Okay. I did not look at them. No, that's okay. That's okay. I am watching but I thought that I would wait for a good stopping point and kind of found one. I thought but okay the first question is from Praveen. I would like to know how if EBPF can help when applications need DPDK networking like some user plane might need DPDK. Can you repeat I often do. Okay, so if I would like to know how if EBPF can help when applications need DPDK. It would have been great I think Praveen could expand what DPDK is but I'm thinking that it's device, some kind of API. DPDK networking, like some user plane might need DPDK. Yeah, DPDK is provides the user space networking programming as I told together so basically I think for the DPDK we have for that from the networking point of view there is the XDP interface is there in part of the BPF where the network traffic acceleration parts are data parts are accelerations are being done. So, BPF has the circle for that as well. So, most of the things that is related to networking, I think that that goes to the XDP interface as such, and that can be leveraged in the DPDK stack as such. Hope that answered your question Praveen if not, please post a question in Q&A or chat. So, it's next question is does EBPF assume a physical machine or can it run well on virtual machine too I think this question has been answered. Yeah, yeah, back and said it can be done on both. As long as Cardinal supports it. So the other question is, does someone know the difference between Dtrace and EBPF, I think this is also answered that Dtrace is only available on Solaris and Oracle's and it's not in the main line Cardinal. I think that's been answered. Let me see. It looks like a hook environment between user mode and Cardinal mode, how does this work with LSM, for example, Cell Linux. So I think this probably in reference to the Python program that you're doing hooking it to the XDP. So that is the question there about specific to that. So, particularly for LSM hooks, there is the new constraints that has been added in the BPF framework that is a BPF or LSM, which can help you to add the hooks into the LSM, which can help you to attach the programs to the LSM. Hooks as such. And that is one of the sub component of the BPF. This is I think the latest governance support start. BPF LSM hook. Yeah. Okay, so that kind of makes it more secure, especially Cell Linux kind of use BPF methods to trace and then attach hooks. Okay. That sounds good. Would Slice be shared? Yes, they will be shared after the session. And we run a fleet of how are you doing on time? Do you want to continue and take the questions later? One enough? There seem to be more questions. Yeah, okay, actually I've lost the track of the questions as such. Should I stop by and look at the questions as such? No, I can keep going reading questions, but I just wanted to see if you want to. I think if you can read out the question, then I would try to elaborate it now, otherwise it will like the list might grow. Absolutely. Okay, thank you for that. Let me read the next one then. We run a fleet of Linux VMs, thinking of building a health monitoring service based on EBPF, complete with EBPF and dashboards. General idea is to provide a periodic pulse of interesting metrics, Brandon Greg's checklist, plus on demand deep dive on interesting processes. Any thoughts on that idea? Yeah, actually, that is like the current requirement of the current trend that I would say that has been used for the security based product assets, where the security big process where the previous traditional security products based products were like using the functionality that's doing the collecting the security infrastructure, the user space applications as such using the oddity framework and other things as they do help to get the information, but they have some drawbacks or downside that not all the information is available. And another reason for that you have to get into the have to be in a kernel space to get all that information. Thanks to this BPO framework, which helps you to run the application from the user space and collect the kernel information as I can go and look at any kernel function and get that information. So, leveraging this functionality is like a key milestone for the security security solutions as so like giving a simple example like a security application like file monitoring through the security solution is basically like I had worked on one of the previous security file system, sorry, file monitoring system, like the application was user space, adding the events, trigger the events for system calls, the distressive system calls and for checking the check, keeping the check on the configuration files as such and all that. And that oddity framework did help to get the information but had some limitations and at that time not all the information was available. But I think with the EPP a framework for you can get a very detailed information about the system calls as well as the whole of the system or the process execution with along with the credentials and the all that information by using this various interfaces and collect huge rich environment data as such and then that can be interpolated and that can be interpolated as well to check not just the security or also like if there are any vulnerabilities or some attack or compromise has been happening as well. So definitely, it is the very right answer for that solution as such. All right. Thank you. So I think it this regard it regards to sell the next LSTM and so on I think it works. The question is, I think it works with selling X as far as I know, unless you hit something that is blocked by the policy, the default selling X is targeted. So, with the help of a sale next week and enforce the different security policies as such. So, so yes, like if the sale Linux is been unable as such and the sale and it's folks they are at the point. The point of system call entries as such so there is a flow of like where the sale Linux goes and checks as with the, with the particular operation is permanent permitted or not based on the policies that have been loaded. The mandatory action access control has been enforced as such. So here in this case, the BP program has to be aligned with the SLA policies that are there running this like suppose you for running a baby program. So, as a, what do you say at the administrator as a conflict of enforcement that you're doing that you would be doing in compliance with what are the configuration that is there as such. What is the significance of our on the program line in the Python program that you showed. Can you repeat the question. Yes, the question is, what is the significance of our quote quotes on the program equal to line. Okay, yeah, basically it is like it is passing the whole of the program to the Python Python basically the Python program at the back end the by the BP of framework tool chain, it takes this code to see code, and it does the work of converting into the byte code and then which would be going through the BP of framework as such to be loaded as such and this like if you go back to this tag. Here in this the Python code basically to load that code BP of program, it goes through the system called BP of system call as I so that code has to be converted has to be passed which would be would be converted into a byte code from the white code into machine learning machine language as such, which would be attached to the event as such so that for that, that is the convention to specify. This is basically a specifying that this is the seat of the program as such, and then we pass it that whole program as a string as such to the BP of object. And then I think this is more of a comment from somebody responding to another question about hooking. If you using ebpf to look at system calls then it looks like a hook environment between user and kernel. Sure. However, you be a ebpf can also be attached to purely kernel level functions is there anything that you would like to add to that one. Other than the system calls sorry. Let me read it again. If using the ebpf to look at the system calls then it looks like a hook environment between user and kernel mode. However, ebpf can also be attached to purely kernel level functions. Yes, yes, it can be attached to not just not limited to the system calls, but to any kind of functions. Okay, so that's that's correct. And this tracing is helpful as such like, we will not only be tracing system calls as such, but any kind of function that like, it might be developing some kernel module some kind of driver as such and you might want to have looking to the functionality maybe some performance issues or some other debugging issues in that case, that can be done through this bpf programs as such. Or maybe you are debugging a problem and you want to see a particular system call is hit. Yes, or not system call particular kernel interfaces hit. Basically, there are like a lot of a lot of help functions are being defined as part of this bpf framework as such that caters to the different system subsystems as such networking block layer volume volume your memory management. We will see that in the following slides as such. So, one more question here. One concern could be performance impacts if used for regular monitoring, instead of occasional debugging, perhaps that will be covered in the stock. I think you probably will touch on that later one then also do you want to address this question now or you have slides on that. Yeah, or you're like bpf programs like they are been loaded means converted from whiteboard to this assembly or machine level. So, so they are the overhead is significantly very less as such. That means not very a lot of things about like if it has been added into the places which is like been triggered like suppose you're adding the bpf program to a schedule or as such are monitoring which would be which would trigger, which would be called consistently based on the workload as such if the if the work load is heavy and potentially the points have been given some intricate from that particular workload related information done in that case there would be some overheads. So there are some books that are generating that there are some books that are potentially having a more overheads but overall the, the way the bpf programs are loaded and kind of done. What do you say in just in time conversion and machine machine instructions are executed so there are minimal minimal overhead in general. There's another question about the security to have any negative impact to bpf program and also any current lockdowns. So, yeah, so one thing is like this programs they are running from the user space that is loaded so as per the. What do you say the architecture bpf architecture it has this very fire that would do the work of a very fine that the court that is going to be run in the kernel that would not put the kernel in the in the infinite looping or lock state as such one of the way that would be triggered is by having the loop as such so if the program has been implemented that is going and taking a loop as such. Then that kind of food is not to be allowed as such to those some some of those checks are being done taken care by the very fire. Okay, I think one more and after that maybe we can hold off on the questions and let you complete. Could any ABI comparable language be used since he seems to be the de facto language to use like they got even see Python. Yeah, we are coming to that slide. Next. All right. So, at this point, I do we do have questions in the chat and then also a few and a, but I would like to have one and a continue representation at this time. So we will get your questions at the next stopping point. Go ahead. Yeah, with having a basic understanding of the BPF program what we would do now is look at the how the that program how that BPF framework can be used for doing the dressing of the current functionality as such. So basically the broader basic broad understanding that the type of tracing it can be used for debugging and troubleshooting the various issues as such and the traditionally there are there are the various tools that are available as such, like the after a during the function tracing of tools as such, and the user space as trace of the system called race and trace as such. So basically these are function these tools are also helpful I based on the use cases as such. But if you want to dive into every aspect of the current system that is going on that's where we make use of the BPF functionality as such. And these tools can be used along with the BPF framework to explore more information as such. So here in this case, so we will look into how the tracing can be done with the help of a BPF framework. And as we know that the tracing gives us the full visibility of the across the full software stack, and it allows to create the data for profiling for debugging and troubleshooting as such. So this is the slide diagram from the Greg Reagan's blogs as such. So here in this case, this notes down the different top systems, any kernel, any kernel, and I also the various layers and application level. So the information from the phrasing can be done as such. So if you talk about the kernel subsystem and the information visibility can be. We can get more visibility from the system called the interface layer as well as file system and the virtual file system layer business pertaining to whole of the file system on block layer as such. Networking at the network layer in socket layer, TCP IP layer, as well as the device driver layer like related to interrupts and as such, and then the other subsystems like the schedule or memory management virtual memory. So, so to get to know what is happening at each of the subsystems in the BPF events, the BPF programs can be attached to those this is subsystems as such and with the help of this support BPF tracing support multiple means which which provides the multiple sources of events as such, and that will provide the visibility to the entire software staff. And this visibility, we can get in by making use of the different instrumentation ways as such. So basically the instrumentation that's a dynamic instrumentation and static instrumentation and that to as a kernel level and at the user level as such. So basically when instrumentation is happening at the user space level, it makes use of this you prove functionality. And that's a dynamic instrumentation dynamic instrumentation means instrumentation instrumentation instrumentation is done at the runtime as such on the live code as such. So with the dynamic instrumentation it provides the ability to to answer the instrumentation points into the top and running code as such, and for the kernel, the cable functionality is been used. And for the user space. And that is done in the with the help of this you, you proves functionality. The dynamic instrumentation that can be used to instrument the, at the starting or end of the kernel function, or the application function as such. You can add the proofs at these different points as such as of course not only start and or you can also add this before pro at some offset of the function as well as such. And as this dynamic as dynamic instrumentation you would go and attach to some particular function as such. And this kernel function API is maybe I might change from version to version as such so that's why we. That's why it's been say about the current forms does not does not provide quite a stable ABM it would change as per the kernel version if the API definite declaration definition changes as such. Whereas in case of static instrumentation, the instrumentation points are added in the code itself, and they're made in by the developers as such, and for for the kernel. It's been implemented using the trace points, which the code has been added since 2006 32 colonel. And for the user space, another, that is the form that is provided through this user statically defined trace points as like. Again, the instrumentation can be done in the kernel and the users applications and as they are been implemented are added by the developers, they are kind of provide a stable API interface as such. So, what we asked for our topic what we are going to look at is the kernel instrumentation that is dynamic as well as static that is through the Cape ropes and the phrase points as my focus is around this. framework as such functionality as such. So what we will do is we will first look at the Cape ropes Cape ropes is the framework functionality that allows to set the folks at any or almost any function or instruction in the current. And basically, and basically the eb programs they are attached to using the Cape ropes that is particularly for the entry of the functions are at the specific offset of the function after the entry. And there are different ways that can be done through the BPF for BCC will come to this for this BCC and BPF tools are there. And these are just to give an idea of what are the functions that have been used and basically also that our instrumentation can be done through the carry. The return proofs that is that is that is attached at the exit of the function. So, when you attach the program to that particular event and reassert exit exit function so through this case written process can collect the information and then exiting from the function as such. Now you're in this example like to take an example like a VFS read or the read read system call like in the case of read what we expect is the output of the read is the number of bytes that has been read so in particular analysis as such you might want to know that application, what kind of read load is happening as such or overall system load that is happening through the read operation so in that case. And return prove a return prove can be used collect and collect the red number of bytes read that is the exit that which at the exit point the kernel code path is aware of that the how much read has happened as such. Particularly this example, it is doing the work of collecting the VFS read function that is executing executed throughout the system and this display displaying the histogram for the read operations that are happening as such. So we'll come to the syntax, how, what is that in a later examples, but this is just to give an idea like at the basically k proofs at the entry is like another example here example I've taken is the open system call. So open system call like if some security has to be enforced in this case, some security tool has been developed and you're trying to monitor or enforce the security as such in that case. So you can add the hope at this open system call and if a particular file has not to be opened by a particular set of users, then in that case, in that BPO program you could have a functionality of this is the file which comes as the first parameter to the open system call corresponding. As you are in the current you have the point to do the task that is running. So from that you can get the credential of that particular task and the process and have some based on that policy decide if that open has to succeed or not. Or you can just do the work of collecting that information for further analysis as long above us like monitor what the things happening around on the system at that particular time. So that's where this probe probe can be used for collecting the information at the start of the function so you're in this case function now you're in this case you will have this do says open is the function particularly corresponding to the open system call and here we are not using the sis open asset but the inside function of the system all assets so you're in this case it's not just restricted to the system files but any function kernel function can be can be proved as such as the BPO program can that as to any function. We'll come come to this what is a BPF trace in a couple of slides next as such. So basically, using this key proofs means for debugging and for developers can write a kernel models and attach the function to the key proofs are use make use of the tools as such or write the BPO programs all together. In the operation. So this key proofs is for dynamic instrumentation. What we'll look as the static instrumentation kernel that is done through the trace points. So trace point is the functionality kernel functionality that allows you to insert the code at a logical places, while implementing that functionality may be or implementing the driver as such. In that case, you, the developer as a developer you would might use the trace point for some of the functionality as such so that case the developer is adding the, adding the static markers into the current path as such. So you can list list on this what are the different trace points that are there on your system by looking into this file as such available events as such. Okay, so if you look at that, then that is trace available events as such. So if you look at that there are, there are, there are the phrase points being added by the different subsystems. Now you're in this case you see this NFS and related to a subsystem and this is the RFS. This, this code has been added by the different, different, different for maintainers or the device drivers as such. And again, you get to see the trace, trace points being added for the system. I just feel it taking the snapshot of some of the system calls that has the trace points with the trace from this like the trace points are there at the exit entry as a, this is about the, this is trace point at the exit of the clone system call entry the clone system call and similar for the fork and fork as such. So basically like if you do, you can count on how many kinds we trace points are there available on the system as such. So you're in this case. I'm having a bit older kernel that is 5.4 kernel. So on 5.4 kernel, currently there are like 15, 1500 plus trace points present in that kernel as such. So basically, what we can do is like use the trace points static trace points to collect the information as such. So you're in this example, we'll go through the example later as such, but here I've just pointed out the example for the read system call as such. So whenever the read system call entry the read system call happens. So you can go and train the count that is the input parameter for the read system call. So here in this case it would print the number of bytes that has to be read. And here in this case the constraint that the process should be SSDSR. So here in this case we get the information about the process, the SHD process doing performing different kinds of doing the reads with the n number of bytes that has to be. So, so to get started with how to use that interfaces will look at this BPF tool chain as such. So basically, in the, in a lot of scenarios or in the current scenarios as the EPBF framework or the programs they are not directly used as such, but they had indirectly been used to via projects of the DCC BPF trace or CDM project as such. So this project provides more abstraction, provides an abstraction on top of the four BPF framework and abstraction in form of tools and script as such so you don't need to write the program, programs directly as such. So to start with is like you can use this program scripts and tools provided by this project. And if they satisfy the requirement, then that is good, but if you have more customized requirement then you should go and write your own program fast. And this is not limited to these two frameworks, these are the most prominently used front ends for the BPF tracing that is a BPF compiler collection, DCC and BPF trace framework as such. And the BPF programs, they can be written either in the pseudo C code, go line, go language or Python as such, and the set of helper functions they are provided for the BPF library. And this code is compiled to bytecode using the LLBM and the C line compiler. And the way you can start is like if you're looking for the tools to run then you start with the DCC tools. And then go for the go with the BPF tool as such if you have great customization and if you want to write your own customized evaluation or adding code or customizing based on the requirement. And in that case, start with the BPF trace, which provides the high level language kind of interface, and then write your own DCC programs as so most of the programs that are there like CDM or the other tools over so tools are there. So, some of them, they use a combination of BPF trace and they own DCC programs as such. And most of the hardware acceleration or network acceleration with the customized requirement that would be done by writing their own DCC programs, BPF programs as such. So this is a collection or a framework that enables to write the program, write a Python program and embed the BPF program inside them the example that we saw as such. And the running the Python program basically generates the bytecode. And that's loads into the column is basically the DCC framework handles the work of converting the program see see like program or pseudo C program that into bytecode and load into the kernel with that the back end. The BPF system call is executed which would load the load this BPF program under the kernel as such. So basically it is useful for some complex tooling or the given kind of application. And BPF trace is a high level tracing language with which is like, I will inspire by a can see language interface. So it provides that kind of interface and then it uses LLVM as a background to compile the scripts into bytecode as such and load and the for loading and interacting with the kernel. The DCC framework has been used as such. And it's the, like one liner kind of interfaces, something that we have seen in the previous example so this is, and this is the BPF trace tool that this whole of this syntax is in its, it's been converted. It does the work of converting into a port that would be loaded into the kernel and then executing as such. To the help of this one liner syntax as such. Even I can write a small syntax short script as such. So there are some syntax syntax and primitive status to based on some pertaining to DCC and some syntax formation for BPF trace as such. Again a slide from Greg, guys, talking showcasing the different tools that are part of this DCC project as such. Okay. So your DCC, let me say, yeah, yeah. These are the, these, these, these are the reports that are from which we get the DCC and BPF trace as such. And all these tools they are, yeah, go ahead. I don't think there was a question, is somebody this question for Vemina? It doesn't look like it, Vemina. So you're like, if you look at this thing, so you would say that we can go and look at each of this subsystem and this information from each of the subsystem is collected by making use of this different tools as such. So as a beginner, what you can do is you can start with using this tools that help you to get the understanding of what is happening at each of the subsystem as I and then as you get comfortable with that then go and implement through them. Programming interface as such. So what I'll do is I plan to take some of the, this tool to demonstrate and then see what are, what are we getting on top of this as such. So here for, let's look at some of the system called interface as such. So here what the system called interface in this case this is some of the tools that are come for that particularly interacts with the system called interface and here the example I would take is the open snow which works the tool that that shows the process. Basically it tries to show which process are opening which files as such. And this tool can be useful for like if some application is failing or if there is some misconfiguration, then in that case this could provide some pointers as such. Similarly, there is a stacked snow. Does the work of snooping on tracing the different does traces stats system calls as the variants of stats system and which would help you to understand like which process trying to read some information about which files as such. And again, you can do the snooping with the help of the exact snoop which does the work of monitoring or snooping the exact system calls. So, like, if you run the application, this tool opens snow. So it trains the information like process ID command with the name file descriptor if there is some error in that and the part which file is been opened so you're in this case this is generally the same rule for the whole of the system as such. So one of the things that you can look up is this the identity just for explanation powers. So Iq balance is a is a is a demon that does the work of the interrupt interrupt load balancing as such so and basically does that goes and checks into the SMB affinity file for balancing the load across multiple CPUs as such. So the activity that is happening in the background as such done by the Iq balance so we just goes and open this files and does whatever the operation so right now we are looking with the open snow is just what are the files that are getting opened by what which process this process named the ID on the path and again like if there are any other errors that would be shown under this column as such and this is the file descriptor ID as such. Similarly, like, we have the start snoopers in the case. Now you're this happen be a BFC that particularly means that it's a VCC tool as such. So you're in this case like you're now like my mind system is just a virtual machine as such and nothing great or anything workload is been generated but based on the application, whatever the testing or some debugging a performance analysis you're doing. Then in that case, while that application is working with this generated these tools can help understand the some of the dynamics going on with that application as such. And similarly we have this exact exact smoke will give you like, is it just not like a performance delivered also like from the security, if the security appliances tends to understand that which process is trying to execute the binaries and all this so from from a security point of view this kind of this interface is quite helpful to track all the know you're in this case there's some of the demos they might be running in the background. So that is printed but some are some some application is running particularly that can be monitored like in that case, or from the security security tools are there. So if there's a lot of things going on, then all that can be taken, looked at a lot that when the exact system call is going as such. So here I am what I'm doing is first just going with the understanding of tools and then we will do the same thing with the examples as such with the programming example. Particularly for the file system, particularly like there are other other tools like file top is particularly used to trace the file read writes as such and it brings the summary and basically the tool what it does is like, it is particularly the VFS related functions as such, particularly the actual kernel function that is a VFS read and we have sprite that they have been using dynamic tracing as such. And it, what it does it brings the reads and writes as such so basically it is the VFS read and VFS write and the read and write workload as such. Basically it does the sort sorting based on the amount of data that has been read as such. And it's been in the default interval of one second that you can change it as such. Okay, so that helps to give gives an understanding like where actually the system is getting utilized and what kind of operation is happening. Similarly, we have this VF, which is that VFS count and VFS that it basically used to for getting an idea of general workload characterization that is happening. And if you want to know, like what all the VFS really functionalities for getting executed in this VFS count basically traces all the kernel function that are starting beginning with VFS underscore so basically all the functions that are part of the VFS of system as such. So all this thing is which brings this VFS count to as such so here in this case it brings this information as such the that this particular VFS function is called so many times so this would give us the idea. Which means under load as such or what what is really causing a blockage and what which are the, which of the kernel code has been more used or executing that can be traced by making use of things if it is particularly a file system load or a block by a load going to the file system and then this is one of the interface that provides this information as such. And VFS start is like it traces some common functions like read write sync and as such. Another that we can look at from the by using the tool is this scheduler, of course like from the performance point of when the system at higher load and is there and schedule is one of the key component as such that helps you to know, like exactly what is the process that are getting that are executing what they are been getting the time slot and who are been starved or as such so here in this case the one of the tool in the CPU distribution to say that does the work of measuring the time that you ask the process spends on the CPU before getting the schedule and basically internet uses the some type of BPF maps, which throws that information and it brings the information in form of histogram as such. So this with the help of this tool basically it helps the like to find a suppose like a task is spending short time on the CPU. So that kind of information is an indication of like, there might be multiple excessive context switching has happening, or the workload distribution is not. Surely poorly been done as such that gives that can that can be estimated by looking at the information that as such are possibly that can be the reason that the locks are being held assess the possibility of the contention of the resources. Like the some process has acquired the mutex and then another process trying to acquire the same mutex and as the lock is not been available there, there's chance of the application application is just context switching a lot many times as such. So that's why the application is not getting the CPU so information can be derived based on the stats collected from this. The task is not spending off time more that before it is getting scheduled as such. So that can happen that the task might be waiting for the IU to completion, or a long IU is happening as such. On this we can derive the information to know the health of the system as such, and all this information for collecting this information storing it uses this internal baths or storing the timestamp and the histograms as such. And this year, this is one of the examples like where there can be more may incur the overheads for some kind of rules because here these traces it goes and traces the scheduler trace point so there can be extensive load. This is the overhead created by this kind of tracing. Tracing events generated in the scheduler as such. So it runs in the background and creates a histogram as such. So here in this case, it is displaying in the information. This is happening through this timestamp. Another one is that we can look at as the virtual memory as such. One of the major concerns with that to have utilization of memory, like the memory is available or not and then that can be traced with the memory leak. The allocation location request and what it does it collects the stack traces stacks for the allocation that happens, and then brings a summary like which stack stack perform the allocation that has not yet been for it from that information and get to know the health of the memory allocation that is happening. You can trace the memory leaks for a particular process by specifying the input parameters that is a specific ID. In that case, the, at least the function from the leapsie that is malloc a lock and the related memory management functions and you can use it for tracing the full processes. So in that case, it goes into more of a kernel function that came along for kernel allocation API as such, along with the page allocations itself. So you're like this list is quite extensive and actually like if you get to a study each of this it's like you get to get into the details. And that's really as part of the kernel subsystems outside extensive very incredible resource information we get from this. So, when then you have about five minutes left and then I'm not sure how many slides you have. And do you want to. Okay, I'll. Okay, should I take questions. Yeah. Okay. So, the question in the q amp a box is, can ebbf be used in in a term of us. In a term of us. Yeah, I think that should be able to be used because BPO framework. And that's part of the kernel. So in the any drama, you tend to load the kernel modules and do some other basic functionality at that time of minute. But I think that by that time because BPO framework should be available. But would you be able to kick off, you won't be able to kick off a user process at that point. So, the user process. Not big kick no in it. So, through the internet, you've been able to load the module module that is you do you do through in small amount of product model probe as such. So, right, right. So here basically we're using BPF trace tool as such like, if you do, if we tend to use BPF trace then I think we should be able to do that functionality, or even BPF application programs been written to be this is the interface I think that should be able to be done. We are at a point that the kernel is up and running, and then you're loading the file system or any such components, but the functionality is up and running. So through BPF trace or BCC tools, I think that would be free that should be able to be done executed at the in it. I think some of the application might have that need as well. And you'll have to make that part of the in it. Right, yes, not not at the command line of course. Right, right. Yes. Okay, so that is great. That's one. Okay, let me see answered live let me get rid of that. Since it has ability to monitor the processes and virtual memory, I think it has the ability to tamper the memory data. Yes, you should be able to do that means if you're attaching those kernel functions as such. So memory dumping would not be a very good way through the system through the BPF but the information related to memory functionality would be good as such but of course you can dump that. There are other interfaces provided for dumping the memory as such because like it would be how much amount of memory one to dump us because the BPF map or the trace pipe, it would have limited size limitation as such. But if you define if you define if you divine your own BPF map functionality. And I think in that case, you should be able to do that. It's not a raw memory number some way to pass on that memory memory contents. There is one question that might be a good one to answer. Can the BBB a program store state or does it simply pass data back, we need to build a state in use of space. So the data that is available like by default we're using the kernel buffer, that is the buff. Then in that case, once it is passed it is emptied but other case it is there in the maps as such. But once it is access I think it gets flushed from the kernel store as such. So user space has to once they obtain the data user space has to keep track of the state and yes, yes, that would make sense, because these are these are. These are running buffers go anywhere. Yeah, or flow buffers. Yes. Okay, I think there are other questions. However, we are out of time. Candice would. I think that yes we are kind of it's where at time now 1030. I mean, 30 minutes. So do you have any more slides. I think I had some examples as such but it's there in the slides like it can be referred on the government, the main concepts and these things I've covered as such. Candice, I'll throw it back to you. Thank you everyone for joining us as a reminder this recording will be on the Linux Foundation's YouTube page later today and a copy of the presentation slides will be added to the Linux Foundation's website. We hope you are able to join us for future mentorship sessions. Have a wonderful day. Thank you guys.