 I recently joined my friends and working on some of the exciting features of memory management, particularly to settle the CXL memory as a software terrain, as a polling asset. And other than that, this is something like an endless line that comes whenever you're to talk about yourself. So I've been working with Linux Kernel around 24 years, I such. And I've worked on product development for networking, storage, advice systems, as well as mobile SOC development, particularly for Android-based kernel development as such. And to talk about my passion, other than coding that we do as a system programmer, I also like passionate about mentoring, guiding people. Those who are interested in all this was technologies, Linux systems and kernel systems as such. And I've been involved in mentoring team members within the company as well as Linux communities. Last, I think, two, three months back, I took some sessions, like free kernel development sessions. And it was very surprising and good to see that a lot of people are really interested in open source technologies, learn the system development as well as the ecosystem and willingness to contribute back to the community as such. And other than that, personally, I like tracking, reading, meditation and of course, work in the systems as such. So what we are going to do to cover up is understand what the BPF is and how it works and then how it is helpful for observable ideas as such when we can talk about security, for my security point of view with some examples using the BPF tools and framework as such. So this is like the very common understanding about the BPF as kernel technology that allows the programmer to run the programs without modifying the kernel or without adding the additional kernel modules as such. And you can consider it as a virtual machine inside the kernel where the programs can be executed and those programs can be customized based on the use cases as such. So the use case is particularly for security. Then the security engineers can implement the programs as per the security policies. The use case is for monitoring or observing the system, particularly for performance or for threats or something as such. Activity in that case, the BPF program can be accordingly programmed as such. So basically this functionality that included quite entry dot 18 kernel where as it's not a new feature as such, it was been there since a long time, particularly target for packet filtering as such. But then it was been extended to support the other operations and that came through this number of the features that got in including the BPF system call where the different BPF programs can be loaded as such. And that's how it evolved to be called as extended BPF as such. So basically these words can be used BPF or extended very exchangeable as such. And how exactly does it work is that these EPF programs, they are even driven and they can be attached to the some kernel control part as such. If the requirement from them is to monitor some activity in suppose in the networking stack or file system stack, then these EPF programs can be attached to that control parts and the necessary data can be collected based on which other intelligence can be applied for the actions as such. And this core part contains the specific triggers that call as hooks as such. And there are various hooks at various subsystems like the hooks can includes like network, even system calls, function entries and trace current response. Of course this is not just the limited but there are various places the hooks can be attached. And when they are triggered the code is the code that is the BPF code is compiled, compiled into bytecode that would get loaded into the kernel memory. And the internal bytecode has to be verified because this BPF programs, they were loading from the user space into the kernel address space. So there has to be some security measurements to be fulfilled that it is not going to temper the system. That is the world that has been done by the verifier, BPF verifier that verifies before it is executed. Some of the things that should be looked up as it does not contain a cyclic dependency or loop as such. So that the web program might end up in executing a loop and causing a system hang as such. So those things are being taken care, are being checked verified by the verifier as such. So basically it does the work of preventing the program from compromising the kernel or the system existently or on purpose as such. And once the programs is triggered at the hook then with the help of the help function, the APBF framework, with the help of this APBF help functions, you can collect the data and then data can be passed to the user space as such. So this BPF framework has a number of data structure constraints for transferring the data from kernel to the user space as such. This is just to give you an example like how exactly the BPF program loading happens. Here in this case we take an example of a BPF program that gets attached to the exact system call as such. And whenever any application that executes the exact system call thus particular BPF program would be executed, triggered as such. And then this is the way like helps you to get the visibility of the overall execution programs getting executed on the system as such. So just this is this again I'm going over you to understand how this BPF program ecosystem as such. So the development tool chain for the APBF as basically most of the applications that are using BPF, they might not directly go and write the BPF programs as such. And they might do it through the other evolved projects such as CLM, BCCR, BPF trace to utilities as such. So basically like if you as a developer as broad development using BPF framework, then in that case the developers can write the programs either in C language or Rust, Lua or Python as such. When programs they take help of this LibBF library as such that helps to like go and access particular kernel data structures and pass this kernel data structures or pass the data once one it has been collected to the user space for all those operations this BPF library is quite handy as such so every BPF program or might not be necessary to know the every kernel data structure, how does it look what are the data elements that it might want to extract so in the office can directly go and do it but using the library functions. This comes at a much handy as such. And the BPF programs they are being compiled into the bytecode and the LLVM C line compiler has been used. And like the two main front ends when you talk of BPF framework as the BPF compiler can collection this set of tools as such BCC and the BPF trace this is basically a utility as such. So basically if you're looking for running the tools for tools to run, then you can use the BCC tools that are there developed to collect this information as such as a first, if you're trying first then go with the BCC tools. And then you can if you want to write your own programs, maybe like some one liners kind of thing than the BPF trace tool can be used. And like if you are going to do analysis on the data that is collected adding more logic to that on that in that case the you can implement the BPF program either using Python go Rust or CS such. And there in that case you can based on the requirement whether you want to use some dynamic instrumentation or static instrumentation either the kernel level or the user space level based on that those programs can be written as such. So basically BCC framework that enables users to write the Python programs as such. And this framework itself will do the work of running the Python program and generate the bytecode as such. And BPF trace is again a high level tracing language. And basically it provides like a kind of syntax to do the fuzzy commands as such. And these are the repository from where you can get the. So, having got that understanding of BPF, let's look at the observability, how can we achieve through the BPF as such. So BPF observability encompasses like a light yet lightweight through implementation programs designed to monitor the events as such observability means what we are with the intention is to observe what is happening and on the system as such. So, again, as we have said that the BPF as the just in time compilers, which has the work of verifying the BPF program and loading to the kernel and once it is attached, once in a long time it gets attached to the kernel functions as such. And every time in that particular event has been triggered the hook would be executed as such. And the events I think is this is bit of a repetition from my previous slide but this is like the events can be the trace points system called events are not only system calls but any kernel function at entry or exit as such. Along with the perf events that is used to collect the performance data. This can attach the programs and the perf events and collect the data of the performance, maybe your schedule or CPU and the other contact switches all those things can be collected. Then there is another event interface where the BPF programs can be out of this is one of the feature recently. And not recently it got introduced in 5.7 kernel that is the LSM LSM interface that whether your programs can be attached to the LSM Linux security model interface where Linux security model interface is the kernel framework that basically provides a way to enforce mandatory access control policies onto the system, more from a security point of view. And other than that again the network interface at the various, like at the various levels in the networking, networking layers as like we can have the BPF hooks can be attached at layer 2, layer 3, L3, L4 or even the application like at the socket layer, TCP IP layer or even at the network driver layer as such. And the programs, BPF programs can be attached. So here, this is just a broad diagram that are elaborating on the storage, storage stack and the network stack as such. So when we say security absorbability means observing the system from a security point of view as like how do we go as like how do you quantify a matrix that to represent the security properties as such. And some of the matrix that comes into more comes broadly defined are the latency, traffic errors and saturation. So what we're going to look at look at these matrix items when we talk about security observability is from this point of view of these properties, we can do the analysis as such. And what, what are the core events that you should monitor you would be monitoring when you're talking about the observability security also it is particularly a process management than the process management in the sense, the life cycle of the whole process how it happening what are the process that are created, and creating executing passing data or passing signals communication all those things as part of the process events from the networking traffic as such at the socket earlier and that that considered file file system or storage or virtual file system all those things. So in that case the file accesses that are happening onto the system. So what we'll look at these four golden signals that governs the matters for security observability is latency is like when you're monitoring latency is one of the metrics that taking care of like that monitoring the time it takes to service the request as such. So here in this case like if you talk about the web service as such than the monitor can be happened on the HTTP traffic as such how the HTTP request, what is the time or how much latency it is taking whether it is the execution request is coming response within the refined time expectations or not as such. And again, the observability from that point distinguishing between the latency of a successful request and latency of the field requests as such. Then another is about the traffic like a measurement of how much demand is placed on to your system like particularly in a high level system as such like for means how much load your system can handle without getting blocked as such or stuck up as such without affecting the latencies. So the system responses like again, take an example of web service like that would measure the HTTP request for seconds as such are in case of audio streaming system. The measurement might focus on the network IO rate or concurrent actions concurrent sessions that are going for the for the different stream audio streams or the application that are running for the audio streams as such. Another criteria matrix would be the errors like the rate of requests means how the monitor the failures that is happening from the applications as such request failures as such. In your I'm for taking a web service application, like the request rate of requests explicitly for example like the HTTP are absolutely by this, the hundred success responses that are happening. Again, the policies for example is committed once you can have the metrics like if the response is not coming within the defined time frame like one second per response time, then that those, those responses can be considered as errors or not. So this can be one of the criteria based on the policies that have been defined as such. And then another is the saturation like how full your services as such like a measure or for system for emphasizing the resources that are most constrained as such like on a system like if you take up a data center as such running different services as such and the service the application various application that running and how much memory it is consuming in more in a more in a memory constraint systems memory workloads as one of the criteria that should be monitored very much. And again it is one again that is like one is the performance or the whole of the system. Health is one thing. Another is from the security that there might be some bad application or some suspicious application that is trying to allocate memory consuming a lot of a lot of memory as such. Similarly, in case of for IE Constraint system monitor the IE load as such that is the load that by being from the from the application to the data to the disk IE as such. So you're looking at those different metrics what I tried to cover up is that the monitoring can be done at the process level exhibition level at the networking socket level and the file access level these are the three examples that have I taken for observability as such. So for things what we will do is like look at the process execution what what during this process show what are what are the what all information can be monitored and what can be done from that information as such like process execution events can give you the full lifecycle process like from the start that is forking or exacting to the exit that is the exiting the process through the exit system call as such and the information that can be collected or the metadata what we can say is that the binary name the command line parameters that defines the runtime behavior of the programs what what were the parameters that been passed to that and then the other information on the process execution as the process ancestry like that would help to identify any process anomalies like in case the shell is trying to malware has to be executed that that would might get exit from the shell so and that case what is the who's the process of that that information can be collected from the process execution as such means what if the application that has been spawned it might be some malware or some suspicious application and other information like current working directory as such that would also help to identify if there are some hidden malware executions happening from some temporary directories as such so this directory information is one of the resource to get more visibility from the security part of you and then other than that the main main one of the main thing is the capabilities as such so the capabilities like that defines the effective permanent or inherent capabilities and those are like very much crucial when the capabilities are passed from the parent process to child process and monitoring all those things from security point of view for process execution is helps miss from the like detecting some privileged escalations can be done through the capabilities information and here while for doing this first time what I will do is I'll use the BCC tools that helps you to get the same information and then I'll show some examples for the B BPF programs through the programming as such so here in this case the BCC BCC tools has a lot of BCC framework has a lot of tools as such and one of them is the exact snoop as such so basically this tool does the work of snooping the process and internally it does is like it raises the exact system call as such like whenever the exact system calls is executed basically the exact system call is like what you do is pass the executable name and the parameters arguments and the environment variable so the exact so basically it goes and attaches the BPF program to the exact system call and collects that information so that helps you to show the shows the files executed its arguments as such okay so let me do one thing for now okay what has happened let me sit okay I need to share I need to share I can do with other applications as well right I'll do it so for now please ignore the errors in warnings I've been doing a lot of so for now just ignore the warnings as such I've been doing a lot of for modification running the program so there's some mismatch with the libraries as such load as such so here in this case what I've done is a started the exact snoop application as such and like you suppose if you do another terminal execute some on the command as such so you would see the exact snoop captures that command that has been executed so basically here what is happening is like the exact system call the show the shell executes the command that is executes the command piece command by executing the exact system call as such so exact snoop basically attaches the BPF program to the exact system call as such okay so here in this case what we see is we are seeing the process that is executed yes but it's PID parent PID and the argument as such so this is one of the two BCC tool that's basically does the work that gets so you're at first hand as a beginner as a starting you can you can use this BCC tools as such and then another step next step is like you can add more one liners using the BPF B5 trace as such and then eventually go and write the actual programs as such so another thing is like network socket monitoring as such which capturing the network socket gives the gives gives the idea improved identity or the network packets like what the network packets in the sense what the data that is flowing through the network interfaces and basically when we capture the information at the socket layer or at the socket system calls as such you're they are more associated with the process metadata rather than with the actual protocol network protocol as such the below layer three protocol as such so if you can capture the information at the sockets as such so that can help you to associate the information related to the process with what process has been passing the this information as such and basically helpful can be used to detect any data explorations or any unusual behavior that is happening using this socket statistics as such so basically the observability can be done at the the program BPF programs can be attached to a various socket system calls and that information can be elements everything related to the IP address where it is passing what is routing information and all that those things as such and basically you can trace the whole life cycle of the packet as such and some of the things that we talked about is visibility a process listening to what all connections it is listening them what if it is accepting any yeah how does that happen so like we can get the visibility of what are the connection that is accepting or listening to the different sockets and how other metrics would be data transfer that is how happening in and out of the connections and there are all the whole life cycle of the socket as such then yeah so you're again we can look at this with an example with the PCC tool that is taking example of TCP connect this is another tool from the PCC framework that helps you to look at the network traffic as such so this tool basically traces the TCP connections as such so and it goes and attaches the BPF program to the connect system call socket system call as such and this basically internally this way it is going and tracing the kernel function that is a TCP v4 connect and TCP v6 connect as such so all you can do is like you can run a simple call come on on one terminal and run this TCP connect on another terminal and you could be able to see the information the TCP connections that are happening as such as such okay okay let me I'll just I'm sorry just a minute oh sorry sorry sorry sorry all that some mistake in the system but basically it has to work the way it has been shown it is currently not working my bad on that spot so basically the TCP connect gives you the information like PID the process this source IP address destination IP address port and all this information as such so here this we talk about the socket information as such then another is the first we go with the process execution process lifecycle then the socket lifecycle and then we'll just look at some of the monitoring that can be done on the file file access as such so this file access monitoring provides the ability to monitor the file access that is happening as such like starting with opening the file performing read and read and write on to that file through the file is closed as such and in this case the accessing a monitoring of the file access can be done using the K probe K probe hooks are through the trace point hooks as such and here you can do the observe the system calls related to the files system call events and other information related to the file that are open that is the file descriptors and the data metadata associated with that file descriptors and the metadata that basically can be the path name that is the file that has been opened then the by the in case of read and write the bytes that is read and write written as such and the size parameter and the other things corresponding to this file access as such and then again this open snoop is the tool that can be used that basically this it attaches the BPA program to the open system call and to the variance of the open system call that is open at and all those things and basically it provides it will it provides the it shows the information at which processes which process are trying to open which set of files as such and from the security first point of view that can that can be very helpful to state that if some unwanted the user or unwanted application has been trying to temper around with the configuration files or some information that should not be accessed by the access by the files so now open snoop is one of the tool that is basically it is what it is doing is internally it is using the trace points and the K probes hooks as such that gets attached at the beginning of this open system calls as such okay so the parameters that gets passed to the open system call is the file name and the param mode and all that information as such similarly for the read and write we pass the file descriptor the buffer and the size as such so collecting this information helps to know that what are the accesses that are happening from which process as such similarly another status snoop basically that that does not provide what internally file is read or written but it provides the information of the file file as such like the metadata related to the files the I know the information and the access access permissions and all those things can be collected to the stats stats snoop tool as such basically it goes and attaches to the various the variants of a stat system call as such so here this is the snippet that I have taken as such like here in this case you see that this is the it tells the path here in this case it also gives you the file list FDs that has been written and come on what has what is going and trying to access that file starting on that file as such okay and you can in case of open snoop you get the PID then the process name as such and then the file descriptor that is written and if there has been error on to that on that file that is when that has been opened as such that has also been given as such okay so this to summarize here just to talk about the BCC tools here I cover only two or three that is open snoop stats snoop and exact snoop covering the file access and the process lifecycle the BCC tools provides a rich set of tools that can go and look at the system at the various levels as such so here in this case the there are the tools that that that can be used at the application level user space and the tools that are used at the kernel level for the different subsystems as such so here in this case what we looked sorry we looked at this to come out that basically interfaces at the system call level as such basically open snoop does the open system call a stats snoop that the stats system call as such similarly information for BFS can be done can capture through this set of tools that is file life file top like something like a top common that gives the information of the files at the BFS layer similarly there are most file system specific one volume manager and block layer so basically discovered the whole of the IO path as such at the network network path the information can be captured or collected at the socket level or at the network what you see layer through the layer for TCP IP layer as well as the network driver or layer to what we can say as such and other than that then we can also collect the information about the memory management like some of the tools like memory leak actually it gives and it gives quite a detailed information as such like provide the stack trace of what are the processes that are allocated the memory but not yet freed as such so this memory like else basically help to collect if some process is trying to just keep on allocating the memory not rain leading to some out of memory kind of situation so this memory is one of the tool that it's quite handy as such similar at the socket at this scheduler level to understand if the other contact switching is happening and what what process taking time as such so the hooks can be attached at the scheduler level as such so overall this is quite a resourceful tools that can be used as a first step explore with these tools and then we can go and write the BPR programs for that so basically like here I just wanted to elaborate that there can be multiple sources of events to provide the visibility as such of the entire stack and to broadly categorize this instrumentation of the programs the as a dynamic instrumentation and the static instrumentation that is happening at the user user space as well as current space as such so particularly if talking about the kernel for a dynamic instrumentation that goes through the K probes function K probes interface where you can attach the BPR program to the K probe as such and static instrumentation at the kernel level is done through the trace points as such so basically trace points are the is the framework where it the code is implemented inside the kernel as the trace point events has been implemented as such okay so the static points are being added through the trace points and similarly for the user space you have the U probes only this usdT that is user statically defined trace points as such so another tool that is the BPC then other is the BEPF trace as such it provides a bit of programming kind of interface but not in a very detailed but one liner kind of commands can be written as such so here in this case you can specify whether you are doing a static instrumentation or a dynamic instrumentation by specifying K probe or trace points as such so here in this in this first example the BPF trace what we are telling is user K probe instrumentation and attached the program to the do this open kernel function so basically this is the function that gets called when an open system call is the basically the kernel code path for the open system call from the application comes to the open system call handler and from the ultimately this is the function that would get executed and here in this case you can write some print of that when this do this open call function kernel function is executing print the print what is the process that is executing this particular kernel function and what is the argument as such similarly for another way is like you can use the trace points and in this example and trace points can you can specify trace points as entering the system call or exiting the system call those can give that probes that time as such so since read enter the collect the information when entering the read system call as such so information can be collected at the beginning of the system call as well as at the end of the system call as such so end of system call would give you the behavior of that whole system call how what what it has done and the return parameters help you to decide on that here in case of read the return value would be the number of bytes that has been read by that system call so that information can be collected as such similarly information can be collected for the process whatever the bit what you see through this exact snow can be done more elaborately through using the BPF trace functionality as such let me show you one example so when you're when you're doing programming now so here in this example this is an example for writing the BPF program using C as such so in case of when you're writing the program as we have seen in the previous slides as such at the big name so here in this case what we see that the BPF program has to be attached to particular hook as such in this example exact system call so there when you're doing the programming there are the two parts one is the BPF program itself and then there is an application that would go and load that BPF program as such and then there is another instance where the actual activity is happening which would trigger this hook as such so when you're doing the programming what we do is we have we implement this BPF program as such and and that is that whole of this program have to be combined into a bytecode and then loaded into the memory as such okay so when you're doing the program we have the two instance one is the BPF program and the other is the application that will go and load the BPF program and that is done with the help of the library function LibBPF as such okay so here in this example and like what kind of hooks that are being used whether it's a trace point or it's a K probe as such based on the macros are being defined as part of the LibBPF as such so here in this case if we look at this thing here in this case this application BPF program is going to get hooked to the kernel function VFS read entry function and it is going to use a K probe mechanism for doing that so that that is done through this macro as such and here in this case I'm just not doing anything just printf but it's a library function that has been used as such so similar here is another example for doing the working at the system call that is a link and link system call as such the APIs that are there so here in this case what we'll do is the for the semantics of this API is that the name of the program and then what are the parameters that this input parameter is one that goes corresponding to this kernel API as such okay and here in this case you can implement what whenever that hook has been triggered as such what has to be what has to what you want to collect as such that that can be done here in this case like we are collecting the process ID and then the file name corresponding to the link unlink unlink tool particularly the system call that passes the file name that has to be unlinked as such so that information and then that can be so here in this case this implementation whatever the logic has to be has to be applied or whatever the information from the observability point of view what information has to be collected that would go as part of this bpf program as such now in case of open snoop or in the exact case what we saw that was the file name its parameters and all those things so that is basically and that's it's has the at the code part of that bpf program as such now here in this case if you look at this is another probe and k return probe is there so probe basically is triggered at the starting of the function and return return probe is at the end exit part of that so that way you can put the put the hooks at the exit entry or exit as such another example I wanted to show okay so here in this case this is an example another example that is there for the LSM hooks as such okay I'm sorry that it has some this is an example I'll just wrap up that is this is an example that will program for LSM attached to the LSM hooks as particularly the LSM hook defined to task free that is when the task gets completed or exited that time you can collect the information as such okay so that's the end of the time as such and as well as the end of the session I know it's not very much in depth but it gives you a elaborate understanding of the observability that you can get started with and then develop on top of that as such okay thank you guys